MAAP-Project · rtapella · May 18, 2023 · Mar 30, 2023 · Mar 30, 2023 · Mar 31, 2023
diff --git a/docs/source/_static/faq_images/check_job_1.png b/docs/source/_static/faq_images/check_job_1.png
diff --git a/docs/source/_static/faq_images/dps_jobs_ui.png b/docs/source/_static/faq_images/dps_jobs_ui.png
diff --git a/docs/source/_static/faq_images/dps_output.png b/docs/source/_static/faq_images/dps_output.png
diff --git a/docs/source/_static/faq_images/get_dps_job_result.png b/docs/source/_static/faq_images/get_dps_job_result.png
diff --git a/docs/source/_static/faq_images/get_dps_job_status.png b/docs/source/_static/faq_images/get_dps_job_status.png
diff --git a/docs/source/_static/faq_images/get_job_result.png b/docs/source/_static/faq_images/get_job_result.png
diff --git a/docs/source/_static/faq_images/get_job_status.png b/docs/source/_static/faq_images/get_job_status.png
diff --git a/docs/source/_static/faq_images/job_result_1.png b/docs/source/_static/faq_images/job_result_1.png
diff --git a/docs/source/_static/faq_images/job_result_2.png b/docs/source/_static/faq_images/job_result_2.png
diff --git a/docs/source/_static/faq_images/jobs.png b/docs/source/_static/faq_images/jobs.png
diff --git a/docs/source/_static/faq_images/submit_job_1.png b/docs/source/_static/faq_images/submit_job_1.png
diff --git a/docs/source/_static/faq_images/submit_job_2.png b/docs/source/_static/faq_images/submit_job_2.png
diff --git a/docs/source/_static/faq_images/submit_job_3.png b/docs/source/_static/faq_images/submit_job_3.png
diff --git a/docs/source/_static/faq_images/submit_job_4.png b/docs/source/_static/faq_images/submit_job_4.png
diff --git a/docs/source/getting_started.rst b/docs/source/getting_started.rst
@@ -5,6 +5,9 @@ Getting Started
   :maxdepth: 2
   :caption: Getting Started:
 
-  getting_started/welcome.ipynb
+  getting_started/about_maap.ipynb
+  getting_started/maap_overview.ipynb
   getting_started/getting_started.ipynb
-  getting_started/git_clone_repo.ipynb
+  getting_started/writing_code.ipynb
+  getting_started/running_at_scale.ipynb
+
diff --git a/docs/source/getting_started/_static/clone_demo1.png b/docs/source/getting_started/_static/clone_demo1.png
diff --git a/docs/source/getting_started/_static/clone_demo2.png b/docs/source/getting_started/_static/clone_demo2.png
diff --git a/docs/source/getting_started/_static/clone_demo3.png b/docs/source/getting_started/_static/clone_demo3.png
diff --git a/docs/source/getting_started/_static/clone_demo4.png b/docs/source/getting_started/_static/clone_demo4.png
diff --git a/docs/source/getting_started/_static/clone_demo5.png b/docs/source/getting_started/_static/clone_demo5.png
diff --git a/docs/source/getting_started/_static/clone_demo6.png b/docs/source/getting_started/_static/clone_demo6.png
diff --git a/docs/source/getting_started/_static/clone_demo7.png b/docs/source/getting_started/_static/clone_demo7.png
diff --git a/docs/source/getting_started/_static/createws1.png b/docs/source/getting_started/_static/createws1.png
diff --git a/docs/source/getting_started/_static/createws2.png b/docs/source/getting_started/_static/createws2.png
diff --git a/docs/source/getting_started/_static/createws3.png b/docs/source/getting_started/_static/createws3.png
diff --git a/docs/source/getting_started/_static/example_alg1.png b/docs/source/getting_started/_static/example_alg1.png
diff --git a/docs/source/getting_started/_static/gitlab1.png b/docs/source/getting_started/_static/gitlab1.png
diff --git a/docs/source/getting_started/_static/gitlab2.png b/docs/source/getting_started/_static/gitlab2.png
diff --git a/docs/source/getting_started/_static/gitlab3.png b/docs/source/getting_started/_static/gitlab3.png
diff --git a/docs/source/getting_started/_static/gitlab4.png b/docs/source/getting_started/_static/gitlab4.png
diff --git a/docs/source/getting_started/_static/interface1.png b/docs/source/getting_started/_static/interface1.png
diff --git a/docs/source/getting_started/_static/interface2.png b/docs/source/getting_started/_static/interface2.png
diff --git a/docs/source/getting_started/_static/loggingin1.png b/docs/source/getting_started/_static/loggingin1.png
diff --git a/docs/source/getting_started/_static/loggingin2.png b/docs/source/getting_started/_static/loggingin2.png
diff --git a/docs/source/getting_started/_static/loggingin3.png b/docs/source/getting_started/_static/loggingin3.png
diff --git a/docs/source/getting_started/_static/maap_overview_diagram.png b/docs/source/getting_started/_static/maap_overview_diagram.png
diff --git a/docs/source/getting_started/_static/maap_tour_notification.png b/docs/source/getting_started/_static/maap_tour_notification.png
diff --git a/docs/source/getting_started/_static/register-as-alg.png b/docs/source/getting_started/_static/register-as-alg.png
diff --git a/docs/source/getting_started/_static/rename_alg.png b/docs/source/getting_started/_static/rename_alg.png
diff --git a/docs/source/getting_started/_static/running_algorithms_overview.png b/docs/source/getting_started/_static/running_algorithms_overview.png
diff --git a/docs/source/getting_started/_static/storage_options_diagram.png b/docs/source/getting_started/_static/storage_options_diagram.png
diff --git a/docs/source/getting_started/_static/storageoptions.png b/docs/source/getting_started/_static/storageoptions.png
diff --git a/docs/source/getting_started/_static/writing_code_overview.png b/docs/source/getting_started/_static/writing_code_overview.png
diff --git a/docs/source/getting_started/about_maap.ipynb b/docs/source/getting_started/about_maap.ipynb
@@ -0,0 +1,41 @@
+{
+ "cells": [
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### About the Multi-Mission Algorithm and Analysis Platform (MAAP)\n",
+    "\n",
+    "**The MAAP platform is designed to combine data, algorithms, and computational abilities for the processing and sharing of data related to NASA’s GEDI, ESA’s BIOMASS, and NASA/ISRO’s NISAR missions.** These missions generate vastly greater amounts of data than previous Earth observation missions. There are unique challenges to processing, storing, and sharing the relevant data due to the high data volume as well as with the data being collected from varied satellites, aircraft, and ground stations with different resolutions, coverages, and processing levels.\n",
+    "\n",
+    "**MAAP aims to address unique challenges by making it easier to discover and use biomass relevant data, integrating the data for comparison, analysis, evaluation, and generation.** An algorithm development environment (ADE) is used to create repeatable and sharable science tools for the research community. The software is open source and adheres to ESA’s and NASA’s commitment to open data.\n",
+    "\n",
+    "**NASA and ESA are collaborating to further the interoperability of biomass relevant data and metadata.** Tools have been developed to support a new approach to data stewardship and there is a data publication workflow for organizing and storing data and generating metadata to be discoverable in a cloud-based centralized location. The platform and data stewardship approaches are designed to ease barriers and promote collaboration between researchers, providers, curators, and experts across NASA and ESA.\n",
+    "\n",
+    "**This guide aims to help users get started with using the platform for searching, visualizing, accessing, processing, querying, and sharing biomass relevant data to the MAAP.** These data, collected from satellites, aircraft, and ground stations, are organized into collections and granules. Collections are a grouping of files that share the same product specification. Granules are the individual files which are independently described, inventoried, and retrieved within a collection. Granules inherit additional attributes from their containing collection. Explanations of the various functions available in MAAP to use in the ADE will also be explored.\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.7.8"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}
diff --git a/docs/source/getting_started/getting_started.ipynb b/docs/source/getting_started/getting_started.ipynb
@@ -1,15 +1,17 @@
 {
  "cells": [
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Getting started with the MAAP\n",
+    "## Setting up your account and workspace\n",
     "\n",
-    "Learn how to sign up for an account and access all the services of the MAAP."
+    "Learn how to sign up for an account and access the MAAP."
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -19,12 +21,13 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
     "### Signing up for a new MAAP account\n",
     "\n",
-    "Once registered, you can register for a MAAP account by navigating to the MAAP ADE at http://ade.ops.maap-project.org. On your first visit, select the \"URS\" login button shown here:\n",
+    "Once registered, you can register for a MAAP account by navigating to the MAAP ADE at http://ade.maap-project.org. On your first visit, select the \"URS\" login button shown here:\n",
     "\n",
     "![ADE Login](../_static/ade_login.png)  \n",
     "  \n",
@@ -39,46 +42,120 @@
     "![ADE Access Denied](../_static/access_denied.png)  \n",
     "  \n",
     "  \n",
-    "At this point, a MAAP administrator will approve your account, which will grant you access to the MAAP ADE. This process typically takes 5-10 minutes. To check on the status of your pending account, contact the MAAP team at support@maap-project.org.\n",
+    "At this point, a MAAP administrator will approve your account, which will grant you access to the MAAP ADE. Access is only granted to known users in the biomass science community and other projects directly related to MAAP. To check on the status of your pending account, contact the MAAP team at support@maap-project.org.\n",
     "\n",
-    "Once your MAAP account is approved, you will receive an email notification using the address of your Earthdata Login account to let you know that your access is enabled."
+    ".. note::\n",
+    "    Once your MAAP account is approved, you will receive an email notification using the address of your Earthdata Login account to let you know that your access is enabled."
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "### Code repository access\n",
+    "### Logging in\n",
     "\n",
-    "After creating your MAAP account, you can create a code repository by navigating to the MAAP GitLab account at https://repo.ops.maap-project.org. This GitLab account is connected to your ADE workspaces automatically when signing into the ADE."
+    "1. Navigate to https://ade.maap-project.org/ in Chrome or Firefox. You should be redirected to a page that looks like this:\n",
+    "![MAAP Project log in](_static/loggingin1.png)\n",
+    "\n",
+    "\n",
+    "2. Click the “Login with EarthData Account” button. If this is your first time logging in, you should be redirected to an EarthData Login page that looks like this:\n",
+    "![Login with EarthData](_static/loggingin2.png)\n",
+    "\n",
+    "\n",
+    "3. Enter your “EarthData Login” account credentials here and click “Log in”. You should see a temporary page that says “Redirecting”, followed by the MAAP showing your Workspaces (which will be empty to start):\n",
+    "![MAAP Jupyter logging in](_static/loggingin3.png)"
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "### Uploading your public SSH key \n",
+    "### Creating a workspace\n",
     "\n",
-    "In order to access your ADE workspaces using SSH, you'll need to upload your public SSH key to your MAAP profile using the MAAP portal at https://ops.maap-project.org. \n",
+    "Workspaces are effectively a JupyterLab \"computer in the cloud\". To get started with Jupyter you need to create a workspace.\n",
     "\n",
-    "Click on your profile name (or the \"Login\" button) in the top right corner shown here:\n",
+    "1. In the top-left corner of the MAAP dashboard, under “NASA MAAP”, click “Get Started”. You should see a menu that looks like this:\n",
+    "![Choose a workspace stack](_static/createws1.png)\n",
     "\n",
-    "![MAAP Portal](../_static/portal.png)\n",
     "\n",
-    "On your profile page, click the \"Choose File\" button to upload your key.\n",
+    "2. Select \"Basic Stable”. This is called a \"Stack\" and represents a type of cloud compute environment that will be set up. If you are interested in seeing more about each Stack, the adjacent link in the left-hand area is where you can see the configuration of each Stack in detail. After choosing \"Basic Stable\", you will see a loading screen that looks like this – wait for it to finish loading.\n",
+    "![Loading Basic Stable](_static/createws2.png)\n",
     "\n",
-    "![SSH Key Upload](../_static/sshkey.png)\n",
     "\n",
-    "After uploading your key, the SSH connection will be enabled **after your next** login into the ADE."
+    "3. Once the workspace has loaded, you should see a Jupyter interface that looks like this (note: You will see fewer environments and items in your root directory — this is normal! You may also see some notifications in the bottom right that look like errors about SSH Keys and other things; that is normal as well. You will also see one asking if you would like to take a guided tour.).\n",
+    "![New Workspace View](_static/createws3.png)\n"
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "### Note on future URL changes\n",
+    "### Jupyter Interface overview\n",
+    "When you first log in you may see a notification in the bottom right about a guided tour. Feel free to view the tour, which will give you a quick overview of the Jupyter user interface. You can also find the MAAP Tour in the Help menu at any time.\n",
+    "![MAAP Tour Notification](_static/maap_tour_notification.png)\n",
+    "\n",
+    "\n",
+    "In addition to [typical JupyterLab menu bar and sidebar](https://jupyterlab.readthedocs.io/en/stable/user/interface.html) configuration:\n",
+    "\n",
+    "#### MAAP Jupyter Menus\n",
+    "![Jupyter Menus](_static/interface1.png)\n",
+    "\n",
+    " - Git: Open repo in terminal, init, or clone repo.\n",
+    " - Jobs: Users may submit jobs through the submit tab and view their jobs through the view tab.\n",
+    " - Help: The help menu has several customized extensions and references to the MAAP documentation.\n",
+    "\n",
+    "#### MAAP Jupyter Sidebar\n",
+    "![Jupyter Sidebar](_static/interface2.png)\n",
+    "\n",
+    " - File Browser\n",
+    " - Running Terminals & Kernels\n",
+    " - Git Repo Interface (if this folder is a Git repo)\n",
+    " - Collaborators\n",
+    " - Table of Contents\n",
+    " - Extension Manager\n",
+    "\n",
+    "#### MAAP Blue Sidebar\n",
+    "\n",
+    " - Workspaces: See workspaces, share them, as well as configure settings\n",
+    " - Stacks: See available platforms for workspaces & required memory\n",
+    " - Administration: Control the configuration & policies for your installation.\n",
+    " - Organizations: allow groups of developers to collaborate with private & shared workspaces. Resources & permissions are controlled & allocated within the organization by admin.\n",
+    " - Profile (bottom, labeled with your name): See account info, logout\n"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### MAAP Storage Options\n",
+    "![Jupyter annotated folders](_static/storageoptions.png)\n",
+    "\n",
+    "#### My root folder (fast cloud storage)\n",
+    " - Your Jupyter home directory (`~`) is mounted to `/projects`.  Files in here persist across sessions and exist across your workspaces.\n",
+    "     - Use this for code-related items, smaller data storage\n",
+    "     - Git is more likely to behave predictably here compared to other storage\n",
+    "     - This is also the place to make persistent conda environments (covered in another section), but make sure to not make a conda env inside a git-tracked folder, or if you do add it to the .gitignore. If git is tracking an env, it could cause your workspace to crash. \n",
+    " - Uses local (to Jupyter) file system; generally faster and more reliable for “normal” file operations, but expensive\n",
+    "\n",
+    "#### Large file storage: my-private-bucket\n",
+    "`~/my-private-bucket` is an S3 bucket with persistent storage, but accessible only to you and others in a shared workspace.\n",
+    " \n",
+    " - Use for large data storage\n",
+    " - It will be slower than the root folder to copy and move files, which is why it is not ideal for storing smaller files that need to be read or written quickly\n",
+    "\n",
+    "#### Sharing files: my-public-bucket and shared-buckets\n",
+    "`~/my-public-bucket` is an S3 bucket with persistent storage. It is the same as `~/shared-buckets/<my_username>/` — anything you put in here will be accessible to other users via `~/shared-buckets/<my_username>` as a read-only file. Likewise, to find shared files from another user, look in `~/shared-buckets/<their_username>`.\n",
+    "\n",
+    " - Use for large data storage for files that you want to share across workspaces\n",
+    "\n",
+    "![Storage options diagram](_static/storage_options_diagram.png)\n",
     "\n",
-    "The \"*.ops.maap-project.org\" URLs referenced in this guide will be converted to \"*.maap-project.org\" once the version one of MAAP is released in the near future. At that time, these URLs will be updated accordingly."
+    "#### Mounting your MAAP workspace on your local computer\n",
+    "If you prefer to work on your local computer, or to drag-and-drop copy files from your computer to/from MAAP, you access the workspace via SSH. The [process for doing this](../system_reference_guide/ssh.ipynb#Accessing-MAAP-workspaces-over-SSH) is in the system guide."
    ]
   }
  ],

diff --git a/docs/source/getting_started/git_clone_repo.ipynb b/docs/source/getting_started/git_clone_repo.ipynb
diff --git a/docs/source/getting_started/maap_overview.ipynb b/docs/source/getting_started/maap_overview.ipynb
@@ -0,0 +1,44 @@
+{
+ "cells": [
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### An overview of the MAAP platform\n",
+    "\n",
+    "The MAAP is a cloud-based system to write science-analysis code and then run it at scale. This lets you keep all of the input and output data “in the cloud”.  It is composed of a few parts:\n",
+    "\n",
+    "![MAAP Overview Diagram](_static/maap_overview_diagram.png) \n",
+    "\n",
+    " -  The **Algorithm Development Environment (ADE)** is a tool that helps with the development of algorithms in a consistent, standardized environment that helps with the development and testing of algorithms and facilitates large scale data processing. MAAP's primary user interface is Jupyterlab, where code is written and tested before pushed to the large scale data processing system. Code is stored and checked out from Git-based repositories, including Github and MAAP's own code repository subsystem.\n",
+    " - The **Data Processing System (DPS)** is where registered algorithms (see Algorithm Catalog) can be run at scale in the cloud. The MAAP system provides a Jupyter GUI to run Jobs, or the maap.py library can be used to run a batch of Jobs in a loop using Python. The DPS also has monitoring capabilities, and again the MAAP system provides a Jupyter GUI to help monitor Jobs. This can also be done using maap.py in Python.\n",
+    " - The **Algorithm Catalog**, where your algorithms from the ADE can be registered and compiled for use by the DPS. The MAAP system provides API and GUI tools to help you register and view your algorithms.\n",
+    "- The **Code Repository** is a git-based repository to store user code. It is also used to store the configuration files necessary for building algorithms to store in the algorithm catalog and for execution in the DPS.\n",
+    " - Input data comes from a few **Data Catalogs**. Currently there is a MAAP [STAC Catalog](https://stacspec.org/en/about/) and the [NASA CMR Catalog](https://www.earthdata.nasa.gov/eosdis/science-system-description/eosdis-components/cmr). More information can be found in the [search tutorials section](https://docs.maap-project.org/en/latest/technical_tutorials/search/catalog.html).\n",
+    " "
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.7.8"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}