Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add missing reference to example yaml files notebook, and clarify section about env update based on a config file #252

Merged
merged 5 commits into from
Jun 30, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 34 additions & 13 deletions docs/source/system_reference_guide/custom-environments.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@
"\n",
"### Package manager\n",
"\n",
"We use `mamba` (a fast `conda` drop-in replacement) as a package manager to install, update or remove packages (libraries). `mamba` works with 'environments' that are directories in your local file system containing a set of packages. When you work 'in a given environment', it means you that your programs will look for dependencies in that environment's `mamba` directory. All workspaces launch with a environment called `base`, which is a `mamba` environment that has all the pre-installed libraries. If you open a terminal launcher after creating a `Basic Stable` workspace : \n",
"We use `mamba` (a fast `conda` drop-in replacement) as a package manager to install, update or remove packages (libraries). `mamba` works with 'environments' that are directories in your local file system containing a set of packages. When you work 'in a given environment', it means that your programs will look for dependencies in that environment's `mamba` directory. All workspaces launch with an environment called `base`, which is a `mamba` environment that has all the pre-installed libraries. If you open a terminal launcher after creating a `Basic Stable` workspace : \n",
"\n",
"\n",
"![Base conda environment location](../_static/base_conda_environment_location.png)\n",
Expand All @@ -49,14 +49,19 @@
"\n",
emileten marked this conversation as resolved.
Show resolved Hide resolved
"**Note : any modification to the `base` environment does not survive a workspace restart. In other words, modifications to `/opt/conda` disappear after a workspace restart**. \n",
"\n",
"Extending an existing `mamba` environment means adding packages on top of what it contains, which works provided there are no dependency conflicts. You can use a configuration file specifying all the new libraries to add. It is recommended for reproducibility and shareability. See the below sections for configuration file usage. \n",
"\n",
"Alternatively, you can install libraries using the `mamba install` command to install additional packages in your current environment (run `mamba --help` to learn more about how to use `mamba` commands). For example :\n",
"Extending an existing `mamba` environment means adding packages on top of what it contains, which works provided there are no dependency conflicts. You can install libraries using the `mamba install` command to install additional packages in your current environment (run `mamba --help` to learn more about how to use `mamba` commands). For example :\n",
"\n",
"```\n",
"mamba install xarray\n",
"```\n",
"\n"
"\n",
"However, it is recommended to use configuration files for reproducibility and shareability. With this approach, assuming your configuration file is named `config.yml`, the command to use is : \n",
"\n",
"```\n",
"mamba env update -f config.yml\n",
"```\n",
"\n",
"For more details on configuration files, see the [Custom environments section](#Custom-environments) and for an example of this command, refer to the [subsection about updating an environment with a configuration file](#Updating-an-existing-environment-with-a-configuration-file).\n"
]
},
{
Expand All @@ -66,19 +71,21 @@
"source": [
"## Custom environments\n",
"\n",
"Note that for the rest of this README, you can find the example configuration files in the `examples-environment-configuration-files` page, in the same section as this page.\n",
"*For the rest of this README, in each section we provide a link to download an example YAML configuration file.*\n",
"\n",
"You can use the `mamba` CLI to create a new, custom environment. The parameters (the list of libraries, the location where to search for them, etc...) can be passed either from a configuration YAML file or directly on the console. We recommend using the first option (a YAML file is easier to share and modify). \n",
"\n",
"### Basic custom environment\n",
"\n",
"See the `env-example` in the examples notebook. This configuration installs specific versions `python`, `pandas` and `geopandas` from `conda-forge`. If versions aren't specified, the latest is installed. We recommend to always specify the version for reproducibility. The basic command to create this environment would be :\n",
"*Example config file for this section [here](./example_conda_configuration_files/env-example.yml).*\n",
"\n",
"This configuration installs specific versions `python`, `pandas` and `geopandas` from `conda-forge`. If versions aren't specified, the latest is installed. We recommend to always specify the version for reproducibility. The basic command to create this environment would be :\n",
"\n",
"```\n",
"mamba env create -f env-example.yml\n",
"```\n",
"\n",
"However, this stores this environment files in `/opt/conda`, which a directory that is reset when the workspace restarts, and so custom environments are lost. Therefore, you want to specify a storage location in your user directory with the `--prefix` parameter\n",
"However, this stores this environment files in `/opt/conda`, which is a directory that is recreated when the workspace restarts, and so custom environments are lost. Therefore, you want to specify a storage location in your user directory with the `--prefix` parameter\n",
"\n",
"```\n",
"mamba env create -f env-example.yml --prefix /projects/env\n",
Expand All @@ -92,19 +99,31 @@
"\n",
"### Updating an existing environment with a configuration file\n",
"\n",
"You can update an existing environment with a configuration file as well. For example, let's assume you have a `mamba` environment with a set of packages already installed in it (for example the `base` environment), but it doesn't have `xarray` and `geopandas`. See the `env-extend` example in configuration examples notebook. Running `mamba env update -f env-extend.yml` will update `base` by adding `xarray` and `geopandas`, provided it does not cause conflicts with the existing libraries. \n",
"*Example config file for this section [here](./example_conda_configuration_files/env-extend.yml).*\n",
"\n",
"You can *update* an existing environment with a configuration file as well. For example, let's assume you have a `mamba` environment with a set of packages already installed in it (for example the `base` environment), but it doesn't have `xarray` and `geopandas`. Using the linked example config : \n",
"\n",
"```\n",
"mamba env update -f env-extend.yml\n",
"```\n",
"\n",
"This command will update `base` by adding `xarray` and `geopandas`, provided it does not cause conflicts with the existing libraries. \n",
"\n",
"\n",
"### Using `pip` for python packages\n",
"\n",
"Some python packages might not be availabe in the channel you are using, or in any `mamba` channel. If that package however is in `PyPI` (the official python package repository), one can use `pip` within a `mamba` environment to download packages. The recommended way is to specify this in the configuration file. An example of this configuration in the examples notebook is `env-with-pip`. In that example, we add `stackstac` as a dependency to install from `PyPI` because it is not available in the `conda-forge` channel. \n",
"*Example config file for this section [here](./example_conda_configuration_files/env-with-pip.yml).*\n",
"\n",
"Some python packages might not be availabe in the channel you are using, or in any `mamba` channel. If that package however is in `PyPI` (the official python package repository), one can use `pip` within a `mamba` environment to download packages. The recommended way is to specify this in the configuration file. In the linked example, we add `stackstac` as a dependency to install from `PyPI` because it is not available in the `conda-forge` channel. \n",
"\n",
"### Using custom environments in jupyter notebooks\n",
"\n",
"*Example config file for this section [here](./example_conda_configuration_files/env-with-ipykernel.yml).*\n",
"\n",
"The following instruction steps are for python kernels.\n",
"\n",
"- Make sure ipykernel is listed as a dependency in your configuration file. See the `env-with-ipykernel` example in the examples notebook.\n",
"- Create your environment using that configuration file.\n",
"- Make sure ipykernel is listed as a dependency in your configuration file.\n",
"- Create your environment using the linked configuration file.\n",
"- Install the environment as a kernel by running the following command (parameter values follow the example mentioned):\n",
" ```\n",
" python -m ipykernel install --user --name env-with-ipykernel --display-name \"Python env-with-ipykernel\".\n",
Expand All @@ -115,7 +134,9 @@
"\n",
"### Suggested packages for custom environment\n",
"\n",
"MAAP users typically use the python `maap-py`. It's pre-installed in all workspaces, in the `base` mamba environment, but any custom environment should specify it, otherwise it is not going to be accessible from that environment. However, `maap-py` is not packaged in a public package repository, like `PyPI` or `conda-forge`. It is possible to install it directly from its github repository with `pip` though. See the `env-with-maap-py` example in the examples notebook. You can note that inside of it `maap-py` is 'versioned' using a commit hash (at the end of the github URL). "
"*Example config file for this section [here](./example_conda_configuration_files/env-with-maap-py.yml)*\n",
"\n",
"MAAP users typically use the python `maap-py`. It's pre-installed in all workspaces, in the `base` mamba environment, but any custom environment should specify it, otherwise it is not going to be accessible from that environment. However, `maap-py` is not packaged in a public package repository, like `PyPI` or `conda-forge`. It is possible to install it directly from its github repository with `pip` though. See the configuration example linked. You can note that in the example, `maap-py` is 'versioned' using a commit hash (at the end of the github URL). "
]
}
],
Expand Down

This file was deleted.