-
Notifications
You must be signed in to change notification settings - Fork 43
Dependency Updates
This wiki page describes how to update the JWQL repository with dependency updates.
A general description of how to do a dependency update for JWQL.
On your local machine:
- Using the environment file for the version of the environment currently running on the JWQL servers, create a copy of the environment file and remove the pinned version numbers for everything except python (and any dependencies that can't be updated to newer versions). Pin the python sub-version to be the latest release for the the python version. (e.g. python=3.9.17 in the python 3.9 env file) Do the same for requirements.txt. Then use those files to create a new environment using:
conda env create -n test -f <your_blank_env_file.yml>
conda activate test
Note that in the final step of creating the environment, jwql will be installed. As part of this, any packages listed in pyproject.toml will be installed via pip. In most cases, these packages should already have been installed from the list of dependencies in the environment file. But pay attention. nodejs was a problem because only version 0.1.1 is available via pip, while much more recent versions are only available on conda. So when nodejs was in pyproject.toml, it was installed via pip while jwql was being installed. This caused the recent (conda-derived) version to be uninstalled, and the old 0.1.1 version to be installed in its place.
-
Use "conda list" to list the versions of all packages installed in the new environment. Populate the environment file and requirements.txt with these versions.
-
Repeat steps 1 and 2 above on the dev server. Compare the resulting package versions between the local machine and the server. If any are different, you'll have to specify an upper or lower bound for the package version so that the environment file works for both server and local installations.
-
On the dev server, run all of the unit tests and make sure they pass. You can ignore or not run anything in
test_api_viewsas the SSO layer between the user and the app will cause these tests to fail. -
Repeat the process for any other environment files in the repo. If there are any differences between the package versions across the supported python versions, make sure requirements.txt has versions that match the environment file for the version of python running on the servers. (i.e. JWQL currently supports python 3.9 and 3.10. But the servers are running the 3.9 environment, so requirements.txt should agree with the package versions in the 3.9 environment file.)
-
In the case where you end up replacing (rather than just updating) the environment on the servers, note that you will have to manually install gunicorn on the servers. It is necessary there, but not when running locally, so we do not put it in the environment files.
On the first day of each month, the pyup tool automatically opens up a new pull request in the jwql repository with suggested dependency updates. Since pyup only works on requirements.txt files and not conda environment files (see the currently open GitHub issue about this), someone must manually go into that branch/pull request and update the appropriate environment files. The files of interest are:
environment_python_3.9.ymlenvironment_python_3.10.ymlrequirements.txt
Ultimately, the goal of this exercise is to (1) make sure the requirements.txt and environment_python_<version>.yml files are all in sync with each other (i.e. all have the same pinned version numbers) and (2) make sure the version numbers reflect that of the latest version available in conda.
Below is a step-by-step procedure for processing dependencies:
The pull request will be named something like "Scheduled monthly dependency update for <month>". Open up the pull request on GitHub. Add yourself as an assignee. Add the GitHub label for "environment".
The pyup tool opens the PR on a branch called pyup/scheduled-update-YYYY-MM-DD. Grab this branch in your local clone of the jwql repository with:
-
git fetch upstream(This assumes you have a remote calledupstreampointing tohttps://github.com/spacetelescope/jwql.git) git checkout -b pyup/scheduled-update-YYYY-MM-DD upstream/pyup/scheduled-update-YYYY-MM-DD
Check which dependency updates are actually updated in conda, update the conda environment files/requirements.txt file as necessary
Some updates that pyup suggests may not yet be available in conda yet, and thus should not be updated in the environment/requirements files. To check this, initialize the jwql-3.9 conda environment (this assumes you have done something like conda env create --file=environment_python_3.9.yml), and try to conda install <dependency> for each suggested dependency. If conda does indeed install an update, make the appropriate update to the corresponding environment_python_3.9.yml file. If conda cannot find the updated dependency, do not make any changes to the conda environment files, and revert the dependency in the requirements.txt file back to what it was before.
Repeat this process for the Python 3.10 environment too.
Once you have finished identifying all of the dependency updates and have updated the conda environment files and requirements.txt file, commit those changes to the branch, and push the branch back to upstream using e.g.:
-
git push upstream pyup/scheduled-update-YYYY-MM-DDThis will automatically update the pull request.
Updating the pull request will trigger new GitHub action builds. Make sure that the changes you (and pyup) have made to the environment/requirement files result in a successful build of the GitHub Actions CI workflow. If the build fails, make any appropriate changes to the files until they pass.
Once the GitHub Actions builds pass, the PR is ready to be reviewed. Assign a reviewer. Once the reviewer has approved, or you have iterated with the reviewer with any necessary changes, the PR can be merged.