Deploying to master from @ ZwickyTransientFacility/scope-ml@c0265a4 🚀

ZwickyTransientFacility · Apr 9, 2024 · 4f4342f · 4f4342f
commit 4f4342f
Show file tree

Hide file tree

Showing 156 changed files with 11,070 additions and 0 deletions.
diff --git a/.buildinfo b/.buildinfo
@@ -0,0 +1,4 @@
+# Sphinx build info version 1
+# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done.
+config: 7e76fb2d63321ec3c347e52311e09cea
+tags: 645f666f9bcd5a90fca523b33c5a78b7
diff --git a/.nojekyll b/.nojekyll
diff --git a/CNAME b/CNAME
@@ -0,0 +1 @@
+scope.ztf.dev
diff --git a/_images/beta_lyr.png b/_images/beta_lyr.png
diff --git a/_images/bogus_1.png b/_images/bogus_1.png
diff --git a/_images/bogus_2.png b/_images/bogus_2.png
diff --git a/_images/cepheid_1O.png b/_images/cepheid_1O.png
diff --git a/_images/cepheid_F.png b/_images/cepheid_F.png
diff --git a/_images/cepheid_F1O.png b/_images/cepheid_F1O.png
diff --git a/_images/cepheid_F_1.png b/_images/cepheid_F_1.png
diff --git a/_images/cv_Novalike.png b/_images/cv_Novalike.png
diff --git a/_images/cv_SU_UMa.png b/_images/cv_SU_UMa.png
diff --git a/_images/cv_U_Gem.png b/_images/cv_U_Gem.png
diff --git a/_images/cv_Z_Cam.png b/_images/cv_Z_Cam.png
diff --git a/_images/delta_scuti.png b/_images/delta_scuti.png
diff --git a/_images/flaring_dwarfnova.png b/_images/flaring_dwarfnova.png
diff --git a/_images/flaring_mdwarf.png b/_images/flaring_mdwarf.png
diff --git a/_images/hr__beta_lyr.png b/_images/hr__beta_lyr.png
diff --git a/_images/hr__cepheid.png b/_images/hr__cepheid.png
diff --git a/_images/hr__cv.png b/_images/hr__cv.png
diff --git a/_images/hr__delta_scuti.png b/_images/hr__delta_scuti.png
diff --git a/_images/hr__flaring.png b/_images/hr__flaring.png
diff --git a/_images/hr__lpv.png b/_images/hr__lpv.png
diff --git a/_images/hr__periodic.png b/_images/hr__periodic.png
diff --git a/_images/hr__rr_lyr.png b/_images/hr__rr_lyr.png
diff --git a/_images/hr__variable.png b/_images/hr__variable.png
diff --git a/_images/hr__w_uma.png b/_images/hr__w_uma.png
diff --git a/_images/mira.png b/_images/mira.png
diff --git a/_images/period__beta_lyr.png b/_images/period__beta_lyr.png
diff --git a/_images/period__cepheid.png b/_images/period__cepheid.png
diff --git a/_images/periodic.png b/_images/periodic.png
diff --git a/_images/radec__cepheid.png b/_images/radec__cepheid.png
diff --git a/_images/radec__delta_scuti.png b/_images/radec__delta_scuti.png
diff --git a/_images/radec__flaring.png b/_images/radec__flaring.png
diff --git a/_images/radec__lpv.png b/_images/radec__lpv.png
diff --git a/_images/radec__periodic.png b/_images/radec__periodic.png
diff --git a/_images/radec__rr_lyr.png b/_images/radec__rr_lyr.png
diff --git a/_images/radec__variable.png b/_images/radec__variable.png
diff --git a/_images/radec__w_uma.png b/_images/radec__w_uma.png
diff --git a/_images/rr_lyr_ab.png b/_images/rr_lyr_ab.png
diff --git a/_images/rr_lyr_c.png b/_images/rr_lyr_c.png
diff --git a/_images/srv.png b/_images/srv.png
diff --git a/_images/variable.png b/_images/variable.png
diff --git a/_images/w_uma.png b/_images/w_uma.png
diff --git a/_sources/allocation.md.txt b/_sources/allocation.md.txt
@@ -0,0 +1,25 @@
+# ACCESS allocation management
+
+This project (and others in the group) are supported in large part by computing resource allocations, e.g. from NSF ACCESS. When our group is granted ACCESS credits, we need to exchange them for the resources our group requires. This guide explains some details about that process.
+
+## Viewing Resources
+
+Users with the Allocation Manager role can view available allocations on the [ACCESS website](https://allocations.access-ci.org/). The different tabs show available resources, current users, and request history. The figure below shows the "Credits + Resouces" tab that is useful for viewing how much of each allocated resource remains.
+
+<img src="data/access_credit_view.png" alt="ACCESS Allocation UI" width="900"/>
+
+## Making Requests
+
+To request more resources, increase the associated "Balance" box on the right to the desired amount. Doing this will preview the exchange of credits for resources in the visualization of the balances. To submit the request, add a 1-2 sentence justification in the explanation box near the bottom of the page. This justification should mention what efforts the exchange will support and the basic reasons why the resources are being requested (at an approximate level). For example, the justification below accompanied a request for 5,000 Expanse GPUh:
+
+*We will continue to perform GPU-accelerated time series feature generation on 20 more ZTF fields containing ~56 million light curves. At ~0.25s per lightcurve, and considering the added memory requirements of some fields leading to greater credit expenditure, we request an additional 5,000 Expanse GPUh.*
+
+Note that it is always possible to turn unused resources back into the equivalent amount of credits. Use the [Exchange Calculator](https://allocations.access-ci.org/exchange_calculator) to calculate the exchange rates between credits and resources. From experience, ACCESS prefers not to allocate vast quantities of a single resource at once. Instead, more manageable requests should be made at regular intervals.
+
+## Anticipating Trends
+
+Clicking the chart buttons under the Usage column will show which users used how much of each resource. Over time, trends will tend to stabilize and make subsequent exchange requests more predictible. When managing the allocation, try to keep resources consistently available so projects do not get interrupted. If a resource is nearing exhaustion, Allocation Managers will receive an email from ACCESS with this information.
+
+### Optimizing resource usage
+
+If users seem to be using more resources than they need for their given project, reach out to ensure they are optimizing their slurm scripts. Many ACCESS resources have a basic unit that they consider to be 1 Service Unit (SU); for example, on SDSC Expanse 1 CPU SU = 1 core with 2GB of memory for one hour. Any additional memory used will scale up the CPU SU accounting charge by the ratio of the requested memory amount to 2 GB. 1 Expanse GPU SU is 1 GPU, <10CPUs, and <92G of memory. Additional GPUs/CPUs/memory beyond these levels will likewise scale up the GPU SUs charged to the group account.
diff --git a/_sources/developer.md.txt b/_sources/developer.md.txt
@@ -0,0 +1,218 @@
+# Installation/Developer Guidelines
+
+## Science users
+- Create and activate a virtual/conda environment with Python 3.11, e.g:
+  ```shell script
+  conda create -n scope-env python=3.11
+  conda activate scope-env
+  ```
+- Install the latest release of `scope-ml` from PyPI:
+  ```shell script
+  pip install scope-ml
+  ```
+- In the directory of your choice, run the initialization script. This will create the required directories and copy the necessary files to run the code:
+  ```shell script
+  scope-initialize
+  ```
+- Change directories to `scope` and modify `config.yaml` to finish the initialization process. This config file is used by default when running all scripts. You can also specify another config file using the `--config-path` argument.
+
+
+## Developers/contributors
+
+- Create your own fork the [scope repository](https://github.com/ZwickyTransientFacility/scope) by clicking the "fork" button. Then, decide whether you would like to use HTTPS (easier for beginners) or SSH.
+- Following one set of instructions below, clone (download) your copy of the repository, and set up a remote called `upstream` that points to the main `scope` repository.
+
+### HTTPS:
+
+```shell script
+git clone https://github.com/<yourname>/scope.git && cd scope
+git remote add upstream https://github.com/ZwickyTransientFacility/scope.git
+```
+
+### SSH:
+
+- [Set up SSH authentication with GitHub](https://help.github.com/en/github/authenticating-to-github/connecting-to-github-with-ssh).
+
+```shell script
+git clone git@github.com:<yourname>/scope.git && cd scope
+git remote add upstream git@github.com:ZwickyTransientFacility/scope.git
+```
+
+### Setting up your environment (Windows/Linux/macOS)
+
+#### Use a package manager for installation
+
+We currently recommend running `scope` with Python 3.11. You may want to begin your installation by creating/activating a virtual environment, for example using conda. We specifically recommend installing miniforge3 (https://github.com/conda-forge/miniforge).
+
+Once you have a package manager installed, run:
+
+```bash
+conda create -n scope-env -c conda-forge python=3.11
+conda activate scope-env
+```
+
+#### (Optional): Update your `PYTHONPATH`
+
+If you plan to import from `scope`, ensure that Python can import from `scope` by modifying the `PYTHONPATH` environment variable. Use a simple text editor like `nano` to modify the appropriate file (depending on which shell you are using). For example, if using bash, run `nano ~/.bash_profile` and add the following line:
+
+```bash
+export PYTHONPATH="$PYTHONPATH:$HOME/scope"
+```
+
+Save the updated file (`Ctrl+O` in `nano`) and close/reopen your terminal for this change to be recognized. Then `cd` back into scope and activate your `scope-env` again.
+
+### Install required packages
+
+Ensure you are in the `scope` directory that contains `pyproject.toml`. Then, install the required python packages by running:
+```bash
+pip install .
+```
+
+#### Install dev requirements, pre-commit hook
+
+We use `black` to format the code and `flake8` to verify that code complies with [PEP8](https://www.python.org/dev/peps/pep-0008/).
+Please install our dev requirements and pre-commit hook as follows:
+
+```shell script
+pip install -r dev-requirements.txt
+pre-commit install
+```
+
+This will check your changes before each commit to ensure that they
+conform with our code style standards. We use black to reformat Python
+code.
+
+The pre-commit hook will lint *changes* made to the source.
+
+#### Create and modify config.yaml
+
+From the included config.defaults.yaml, make a copy called config.yaml:
+
+```bash
+cp config.defaults.yaml config.yaml
+```
+
+Edit config.yaml to include Kowalski instance and Fritz tokens in the associated empty `token:` fields.
+
+#### Testing
+Run `scope-test` to test your installation. Note that for the test to pass, you will need access to the Kowalski database. If you do not have Kowalski access, you can run `scope-test-limited` to run a more limited (but still useful) set of tests.
+
+### Troubleshooting
+Upon encountering installation/testing errors, manually install the package in question using  `conda install xxx` , and remove it from `.requirements/dev.txt`. After that, re-run `pip install -r requirements.txt` to continue.
+
+#### Known issues
+- If using GPU-accelerated period-finding algorithms for feature generation, you will need to install [periodfind](https://github.com/ZwickyTransientFacility/periodfind) separately from the source.
+- Across all platforms, we are currently aware of `scope` dependency issues with Python 3.12.
+- Anaconda continues to cause problems with environment setup.
+- Using `pip` to install `healpy` on an arm64 Mac can raise an error upon import. We recommend including `h5py` as a requirement during the creation of your `conda` environment.
+- On Windows machines, `healpy` and `cesium` raise errors upon installation.
+   - For `healpy`, see [this](https://healpy.readthedocs.io/en/latest/install.html#installation-on-windows-through-the-windows-subsystem-for-linux) guide for a potential workaround.
+   - For `cesium`, try to install from the source (https://cesium-ml.org/docs/install.html#from-source) within `scope`. If you will not be running feature generation, this is not a critical error, but there will be points in the code that fail (e.g. `scope.py test`, `tools/generate_features.py`)
+
+If the installation continues to raise errors, update the conda environment and try again.
+
+### How to contribute
+
+Contributions to `scope` are made through [GitHub Pull Requests](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/about-pull-requests), a set of proposed commits (or patches):
+
+1. Download the latest version of `scope`, and create a new branch for your work.
+
+   Here, let's say we want to contribute some documentation fixes; we'll call our branch `rewrite-contributor-guide`.
+
+   ```shell script
+   git checkout main
+   git pull upstream main
+   git checkout -b rewrite-contributor-guide
+   ```
+
+2. Make modifications to `scope` and commit your changes using `git add` and `git commit`.
+Each commit message should consist of a summary line and a longer description, e.g.:
+
+   ```text
+   Rewrite the contributor guide
+   While reading through the contributor guide, I noticed several places
+   in which instructions were out of order. I therefore reorganized all
+   sections to follow logically, and fixed several grammar mistakes along
+   the way.
+   ```
+
+1. When ready, push your branch to GitHub:
+
+   ```shell script
+   git push origin rewrite-contributor-guide
+   ```
+
+   Once the branch is uploaded, GitHub should print a URL for turning your branch into a pull request.
+   Open that URL in your browser, write an informative title and description for your pull request, and submit it.
+
+2. The team will now review your contribution, and suggest changes.
+*To simplify review, please limit pull requests to one logical set of changes.*
+To incorporate changes recommended by the reviewers, commit edits to your branch, and push to the branch again
+(there is no need to re-create the pull request, it will automatically track modifications to your branch).
+
+1. Sometimes, while you were working on your feature, the `main` branch is updated with new commits, potentially
+resulting in conflicts with your feature branch. The are two ways to resolve this situation - merging and rebasing,
+please look [here](https://www.atlassian.com/git/tutorials/merging-vs-rebasing) for a detailed discussion.
+While both ways are acceptable, since we are squashing commits from a PR before merging, we prefer the first option:
+
+    ```shell script
+    git merge rewrite-contributor-guide upstream/main
+    ```
+Developers may merge `main` into their branch as many times as they want to.
+
+1. Once the pull request has been reviewed and approved by at least one team member, it will be merged into `scope`.
+
+#### Releasing on PyPI
+
+As new features are added to the code, maintainers should occasionally initiate a new release of the `scope-ml` [PyPI](https://pypi.org/project/scope-ml/) package. To do this, first bump the version of the package in `pyproject.toml` and `scope/__init__.py` to the intended `vX.Y.Z` format. Then, navigate to "Releases" in the SCoPe GitHub repo and click "Draft a new release". Enter the version number in "Choose a tag" and click "Generate release notes". It is also advisable to check the box creating a discussion for the release before clicking "Publish release".
+
+Upon release, the `publish-to-pypi.yml` workflow will use GitHub Actions to publish a new version of the package to PyPI automatically. **Note that if the version number has not yet been manually updated in `pyproject.toml`, this workflow will fail.**
+
+### Contributing Field Guide sections
+
+If you would like to contribute a Field Guide section, please follow the steps below.
+
+- Make sure to follow the steps described above in the "How to contribute" section!
+
+- Add sections to `config.defaults.yaml` under `docs.field_guide.<object_class_type>`.
+  - Use `docs.field_guide.rr_lyr_ab` as an example. You need to specify the object's
+    coordinates and a title for the generated light curve plot. Optionally,
+    you may specify a period [days] - then a phase-folded light curve will also be rendered.
+
+- Make sure your `config.yaml` file contains a valid Kowalski token.
+  - See [here](https://github.com/dmitryduev/penquins) on how to generate one
+  (Kowalski account required).
+  - You can use `config.defaults.yaml` as a template.
+
+- Make sure the structure of your config file is the same as the default,
+  i.e. you propagated the changes in `config.defaults.yaml`.
+  (otherwise the `scope.py` utility will later complain and ask you to fix that).
+
+- Add a Markdown file to `doc/` and call it `field_guide__<object_class>.md`.
+  - Use [`doc/field_guide__rr_lyrae.md`](field_guide__rr_lyrae.md) as a template.
+  - Light curve examples will be generated as `data/<object_class_type>.png` files using the info
+    you provided in the config.
+  - Add the following include statements to [`doc/field_guide.md`](field_guide.md):
+
+```
+{include} ./field_guide__<object_class>.md
+```
+
+- If you wish to render a sample Gaia-based HR diagram, you need to create a "Golden" data set
+  for that class of objects and put it under `data/golden` as `<object_class>.csv`
+  - The `csv` file must follow the same structure as [`data/golden/rr_lyr.csv`].
+   Please keep the `csv` header ("ra,dec") and provide object coordinates in degrees.
+  - The HR diagram will be generated as `data/hr__<object_class>.png`
+
+- Run the `./scope.py doc` command to generate the imagery and build the documentation.
+  - If the build is successful, you can check the results in
+    [`doc/_build/html/index.html`](_build/html/index.html)
+
+- Once you're happy with the result, commit the changes to a branch on your fork
+  and open a pull request on GitHub (see the "How to contribute" section above).
+  - The GitHub Actions CI will run a subset of the testing/deployment pipeline
+    for each commit you make to your branch -
+    make sure you get a green checkmark next to the commit hash.
+  - Once the PR is reviewed, approved, and merged,
+    the CI will automatically build and deploy the docs to
+    [`https://scope.ztf.dev`](https://scope.ztf.dev)
diff --git a/_sources/field_guide.md.txt b/_sources/field_guide.md.txt
@@ -0,0 +1,40 @@
+# Field guide
+
+This guide provides detailed information about the different types of variable sources along with examples of bogus variability.
+
+Proceed <a href="_static/taxonomy.html">here</a> to interactively inspect the taxonomy
+tree we are employing in SCoPe. Please refer to [arXiv:2102.11304](https://arxiv.org/pdf/2102.11304.pdf)
+for more details on the taxonomy.
+
+```{include} ./field_guide__variable.md
+```
+
+```{include} ./field_guide__periodic.md
+```
+
+```{include} ./field_guide__rr_lyrae.md
+```
+
+```{include} ./field_guide__w_uma.md
+```
+
+```{include} ./field_guide__delta_scuti.md
+```
+
+```{include} ./field_guide__cepheid.md
+```
+
+```{include} ./field_guide__CVs.md
+```
+
+```{include} ./field_guide__flaring.md
+```
+
+```{include} ./field_guide__beta_lyr.md
+```
+
+```{include} ./field_guide__lpv.md
+```
+
+```{include} ./field_guide__bogus.md
+```
diff --git a/_sources/field_guide__CVs.md.txt b/_sources/field_guide__CVs.md.txt
@@ -0,0 +1,55 @@
+## Cataclysmic Variables (cv)
+Cataclysmic variables are close binaries with active mass transfer from a late
+type main sequence star or brown dwarf to a white dwarf. There are several different
+types of CVs, depending on their causes of variability. These include novae (which
+have 9-15 mag outbursts from thermonuclear events on the white dwarf surface), dwarf novae which have 2-9 mag outbursts on weeks to decades timescales due to disk instabilities) and novalikes which do not have outbursts but have high and low states of accretion which cause several magnitudes of brightness change.
+
+### Classification and numbers
+- Supertypes
+  - variable
+  - aperiodic outbursts
+  - aperiodic high and low states
+  - periodic orbital variability
+- Subtypes
+  - Nova
+  - U Gem dwarf nova
+  - Z Cam dwarf nova
+  - SU UMa/WZ Sge dwarf nova
+  - Novalike
+
+- Occurrence rate: common, several thousand expected in ZTF data
+
+
+### ZTF light curves
+![ZTF CV U Gem](data/cv_U_Gem.png)
+![ZTF CV Z Cam](data/cv_Z_Cam.png)
+![ZTF CV SU UMa](data/cv_SU_UMa.png)
+![ZTF CV Novalike](data/cv_Novalike.png)
+
+#### Description
+CVs are easy to recognize by their distinctive light curve shape, colors and
+high amplitude variability. The outbursts are non-periodic.
+They can sometimes be confused with supernovae, flare stars or long period variables.
+
+#### Light curve characteristics
+- non-periodic variable but recur on some timescale
+- outburst timescale range: days to decades
+- amplitude: 9-15 mag (nova), 2-9 mag (dwarf nova), 1-4 mag (novalike)
+- light curve shape:
+    - sawtooth; steep rise and slow decay (subtype nova)
+    - symmetrical to slightly sawtooth (subtype U Gem)
+    - standstills about 1 mag below outburst level for weeks (subtype Z Cam)
+    - fast rise and extended plateau for 1-2 weeks followed by steeper decline     (subtype SU UMa/WZ Sge)
+    - extended (days-weeks) states at either high or low brightness
+- can show periodic modulation of the light curve on orbital timescales of hours
+
+
+#### Other characteristics and selection methods
+- intrinsic CV colors: blue, (g-r < 0.6).
+  Reddening is usually not important except for places in the galactic plane.
+- absolute magnitude at quiescence: 8<G<14
+
+![HR diagram of CVs](data/hr__cv.png)
+
+### References and further reading:
+- Warner 1995: Cataclysmic Variable Stars
diff --git a/_sources/field_guide__beta_lyr.md.txt b/_sources/field_guide__beta_lyr.md.txt
@@ -0,0 +1,30 @@
+## Beta Lyr (blyr)
+
+Semi-detached eclipsing binaries (also called [beta Lyrae variables](https://en.wikipedia.org/wiki/Beta_Lyrae_variable)) are binary star systems (i.e. two stars that are in orbit around each other) that are aligned such that they eclipse each other relative to our line of sight from the Earth. Unlike detached EBs, semi-detatched systems are so close to each other that the shape of (at least one of) the stars in the system are distorted, giving rise to the smooth periodic variations even when the system is not eclipsing.
+
+### Classification and numbers
+- Supertypes
+  - variable
+  - periodic
+  - binary
+  - eclipsing
+- Occurrence rate: very common, about 10<sup>5</sup> expected in ZTF data
+
+### ZTF light curves
+![ZTF betalyr](data/beta_lyr.png)
+
+#### Description
+Any type of star can be in an eclipsing binary (so they may be small, large, hot, cool, etc.), which in turn can lead to a large range in eclipse amplitudes (though typically these are between 0.1 and ~1 mag, with most in the ~0.3 to 0.75 mag range) or eclipse periods (though our observations are typically only sensitive to periods between 0.1 to ~20 days).
+
+#### Light curve characteristics
+- periodic variable
+- Range of amplitudes (~0.2 to >1 mag)
+- Intermediate periods (log Period between -0.5 and 1.4)
+- light curve shape: EB, round or sinusoidal light curves, with imposed "V-shape" dips in the phase folded light curve
+
+![HR diagram of Beta Lyrae](data/hr__beta_lyr.png)
+
+![Period histogram of Beta Lyrae](data/period__beta_lyr.png)
+
+### References and further reading:
+- Sterken & Jasschek: Light curves of variable stars