Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for Python 3.12, pin proj back to 9.3.x, require geopandas>=1.0 #1453

Merged
merged 8 commits into from
Sep 9, 2024
Merged

Add support for Python 3.12, pin proj back to 9.3.x, require geopandas>=1.0 #1453

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
89617dd
Add support for Python 3.12.
jameslamb Sep 4, 2024
9bb0ae5
Merge branch 'branch-24.10' into python-3.12
jameslamb Sep 4, 2024
4361f9e
Merge branch 'branch-24.10' into python-3.12
bdice Sep 6, 2024
0e25119
Pin proj to >=9.3.0,<9.3.1.0a0
bdice Sep 7, 2024
1a5cd18
Pin more instances of proj.
bdice Sep 7, 2024
eaac568
Update branches for new C++ wheels workflows.
bdice Sep 9, 2024
9295c3f
bump geopandas to 1.x, drop fiona
jameslamb Sep 9, 2024
b4d51cf
Merge branch 'branch-24.10' into python-3.12
jameslamb Sep 9, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 10 additions & 10 deletions .github/workflows/build.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ concurrency:
jobs:
cpp-build:
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-build.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-build.yaml@python-3.12
with:
build_type: ${{ inputs.build_type || 'branch' }}
branch: ${{ inputs.branch }}
Expand All @@ -37,7 +37,7 @@ jobs:
python-build:
needs: [cpp-build]
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/conda-python-build.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/conda-python-build.yaml@python-3.12
with:
build_type: ${{ inputs.build_type || 'branch' }}
branch: ${{ inputs.branch }}
Expand All @@ -46,7 +46,7 @@ jobs:
upload-conda:
needs: [cpp-build, python-build]
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/conda-upload-packages.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/conda-upload-packages.yaml@python-3.12
with:
build_type: ${{ inputs.build_type || 'branch' }}
branch: ${{ inputs.branch }}
Expand All @@ -56,7 +56,7 @@ jobs:
if: github.ref_type == 'branch'
needs: python-build
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@python-3.12
with:
arch: "amd64"
branch: ${{ inputs.branch }}
Expand All @@ -68,7 +68,7 @@ jobs:
sha: ${{ inputs.sha }}
wheel-build-libcuspatial:
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@python-3.12
with:
# build for every combination of arch and CUDA version, but only for the latest Python
matrix_filter: group_by([.ARCH, (.CUDA_VER|split(".")|map(tonumber)|.[0])]) | map(max_by(.PY_VER|split(".")|map(tonumber)))
Expand All @@ -80,7 +80,7 @@ jobs:
wheel-publish-libcuspatial:
needs: wheel-build-libcuspatial
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-publish.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/wheels-publish.yaml@python-3.12
with:
build_type: ${{ inputs.build_type || 'branch' }}
branch: ${{ inputs.branch }}
Expand All @@ -91,7 +91,7 @@ jobs:
wheel-build-cuspatial:
needs: wheel-publish-libcuspatial
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@python-3.12
with:
build_type: ${{ inputs.build_type || 'branch' }}
branch: ${{ inputs.branch }}
Expand All @@ -101,7 +101,7 @@ jobs:
wheel-publish-cuspatial:
needs: wheel-build-cuspatial
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-publish.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/wheels-publish.yaml@python-3.12
with:
build_type: ${{ inputs.build_type || 'branch' }}
branch: ${{ inputs.branch }}
Expand All @@ -111,7 +111,7 @@ jobs:
package-type: python
wheel-build-cuproj:
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@python-3.12
with:
build_type: ${{ inputs.build_type || 'branch' }}
branch: ${{ inputs.branch }}
Expand All @@ -121,7 +121,7 @@ jobs:
wheel-publish-cuproj:
needs: wheel-build-cuproj
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-publish.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/wheels-publish.yaml@python-3.12
with:
build_type: ${{ inputs.build_type || 'branch' }}
branch: ${{ inputs.branch }}
Expand Down
30 changes: 15 additions & 15 deletions .github/workflows/pr.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,47 +27,47 @@ jobs:
- wheel-tests-cuproj
- devcontainer
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/pr-builder.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/pr-builder.yaml@python-3.12
checks:
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/checks.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/checks.yaml@python-3.12
with:
enable_check_generated_files: false
conda-cpp-build:
needs: checks
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-build.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-build.yaml@python-3.12
with:
build_type: pull-request
conda-cpp-checks:
needs: conda-cpp-build
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-post-build-checks.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-post-build-checks.yaml@python-3.12
with:
build_type: pull-request
enable_check_symbols: true
conda-cpp-tests:
needs: conda-cpp-build
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-tests.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-tests.yaml@python-3.12
with:
build_type: pull-request
conda-python-build:
needs: conda-cpp-build
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/conda-python-build.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/conda-python-build.yaml@python-3.12
with:
build_type: pull-request
conda-python-tests:
needs: conda-python-build
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/conda-python-tests.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/conda-python-tests.yaml@python-3.12
with:
build_type: pull-request
conda-notebook-tests:
needs: conda-python-build
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@python-3.12
with:
build_type: pull-request
node_type: "gpu-v100-latest-1"
Expand All @@ -77,7 +77,7 @@ jobs:
docs-build:
needs: conda-python-build
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@python-3.12
with:
build_type: pull-request
node_type: "gpu-v100-latest-1"
Expand All @@ -87,7 +87,7 @@ jobs:
wheel-build-libcuspatial:
needs: checks
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@python-3.12
with:
# build for every combination of arch and CUDA version, but only for the latest Python
matrix_filter: group_by([.ARCH, (.CUDA_VER|split(".")|map(tonumber)|.[0])]) | map(max_by(.PY_VER|split(".")|map(tonumber)))
Expand All @@ -96,34 +96,34 @@ jobs:
wheel-build-cuspatial:
needs: [checks, wheel-build-libcuspatial]
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@python-3.12
with:
build_type: pull-request
script: ci/build_wheel_cuspatial.sh
wheel-tests-cuspatial:
needs: wheel-build-cuspatial
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@python-3.12
with:
build_type: pull-request
script: ci/test_wheel_cuspatial.sh
wheel-build-cuproj:
needs: checks
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@python-3.12
with:
build_type: pull-request
script: ci/build_wheel_cuproj.sh
wheel-tests-cuproj:
needs: [wheel-build-cuspatial, wheel-build-cuproj]
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@python-3.12
with:
build_type: pull-request
script: ci/test_wheel_cuproj.sh
devcontainer:
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/build-in-devcontainer.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/build-in-devcontainer.yaml@python-3.12
with:
arch: '["amd64"]'
cuda: '["12.5"]'
Expand Down
10 changes: 5 additions & 5 deletions .github/workflows/test.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ on:
jobs:
conda-cpp-checks:
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-post-build-checks.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-post-build-checks.yaml@python-3.12
with:
build_type: nightly
branch: ${{ inputs.branch }}
Expand All @@ -25,23 +25,23 @@ jobs:
enable_check_symbols: true
conda-cpp-tests:
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-tests.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-tests.yaml@python-3.12
with:
build_type: nightly
branch: ${{ inputs.branch }}
date: ${{ inputs.date }}
sha: ${{ inputs.sha }}
conda-python-tests:
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/conda-python-tests.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/conda-python-tests.yaml@python-3.12
with:
build_type: nightly
branch: ${{ inputs.branch }}
date: ${{ inputs.date }}
sha: ${{ inputs.sha }}
wheel-tests-cuspatial:
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@python-3.12
with:
build_type: nightly
branch: ${{ inputs.branch }}
Expand All @@ -50,7 +50,7 @@ jobs:
script: ci/test_wheel_cuspatial.sh
wheel-tests-cuproj:
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@python-3.12
with:
build_type: nightly
branch: ${{ inputs.branch }}
Expand Down
42 changes: 3 additions & 39 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -119,13 +119,13 @@ docker run --gpus all --pull always --rm -it \
### Install with Conda

To install via conda:
> **Note** cuSpatial is supported only on Linux or [through WSL](https://rapids.ai/wsl2.html), and with Python versions 3.10 and 3.11.
> **Note** cuSpatial is supported only on Linux or [through WSL](https://rapids.ai/wsl2.html), and with Python versions 3.10, 3.11, and 3.12.

cuSpatial can be installed with conda (miniconda, or the full Anaconda distribution) from the rapidsai channel:

```shell
conda install -c rapidsai -c conda-forge -c nvidia \
cuspatial=24.10 python=3.11 cudatoolkit=11.8
cuspatial=24.10 python=3.12 cudatoolkit=11.8
```
We also provide nightly Conda packages built from the HEAD of our latest development branch.

Expand All @@ -134,7 +134,7 @@ See the [RAPIDS installation documentation](https://docs.rapids.ai/install) for
### Install with pip

To install via pip:
> **Note** cuSpatial is supported only on Linux or [through WSL](https://rapids.ai/wsl2.html), and with Python versions 3.10 and 3.11.
> **Note** cuSpatial is supported only on Linux or [through WSL](https://rapids.ai/wsl2.html), and with Python versions 3.10, 3.11, and 3.12.

The cuSpatial pip packages can be installed from NVIDIA's PyPI index. pip installations require using the matching wheel to the system's installed CUDA toolkit.
- For CUDA 11 toolkits, install the `-cu11` wheels
Expand All @@ -145,42 +145,6 @@ pip install cuspatial-cu12 --extra-index-url=https://pypi.nvidia.com
pip install cuspatial-cu11 --extra-index-url=https://pypi.nvidia.com
```

#### Troubleshooting Fiona/GDAL versions
Copy link
Member Author

@jameslamb jameslamb Sep 9, 2024 •
edited by bdice
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Starting a threaded conversation for this one.

Proposal

pin to geopandas >= 1.0.0 so we can drop this project's reliance on fiona entirely

commit: 9295c3f

Benefits

No more need to install specific versions of fiona based on the compatible GDAL version provided by your operating system's package manager (see these deleted notes in the README).

Risks

geopandas==1.0.0 is only a few months old... it was released in June 2024, so it's only been out about 3 months (https://pypi.org/project/geopandas/1.0.0/). Being so new, it might have other dependency constraints that make it harder for users on older systems to use cuspatial.

Notes

In practice, most of this project's tests have been running against geopandas>=1.0.0 for a few months now. Choose any random successful pr workflow from the last 3 months at https://github.com/rapidsai/cuspatial/actions/workflows/pr.yaml?query=is%3Asuccess and look in the wheel-tests-* or conda-python-tests* jobs. You'll see geopandas>=1.0.0 getting pulled in. For example, here's one from July 8th (chosen randomly): https://github.com/rapidsai/cuspatial/actions/runs/9850995522/job/27197475692#step:7:348

more context on why this is coming up now as part of the Python 3.12 work: #1453 (comment)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(cc @mroeschke )

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't have the strongest opinion on geopandas support, but given your breakdown, +1 for requiring >= 1.0.0

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks very much!

Ok, I'm going to merge this as-is then. @harrism @-ing you for awareness... if you have any concerns about this, let me know and I'd be happy to address them in a follow-up PR.


cuSpatial depends on [`geopandas`](https://github.com/geopandas/geopandas), which uses [`fiona >= 1.8.19`](https://pypi.org/project/Fiona/), to read common GIS formats with GDAL.

Fiona requires GDAL is already present on your system, but its minimum required version may be newer than the version of GDAL in your OS's package manager.

Fiona checks the GDAL version at install time and fails with an error like this if a compatible version of GDAL isn't installed:
```
ERROR: GDAL >= 3.2 is required for fiona. Please upgrade GDAL.
```

There are two ways to fix this:

1. Install a version of GDAL that meets fiona's minimum required version
* Ubuntu users can install a newer GDAL with the [UbuntuGIS PPA](https://wiki.ubuntu.com/UbuntuGIS):
```shell
sudo -y add-apt-repository ppa:ubuntugis/ppa
sudo apt install libgdal-dev
```
* Rocky Linux users can install a newer GDAL as follows
```shell
yum update -y
yum config-manager --set-enabled powertools
yum update -y
yum install -y gdal-devel
```
2. Pin fiona's version to a range that's compatible with your version of `libgdal-dev`
* For Ubuntu20.04 ([GDAL v3.0.4](https://packages.ubuntu.com/focal/libgdal-dev)):
```shell
pip install --no-binary fiona --extra-index-url=https://pypi.nvidia.com cuspatial-cu12 'fiona>=1.8.19,<1.9'
```
* For Ubuntu22.04 ([GDAL v3.4.1](https://packages.ubuntu.com/jammy/libgdal-dev)):
```shell
pip install --no-binary fiona --extra-index-url=https://pypi.nvidia.com cuspatial-cu12 'fiona>=1.9'
```

### Build/Install from source

To build and install cuSpatial from source please see the [build documentation](https://docs.rapids.ai/api/cuspatial/stable/developer_guide/build.html).
Expand Down
16 changes: 1 addition & 15 deletions ci/test_wheel_cuproj.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,30 +6,16 @@ set -eou pipefail
mkdir -p ./dist
RAPIDS_PY_CUDA_SUFFIX="$(rapids-wheel-ctk-name-gen ${RAPIDS_CUDA_VERSION})"

# install build dependencies for fiona
if type -f yum > /dev/null 2>&1; then
yum update -y
# some of gdal-devel's dependencies, like 'libdap', come from the powertools repo
yum config-manager --set-enabled powertools
yum update -y
yum install -y gdal-devel
else
apt update
DEBIAN_FRONTEND=noninteractive apt install -y --no-install-recommends libgdal-dev
fi

# Download the cuproj and cuspatial built in the previous step
RAPIDS_PY_WHEEL_NAME="cuproj_${RAPIDS_PY_CUDA_SUFFIX}" rapids-download-wheels-from-s3 python ./dist
RAPIDS_PY_WHEEL_NAME="cuspatial_${RAPIDS_PY_CUDA_SUFFIX}" rapids-download-wheels-from-s3 python ./dist
RAPIDS_PY_WHEEL_NAME="libcuspatial_${RAPIDS_PY_CUDA_SUFFIX}" rapids-download-wheels-from-s3 cpp ./dist

# echo to expand wildcard before adding `[extra]` requires for pip
python -m pip install \
--no-binary 'fiona' \
"$(echo ./dist/cuspatial*.whl)" \
"$(echo ./dist/cuproj*.whl)[test]" \
"$(echo ./dist/libcuspatial*.whl)" \
'fiona>=1.8.19,<1.9'
"$(echo ./dist/libcuspatial*.whl)"

rapids-logger "pytest cuproj"
pushd python/cuproj/cuproj
Expand Down
16 changes: 1 addition & 15 deletions ci/test_wheel_cuspatial.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,28 +6,14 @@ set -eou pipefail
mkdir -p ./dist
RAPIDS_PY_CUDA_SUFFIX="$(rapids-wheel-ctk-name-gen ${RAPIDS_CUDA_VERSION})"

# install build dependencies for fiona
if type -f yum > /dev/null 2>&1; then
yum update -y
# some of gdal-devel's dependencies, like 'libdap', come from the powertools repo
yum config-manager --set-enabled powertools
yum update -y
yum install -y gdal-devel
else
apt update
DEBIAN_FRONTEND=noninteractive apt install -y --no-install-recommends libgdal-dev
fi

# Download the cuspatial and libcuspatial built in the previous step
RAPIDS_PY_WHEEL_NAME="cuspatial_${RAPIDS_PY_CUDA_SUFFIX}" rapids-download-wheels-from-s3 python ./dist
RAPIDS_PY_WHEEL_NAME="libcuspatial_${RAPIDS_PY_CUDA_SUFFIX}" rapids-download-wheels-from-s3 cpp ./dist

# echo to expand wildcard before adding `[extra]` requires for pip
python -m pip install \
--no-binary 'fiona' \
"$(echo ./dist/cuspatial*.whl)[test]" \
"$(echo ./dist/libcuspatial*.whl)" \
'fiona>=1.8.19,<1.9'
"$(echo ./dist/libcuspatial*.whl)"

rapids-logger "pytest cuspatial"
pushd python/cuspatial/cuspatial
Expand Down
Loading
Loading