[RELEASE] cucim v21.12 #165
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
[gpuCI] Forward-merge branch-21.10 to branch-21.12 [skip gpuci]
[gpuCI] Forward-merge branch-21.10 to branch-21.12 [skip gpuci]
[gpuCI] Forward-merge branch-21.10 to branch-21.12 [skip gpuci]
[gpuCI] Forward-merge branch-21.10 to branch-21.12 [skip gpuci]
We do not need anymore to force install `linux-64` version of `openslide-python` as `noarch` packages are not available anymore. This will help to add `arm64` support cc @gigony @jakirkham Authors: - Jordan Jacobelli (https://github.com/Ethyling) Approvers: - https://github.com/jakirkham - Gigon Bae (https://github.com/gigony) URL: #124
- Update Jupyter notebook to include the correct GDS link -- <https://developer.nvidia.com/gpudirect-storage> Authors: - Gigon Bae (https://github.com/gigony) Approvers: - Benjamin Zaitlen (https://github.com/quasiben) - https://github.com/jakirkham URL: #131
Follow-up change for #131 (comment) . `branch-${CURRENT_MAJOR}.${CURRENT_MINOR}` => `branch-${NEXT_MAJOR}.${NEXT_MINOR}` in README.md Authors: - Gigon Bae (https://github.com/gigony) Approvers: - AJ Schmidt (https://github.com/ajschmidt8) URL: #132
Update Jupyter notebook that shows the interoperability with DALI, to use the recommended functional APIs (object-oriented API are deprecated). Existing notebook: - https://github.com/rapidsai/cucim/blob/branch-21.12/notebooks/Working_with_DALI.ipynb Updated notebook: - https://nbviewer.org/github/gigony/cucim/blob/update_dali_notebook/notebooks/Working_with_DALI.ipynb Authors: - Gigon Bae (https://github.com/gigony) Approvers: - Benjamin Zaitlen (https://github.com/quasiben) URL: #127
Fixes #133 Authors: - Benjamin Zaitlen (https://github.com/quasiben) Approvers: - https://github.com/jakirkham URL: #138
Update existing unit test code to use generated test image and allow external image specified by the parameter. Updating `run` script to pre-generate the necessary test images and running C++ unit tests would be set up in a separate PR. Authors: - Gigon Bae (https://github.com/gigony) Approvers: - Benjamin Zaitlen (https://github.com/quasiben) - https://github.com/jakirkham URL: #128
This change adds a skeleton plugin named `cucim.kit.cumed` for medical-related file loaders and image operations (e.g., .mhd file format support). The image loader to use is determined by `is_valid()` method implemented in each plugin and CuImage object holds the reference to the image format descriptor (`cucim::io::format::ImageFormatDesc*`). cuCIM treats `cucim.kit.cuslide` and `cucim.kit.cumed` plugins as built-in plugins, and it would support custom plugins in the future (by refactoring the framework code, according to the recent update of Carbonite (minimal) SDK library and use the library as it is). Authors: - Gigon Bae (https://github.com/gigony) Approvers: - AJ Schmidt (https://github.com/ajschmidt8) - Benjamin Zaitlen (https://github.com/quasiben) - https://github.com/jakirkham URL: #129
Instead of letting `host`'s `numpy` have a minimum, which could float to a high version that is in turn pinned to a high version in `run`, build using the minimum required `numpy` version. That way the minimum is set similarly in `host` and a wide range of `numpy`'s will still work at runtime.
Fix Changelog Merge Conflicts for `branch-21.12`
* Fix a segmentation fault due to libcufile's atexit Do not unload the cufile (GDS) library as libcufile registers a cleanup function with atexit() and unloading the library will cause a segfault (calling the cleanup function that doesn't exist anymore). - Leave the libcufile library loaded * Move CuFileStub declaration to header
**Add tif_lzw.c from libtiff as it is** Copy tif_lzw.c file from libtiff for porting lzw decoder. The code is from the following link: https://gitlab.com/libtiff/libtiff/-/blob/8546f7ee994eacff0a563918096f16e0a6078fa2/libtiff/tif_lzw.c , which is after v4.3.0 **Support Aperio SVS with CPU LZW and jpeg2k decoder** - Update Jpeg decoder to consider colorspace of the JPEG image - We need to set a proper color space from TIFF metadata to the decoder - Update tjDecompress2() method call with jpeg_decode_buffer() that modified tjDecompress2() implementation - Add `jpeg_color_space_` to IFD Class. - Move additional includes to their own CMake files - Add Jpeg2k decoder with OpenJpeg - Added fast color conversion logic with pre-calculated table - Add LZW decoder with libtiff's implementation - Take part of the code to provide only LZW decoder part - Add `rows_per_strip_` and `predictor_` attributes to IFD class, to support LZW - Refactor TIFF:resolve_vendor_format() to support Aperio SVS metadata - Support LZW compressed image (with multi strips) for the associated image. Addresses #17    Authors: - Gigon Bae (https://github.com/gigony) Approvers: - https://github.com/jakirkham URL: #141
- Update changelog and readme for PyPI Package (v21.10.01) - Update `update-version.sh` script to handle links in PyPI README.md Authors: - Gigon Bae (https://github.com/gigony) Approvers: - AJ Schmidt (https://github.com/ajschmidt8) - https://github.com/jakirkham URL: #142
- Update `notebooks/Welcome.ipynb` - Add `notebooks/Supporting_Aperio_SVS_Format.ipynb` to show loading Aperio SVS images Authors: - Gigon Bae (https://github.com/gigony) Approvers: - https://github.com/jakirkham URL: #147
Optimizing zoom-out image transform with work efficient padding kernel. Changes are simple on python side, existing unit tests will suffice and are passing. CC: @shekhardw @gigony Authors: - https://github.com/chirayuG-nvidia - https://github.com/jakirkham Approvers: - https://github.com/shekhardw - Gregory R. Lee (https://github.com/grlee77) - https://github.com/jakirkham URL: #125
When multiple threads are used, it can cause thread contentions so performance can be degraded. This patch fixes the issue by using a hash value for the index.
Similar to rapidsai/cudf#9456. This PR updates the `conda` recipe build strings and `cudatoolkit` version specifications as part of the Enhanced Compatibility efforts. The build strings have been updated to only include the major CUDA version (i.e. `librmm-21.12.00a-cuda11_gc781527_12.tar.bz2`) and the `cudatoolkit` version specifications will now be formatted like `cudatoolkit >=x,<y.0a0` (i.e. `cudatoolkit >=11,<12.0a0`). Moving forward, we'll build the packages with a single CUDA version (i.e. `11.5`) and test them in environments with different CUDA versions (i.e. `11.0`, `11.2`, `11.4`, `11.5` etc.).
Fix thread contention in Cache
- Instrumented important methods in cuCIM with NVTX so can see performance bottlenecks easily with NSight systems. - Tracing can be enabled through config file or environment variable or through API and less than 1% performance overheads in normal execution.
Currently, cuCIM code links CUDA runtime by creating PTX code explicitly (changing LANGUAGE to CUDA) even though cuCIM code (in C++) currently doesn't have any CUDA kernels. ```cmake # At least one file needs to be compiled with nvcc. # Otherwise, it will cause `/usr/bin/ld: cannot find -lcudart` error message. set_source_files_properties(src/cucim.cpp src/filesystem/cufile_driver.cpp PROPERTIES LANGUAGE CUDA) ``` That causes the issue with CuPy, causing the following error message (#170) ```python >>> import cupy as cp >>> import cucim.clara >>> a = cp.zeros((3,3)) ... cupy/cuda/memory.pyx in cupy.cuda.memory.Memory.__init__() cupy_backends/cuda/api/runtime.pyx in cupy_backends.cuda.api.runtime.malloc() cupy_backends/cuda/api/runtime.pyx in cupy_backends.cuda.api.runtime.check_status() CUDARuntimeError: cudaErrorUnsupportedPtxVersion: the provided PTX was compiled with an unsupported toolchain. ``` The error is related to the nvcc version (from cuda 11.5) used in RAPIDS GPUCI. https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html#group__CUDART__TYPES_1gg3f51e3575c21[…]e003844fd2df23a72a9ee7364d30150ae7e9e >cudaErrorUnsupportedPtxVersion = 222 >This indicates that the provided PTX was compiled with an unsupported toolchain. The most common reason for this, is the PTX was generated by a compiler newer than what is supported by the CUDA driver and PTX JIT compiler. With CUDA toolkit (nvcc) v11.4, it doesn't cause the error. Fix the error by linking libcudart.so explicitly. Also fixes warnings. Close #170
Including [rmm](https://github.com/rapidsai/rmm)'s CMakeLists.txt (by using add_subdirectory() method with FetchContent in CMakeList.txt), though it is not used/linked to `libcucim.so`, polluted main libcucim's CMake environment variables (cuCIM was including old `rmm` version whose CMakeLists.txt was not modernized) so PTX code was always included in libcucim.so causing the issue in #170. Since cuCIM currently doesn't use `rmm`, This patch removes rmm dependency completely and makes sure that `libcucim.so` doesn't have PTX code. - Remove `superbuild_depend(rmm)` and add `superbuild_depend(googletest)` - Remove CUDA language in CMakeLists.txt - Fix compilation warnings/errors caused by switching to GCC compiler (instead of nvcc).
CUDA based normalization operation integrated into cucim. Normalization supports following types. 1- simple range based normalization 2- Atangent based normalization closes #151
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
❄️ Code freeze for
branch-21.12and v21.12 releaseWhat does this mean?
Only critical/hotfix level issues should be merged into
branch-21.12until release (merging of this PR).What is the purpose of this PR?
branch-21.12intomainfor the release