Merge pull request #260 from rapidsai/branch-22.04

[RELEASE] cucim v22.04

Author: raydouglass

.github/ops-bot.yaml

@@ -0,0 +1,8 @@
+# This file controls which features from the `ops-bot` repository below are enabled.
+# - https://github.com/rapidsai/ops-bot
+
+auto_merger: true
+branch_checker: true
+label_checker: true
+release_drafter: true
+external_contributors: false

CHANGELOG.md

@@ -1,116 +1,68 @@
-# cuCIM 22.02.00 (2 Feb 2022)
-
-## 🚨 Beaking Changes
-
-- Update cucim.skimage API to match scikit-image 0.19 ([#190](https://github.com/rapidsai/cucim/pull/190)) [@glee77](https://github.com/glee77)
-
-## 📖 Documentation
-
-- PyPI v21.12.00 elease ([#182](https://github.com/rapidsai/cucim/pull/182)) [@gigony](https://github.com/gigony)
-
-## 🚀 New Featues
-
-- Allow CuPy 10 ([#195](https://github.com/rapidsai/cucim/pull/195)) [@jakikham](https://github.com/jakikham)
-- Suppot multi-theads and batch, and suppot nvJPEG fo JPEG-compessed images ([#191](https://github.com/rapidsai/cucim/pull/191)) [@gigony](https://github.com/gigony)
-- Update cucim.skimage API to match scikit-image 0.19 ([#190](https://github.com/rapidsai/cucim/pull/190)) [@glee77](https://github.com/glee77)
-
-## 🛠️ Impovements
-
-- Upgade Numpy to 1.19 fo Python 3.9 suppot ([#203](https://github.com/rapidsai/cucim/pull/203)) [@Ethyling](https://github.com/Ethyling)
-- Allow installation with CuPy 10 ([#197](https://github.com/rapidsai/cucim/pull/197)) [@glee77](https://github.com/glee77)
-- Upgade Numpy to 1.18 fo Python 3.9 suppot ([#196](https://github.com/rapidsai/cucim/pull/196)) [@Ethyling](https://github.com/Ethyling)
-- Add missing impots tests ([#183](https://github.com/rapidsai/cucim/pull/183)) [@Ethyling](https://github.com/Ethyling)
-
-# cuCIM 21.12.00 (9 Dec 2021)
-
-## 🚀 New Features
-
-1. Support Aperio SVS with CPU LZW and jpeg2k decoder ([#141](https://github.com/rapidsai/cucim/pull/141)) [[@gigony](https://github.com/gigony)](https://github.com/gigony](https://github.com/gigony))
-2. Add NVTX support for performance analysis ([#144](https://github.com/rapidsai/cucim/pull/144)) [[@gigony](https://github.com/gigony)](https://github.com/gigony](https://github.com/gigony))
-3. Normalize operation ([#150](https://github.com/rapidsai/cucim/pull/150)) [[@shekhardw](https://github.com/shekhardw)](https://github.com/shekhardw](https://github.com/shekhardw))
-
-### 1. Support Aperio SVS (.svs)
-
-cuCIM now supports [Aperio SVS format](https://openslide.org/formats/aperio/) with help of [OpenJpeg](https://www.openjpeg.org/) for decoding jpeg2k-compressed data.
-
-Please check [this notebook](https://nbviewer.org/github/rapidsai/cucim/blob/branch-21.12/notebooks/Supporting_Aperio_SVS_Format.ipynb) to see how to use the feature.
+# cuCIM 22.04.00 (6 Apr 2022)
 
-#### Unaligned Case (`per_process`, JPEG-compressed SVS file)
+## 🚨 Breaking Changes
 
-![image](https://user-images.githubusercontent.com/1928522/141350490-06fdd8cb-5be2-42e4-9774-c7b76fab6f9a.png)
+- Apply fixes to skimage.transform scheduled for scikit-image 0.19.2 ([#208](https://github.com/rapidsai/cucim/pull/208)) [@grlee77](https://github.com/grlee77)
 
-#### Unaligned Case (`per_process`, JPEG2000 RGB-compressed SVS file)
-
-![image](https://user-images.githubusercontent.com/1928522/141093324-574b532e-ad42-4d61-8473-4c3e07e3feae.png)
-
-#### Unaligned Case (`per_process`, JPEG2000 YCbCr-compressed SVS file)
-
-![image](https://user-images.githubusercontent.com/1928522/141093381-8ab0161d-1b17-4e80-a680-86abfbf2fa65.png)
-
-The detailed data is available [here](https://docs.google.com/spreadsheets/d/15D1EqNI_E9x_S8i3kJLwBxMcEmwk8SafW0WryMrAm6A/edit#gid=369408723).
-
-### 2. Add NVTX support for performance analysis
-
-Important methods in cuCIM are instrumented with [NVTX](https://docs.nvidia.com/gameworks/index.html#gameworkslibrary/nvtx/nvidia_tools_extension_library_nvtx.htm) so can see performance bottlenecks easily with [NSight systems](https://developer.nvidia.com/nsight-systems).
-
-Tracing can be enabled through config file or environment variable or through API and less than 1% performance overheads in normal execution.
+## 🐛 Bug Fixes
 
-#### Enabling Tracing
-##### Through `.cucim.json` file
+- Fix ImportError from vendored code ([#252](https://github.com/rapidsai/cucim/pull/252)) [@grlee77](https://github.com/grlee77)
+- Fix wrong dimension in metadata ([#248](https://github.com/rapidsai/cucim/pull/248)) [@gigony](https://github.com/gigony)
+- Handle file descriptor ownership and update documents for GDS ([#234](https://github.com/rapidsai/cucim/pull/234)) [@gigony](https://github.com/gigony)
+- Check nullptr of handler in CuFileDriver::close() ([#229](https://github.com/rapidsai/cucim/pull/229)) [@gigony](https://github.com/gigony)
+- Fix docs builds ([#218](https://github.com/rapidsai/cucim/pull/218)) [@ajschmidt8](https://github.com/ajschmidt8)
+- Apply fixes to skimage.transform scheduled for scikit-image 0.19.2 ([#208](https://github.com/rapidsai/cucim/pull/208)) [@grlee77](https://github.com/grlee77)
 
-```json
-{
-        "profiler" : { "trace": true }
-}
-```
+## 📖 Documentation
 
-##### Through Environment variable
+- Update PyPI cuCIM v22.02.01 CHANGELOG.md ([#249](https://github.com/rapidsai/cucim/pull/249)) [@gigony](https://github.com/gigony)
+- Update GTC 2021 Spring video links ([#227](https://github.com/rapidsai/cucim/pull/227)) [@gigony](https://github.com/gigony)
+- Update documents for v22.02.00 ([#226](https://github.com/rapidsai/cucim/pull/226)) [@gigony](https://github.com/gigony)
+- Merge branch-22.02 into branch-22.04 ([#220](https://github.com/rapidsai/cucim/pull/220)) [@jakirkham](https://github.com/jakirkham)
 
-```bash
-CUCIM_TRACE=1 python
-```
+## 🛠️ Improvements
 
-##### Through API
+- Expose data type of CuImage object for interoperability with NumPy ([#246](https://github.com/rapidsai/cucim/pull/246)) [@gigony](https://github.com/gigony)
+- Temporarily disable new `ops-bot` functionality ([#239](https://github.com/rapidsai/cucim/pull/239)) [@ajschmidt8](https://github.com/ajschmidt8)
+- Add `.github/ops-bot.yaml` config file ([#236](https://github.com/rapidsai/cucim/pull/236)) [@ajschmidt8](https://github.com/ajschmidt8)
+- randomization per image per batch ([#231](https://github.com/rapidsai/cucim/pull/231)) [@shekhardw](https://github.com/shekhardw)
 
-```python
-from cucim import CuImage
+# cuCIM 22.02.00 (2 Feb 2022)
 
-CuImage.profiler(trace=True)
-#or
-CuImage.profiler().trace(True)
+## 🚨 Breaking Changes
 
-CuImage.profiler().config
-# {'trace': True}
-CuImage.profiler().trace()
-# True
-CuImage.is_trace_enabled # this is simpler method.
-# True
-```
+- Update cucim.skimage API to match scikit-image 0.19 ([#190](https://github.com/rapidsai/cucim/pull/190)) [@glee77](https://github.com/glee77)
 
-#### Profiling with NVIDIA Nsight Systems
+## 🐛 Bug Fixes
 
-```bash
-nsys profile -f true -t cuda,nvtx,osrt -s cpu -x true --trace-fork-before-exec true -o my_profile `which python` benchmark.py
-# can add `--stats true`
-```
+- Fix a bug in [v21.12.01](https://github.com/rapidsai/cucim/wiki/release_notes_v21.12.01) ([#191](https://github.com/rapidsai/cucim/pull/191)) [@gigony](https://github.com/gigony)
+  - Fix GPU memory leak when using nvJPEG API (when `device='cuda'` parameter is used in `read_region` method).
+- Fix segfault for preferred_memory_capacity in Python 3.9+ ([#214](https://github.com/rapidsai/cucim/pull/214)) [@gigony](https://github.com/gigony)
 
-Then, execute `nsight-sys` to open the profile results (my_profile.qdrep).
+## 📖 Documentation
 
-![image](https://user-images.githubusercontent.com/1928522/141221297-2ff5224b-e99b-4fe6-af7d-69452141d71d.png)
+- PyPI v21.12.00 release ([#182](https://github.com/rapidsai/cucim/pull/182)) [@gigony](https://github.com/gigony)
 
-With this feature, a bug in cuCIM [v21.10.01](https://github.com/rapidsai/cucim/wiki/release_notes_v21.10.01) (thread contention in Cache) was found and fixed ([#145](https://github.com/rapidsai/cucim/pull/145)).
+## 🚀 New Features
 
-### 3. Normalize operation
+- Update cucim.skimage API to match scikit-image 0.19 ([#190](https://github.com/rapidsai/cucim/pull/190)) [@glee77](https://github.com/glee77)
+- Support multi-threads and batch, and support nvJPEG for JPEG-compressed images ([#191](https://github.com/rapidsai/cucim/pull/191)) [@gigony](https://github.com/gigony)
+- Allow CuPy 10 ([#195](https://github.com/rapidsai/cucim/pull/195)) [@jakikham](https://github.com/jakikham)
 
-CUDA-based normalization operation is added. Normalization supports the following types.
+## 🛠️ Improvements
 
-1. Simple range based normalization
-2. Arctangent based normalization
+- Add missing imports tests ([#183](https://github.com/rapidsai/cucim/pull/183)) [@Ethyling](https://github.com/Ethyling)
+- Allow installation with CuPy 10 ([#197](https://github.com/rapidsai/cucim/pull/197)) [@glee77](https://github.com/glee77)
+- Upgrade Numpy to 1.18 for Python 3.9 support ([#196](https://github.com/rapidsai/cucim/pull/196)) [@Ethyling](https://github.com/Ethyling)
+- Upgrade Numpy to 1.19 for Python 3.9 support ([#203](https://github.com/rapidsai/cucim/pull/203)) [@Ethyling](https://github.com/Ethyling)
 
-Arctangent-based normalization helps to stretch lower intensity pixels in the image slightly better than range-based normalization. If you look at its [graph](https://mathworld.wolfram.com/InverseTangent.html), there is a huge variation at certain lower intensities, but as intensities become higher, the curve becomes flatter. This helps in isolating regions like lungs (and regions within lungs) more efficiently. There can be separate use cases depending on the modality and the application.
+# cuCIM 21.12.00 (9 Dec 2021)
 
-Please check the [test cases](https://github.com/rapidsai/cucim/blob/branch-21.12/python/cucim/src/cucim/core/operations/intensity/tests/test_normalize.py) to see how you can use the operation.
+## 🚀 New Features
 
+- Support Aperio SVS with CPU LZW and jpeg2k decoder ([#141](https://github.com/rapidsai/cucim/pull/141)) [[@gigony](https://github.com/gigony)](https://github.com/gigony](https://github.com/gigony))
+- Add NVTX support for performance analysis ([#144](https://github.com/rapidsai/cucim/pull/144)) [[@gigony](https://github.com/gigony)](https://github.com/gigony](https://github.com/gigony))
+- Normalize operation ([#150](https://github.com/rapidsai/cucim/pull/150)) [[@shekhardw](https://github.com/shekhardw)](https://github.com/shekhardw](https://github.com/shekhardw))
 
 ## 🐛 Bug Fixes
 
@@ -203,4 +155,4 @@ Please check the [test cases](https://github.com/rapidsai/cucim/blob/branch-21.1
 
 # cuCIM 0.19.0 (15 Apr 2021)
 
-Initial release of cuCIM including cuClaraImage and [cupyimg](https://github.com/mritools/cupyimg).
+- Initial release of cuCIM including cuClaraImage and [cupyimg](https://github.com/mritools/cupyimg).

README.md

@@ -5,8 +5,8 @@
 
 **NOTE:** For the latest stable [README.md](https://github.com/rapidsai/cucim/blob/main/README.md) ensure you are on the `main` branch.
 
-- [GTC 2021 cuCIM: A GPU Image I/O and Processing Toolkit [S32194]](https://www.nvidia.com/en-us/gtc/catalog/?search=cuCIM#/)
-  - [video](https://gtc21.event.nvidia.com/media/cuCIM%3A%20A%20GPU%20Image%20I_O%20and%20Processing%20Toolkit%20%5BS32194%5D/1_fwfxd0iu)
+- [GTC 2021 cuCIM: A GPU Image I/O and Processing Toolkit [S32194]](https://www.nvidia.com/en-us/on-demand/search/?facet.mimetype[]=event%20session&layout=list&page=1&q=cucim&sort=date)
+  - [video](https://www.nvidia.com/en-us/on-demand/session/gtcspring21-s32194/)
 - [SciPy 2021 cuCIM - A GPU image I/O and processing library](https://www.scipy2021.scipy.org/)
   - [video](https://youtu.be/G46kOOM9xbQ)
 
@@ -39,7 +39,7 @@ pip install scipy scikit-image cupy-cuda110
 
 ### Notebooks
 
-Please check out our [Welcome](notebooks/Welcome.ipynb) notebook ([NBViewer](https://nbviewer.jupyter.org/github/rapidsai/cucim/blob/branch-22.02/notebooks/Welcome.ipynb))
+Please check out our [Welcome](notebooks/Welcome.ipynb) notebook ([NBViewer](https://nbviewer.jupyter.org/github/rapidsai/cucim/blob/branch-22.04/notebooks/Welcome.ipynb))
 
 #### Downloading sample images
 

VERSION

@@ -1 +1 @@
-22.02.00
+22.04.00

cpp/include/cucim/cuimage.h

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2020-2021, NVIDIA CORPORATION.
+ * Copyright (c) 2020-2022, NVIDIA CORPORATION.
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.
@@ -37,6 +37,14 @@
 #include <string>
 #include <vector>
 
+
+// Forward declarations for DLDataType's equality operator.
+template <>
+struct std::hash<DLDataType>;
+
+EXPORT_VISIBLE bool operator==(const DLDataType& lhs, const DLDataType& rhs);
+EXPORT_VISIBLE bool operator!=(const DLDataType& lhs, const DLDataType& rhs);
+
 namespace cucim
 {
 
@@ -148,6 +156,8 @@ class EXPORT_VISIBLE CuImage : public std::enable_shared_from_this<CuImage>
 
     DLDataType dtype() const;
 
+    std::string typestr() const;
+
     std::vector<std::string> channel_names() const;
 
     std::vector<float> spacing(std::string dim_order = std::string{}) const;

cpp/include/cucim/filesystem/cufile_driver.h

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2020-2021, NVIDIA CORPORATION.
+ * Copyright (c) 2020-2022, NVIDIA CORPORATION.
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.
@@ -183,6 +183,9 @@ class EXPORT_VISIBLE CuFileDriver : public std::enable_shared_from_this<CuFileDr
 
     ~CuFileDriver();
 
+    // To allow 'handle_' field
+    friend std::shared_ptr<CuFileDriver> open(const char* file_path, const char* flags, mode_t mode);
+
 private:
     static Mutex driver_mutex_; // TODO: not used yet.
 

cpp/include/cucim/filesystem/file_handle.h

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2020, NVIDIA CORPORATION.
+ * Copyright (c) 2020-2022, NVIDIA CORPORATION.
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.
@@ -19,6 +19,9 @@
 
 
 #include "../macros/defines.h"
+
+#include <unistd.h>
+
 #include <cstdio>
 #include <cstdint>
 #include <memory>
@@ -32,7 +35,7 @@ typedef void* CuCIMFileHandle_share;
 typedef void* CuCIMFileHandle_ptr;
 typedef bool (*CuCIMFileHandleDeleter)(CuCIMFileHandle_ptr);
 
-enum class FileHandleType: uint16_t
+enum class FileHandleType : uint16_t
 {
     kUnknown = 0,
     kPosix = 1,
@@ -55,7 +58,8 @@ struct EXPORT_VISIBLE CuCIMFileHandle : public std::enable_shared_from_this<CuCI
                     void* client_data,
                     uint64_t dev,
                     uint64_t ino,
-                    int64_t mtime);
+                    int64_t mtime,
+                    bool own_fd);
 
     ~CuCIMFileHandle()
     {
@@ -70,6 +74,13 @@ struct EXPORT_VISIBLE CuCIMFileHandle : public std::enable_shared_from_this<CuCI
             deleter(this);
             deleter = nullptr;
         }
+
+        if (own_fd && fd >=0)
+        {
+            ::close(fd);
+            fd = -1;
+            own_fd = false;
+        }
     }
 
     CuCIMFileHandleDeleter set_deleter(CuCIMFileHandleDeleter deleter)
@@ -86,6 +97,7 @@ struct EXPORT_VISIBLE CuCIMFileHandle : public std::enable_shared_from_this<CuCI
     uint64_t dev = 0;
     uint64_t ino = 0;
     int64_t mtime = 0;
+    bool own_fd = false; /// whether if the file descriptor is created internally by the driver
     CuCIMFileHandleDeleter deleter = nullptr;
 };
 #else