Experimental Blosc2 compression, benchmark on real CDF files, and in-browser converter - #106
Merged
Merged
Conversation
Saving a variable or file with zstd_compression threw "Unsupported compression algorithm": make_cpr() and compression::deflate() had no zstd case. Only the codec itself was tested, never save then reload. deflate() now also takes the element size, so typed codecs (byte-shuffling ones such as blosc2) can use it. Adds a round-trip test over every compiled-in codec, for both per-variable and whole-file compression. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
New cdf_compression_type::blosc2_compression (id 17, non-standard like zstd's 16), behind -Dwith_experimental_blosc2=true. Blosc2 is used with its zstd codec, byte shuffle and the CDF element size as typesize. Chunks are self-describing, so decoding needs no parameters. c-blosc2 comes from a wrap with a native meson overlay. Its CMake build cannot go through meson's cmake module: that applies the AVX512 flags of one file to the whole library, which crashes with SIGILL on CPUs without AVX512. The overlay also takes lz4/zstd from wraps instead of CMake FetchContent downloads at configure time. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Blosc2's typesize is now the record size (flattened record shape times element size) when it is at most 255 bytes, otherwise the element size. Each record component then gets its own shuffled byte streams, e.g. the x, y, z of a vector. On a 39-file CDAWeb/NASA corpus (1011 variables, 1.6 GiB raw) this is about 3% smaller than using the element size, at the same speed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Corpus: 23 current CDAWeb datasets (one day each, 11 missions) plus 16 historical NASA CDF test files, 1.1 GiB. Two benchmarks: - sweep.py: per-variable grid of gzip, zstd and blosc2 settings (codec, filters, element vs record typesize) on raw variable bytes. - end_to_end.py: CDFpp save and eager load of every file with each codec, values checked bit-exact. End to end, blosc2 files are 38% smaller than gzip, saved 1.75x and loaded 2.8x faster. results/summary.json holds the full numbers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
New "Convert codec" panel in the Explorer. It re-encodes every variable of the loaded file with each codec the build offers (none, GZIP, and the experimental Zstd and Blosc2), reloads each output eagerly, checks every value is bit-identical, and shows size, change vs the original and vs GZIP, write and read time, with a download per codec. WASM API: CdfFile.save_as(codec), CdfFile.same_values(other) and load_eager(bytes), plus zstd/blosc2 in CompressionType when enabled. The Pages build now enables zstd and blosc2. Both must build without threads under emscripten: -pthread would turn the whole link into a shared-memory build, so zstd gets multi_thread=disabled and the blosc2 overlay drops its threads dependency there (it runs with nthreads=1). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
meson wrap update: catch2 3.16.0-1, zlib 1.3.2-1. xsimd's wrap has no wrapdb_version, so meson leaves it alone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The converter blocked the page for each codec: 1.9 s for GZIP on a 74 MB MMS file. It now runs in a module worker with its own WASM instance; the page's longest frame during a conversion is ~70 ms. The worker loads the original eagerly so decoding it is not billed to the first codec's write time, and transfers each output back instead of copying it. A new conversion, or leaving the panel, stops the worker. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #106 +/- ##
==========================================
- Coverage 93.73% 93.34% -0.39%
==========================================
Files 62 63 +1
Lines 4802 4853 +51
==========================================
+ Hits 4501 4530 +29
- Misses 301 323 +22
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
pycdfpp.save() now emits ExperimentalCompressionWarning when the CDF or any of its variables uses zstd_compression or blosc2_compression: the file is not standard CDF and only CDFpp can read it. save() becomes a thin Python wrapper over the binding; its signature is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Wheels now build with -Dwith_experimental_zstd=true and -Dwith_experimental_blosc2=true, with zstd, lz4 and blosc2 forced to their static fallbacks. Reading such files just works; writing them stays opt-in and warns. Cost: ~1.7 MB unpacked. - zstd's multi_thread is disabled from meson.build for both callers (CDFpp only uses the single-shot API, and -pthread breaks emscripten builds: the web app and Pyodide wheels), replacing the CLI flag in gh-pages.yml. - The blosc2 overlay no longer pins c_std=gnu99, which MSVC rejects, and passes /arch:AVX2 and /arch:AVX512 to MSVC so Windows wheels get the SIMD kernels too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The zstd wrap builds its command-line programs by default, and meson-python installed them into the wheel's scripts (zstd, zstdgrep, zstdless...). On macOS the zstd program links Homebrew's liblzma, which delocate rejects for the macOS 13 target, failing every wheel job. zstd is now pulled in as a library only: bin_programs=false, and its zlib/lzma/lz4 options, only used by the programs, are disabled. Also compare floats with a tolerance in the converter model test (SonarCloud S1244). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
benchmarks/ holds developer tools, not shipped code. The compression benchmark scripts write their CSV to a path given on the command line, which SonarCloud flags as path traversal (S8707) and fails the gate on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
blosc2.h includes <windows.h>, whose min/max macros broke every later std::min/std::max in the MSVC wheel build (C2589 in blosc2.hpp, rle.hpp, create_records.hpp, cpp_utils...). The blosc2 library itself, SIMD kernels included, already compiled fine with MSVC. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Since the catch2 3.16 wrap update, tests/simple_open exceeds MSVC's default COFF section limit (fatal error C1128) in the Windows test workflow. /bigobj is Catch2's documented remedy; other compilers are unaffected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The previous fix only covered the Catch2 test loop; python_wrapper_cpp hit the same C1128 section limit next. Apply the flag to all test, manual and example executables. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds Blosc2 as an experimental CDF compression codec, a benchmark on real files, and an in-browser converter in the Explorer so people can try it on their own files.
The goal is to give CDAWeb / the NASA CDF team numbers, and a way to test, for adding Blosc2 to the CDF specification.
Results
Corpus: 23 current CDAWeb datasets (one day each, 11 missions) and the 16 NASA CDF test files, 1.1 GiB. Every file was saved and reloaded through CDFpp with each codec, and every value was checked bit-identical.
On the CDAWeb files, compared with GZIP level 6:
Across all 39 files: 38% smaller, 2.8× faster reads, 1.75× faster writes. Blosc2 is larger than GZIP on 7 of the 39 files (small or noisy-float files), by at most 24%.
What's in it
53c6bea): saving with the experimental zstd codec threw "Unsupported compression algorithm". A new test round-trips every compiled-in codec, per variable and for the whole file.bda2b33,725964a):-Dwith_experimental_blosc2=true, compression id 17 (non-standard, like zstd's 16). Zstd level 5 inside Blosc2, byte shuffle. The typesize is the whole record when it fits in 255 bytes, otherwise the element size. That gives each vector component its own byte streams.subprojects/packagefiles/c-blosc2). Meson's cmake module can't be used here: it applies the AVX-512 flags of one file to the whole library, which crashes with SIGILL on CPUs without AVX-512. lz4 comes from wrapdb, zstd from the existing wrap. Nothing is downloaded at configure time.56a1dad):benchmarks/compression/, with a README to reproduce.results/summary.jsonholds the numbers.bcd6b00,4a7b232): a "Convert codec" panel in the Explorer re-encodes the loaded file with each codec, reloads it, checks the values and offers downloads. It runs in a Web Worker: the page's longest frame during a conversion is ~70 ms (it was 1.9 s on the main thread for a 74 MB file). The Pages build enables zstd and Blosc2.089cf18,15cd32e): wheels now include zstd and Blosc2, statically linked (+~1.7 MB unpacked). Reading just works. Writing stays opt-in, andpycdfpp.save()emitsExperimentalCompressionWarningwhen a file uses an experimental codec.multi_threadis disabled frommeson.build(CDFpp only uses the single-shot API, and-pthreadbreaks emscripten: the web app and the Pyodide wheels). The blosc2 overlay is MSVC-aware (noc_std=gnu99,/arch:AVX2,/arch:AVX512).f7f990e):meson wrap update(catch2 3.16.0, zlib 1.3.2).Testing
meson test --suite pycdfpp, 36/36 passwasm_convertpyproject.toml(Linux x86_64) exposes both codecs, warns on save, round-trips, and has no new shared-library dependenciesNot in this PR
🤖 Generated with Claude Code