KHR_meshopt_compression proposal

[zeux]

This PR proposes a new extension, KHR_meshopt_compression, which is a successor to the existing extension EXT_meshopt_compression.

Motivation

KHR_meshopt_compression provides general functionality to compress bufferViews. This compression is tailored to the common types of data seen in glTF buffers, but it's specified independently and is transparent to the rest of the loading process - implementations only need to decompress compressed bufferViews and the accessors behave as usual after that. The compression is designed to make it possible for optimized implementations to reach decompression throughput of multiple gigabytes per second on modern desktop hardware (through native or WebAssembly code), to ensure that the compression is not a bottleneck even when the transmission throughput is high.

As a result, it's possible to compress mesh vertex and index data (including morph targets), point clouds, animation keyframes, instancing transforms, Gaussian Splat data and other types of binary data. Compression can be lossless, taking advantage of preprocessing to optimally order and optionally pre-quantize the data, or lossy with the use of filters, which allows arbitrary degree of tradeoff between transmission size and quality for multiple types of common glTF data.

The compression process is versatile and different data processing tools may make different tradeoffs with how to preprocess the data before compression, whether to use lossy compression and of what kind, etc. - by comparison, the decompression process is straightforward and fast. This is by design, and means that it's comparatively easier to implement this extension in renderers compared to processing pipelines, which removes as many barriers to adoption as possible.

Compared to EXT_meshopt_compression, this extension uses an improved method of compressing attribute data, which delivers improved compression ratios at the same decompression speeds for a variety of different use cases, and incorporates special support for lossy color encoding which allows to reduce the size further for cases where color streams are a significant fraction of the asset (some 3D scanned meshes, point clouds, Gaussian splats). All existing use cases of EXT_meshopt_compression are supported well, and no significant performance compromises are being made - as such, all existing users of EXT_meshopt_compression should be able to upgrade (pending ecosystem adoption) to KHR_meshopt_compression.

Why a new extension?

EXT_meshopt_compression has been completed ~4 years ago; it serves as a good and versatile compression scheme that transparently supports geometry, animation, instancing data and allows to maintain maximum rendering efficiency and in-memory size while using additional compression during transfer, with decompression throughput in gigabytes/second on commodity hardware.

Since then, the underlying compression implementation for attributes in meshoptimizer has been revised to version 1 (from version 0 that's used in the EXT extension) for better compression; this is currently used outside of glTF ecosystem by some native and web applications. Additionally, some use cases like point clouds and 3D scans have emerged after the extension was initially standardized, that benefit from better color compression (which has been considered for EXT but not included since at the time it was focused more on "traditional" 3D geometry).

Because of the change in the underlying compression bitstream, the bitstream specification needs to be revised as implementations of EXT_meshopt_compression may not be able to decode the new format - thus, a new extension name is necessary.

What changed?

KHR_meshopt_compression uses the same JSON structure as EXT_ extension, keeps the same three filters and keeps two existing schemes for index compression as is. It upgrades attribute compression to use a more versatile encoding (v1), which supports enhanced bit specification for deltas and customizable per-channel delta modes that improve compression further for 16-bit as well as some types of 32-bit data. For compatibility, v0 encoding is still supported. It also adds one new filter, COLOR, which applies additional lossy transform to quantized RGBA color data to decorrelate input channels (similarly to OCTAHEDRAL encoding for vectors, this improves compression and provides more optionality wrt using variable number of bits, without any changes needed to the renderer as the filter unpacks data back into quantized normalized RGBA).

On typical geometry data, enhanced attribute compression provides approximately 10% geomean reduction in vertex data size; the gains obviously highly depend on the content. On point clouds, as well as some 3D scanned models that use vertex colors, the new color filter together with the new attribute compression results in ~15% geomean reduction, with even stronger (20-25%) gains when non-aligned bit counts are used (e.g. 6-bit or 10-bit colors).

Why KHR?

These improvements require a new extension, as they update the data format / bitstream as well as the JSON enum for color filter. While it's possible to specify this as a new EXT extension, like EXT_meshopt_compression2, it seemed better to promote to KHR:

• This matches existing compression formats, like Draco and Basis Universal, as well as some formats pending discussion like SPZ
• This makes the compression scheme, which has proven to be versatile and useful since it got standardized, more first class, with associated ecosystem benefits in the coming years (more support)

Since a lot of different parts of glTF ecosystem can be supported by meshopt compression, including core functionality and extensions like Gaussian splats or mesh instancing, specifying a KHR version provides a more comprehensive/coherent story wrt compression for glTF ecosystem.

The minimal "upgrade" path from EXT to KHR would involve just changing the extension name, as the original bitstream should be fully compatible with the new bitstream. The ideal upgrade path would involve re-encoding the original data (helpful if COLOR filter is useful), or at least losslessly re-encoding the attribute data from v0 to v1 - this doesn't require parsing any accessor data, and merely requires decompressing buffer views and re-compressing them with new encoding.

Implementations

Since this is a proposal that just got created, this extension obviously does not have implementations yet :) Having said that, because the JSON structure is exactly the same, except the addition of a COLOR filter, and most implementations of EXT_meshopt_compression use meshoptimizer library which supports the new additions (color filter was released in 0.25), I'd expect that existing implementations for EXT_ can be made compatible with KHR_ with minimal effort:

• For loaders that currently implement support for EXT_ (three.js, Babylon.JS), updating meshoptimizer module and tweaking the JSON parsing code to recognize KHR_ extension should be only a few lines of changes and should be sufficient (for reference, full three.js implementation of EXT variant);
• For data processors that currently implement support for EXT_ (gltfpack, glTF-Transform), updating meshoptimizer module and exposing a user option that would serialize KHR_ extension and encode attribute data using new attribute encoding, plus optionally support color filter, should be easy
• For any loader that wants to implement this without relying on meshoptimizer library for some reason, I've updated the reference decoder by following the updates made to this specification, so it should be comprehensive and while it's more changes than you'd need if you were just using the library, it's a manageable amount of extra complexity.

[lexaknyazev]

it seemed better to promote to KHR

KHR vs EXT prefix choice generally means different treatment of the extension's IP besides perceived "ecosystem support". In particular the following details should be clarified upfront (not a legal advice, though):

• Khronos would be the extension text copyright holder
• The extension should not include 3rd-party trademarks or their use should be explicitly allowed (is "meshopt" registered in some way?)
• Any technology needed by the extension may become included in the Khronos IP framework (with some caveats)

[zeux]

Understood. I'm not a lawyer, but none of these seem like blockers to me (pending closer review). meshopt is not a registered trademark.

[zeux]

Some updates!

1. There should not be any copyright/trademark/IP issues as far as I'm concerned.
2. meshoptimizer 0.25 released last week includes support for color filter and v1 vertex codec in JS module, as well as an update to the reference decoder that supports both as well.
3. While no tools exist that can produce files with this extension, and I intentionally deferred official support in gltfpack for it, I have an implementation of encoding that requires just a few small tweaks to the source code (to use the correct extension name), so it should be very easy to produce test assets if necessary
4. I took a stab at seeing what it takes to upgrade an implementation of EXT_meshopt_compression to KHR; since the heavy-lifting is done by meshoptimizer library, the actual change should only require a few lines of JSON parsing changes. See mrdoob/three.js@dev...zeux:three.js:khr-meshopt - the first commit there is just updating the meshopt_decoder to latest version.

It would be great to understand what the next steps could be here, as from my perspective all blockers have been resolved. Quite confident the actual implementations are going to be quick to finalize if there is agreement from Khronos in principle that the extension should be included. I'm obviously also happy to adjust the proposed text if corrections or clarifications are necessary.

[lexaknyazev]

The three.js update suggests that the decoder does not need to know whether the glTF asset uses the original EXT or the new KHR extension name. This implies that existing EXT (v0) files could be upgraded to KHR (v1) without re-encoding anything at all.

Since v0 support is not going anywhere (because the EXT extension has been ratified and tools are expected to accept such files indefinitely), I'd suggest allowing v0 in KHR as well (with a note about v1 benefits), given that the intention is to keep using the same extension name.

[zeux]

Since v0 support is not going anywhere (because the EXT extension has been ratified and tools are expected to accept such files indefinitely), I'd suggest allowing v0 in KHR as well (with a note about v1 benefits), given that the intention is to keep using the same extension name.

That sounds good to me. I originally specified KHR as just accepting v1 in order to have the smallest/simplest possible specification. But indeed, this restriction is not strictly necessary - v0 support is never going away, and the format version is encoded in the first byte of the stream so the meshoptimizer library can decode both. The reference decoder decodes both as well. I can change this, it would just require a more complicated bitstream description where specific additions are called out as only being there if version is 1.

[lexaknyazev]

Looks good. We'll likely need further editorial and/or formatting updates before publishing.