Support for Property Tables in Custom Shaders #13124
Description
Overview
Currently, custom shaders can access metadata stored in property attributes or property textures (see the EXT_Structural_Metadata spec). They cannot access metadata stored in property tables.
Representing property tables in shaders pose challenges because not all data types can be natively represented on the GPU - strings and variable length arrays, to name two. These are challenges faced in property textures as well, and CesiumJS support currently limits property textures to UINT8 component types (this might be to ensure support for WebGL 1?).
Since property tables are 2D, it makes sense to represent them as textures as well. Thus, I've focused my investigation efforts so far on understanding how property textures are implemented in Cesium. I've summarized some important code paths below (they're not exhaustive, but hit the main points of how a property texture goes from glTF to on-screen):
Loading + parsing the glTF into CPU memory
I'm not going to go into too much detail here because Cesium's glTF loading process is extremely complex, and it already handles property tables, but here's an image of the call stack at the point where a vertex attribute buffer is actually (finally) loaded into CPU memory:
This can go even deeper depending on the buffer type (e.g. whether the buffer is embedded in the gltf or external, in a binary). But after this call, the buffer is officially how we want it on the CPU. For textures, it's a similar call stack:
Processing the model resources + uploading to GPU
The upload from CPU to GPU happens during the main update loop - specifically, from Model.update. Each frame, the model calls its loader to do some processing. The gltf loader, in turn, calls its component loaders (buffer loader, texture loader, structural metadata loader, etc.) to process. Since we can't hang the browser by spending too much time processing in a single frame, these loaders give a job to the JobScheduler which picks up work where it left it off each frame.
The call stack looks like this:
(And a similar call stack for vertex attribute upload)
Creating draw commands using those GPU handles
The GPU handles are stored on the GltfLoader, which the Model class has a reference to. Those handles are bundled up into PrimitiveRenderResources in ModelSceneGraph.buildDrawCommands > ModelSceneGraph.buildRenderResources, and then finally become a DrawCommand, call stack:
Shader code to facilitate metadata
Okay, so the glTF model has been parsed, stored in CPU memory, uploaded to GPU memory, and bound to the GPU using a DrawCommand. To actually be able to do custom shading based on metadata, however, there's still one more facet: building the vertex and fragment shaders dynamically for the given model's metadata properties.
This work happens in MetadataPipelineStage.js, during the process function - called from ModelSceneGraph.buildRenderResources:
It's easiest to understand what this stage does by viewing the output vertex and fragment shaders, preprocessed, directly. We can get that from SpectorJS:
propertytexture_preprocessed_fragshader.txt
propertytexture_preprocessed_vertshader.txt
I generated these for the Simple Property Texture example. You can see the relevant structs and functions added to support metadata. Some snippets:
struct Metadata {
int insideTemperature;
int outsideTemperature;
float insulation;
};
struct MetadataClass {
intMetadataClass insideTemperature;
intMetadataClass outsideTemperature;
floatMetadataClass insulation;
};
void initializeMetadata(out Metadata metadata, out MetadataClass metadataClass, out MetadataStatistics metadataStatistics, ProcessedAttributes attributes) {
metadata.insideTemperature = int(255.0 * texture(u_propertyTexture_1, attributes.texCoord_0).r);
metadata.outsideTemperature = int(255.0 * texture(u_propertyTexture_1, attributes.texCoord_0).g);
metadata.insulation = texture(u_propertyTexture_1, attributes.texCoord_0).b;
}
These three properties (insideTemperature, outsideTemperature, insulation) are specific to this tileset / property textures and the struct fields and functions that use them are generated dynamically.