Supporting picking properties from property textures

[sanjeetsuhag]

Unlike Feature ID attributes and Feature ID textures, Feature Textures present a challenge for implementing picking because both the attribution and the metadata is on the GPU side, in the texture.

An initial, and admittedly flawed, approach to picking certain values from feature textures was accomplished by exploiting how the pickColor is generated, with the rgba channels populated from left-to-right, with each channel accounting for 32768 pickable objects. The feature ID texture values were sampled (by modifying the pickId) and encoded in the unused (arbitrarily set to b and a) channels, and PickFramebuffer.js was modified to extract those values.

A more stable approach would be to add a pickFeatureTexture() function to Scene.js:

1. This function would first call pick, check if the object returned has the requested feature texture.
2. The primitive's pickId would be modified to sample the values from the feature texture.
3. The draw commands would be updated.
4. pick would be called again, and values would be read from the channels specified by the feature texture
5. The pickId would be reset back to czm_pickColor
6. The draw commands would be updated

CC: @lilleyse @ptrgags

[lilleyse]

with each channel accounting for 32768 pickable objects

This is not quite correct, it's exponential not additive. If you use just the red channel you get 256 pickable objects (2^8), if you use the red and green you get 65536 (2^16), and so on.

[lilleyse]

The suggested approach still feels like it is fighting the derived command system. Also we can think bigger than feature textures. Really they're just an implementation detail and pickFeatureTexture feels like a leaky abstraction.

Maybe something more generic like pickProperty(propertyId)? The function should be able to pick any property that's under the cursor whether it's stored in a feature texture or in a feature table or as a vertex attribute (like for point clouds where we don't have feature tables on the CPU).

I think this would require a new type of derived command that's separate from the pick command. I haven't thought much more about that yet.

[ptrgags]

I was talking with @lilleyse today about this. One idea (requires MRT) is to have 1 float render target for the property texture value (that way we can render properties up to vec4 in size). Then have another render target include other information such as whether the pick color is a property value, a pickId, or something else.

Such a change to the picking system would also be helpful for implementing picking for feature IDs without property tables (#9884) and per-vertex feature Ids (if it can be done without allocating pick objects for everything on the CPU)

[javagl]

One very fundamental question is still open. This applies to all sorts of picking, and metadata handling in general. So this could therefore be brought up in many other issues. It could even be escalated into a new, overarching issue, which then would have a certain overlap to #10085 ...

But I'll start here:

How can a user select what is supposed to be picked?

There currently is the featureIdLabel for feature IDs. This may not be sufficient for certain usage patterns, even though it only refers to feature IDs. But for EXT_structural_metadata, it's far more complex.

The user may load two models (maybe just GLB that is tile content). And the structural metadata may have the following structure:

• Model A
  • a property attribute that refers to propertyA0 of classA0
  • a property texture that refers to propertyA1 of classA0
• Model B
  • a property attribute that refers to propertyB0 of classB0
  • a property texture that refers to propertyB1 of classB1

Now, the user wants to ... ~"pick something". And the user has to explicitly say whether the "picked thing" should be classA0.propertyA0 or classB1.propertyB1.
_{(Yeah, ... maybe there are different classes that have the same name, so that will have to include the schema.id... but let's ignore that for now...)}

One problem is that there is no way for "transporting" information about available metadata to the user. Every form of API call (and even more so: UI elements) will have to involve deep, specific knowledge (or ... assumptions) about the structure of the metadata that will be available.

One could think about this with pesudocode: Imagine a user wants to pick something that is stored as
"exampleSchema.exampleClass.exampleProperty"

Then this would require something like

updateOrProcessOrBuildDrawCommandsOrWhateverFor(model) {
    const p = gatherProperties(model); 

    const whatUserWantsToPick = "exampleSchema.exampleClass.exampleProperty";
    if (p.includes(whatUserWantsToPick ) {
        magic.setPickedProperty(whatUserWantsToPick);
    }
}

^{(and one could wish for a populateUI(p); there...)}

On the highest level, this would probably involve something like the SelectedFeatureIdPipelineStage, but as SelectedPropertyPipelineStage (but I have no clue (yet) what that would entail...).