scene.pickPosition returns incorrect position

[hpinkos]

Reported on the forum: https://groups.google.com/forum/?hl=en#!topic/cesium-dev/0E-aKBTLESk

The user created this sandcastle example to demonstrate the problem: http://hosting.virtualcitysystems.de/demos/temp/pickProblem/Apps/Sandcastle/?src=3D%20Tiles.html&label=undefined

The demo is using the latest 3d-tiles branch

pickPosition works fine if you run the example inside sandcastle, but if you click 'Open in New Window' sometimes the returned position sits in front of the building you clicked instead of where the click intersects with it. Maybe it's related to the canvas ratio? See the forum post for more details.

I couldn't reproduce this with any of our sample models, but I asked the user if he could share a tile for us to test with.

[lucasvw]

Here is the tileset which is used in the sandcastle demo: http://hosting.virtualcitysystems.de/demos/temp/b3dm

Here is a video which shows the behavior: https://youtu.be/9twwwMHbjKU
And here is another that shows that it has something to do with the size of the screen: https://youtu.be/IbDNwqKVOik

And zooming : https://youtu.be/nJpHGOdTssY

[lucasvw]

3D tileset zipped: http://hosting.virtualcitysystems.de/demos/temp/b3dm/3dTiles.zip

[hpinkos]

Thanks @lucasvw!

[lilleyse]

I'll take a look at this soon.

[lilleyse]

Just to update this issue, it looks like this is related to inconsistencies with depth picking with multifrustum rendering, so it's a bug in the core engine. I still need to investigate more.

[lucasvw]

Is this going to be part of the bug-bash next week? +1 for yes ;)

[mramato]

@lucasvw I marked us to at least look at it as part of the bash, but depending on how involved the fix is, I can't promise it will be resolved. We'll try our best!

[lucasvw]

@mramato @lilleyse do you perhaps have an idea when you will be able to fix this ?

[lilleyse]

Sorry about the delay, I will spend some more time looking into this.

[lucasvw]

That would be great! Thanks a lot @lilleyse

[lucasvw]

@lilleyse I barely dare to ask... but eh.. any updates on this ?

;)

[lilleyse]

Sorry but not yet. I did find a slight bug with the calculated pick position (#4615) but it didn't solve the problem unfortunately.

If I remember correctly the core issue here is that Cesium uses multi-frustum rendering and when the pick position is calculated it looks to see if a depth exists starting from the first frustum. The problem is that each frustum may have a different depth at that pixel and it doesn't know which frustum to use. That's just a theory though, but it could be tough to solve. I promise I'll get to this eventually.

[lucasvw]

Thanks for the update and the explanation!

[kring]

In looking at this while debugging #4855, I saw a case where I had only a single frustum, and yet the position returned by pickPosition was wildly wrong. For example, values in pickGlobe in ScreenSpaceCameraController.js:

That's without anything in the scene except for the globe (and sun and moon and such I suppose, but not explicitly-added primitives).

So I don't think it's necessarily multi-frustum related.

[emackey]

Our "pick position" Sandcastle demo limits your picking to just a 3D model of the milk truck. Check out what happens if you open it up to allow picking the entire globe. It works at close range to the truck, but the more you zoom out, the worse it gets.

var viewer = new Cesium.Viewer('cesiumContainer', {
    selectionIndicator : false,
    infoBox : false
});

var scene = viewer.scene;
var handler;

var modelEntity = viewer.entities.add({
    name : 'milktruck',
    position : Cesium.Cartesian3.fromDegrees(-123.0744619, 44.0503706),
    model : {
        uri : '../../SampleData/models/CesiumMilkTruck/CesiumMilkTruck-kmc.gltf'
    }
});
viewer.zoomTo(modelEntity);

var labelEntity = viewer.entities.add({
    label : {
        show : false,
        showBackground : true,
        font : '14px monospace',
        horizontalOrigin : Cesium.HorizontalOrigin.LEFT,
        verticalOrigin : Cesium.VerticalOrigin.TOP,
        pixelOffset : new Cesium.Cartesian2(15, 0)
    }
});

var sceneModeWarningPosted = false;

// Mouse over the globe to see the cartographic position
handler = new Cesium.ScreenSpaceEventHandler(scene.canvas);
handler.setInputAction(function(movement) {
    var foundPosition = false;

    var scene = viewer.scene;
    var pickedObject = scene.pick(movement.endPosition);
    if (scene.pickPositionSupported) {
        if (scene.mode === Cesium.SceneMode.SCENE3D) {
            var cartesian = viewer.scene.pickPosition(movement.endPosition);

            if (Cesium.defined(cartesian)) {
                var cartographic = Cesium.Cartographic.fromCartesian(cartesian);
                var longitudeString = Cesium.Math.toDegrees(cartographic.longitude).toFixed(2);
                var latitudeString = Cesium.Math.toDegrees(cartographic.latitude).toFixed(2);
                var heightString = cartographic.height.toFixed(2);

                labelEntity.position = cartesian;
                labelEntity.label.show = true;
                labelEntity.label.text =
                    'Lon: ' + ('   ' + longitudeString).slice(-7) + '\u00B0' +
                    '\nLat: ' + ('   ' + latitudeString).slice(-7) + '\u00B0' +
                    '\nAlt: ' + ('   ' + heightString).slice(-7) + 'm';

                var camera = scene.camera;
                labelEntity.label.eyeOffset = new Cesium.Cartesian3(0.0, 0.0, camera.frustum.near * 1.5 - Cesium.Cartesian3.distance(cartesian, camera.position));

                foundPosition = true;
            }
        } else if (!sceneModeWarningPosted) {
            sceneModeWarningPosted = true;
            console.log("pickPosition is currently only supported in 3D mode.");
        }
    }

    if (!foundPosition) {
        labelEntity.label.show = false;
    }
}, Cesium.ScreenSpaceEventType.MOUSE_MOVE);

[duvifn]

@emackey

1. I'm not sure that the 'jittering' in your animation is due to the pickPosition results . On my machine, If I set eyeOffset.z to a fixed value this doesn't happen.

2. I think it's worth adding scene.globe.depthTestAgainstTerrain = true; to the code in your comment, because this is not the default in the sandcastle.
   pickPosition returns different values on terrain if depthTestAgainstTerrain == false , usually below the ellipsoid surface (maybe of the DepthPlane).

3. The Alt string in this example code is truncated so it shows an wrong height.
   for example: -143976.41485355987 becomes 3976.41.

[denverpierce]

What GPUs/Browsers does everyone run into this on? I get this pretty consistently on a discrete AMD GPU, which makes me think that @kring idea (caught up with the zoom issues which were exacerbated by this issue) that it's depth buffer related with AMD cards may have something to do with it.

[jbo023]

@lilleyse
After some hours of debugging i found a possible solution for this.

The problem is that scene.pickPosition sometimes returns a negative height value if used in combination with scene.pick.

I noticed that the number of frustums in the pickPosition rendering pass differs from the number of frustums in the normal rendering update, when pickPosition delivers a corrupt result.

After some digging I figured that the scene.pick function changes the framestate cullingVolume which is then also used from the pickPosition function.

So when I use the pickPosition function without calling any scene.pick() in the same frame, pickPosition will use the cullingVolume from the last rendering pass and delivers the correct position result.

var viewer = new Cesium.CesiumWidget('cesiumContainer', {
        scene3DOnly : true,
        shadows : true
    });

    //var viewer = cesium.getViewer();

    var scene = viewer.scene;

    viewer.camera.up = new Cesium.Cartesian3(0.602242291265668, 0.47745810596428245, 0.6397952638618689);
    viewer.camera.direction = new Cesium.Cartesian3(-0.09199309329813982, 0.8376012488044867, -0.5384806577646075);
    viewer.camera.position = new Cesium.Cartesian3(3785056.8752774675, 901611.4457953185, 5037174.477028298);


    var tileset = scene.primitives.add(new Cesium.Cesium3DTileset({
        url : 'https://d35ei6ur3bjvr1.cloudfront.net/berlin/3c2a32d3-633c-462d-b79c-7215dcfbc44f',
    }));

    var dataSourceCollection = new Cesium.DataSourceCollection();
    var dataSourceDisplay = new Cesium.DataSourceDisplay({
          scene : scene,
          dataSourceCollection : dataSourceCollection
        });

        var clock = viewer.clock;

        var eventHelper = new Cesium.EventHelper();
        clock.onTick.addEventListener(function(clock){
          var time = clock.currentTime;
          dataSourceDisplay.update(time);
        }.bind(this));

    // Mouse over the globe to see the cartographic position
    var handler = new Cesium.ScreenSpaceEventHandler(viewer.scene.canvas);
    handler.setInputAction(function(click) {
        var foundPosition = false;

        var scene = viewer.scene;
        var pickedOject = scene.pick(click.position);        
        if (true) {
            var position = viewer.scene.pickPosition(click.position);            
            var carto = Cesium.Cartographic.fromCartesian(position);
            console.log('pos' + carto);
            
            
            if (Cesium.defined(position)) {
                dataSourceDisplay.defaultDataSource.entities.add({
                    position : position,
                    point : {
                        pixelSize : 10,
                        color : Cesium.Color.YELLOW
                    }
                });
            }
        }
    }, Cesium.ScreenSpaceEventType.LEFT_CLICK);

With this Sandcastle code i could reproduce the corrupt height value if i click on the building in the center of the screen. If you just remove the line var pickedOject = scene.pick(click.position); pickPosition works correct.

[lilleyse]

This helps a lot @jbo023.

So why does it matter that the pick pass renders lesser frustums if it is still writing out depth correctly? Is the pick depth-copy broken? Just questions I'm thinking about right now...

[bn-dignitas]

@jbo023 Thanks for that analysis. It has been a couple years since your post, but I ran into something similar with Cesium 1.58, calling drillPick and then pickPosition right after with incorrect results. Luckily I was able to just swap the order of operations to work around it.

[1nstinct]

I faced the same issue - pickPosition returns wrong position. It can be seen while moving camera around. Tried to use globe.depthTestAgainstTerrain = true - it works but all entities that were on the surface dive into it now. The only solution that I can see at this moment is adding height: 10 to all the surface entities to make them lying above the surface.

[ggetz]

Also reported in #11043

[worldpeaceenginelabs]

@ggetz Hi, i am the guy from #11043
Video: https://twitter.com/peace_engine/status/1615317374009511937

You wrote "The workaround for your case would be to use either scene.camera.pickEllipsoid if you don't have terrain or scene.globe.pick if you do have terrain."

I understand the solution. Thanks a lot.

But don't understand how my actual code picks like 10 times the right position and than randomly 1 wrong position, then again like 10 times alright.

How is that possible? Shouldn't it return all positions right, or all wrong, but not both?

[ggetz]

Hi @worldpeaceenginelabs, I think we're still trying to pin down the exact cause, however there are several possible contributing factors, such as the multiple render passes performed for rendering and picking, multiple frustums, and asynchronous terrain tile loading. It could be a possible race condition related to any of the above, which would explain why the error can occur intermittantly.

[worldpeaceenginelabs]

@ggetz hmm, thats bad. This bug and the "touchscreen two finger pinch flip" bug are there for years now.

1. Does the Cesium staff not work on this repositories? Is it completly community dependant?
   Is the Cesium company still successful or on a way down? (asking to estimate the actual motivation behind CesiumJS)

2. Could i donate money to someone for making this two bugs a high priority?

3. Is the solution you gave me mature enough that i can go into production?
   I am a bit afraid of these unsolved bugs coming up while in production... (they are the only ones i found while testing, and they really su##...) 😅

[ggetz]

@worldpeaceenginelabs I understand the frustration with these long-running bugs. I'm assuming you're also referring to #4363?

Does the Cesium staff not work on this repositories? Is it completely community dependent?
Is the Cesium company still successful or on a way down?

Cesium as a company has been growing, though we've been a bit of a victim of our own success with the amount of inbound and opportunity. As such, we've needed to prioritize accordingly. CesiumJS is still very much a cornerstone of Cesium, and where most of our growth and revenue comes from. We invested more resources in CesiumJS this past year in 2022. Though many of the recent improvements have been focused on modernizing the codebase rather than bug fixes, we're trying to tackle more user-facing improvements and bug fixes in the near future.

Could i donate money to someone for making this two bugs a high priority?

I'm checking if its policy to take donations. In the meantime, I noticed this rear its head in #11033 which may end up affecting priority as well.

Is the solution you gave me mature enough that i can go into production?

IMO yes. pickPosition and globe.pick/camera.pickEllipsoid work through different mechanisms.

[ggetz]

Also reported in #10806 with globe.translucency.enabled = true.

[ggetz]

Possible fixes discussed in #8179.

[ggetz]

Thank you @worldpeaceenginelabs for the kind offer, but we cannot accept donations. We happily accept contributions from the community, including fixes and test cases. Additionally, we do consider service engagements that benefit the community as a whole. If that sounds applicable to your case, please fill out our contact form with any details and someone can follow up with you: https://cesium.com/contact/.

[lilleyse]

Related issue: #11031

[ggetz]

#11031 is likely caused by this issue directly, but only arose after the Model refactor changed how point clouds were handled.

There are some good debugging notes from @UniquePanda in that issue.

[ggetz]

Based on the debugging that @UniquePanda did, along with some of our own, the problem here might be linked to a bad depth copy around here, where existing values are being overwritten. That could be affecting other picking functions as well, such as sampleHeight.

To solve, either pickPosition should look both at the pick and depth textures and chose the the closest value, or we could possibly move pick depth earlier and composite in later values.

Additionally, this does not eliminate the possibility that multifrustum may still be playing into the issue.

[ggetz]

Also reported in #11313.