Mismatch between gt_mask and pred_mask for instance segmentation visualization

[zeyu659]

Hi：@filaPro @oneformer3d-contributor
Thank you for your wonderful work.
However I encountered a problem in the process of visualizing the results, whether it is my own dataset or the scannet dataset, I successfully ran test.py and got the gt_mask and pred_mask of instance_segmentation, but when I used the indexes corresponding to the same labels in the 2 above mentioned masks, search the points in point cloud and visualize it, I found that the instance objects corresponding to the point cloud representations of gt_(red)_ and pred_(blue)_ are not the same, as shown below:

When the results such as IOU and AP have been 80% or so, the instance segmentation results should be almost correct, am I doing the visualization correctly: that the point cloud corresponding to the same label in gt and pred_mask is the same target object? Or is my calculation wrong?

[oneformer3d-contributor]

Don't quite understand how you match gt and pred? The same object should have different indexes in gt and pred.

[zeyu659]

@oneformer3d-contributor
Thanks for your prompt answer. Could you please answer the next 2 questions?
The final labels of gt and pred for the same target object are not corresponding, is it because the predicted instances were re-ranking using matrix-NMS as mentioned in 3.4 Inference in the paper? Then how are the inconsistent labels calculated when calculating the loss function, and the AP0.5 metric for the final instance segmentation?

Another question: I ran the model in my own dataset (scaffolding in construction), expecting to segment the cylindrical poles for instance_segmentation, as shown in the figure for gt_mask.

But the results obtained are AP_0.5=0.57 for the 'horizontal poles' class, and AP_0.5 for the other 2 classes are 0.90 and 0.82.
My current analysis is that some of the rods in this category are too close together resulting in inaccurate instance segmentation results, as shown in the figure with short blue lines indicating different rods. Is there any way to improve the AP_0.5 results for this category?