increased latency with multi camera in the same process

[svensht2]

We're using RS415 with FW 5.8.14.0. librealsense 2.8.3, Windows 10 1709.
When using one RS415 we're seeing low latency, but when we connect 3 cameras we see a very obvious increase in latency. We can reproduce the issue in the multi-cam example app.
I'm attaching links to 2 videos showing the issue:
one camera: https://drive.google.com/open?id=1eTAxBEujdxrIbn6UgkqmwUVRpPiCoJNx
three cameras: https://drive.google.com/open?id=1Aqve0U4FBlgJPOMnHyQRbiwyz5tRMMor

We also ran an experiment where we run each camera in a separate process at the same time, and the latency was low, just like in one cameras case.
We have a dedicated USB controller per cameras.

We are seeing increased latency with DS430 when running 3 in parallel as well.

[dorodnic]

Hi @svensht2
Thanks for reporting, we will try to reproduce and localize the source of the latency.
Just to clarify, are you seeing this on just Windows 10 1709 or also on Linux?
In the meantime, can you please change the boolean at image.cpp:846 from true to false and see if it has any effect?
I suspect that since depth frames are provided to the application without being copied to a dedicated buffer, and the main-loop is running much slower, we don't return depth frames to the underlying subsystem (be it Media Foundation or V4L2) fast enough, introducing latency.

[svensht2]

@dorodnic sorry, copy paste error. The system we're using is Ubuntu 16.04, we did not test this on windows.
we will try it out

[svensht2]

Tried your suggestion, seem to improve things a bit, but issue is still there. You suspicion didn't quite made sense to me for our application (in which we still see latency) since we hold the frame for a very short time, however we do run at a slightly slower FPS than 30. I looked into the user space queue/dequeue and I did find that you allow for a queue of size QUEUE_MAX_SIZE = 10; When I set this number to 1 the issue goes away, even without your suggestion.
In a previous iteration of your API you allowed the user to set the queue depth, but I don't see the option for that anymore.

[aangerma]

Hi
I tried to reproduce this issue locally, can you make sure you build the library with CMAKE_BUILD_TYPE=Release ?

[aangerma]

I have localized the problematic queue to pipeline.h:19. When main is taking too long, this queue was reaching its maximum capacity.
Please disregard the comment about image.cpp.
I was only able to reproduce the problem in Debug.

[svensht2]

@aangerma you're right, I think I was running multi-cam example in Debug, but my application was running in Release.
What is the expected behavior if my application is running at 25 FPS and the stream is set to 30? currently I think it will build up a 10 frames latency.

[svensht2]

nm I see #938

[dorodnic]

The PR should improve the situation, even in Debug. Please close the issue if you have no additional questions on this topic