-
Notifications
You must be signed in to change notification settings - Fork 15
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Arrow Exporter load prioritizer apparatus #178
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The documentation and diagram on this significant addition of a new load balancing policy between streams have been very useful. However, I did not see the benchmark results. Were they displayed somewhere?
Thank you for this PR.
TL;DR the first numeric column is number of repetitions to meet the benchmark time threshold (where larger numbers == better), the second column is timing per repetition (smaller == better). There is a lot of synthetic aspect to this test, many additional channels created by the test apparatus--with a specific and unrealistic latency curve, but it's still meaningful. For the case of 128 streams, FIFO does 90 reps @ 12ms, LeastLoadedTwo does 162 reps @ 7.1ms, LeastLoadedFour does 142 reps @ 7.0ms |
@moh-osman3 Nice catch. The benchmarks improve. :-)
|
@moh-osman3 Please take another look. |
Adds a configurable prioritization scheme to the Arrow exporter, leaving the existing "FIFO" policy the default and adding a new "leastloadedN" policy which considers the least-loaded among N randomly selected streams.
Adds a README.md explaining the internals of the Arrow streaming component. Updates the original diagram.
Adds a benchmark that exercises the different prioritizers and demonstrates different performance characteristics.
This required a substantial redesign of the existing prioritization scheme, which had some unnecessary complexity. The new design:
streamWorkState
object across streamsExtends most tests to cover all prioritizers.
Fixes #147.