-
Notifications
You must be signed in to change notification settings - Fork 269
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
configs which contain jaeger config will hang the router when updated #331
Labels
Comments
garypen
changed the title
changing config to remove and re-add jaeger opentelemetry details hangs telemetry
configs which contain jaeger config will hang the router when updated
Jan 18, 2022
garypen
added a commit
that referenced
this issue
Jan 18, 2022
Incorporate the fix for #331 and re-structure the code a little. Framework for statistic reporting is now working reliably with router configuration changes. Main outstanding work is: - Implement normalization - Implement real statistics - Add support for OTLP configuration
garypen
added a commit
that referenced
this issue
Jan 19, 2022
- use spawn_blocking() in preference to spawning a thread - update the comment block to refer directly to issue #331
garypen
added a commit
that referenced
this issue
Jan 19, 2022
…#337) * configs which contain jaeger config will hang the router when updated If a configuration is updated which contains jaeger configuration, then at some point a synchronous call to: opentelemetry::global::set_tracer_provider() will arise. Even though this call is made from a synchronous function, the root context is asynchronous and it seems to cause the runtime to mis-behave: i.e.: hang... If we make the call from a newly spawned thread (no performance concerns here because re-configuration is not hot code) then the problem goes away. * Address review feedback - use spawn_blocking() in preference to spawning a thread - update the comment block to refer directly to issue #331 fixes: #337
tinnou
pushed a commit
to Netflix-Skunkworks/router
that referenced
this issue
Oct 16, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
To Reproduce
Create a configuration for the router which contains jaeger opentelemetry. e.g.:
Save this in a yaml config file, e.g.: config_with_jaeger.yml
Now run the router as follows:
The important part is to enable config watching (
-w
) so that changes to the config file cause the router to respond. With the router running, edit the config file to remove theopentelemetry
entry. Observe the server respond. Now add theopentelemetry
entry back into the configuration and note that the router does not respond. At this point if you try to shut down the router,Ctrl-C
, it will not respond and you can only shut down the router withkill -9
.Expected behavior
I expect to be able to change configuration details and have the router respond when it is watching configuration. I also expect to be able to cause the router to shut down with
Ctrl-C
at any time.Output
Sample interaction with router (configuration edits performed in a separate terminal)
Desktop (please complete the following information):
Additional context
It may be that this is an issue with OTLP telemetry configuration as well. It may be a more general problem, which is not telemetry specific - hard to say without more investigation.
The text was updated successfully, but these errors were encountered: