Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Otel grpc reconnection issue #1524

Closed
kstasik opened this issue Feb 10, 2021 · 3 comments
Closed

Otel grpc reconnection issue #1524

kstasik opened this issue Feb 10, 2021 · 3 comments
Labels
bug Something isn't working

Comments

@kstasik
Copy link
Contributor

kstasik commented Feb 10, 2021

Otel grpc client never gets reconnected:

2021/02/10 22:13:12 rpc error: code = Unavailable desc = connection closed
2021/02/10 22:13:14 rpc error: code = Unavailable desc = connection closed
2021/02/10 22:13:16 exporter disconnected
2021/02/10 22:13:18 exporter disconnected
2021/02/10 22:13:20 exporter disconnected
2021/02/10 22:13:22 exporter disconnected
2021/02/10 22:13:24 exporter disconnected
2021/02/10 22:13:26 exporter disconnected
2021/02/10 22:13:28 exporter disconnected
2021/02/10 22:13:30 exporter disconnected
2021/02/10 22:13:32 exporter disconnected
2021/02/10 22:13:34 exporter disconnected
2021/02/10 22:13:36 exporter disconnected
2021/02/10 22:13:38 exporter disconnected
2021/02/10 22:13:40 exporter disconnected
2021/02/10 22:13:42 exporter disconnected
2021/02/10 22:13:44 exporter disconnected
2021/02/10 22:13:46 exporter disconnected
2021/02/10 22:13:48 exporter disconnected
2021/02/10 22:13:50 exporter disconnected
2021/02/10 22:13:52 exporter disconnected
2021/02/10 22:13:54 exporter disconnected
2021/02/10 22:13:56 exporter disconnected
2021/02/10 22:13:58 exporter disconnected
2021/02/10 22:14:00 exporter disconnected
2021/02/10 22:14:02 exporter disconnected
2021/02/10 22:14:04 exporter disconnected
2021/02/10 22:14:06 exporter disconnected

How to reproduce:
a) otel-collector with replicas=1
b) application with configuration similar to:

  OTEL_EXPORTER_OTLP_TIMEOUT=10s \
  OTEL_RESOURCE_ATTRIBUTES=deployment.environment=test \
  OTEL_PROPAGATORS=tracecontext,baggage \
  OTEL_EXPORTER=otlp \
  OTEL_LOG_LEVEL=error \
  OTEL_EXPORTER_OTLP_PROTOCOL=grpc \
  OTEL_EXPORTER_OTLP_INSECURE=true \
  OTEL_EXPORTER_OTLP_ENDPOINT=127.0.0.1:32451

c) restart otel-collector pod (in local environment it needs few seconds to initialize)
d) after first try application stops trying to reconnect

Suggested solution:
#1521
@kstasik kstasik mentioned this issue Feb 10, 2021
Merged
@jmacd
Copy link
Contributor

jmacd commented Feb 10, 2021

Lightstep has been having trouble with these errors, too. Thank you @kstasik.
See also https://gitter.im/open-telemetry/opentelemetry-go?at=601d86aea0246860dc340034

@MrAlias MrAlias added the bug Something isn't working label Feb 11, 2021
kstasik added a commit to kstasik/opentelemetry-go that referenced this issue Feb 12, 2021
@kstasik
fix: grpc reconnection open-telemetry#1524
6492cf2
@MrAlias MrAlias mentioned this issue Feb 12, 2021
Closed
Aneurysm9 pushed a commit that referenced this issue Feb 17, 2021
@kstasik
fix: grpc reconnection (#1521)
8da5299 
* fix: grpc reconnection fixed

* chore: changelog update

* fix: grpc reconnection issue - red test

* fix: grpc reconnection #1524

* fix: grpc reconnection issue cleanup
@pkwarren
Copy link
Contributor

pkwarren commented Mar 3, 2021

Is this issue now fixed that #1521 has been merged?

@MrAlias
Copy link
Contributor

MrAlias commented Mar 3, 2021

@pkwarren it should be, yes. Thanks for double checking this.

@MrAlias MrAlias closed this as completed Mar 3, 2021
ldelossa pushed a commit to ldelossa/opentelemetry-go that referenced this issue Mar 5, 2021
@kstasik @ldelossa
fix: grpc reconnection (open-telemetry#1521)
a6903a7 
* fix: grpc reconnection fixed

* chore: changelog update

* fix: grpc reconnection issue - red test

* fix: grpc reconnection open-telemetry#1524

* fix: grpc reconnection issue cleanup
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
Go: Bugs
Archived in project
Milestone
No milestone
Development

No branches or pull requests
4 participants
@pkwarren @kstasik @jmacd @MrAlias