TCP Socket errors - Transport endpoint is already connected

[oini]

Hi there! We're using this module to log metrics for a high-frequency ETL service that is running on ECS where we have been noticing TCP socket 106 errors. A CloudWatch Agent container is setup as a sidecar alongside the application server in the ECS Task Definition that is responsible for publishing the metrics.

I'm trying to understand the root cause of these errors and how to avoid them. Any help is appreciated. Thanks!

Exact error:

[ERROR][tcp_client] Failed to connect to the socket. [Errno 106] Transport endpoint is already connected

[jaredcnance]

This looks like a bug to me. TcpClient is not thread-safe, so if you have multiple threads concurrently sending metrics on startup, then this seems possible. We should catch and swallow this particular error on connect. Do you know if you have a single TcpClient or if the same client is shared across your application?

aws-embedded-metrics-python/aws_embedded_metrics/sinks/tcp_client.py

Lines 34 to 35 in c85dcdd

	self._sock.connect((self._endpoint.hostname, self._endpoint.port))
	self._should_connect = False

[oini]

Do you know if you have a single TcpClient or if the same client is shared across your application?

While MetricsLoggers are not shared across threads, it looks like each instance of MetricsLogger is pointing to the same Environment which has a single AgentSink which has a single TcpClient. So yes, I would say it's shared across the application with multiple loggers flushing to the CW agent at once.

[ben51]

I've had a similar issue running an multi-threaded application on ECS. It seems that there was a temporary Network issue between the CW agent container and my application that lead to a socket error but the metric logger was never able to recover from it.

An extract from the logs:

2020-06-22T01:54:53.496 [INFO] aws_embedded_metrics.sinks.tcp_client:51 - Submitted metrics to agent over TCP.
2020-06-22T01:54:58.449 [INFO] aws_embedded_metrics.sinks.tcp_client:51 - Submitted metrics to agent over TCP.
2020-06-22T01:58:36.412 [ERROR] aws_embedded_metrics.sinks.tcp_client:56 - Failed to write metrics to the socket due to socket.error. [Errno 113] No route to host
2020-06-22T01:58:36.414 [ERROR] aws_embedded_metrics.sinks.tcp_client:40 - Failed to connect to the socket. [Errno 106] Transport endpoint is already connected
2020-06-22T01:58:36.418 [ERROR] aws_embedded_metrics.sinks.tcp_client:56 - Failed to write metrics to the socket due to socket.error. [Errno 32] Broken pipe
2020-06-22T01:58:36.418 [ERROR] aws_embedded_metrics.sinks.tcp_client:40 - Failed to connect to the socket. [Errno 106] Transport endpoint is already connected
2020-06-22T01:58:36.421 [ERROR] aws_embedded_metrics.sinks.tcp_client:56 - Failed to write metrics to the socket due to socket.error. [Errno 32] Broken pipe
2020-06-22T01:58:36.421 [ERROR] aws_embedded_metrics.sinks.tcp_client:40 - Failed to connect to the socket. [Errno 106] Transport endpoint is already connected
2020-06-22T01:58:36.422 [ERROR] aws_embedded_metrics.sinks.tcp_client:40 - Failed to connect to the socket. [Errno 106] Transport endpoint is already connected
2020-06-22T01:58:36.422 [ERROR] aws_embedded_metrics.sinks.tcp_client:56 - Failed to write metrics to the socket due to socket.error. [Errno 32] Broken pipe
2020-06-22T01:58:36.422 [ERROR] aws_embedded_metrics.sinks.tcp_client:40 - Failed to connect to the socket. [Errno 106] Transport endpoint is already connected
2020-06-22T01:58:36.423 [ERROR] aws_embedded_metrics.sinks.tcp_client:40 - Failed to connect to the socket. [Errno 106] Transport endpoint is already connected
2020-06-22T01:58:36.423 [ERROR] aws_embedded_metrics.sinks.tcp_client:56 - Failed to write metrics to the socket due to socket.error. [Errno 32] Broken pipe