Suspected Memory Leak in linkerd-proxy

[christianhuening]

Bug Report

What is the issue?

linkerd-proxy container is using more than 35GB of memory
We're using TLS and auto-inject, running with Linkerd2 2.2 stable

How can it be reproduced?

Run a Pod with TLS. Happens occasionally

Logs, error output, etc

linkerd-proxy container output:

WARN admin={bg=tls-config} linkerd2_fs_watch::inotify watch error: Os { code: 22, kind: InvalidInput, message: "Invalid argument" }, polling the fs until next change

Metrics output (process_* only):

process_start_time_seconds 1550772831
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 7780
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 547
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1048576
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 52501831680
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 39958102016

(If the output is long, please create a gist and
paste the link here.)

linkerd check output

kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API

kubernetes-version
------------------
√ is running the minimum Kubernetes API version

linkerd-existence
-----------------
√ control plane namespace exists
√ controller pod is running
√ can initialize the client
√ can query the control plane API

linkerd-api
-----------
√ control plane pods are ready
√ can query the control plane API
√ [kubernetes] control plane can talk to Kubernetes
√ [prometheus] control plane can talk to Prometheus

linkerd-service-profile
-----------------------
√ no invalid service profiles

linkerd-version
---------------
√ can determine the latest version
‼ cli is up-to-date
    is running version 2.2.0 but the latest stable version is 2.2.1
    see https://linkerd.io/checks/#l5d-version-cli for hints

control-plane-version
---------------------
‼ control plane is up-to-date
    is running version 2.2.0 but the latest stable version is 2.2.1
    see https://linkerd.io/checks/#l5d-version-control for hints
√ control plane and cli versions match

Status check results are √

Environment

• Kubernetes Version: 1.13.2
• Cluster Environment: (GKE, AKS, kops, ...): bare-metal
• Host OS: Container Linux 1967.6.0
• Linkerd version: 2.2.0

Possible solution

N/A

Additional context

[hawkw]

Fairly sure this is caused by #2331. The polling-based fs watch implementation probably has a slow leak in it somewhere.

[ihcsim]

@christianhuening Can you try out the latest edge-19.2.5 release to see if it resolves this issue? Let us know how it goes.

[christianhuening]

@ihcsim updated, I'll report back, should I see another high mem use

[christianhuening]

That one didn't reappear. Closing for now

[hawkw]

@christianhuening Great, I'm glad to hear that this was fixed!

We should probably open a new ticket to resolve the underlying issue with the polling fs watch implementation. However, since fixing #2331 has resolved the issue that was causing us proxies to fall back to polling unnecessarily, it might be somewhat less urgent.