Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

proxy: inotify watch error on Debian stretch #2331

Closed
siggy opened this issue Feb 19, 2019 · 3 comments
Closed

proxy: inotify watch error on Debian stretch #2331

siggy opened this issue Feb 19, 2019 · 3 comments
Labels
area/proxy bug

Comments

@siggy
Copy link
Member

siggy commented Feb 19, 2019

#2241 bumps the proxy's base Docker image from debian:jessie-slim to debian:stretch-20190204-slim. When Linkerd is deployed via TLS, we observe repeated inotify watch errors:

linkerd linkerd-prometheus-86b4df8585-2ljml linkerd-proxy WARN admin={bg=tls-config} linkerd2_fs_watch::inotify watch error: Os { code: 22, kind: InvalidInput, message: "Invalid argument" }, polling the fs until next change
linkerd linkerd-prometheus-86b4df8585-2ljml linkerd-proxy WARN admin={bg=tls-config} linkerd2_fs_watch::inotify watch error: Os { code: 22, kind: InvalidInput, message: "Invalid argument" }, polling the fs until next change
linkerd linkerd-web-769c446cdb-f4br4 linkerd-proxy WARN admin={bg=tls-config} linkerd2_fs_watch::inotify watch error: Os { code: 22, kind: InvalidInput, message: "Invalid argument" }, polling the fs until next change
linkerd linkerd-grafana-5dd4875c88-rv4hl linkerd-proxy WARN admin={bg=tls-config} linkerd2_fs_watch::inotify watch error: Os { code: 22, kind: InvalidInput, message: "Invalid argument" }, polling the fs until next change
linkerd linkerd-controller-59bf7fb9cf-rj6pd linkerd-proxy WARN admin={bg=tls-config} linkerd2_fs_watch::inotify watch error: Os { code: 22, kind: InvalidInput, message: "Invalid argument" }, polling the fs until next change

Steps to reproduce

git checkout siggy/bump-all-the-dockers
DOCKER_TRACE=1 bin/docker-build
bin/linkerd install --tls optional | kubectl apply -f -
bin/linkerd logs

First reported at #2241 (comment)
@siggy siggy mentioned this issue Feb 19, 2019
Merged
@hawkw hawkw mentioned this issue Feb 21, 2019
Closed
@hawkw
Copy link
Member

hawkw commented Feb 21, 2019

It looks like this has been observed on stable-2.2 (see #2345) so I'm not convinced it's caused by switching from jessie to stretch?

@hawkw hawkw added the bug label Feb 21, 2019
@klingerf
Copy link
Member

@hawkw You're correct. I just verified that these warnings show up in the linkerd-proxy logs of the control plane components of a stable-2.2.1 deployment with TLS enabled on GKE. Am not seeing the corresponding increase in memory, however, as reported in #2345. And interestingly, I don't see these warnings when deploying demo to the same cluster -- in my case the warnings only show up in the control plane proxies.

@klingerf klingerf mentioned this issue Feb 22, 2019
Closed
@hawkw hawkw mentioned this issue Feb 22, 2019
Merged
@hawkw
Copy link
Member

hawkw commented Feb 22, 2019

linkerd/linkerd2-proxy#195 fixes this; we'll have to investigate whether or not this resolves the memory leak as well.

hawkw added a commit to linkerd/linkerd2-proxy that referenced this issue Feb 22, 2019
@hawkw
fs-watch: Use a properly sized buffer for inotify events (#195)
c54377f 
The buffer passed to `read(2)` to read inotify events from the inotify
fd must be long enough to fit events with filenames of the maximum
allowable length. The `inotify-rs` library previously always constructed
such a buffer, but as of 1fe4e38, we
are required to provide our own buffer. The buffer we were providing was
insufficiently large for some filenames.

See also the `man` page for `inotify`:
> The behavior when the buffer given to read(2) is too small to return
> information about the next event depends on the kernel version: in
> kernels before 2.6.21, read(2) returns 0; since kernel 2.6.21,
> read(2) fails with the error EINVAL.  Specifying a buffer of size
>
>         sizeof(struct inotify_event) + NAME_MAX + 1
>
> will be sufficient to read at least one event.

(http://man7.org/linux/man-pages/man7/inotify.7.html)

This branch updates the `fs-watch` lib to calculate the necessary buffer
length correctly, and use a properly-sized buffer. I've also added tests
that fail when the buffer is too short.

Fixes linkerd/linkerd2#2331

Signed-off-by: Eliza Weisman <eliza@buoyant.io>
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jul 18, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
area/proxy bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@klingerf @siggy @hawkw