Do not stop collecting events when journal entries change

[kvch]

Previously sd_journal_wait was not used. From now on all changes to journals are detected.

I also added custom seccomp policy to Journalbeat.

Closes #9533

[kvch]

Failing tests are unrelated.

[urso]

Would we need a seccomp policy for ARM as well?

[urso]

this looks like waitForChange deadlocking -> go-routine/mem leak if waitUntilNewEventOrError returns after waitForChange is started but returns before it reads from changesChan. Due to the Wait on shutdown of waitUntilNewEventOrError, this function might be deadlocked as well.

[kvch]

Failing tests are unrelated on Travis.

[amencarini]

Hi! I'm trying to use the changes of this PR and I'm getting a massive wall of messages such as:

2019-01-15T16:26:39.084Z ERROR [input] input/input.go:165 Error while reading event: error while waiting for event: operation not permitted {"id": "9dde1bfa-fe25-44f6-9869-1a8f7beaed99"}

I'm building the journalbeat in an ubuntu container, and deploy it in an ubuntu container. Any suggestion as to what I might be doing incorrectly? My initial suspicion was that I'm not building it correctly but I am able to build v6.5.4.

[kvch]

@amencarini The errors show up because Journalbeat tries to run syscalls which are not listed in the seccomp policy. As a workaround, you could set seccomp.enabled to false. Alternatively, you could pass a seccomp profile to the container: https://docs.docker.com/engine/security/seccomp/

We need to fix this problem before releasing the new official Journalbeat image.

[amencarini]

@kvch thanks for your reply! Let me know if there is there anything I can do to help on this.

[amencarini]

Me again! I tried to put a 10s sleep before error while waiting for event: operation not permitted, and journalbeat in that scenario keeps shipping logs to Elasticsearch.

2019-01-17T15:02:50.002Z	ERROR	[input]	input/input.go:166	Error while reading event: error while waiting for event: operation not permitted	{"id": "2f872e4b-efba-4b54-bc40-68b03123ef52"}
2019-01-17T15:03:00.002Z	INFO	[input]	input/input.go:132	journalbeat successfully published 1 events	{"id": "2f872e4b-efba-4b54-bc40-68b03123ef52"}
2019-01-17T15:03:00.003Z	INFO	[input]	input/input.go:132	journalbeat successfully published 1 events	{"id": "2f872e4b-efba-4b54-bc40-68b03123ef52"}
2019-01-17T15:03:00.003Z	INFO	[input]	input/input.go:132	journalbeat successfully published 1 events	{"id": "2f872e4b-efba-4b54-bc40-68b03123ef52"}
2019-01-17T15:03:00.003Z	INFO	[input]	input/input.go:132	journalbeat successfully published 1 events	{"id": "2f872e4b-efba-4b54-bc40-68b03123ef52"}
2019-01-17T15:03:00.003Z	INFO	[input]	input/input.go:132	journalbeat successfully published 1 events	{"id": "2f872e4b-efba-4b54-bc40-68b03123ef52"}
2019-01-17T15:03:01.022Z	INFO	[input]	input/input.go:132	journalbeat successfully published 56 events	{"id": "2f872e4b-efba-4b54-bc40-68b03123ef52"}
2019-01-17T15:03:10.073Z	ERROR	[input]	input/input.go:166	Error while reading event: error while waiting for event: operation not permitted	{"id": "2f872e4b-efba-4b54-bc40-68b03123ef52"} 

So my impression is that whatever is involved in the sd_journal_wait C call is what's signalling a lack of permissions.

On the other hand, forcing the rotation of the logs via journalctl --rotate makes journalbeat stop shipping logs (only the go metrics and the operation not permitted lines get logged). Hope this helps!

[kvch]

Were you able to make it work using a seccomp profile?

[amencarini]

I haven't tried but I'm running this off Kubernetes and I can't find the equivalent option of security-opt. Happy to try it out if you can point me to it!

[urso]

Note: It's not too bad to wait for a few 100 millis, and I don't think we should change it here. But I'd prefer if we wouldn't have to wait. The Wait call uses sd_journal_wait, which itself uses ppoll. One should not close a file descriptor a poll operation is active on, but an alternate wait-without-wakeup-schema could be created by wasting a second polling file descriptor:

• create close-signaling file descriptor (e.g. create pipe, wastes 2 FDs)
• get journald fd via sd_journal_get_fd (it's the same journald polls on)
• poll on close-fd + journal-fd
• for handling async shutdown trigger close-fd (e.g. send some bytes)
• check who did send poll signal and handle accordingly

Requested changes have been applied. We should test if PR really helps with collector stopping and in which cases it might still fail.

[kvch]

I have just tested the seccomp policy for ARM in docker image balenalib/raspberrypi3-debian-golang and balenalib/raspberry-pi-debian-golang. It works as expected.