-
-
Notifications
You must be signed in to change notification settings - Fork 1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Own traefik proxy: syslog flooded with sync calls (10 GB per day) #2664
Comments
Are you using mautrix-signal too? |
Please let's not confuse probably unrelated issues.
You mean a loop? Requests are passed from proxy to proxy companion, could also be massive spam from mautrix-signal, although all requests looking exactly the same surely looks fishy. Does it disappear if you stop just mautrix-signal? |
You're right... I looked more closely at OP's logs. I too have the GET requests to /sync failing but with different specifics. Also, as I noted above (now deleted), my issues are not related to mautrix-signal. Sorry for the consuion. |
havin the same issue. server gets stuck every few hours due to extreme logfile flooding, looks the same as OP |
Indeed, the heavy log files seem to be related to using mautrix-signal in my setup. If I stop two signal containers (matrix-mautrix-signal and matrix-mautrix-signal-daemon), the error messages stop appearing. They appear again when the services are automatically relaunched. Not sure 100% though though whether the calls are indeed recursive... As a temporary workaround, I have disabled the log messages from these two services in /etc/rsyslog.d/matrix.conf:
Nevertheless, the problem is not solved and leads to client applications (e.g. MacOS element) to hang (probably due to countless queued backend calls) after having run for some time such that they need to be killed and restarted in order to work again. It would be great if I would know how to continue debugging / getting more information in order to help solve this. |
It might not be relevant, but what you describe sounds similar to what happens if Postgres runs out of connections. You could try to increase the max number of connections as described here: |
I am having this issue as well. I upgraded to the latest version of the playbook today and upgraded postgres. I also increased the number of max connections as per the link from QEDed. I disabled all bridges (slack, whatsapp, discord) but the issue remains. The logs indicate that it is Element itself that is causing the issue. I've attached a snippet that pretty much repeats until the log is full. Here you can see different clients accessing via different browsers and the app as well. Everything works normally until the logs fill up and there's no more space on the server. |
@geoidwalker that looks like presence spam caused by |
Indeed, this might have been the case - strange that the two issues describing the presence spam have been created months later after my observation. I have just enabled syslog logging again and log messages seem to be fine now. If this persists for the next week or so, I am going to close this issue. Thanks for the info! |
I'm using this playbook reverse-proxied behind apache.
30% of them look like this:
|
There was no more activity - in my case, the presence spam was the reason, updating to newer clients solved this issue |
An old client can still cause this issue? So... they didn't actually fix it, did they? |
See above, it has been fixed, this issue is closed |
Describe the bug
I am using the current version of the playbook with an own traefik frontend, configuration has been done according to the guidelines in this repo. However, after some time, my system's syslog messages start filling up with "circular" sync calls between the matrix-nginx-proxy (which I have understood is still there) and the matrix-synapse-reverse-proxy-companion. They fill up about at least 10 GB per day, so currently I have to manually delete them and reboot my server in order not to fill up the hard disk.
To Reproduce
My
vars.yml
file looks like this:Expected behavior
Earlier, my matrix installation was running as expected, without gigabytes of log messages containing apparent "circular" calls.
Matrix Server:
Ansible:
(not the issue)
Client:
(probably not the issue, has worked flawlessly before the move to traefik)
Additional context
/var/log/syslog and /var/log/daemon.log look like (small snippet, this fills up gigabytes - with a few other messages obviously):
The text was updated successfully, but these errors were encountered: