Pusher always fails with DNSLookupError if hostname is in hosts file

[MayeulC]

Description

Original report by @xundeenergie in #7113 (comment)

I think I have the same symptoms, here are a few notes on my install:

• self-hosted synapse, on bare-metal (yunohost, debian buster)
• unifiedpush pusher (ntfy) on the same host (in a docker container)
• I can push via curl just fine from that host
• I have confirmed DNS resolution works on that host, including via that twisted testdns.py script
• SchildiChat's notification troubleshooting is all green, but it probably calls the endpoint itself, not relying on synapse
• I have not received a push notification in months
• In contrast with Pusher stops pushing when it falls behind, with lots of DNSLookupFailed errors #7113, server is not overloaded as I am the only user
• My pusher is also a cname, @reivilibre suggested it might be linked, I'll try changing it to an A record.
• The push hostname is present in /etc/hosts <- later identified as the issue

Metrics screenshots:

Synapse logs:

python[18681]: 2022-05-13 09:31:05,397 - synapse.push.httppusher - 402 - WARNING - httppush.process-48492- Failed to push event $ZZZZZZZZZZZZZZZZ to @mayeul:mayeul.net/de.spiritcroc.riotx.up/https://push.example.com/upYYYYYYYYYY?up=1: <class 'twisted.internet.error.DNSLookupError'> DNS lookup failed: no results for hostname lookup: push.example.com.
python[18681]: 2022-05-13 09:31:05,396 - synapse.push.httppusher - 402 - WARNING - httppush.process-48491- Failed to push event $ZZZZZZZZZZZZZZZZ to @mayeul:mayeul.net/chat.fluffy.fluffychat.CVJMXCBTKU/https://push.example.com/upXXXXXXX?up=1: <class 'twisted.internet.error.DNSLookupError'> DNS lookup failed: no results for hostname lookup: push.example.com.
python[18681]: 2022-05-13 09:30:22,066 - synapse.push.httppusher - 402 - WARNING - httppush.process-48490- Failed to push event $ZZZZZZZZZZZZZZZZ to @mayeul:mayeul.net/de.spiritcroc.riotx.up/https://push.example.com/upYYYYYYYYYY?up=1: <class 'twisted.internet.error.DNSLookupError'> DNS lookup failed: no results for hostname lookup: push.example.com.
python[18681]: 2022-05-13 09:30:22,064 - synapse.push.httppusher - 402 - WARNING - httppush.process-48489- Failed to push event $ZZZZZZZZZZZZZZZZ to @mayeul:mayeul.net/chat.fluffy.fluffychat.CVJMXCBTKU/https://push.example.com/upXXXXXXX?up=1: <class 'twisted.internet.error.DNSLookupError'> DNS lookup failed: no results for hostname lookup: push.example.com.
python[18681]: 2022-05-13 09:29:56,230 - synapse.push.httppusher - 402 - WARNING - httppush.process-48487- Failed to push event $ZZZZZZZZZZZZZZZZ to @mayeul:mayeul.net/de.spiritcroc.riotx.up/https://push.example.com/upYYYYYYYYYY?up=1: <class 'twisted.internet.error.DNSLookupError'> DNS lookup failed: no results for hostname lookup: push.example.com.

I replaced my pusher host name as the ntfy instance is public, and I am not sure of the trade-offs.

Steps to reproduce

TBD

Version information

If not matrix.org:

• Version: 1.56.0
• Install method: yunohost app
• Platform: yunohost (debian buster bare metal)

[MayeulC]

After more testing:

• Setting it to an A record has no effect
• Removing the line in /etc/hosts fixes it.

This is probably a bug with DNS resolution inside twisted.

[rami-dabain]

I think i have similar problem.
I am hosting a synpase + postgres + mautrix-telegram over docker-compose
everything works fine for 5 minutes, then i start receiving those errors:

synapse    | 2022-07-27 18:55:16,103 - synapse.http.client - 455 - INFO - as-recoverer-telegram-2 - Error sending request to  PUT http://mtele:29317/transactions/1128?access_token=<redacted>: DNSLookupError Couldn't find the hostname 'mtele'
synapse    | 2022-07-27 18:55:16,103 - synapse.appservice.api - 323 - WARNING - as-recoverer-telegram-2 - push_bulk to http://mtele:29317/transactions/1128 threw exception(DNSLookupError) DNS lookup failed: Couldn't find the hostname 'mtele'. args=("Couldn't find the hostname 'mtele'",)

but first 5 minutes or so, everything is working fine:

synapse    | 2022-07-27 18:49:39,952 - synapse.http.client - 446 - INFO - as-sender-telegram-108 - Received response to PUT http://mtele:29317/transactions/1126?access_token=<redacted>: 200
synapse    | 2022-07-27 18:49:44,340 - synapse.http.client - 446 - INFO - as-sender-telegram-109 - Received response to PUT http://mtele:29317/transactions/1127?access_token=<redacted>: 200

it's as if DNS entries are lost after some time!

[richvdh]

everything works fine for 5 minutes, then i start receiving those errors:

this sounds like #7113.

[MayeulC]

I think this was just an issue where 127.0.0.1 was blacklisted by default: #7113 (comment) #10224 . This is the address I was putting in my host file, while the DNS was returning my (hairpinned) external IP.

I just had another issue where the DNS lookup would "fail" according to the log (while using testdns.py from twisted was returning 127.0.0.1 without complaining). Adding the loopback address to /etc/hosts explicitly worked around that other issue, somehow, so this confirms that the current issue is, if not mistaken, at least fixed.