support federation queries through http connect proxy

[dklimpel]

Can be specified by HTTPS_PROXY env var.
pass unfiltered reactor to federation agent for proxy support

Sorry for so much lines of code in one PR.
I have tried to do smaller commits.

Replaces: #9306
Fixes: #8660
Blocked by:

• allow specifying https:// proxy #10411
• Add documentation for forward proxy #10443

ToDo:

• Fix tests, hopefully there will be a solution in allow specifying https:// proxy #10411
• Update documentation from Add documentation for forward proxy #10443

Need help / review

• Solve task / comment: support federation queries through http connect proxy #9306 (comment)
  Unfortunately I do not know how to do this.

given that we already pass the ip blacklist into MatrixFederationAgent, why not move the construction of BlacklistingReactorWrapper down to it, rather than having to pass in two reactors?

• Choose the right TLS policy for connection with https proxy.

Pull Request Checklist

• Pull request is based on the develop branch
• Pull request includes a changelog file. The entry should:
  • Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
  • Use markdown where necessary, mostly for code blocks.
  • End with either a period (.) or an exclamation mark (!).
  • Start with a capital letter.
• Pull request includes a sign off
• Code style is correct (run the linters)

Signed-off-by: Dirk Klimpel dirk@klimpel.org

[dklimpel]

I am not sure if BrowserLikePolicyForHTTPS is the best default policy.

[richvdh]

I don't think we need a default policy here. tls_client_options_factory=None is supposed to disable TLS, not fall back to a default. I would make the tls_options_factory parameter to _http_proxy_endpoint Optional, and raise an Exception if the scheme is https but there is no tls factory.

[dklimpel]

Im not sure what is the best error to raise. ValueError, ConfigError, RuntimeError or anything else?

[richvdh]

looks generally great, thank you! Particular thanks for taking the time to figure out the tests.

A few minor suggestions here.

[richvdh]

per #9306 (comment):

I suggest you move the code at lines 330-334 which builds a BlacklistingReactorWrapper into MatrixFederationAgent. There is no need for MatrixFederationHttpClient.reactor to be a BlacklistingReactorWrapper.

[richvdh]

I don't think we need a default policy here. tls_client_options_factory=None is supposed to disable TLS, not fall back to a default. I would make the tls_options_factory parameter to _http_proxy_endpoint Optional, and raise an Exception if the scheme is https but there is no tls factory.

[richvdh]

should we just move this into HTTPConnectProxyEndpoint, to save doing it each time we construct one?

(in other words: make HTTPConnectProxyEndpoint take an Optional[ProxyCredentials] parameter instead of a custom headers parameter)

[dklimpel]

This change was larger.
I had to move ProxyCredentials from proxyagent to connectproxyclient. The reason was a circular import.
I have replaced headers parameter because it was introduced in #9657 only for proxy connections and is not needed anymore.

[richvdh]

I think it would be better not to create an agent here at all, than to create it and then recreate for some of the tests.

[sim0nx]

Am I correct to assume that this also solves #8859 , i.e. http proxy outbound federation ?

[dklimpel]

Am I correct to assume that this also solves #8859 , i.e. http proxy outbound federation ?

#8859 is a duplictae of #8660 (#8859 (comment))
With this PR you can make outbound federation requests via proxy. You can connect to this proxy either with http or https.
All federation request uses https (https_proxy env var).

[richvdh]

lgtm. Thank you so much for working on this!

[f1-outsourcing]

@dklimpel @richvdh
Do I understand correctly that this makes it possible to run synapse only through proxies (forward and reverse)? Without eg. having a default gateway available? Hi how can I test this?

Yes. That is the purpose for this PR.
You can do a pip install https://matrix-org.github.io/synapse/latest/setup/installation.html#installing-from-source
The source code for a test is: https://github.com/dklimpel/synapse/tree/federation_through_proxy

I am using tinyproxy currently for this test environment. I have connected the element webinterface and it authenticated against the synapse server. I am able to send messages between a user on the webinterface and a user on a mobile device. I have configured synapse to run with these environment variables:

HTTP_PROXY=http://xxxx.xxxx.xxxxxxxx.xxxxx:53245"
HTTPS_PROXY=https://xxxx.xxxx.xxxxxxxx.xxxxx:53245"
NO_PROXY=".local,localhost,127.0.0.1"

I am using firefox, and that seems to work for http and https sites with this proxy.

However when I 'explore public rooms' search for test on matrix.org, I am getting this error

Aug 11 21:32:14 xxx journal: matrix-synapse synapse.http.federation.matrix_federation_agent - 361 - INFO - POST-261 - Failed to connect to matrix-federation.matrix.org.cdn.cloudflare.net:8443: [('SSL routines', 'ssl3_get_record', 'wrong version number')]

To be sure I also used curl to test the connections, which seem ok.

curl --proxy http://xxxx.xxxx.xxxxxxxx.xxxxx:53245 http://httpbin.org/get
curl --proxy https://xxxx.xxxx.xxxxxxxx.xxxxx:53245 https://httpbin.org/get

If this is related to the certificate being used for outgoing connections (as maybe mentioned here #5684) How should I configure multiple certificates.

I have my synapse container running with task/host names '111.222.333.444.555' while my 'front end' is using a 'aaa.bbb.cccc' hostname. Currently most containers are running on my own CA so, multiple instances of synapse (workers?) will communicate via such hostnames 111.222.333.444.555.

[dklimpel]

Are you sure that tinyproxy is able to use it with HTTPS_PROXY=https://xxxx.xxxx.xxxxxxxx.xxxxx:53245"? I could not find an option to set a certificate for tinyproxy. HTTPS_PROXY=https:// means that Synapse connect to proxy via https and then request the target. But to connect to proxy via https the proxy needs a certificate. I would recommend to use HTTPS_PROXY=http://xxxx.xxxx.xxxxxxxx.xxxxx:53245"

[f1-outsourcing]

But incoming requests are going to my reverse proxy, haproxy and there I have the correct certificate. With the outgoing request to matrix-federation.matrix.org.cdn.cloudflare.net:8443 is there some client certificate being used and verified?

What does this exactly mean?
SSL routines', 'ssl3_get_record', 'wrong version number'

[f1-outsourcing]

Oh oh, looks like maybe cloudflare issue or where do you get this hostname from?

This is what I get when I open the url in firefox.

https://matrix-federation.matrix.org.cdn.cloudflare.net:8443/

Websites prove their identity via certificates. Firefox does not trust this site because it uses a certificate that is not valid for matrix-federation.matrix.org.cdn.cloudflare.net:8443. The certificate is only valid for the following names: about.riot.im, arewereadyyet.com, lfs.matrix.org, matrix-client.matrix.org, matrix-federation.matrix.org, matrix.org, matrix.to, modular.im, riot.im, spec.matrix.org, status.matrix.org, vector.im, www.matrix.org, www.modular.im

Error code: SSL_ERROR_BAD_CERT_DOMAIN

[f1-outsourcing]

synapse log

Aug 11 23:26:10 c04 journal: matrix-synapse synapse.handlers.presence - 755 - INFO - persist_presence_changes-0 - Persisting 1 unpersisted presence updates
Aug 11 23:26:10 c04 journal: matrix-synapse synapse.http.federation.well_known_resolver - 253 - INFO - POST-72 - Fetching https://matrix.org/.well-known/matrix/server
Aug 11 23:26:10 c04 journal: matrix-synapse synapse.http.federation.well_known_resolver - 284 - INFO - POST-72 - Error fetching https://matrix.org/.well-known/matrix/server: [('SSL routines', 'ssl3_get_record', 'wrong version number')]

Curl

[@test2 ~]#  curl --proxy https://proxy.xxxxxxxxxxxx:53245 'https://matrix.org/.well-known/matrix/server'
{ "m.server": "matrix-federation.matrix.org:443" }
[@test2 ~]#  curl --proxy http://proxy.xxxxxxxxxxxx:53245 'https://matrix.org/.well-known/matrix/server'
{ "m.server": "matrix-federation.matrix.org:443" }

[dklimpel]

How did you configure that tinyproxy supports https://proxy.xxxxxxxxxxxx:53245?
Whe I am use it. I get this:

curl --proxy https://localhost:8888 'https://matrix.org/.well-known/matrix/server' -v
*   Trying 127.0.0.1:8888...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8888 (#0)
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* error:1408F10B:SSL routines:ssl3_get_record:wrong version number
* Closing connection 0
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number

Tinyproxy log:

CONNECT   Aug 12 07:02:58 [267779]: Connect (file descriptor 6): localhost [127.0.0.1]
CONNECT   Aug 12 07:02:58 [267779]: Request (file descriptor 6): ^V^C^A^B
WARNING   Aug 12 07:02:58 [267779]: Could not retrieve all the headers from the client
INFO      Aug 12 07:02:58 [267779]: Read request entity of 341 bytes

This what I am expect. Because tinyproxy cannot create a connection with https. It has no TLS certificate for this.

A normal http connection with tinyproxy:

curl --proxy http://localhost:8888 'https://matrix.org/.well-known/matrix/server' -v
*   Trying 127.0.0.1:8888...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8888 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to matrix.org:443
> CONNECT matrix.org:443 HTTP/1.1
> Host: matrix.org:443
> User-Agent: curl/7.68.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.0 200 Connection established
< Proxy-agent: tinyproxy/1.10.0
<
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* CONNECT phase completed!
* CONNECT phase completed!
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=www.matrix.org
*  start date: Jun 12 02:08:30 2021 GMT
*  expire date: Sep 10 02:08:29 2021 GMT
*  subjectAltName: host "matrix.org" matched cert's "matrix.org"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x56166ddfc860)
> GET /.well-known/matrix/server HTTP/2
> Host: matrix.org
> user-agent: curl/7.68.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
< HTTP/2 200
< date: Thu, 12 Aug 2021 07:03:31 GMT
< content-length: 52
< last-modified: Thu, 12 Aug 2021 02:17:28 GMT
< etag: "34-5c95356209e00"
< x-xss-protection: 1; mode=block
< x-content-type-options: nosniff
< strict-transport-security: max-age=31536000; includeSubDomains; preload
< x-frame-options: sameorigin
< referrer-policy: strict-origin-when-cross-origin
< cache-control: max-age=14400
< permissions-policy: interest-cohort=()
< cf-cache-status: HIT
< age: 1346
< accept-ranges: bytes
< expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< server: cloudflare
< cf-ray: 67d7d825fe7134e4-DUB
<
{ "m.server": "matrix-federation.matrix.org:443" }
* Connection #0 to host localhost left intact

[f1-outsourcing]

btw thanks for helping and testing this, I really appreciate it. ;) I am really looking forward to having this synapse in my container environment.

This is the config I have. I just redacted some more allow lines. I have tinyproxy running on two networks one connected to the internet and one connected to a container/vm environment.

Maybe running this on localhost is not the best way to test, because local routing could mess up the outgoing traffic? My tinyproxy is running separate, from my test2 and synapse. All have their own ip addresses. I know for sure my environment is getting the ssl websites. Yesterday I even checked if the container was validating the certificates of matrix.org correctly, which are letsencrypt and curl just worked fine. To me it starts to look like some client certificate is being used communicating with matrix.org. But this error message is just not clear.

[@ tinyproxy]# cat tinyproxy.conf  | grep -v '^#'|sed  '/^$/d'
User tinyproxy
Group tinyproxy
Port 53245
Timeout 600
DefaultErrorFile "/usr/share/tinyproxy/default.html"
StatFile "/usr/share/tinyproxy/stats.html"
Syslog On
LogLevel Info
MaxClients 100
MinSpareServers 5
MaxSpareServers 20
StartServers 1
MaxRequestsPerChild 0
Allow 127.0.0.1
ViaProxyName "tinyproxy"
DisableViaHeader Yes

[@test2 root]$ curl --proxy http://xxxxxxxx:53245 'https://matrix.org/.well-known/matrix/server' -v
* About to connect() to xxxxxxxxs port 53245 (#0)
*   Trying 192.168.122.21...
* Connected to xxxxxxxxx (192.168.122.21) port 53245 (#0)
* Establish HTTP proxy tunnel to matrix.org:443
> CONNECT matrix.org:443 HTTP/1.1
> Host: matrix.org:443
> User-Agent: curl/7.29.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.0 200 Connection established
< Proxy-agent: tinyproxy/1.10.0
<
* Proxy replied OK to CONNECT request
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
*       subject: CN=www.matrix.org
*       start date: Jun 12 02:08:30 2021 GMT
*       expire date: Sep 10 02:08:29 2021 GMT
*       common name: www.matrix.org
*       issuer: CN=R3,O=Let's Encrypt,C=US
> GET /.well-known/matrix/server HTTP/1.1
> User-Agent: curl/7.29.0
> Host: matrix.org
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Thu, 12 Aug 2021 10:43:11 GMT
< Content-Length: 52
< Connection: keep-alive
< Last-Modified: Thu, 12 Aug 2021 02:17:28 GMT
< ETag: "34-5c95356209e00"
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
< X-Frame-Options: sameorigin
< Referrer-Policy: strict-origin-when-cross-origin
< Cache-Control: max-age=14400
< Permissions-Policy: interest-cohort=()
< CF-Cache-Status: HIT

PS. If you want access to my test environment, you can have it.

[dklimpel]

I think you env var is not correct.
You use

HTTP_PROXY=http://xxxx.xxxx.xxxxxxxx.xxxxx:53245"
HTTPS_PROXY=https://xxxx.xxxx.xxxxxxxx.xxxxx:53245"
NO_PROXY=".local,localhost,127.0.0.1"

You have to use:

HTTP_PROXY=http://xxxx.xxxx.xxxxxxxx.xxxxx:53245"
HTTPS_PROXY=http://xxxx.xxxx.xxxxxxxx.xxxxx:53245"
NO_PROXY=".local,localhost,127.0.0.1"

You can connect to your proxy via http and not https.
The error message is not a message from Synapse.
[('SSL routines', 'ssl3_get_record', 'wrong version number')] is from SSL library.
You try to connect with TLS (HTTPS_PROXY=https://xxxx.xxxx.xxxxxxxx.xxxxx:53245") to your proxy. But the proxy does not support TLS/https. It supports only http (HTTPS_PROXY=http://xxxx.xxxx.xxxxxxxx.xxxxx:53245").

[f1-outsourcing]

I think tinyproxy by default supports ssl/tls and just switches automatically based on the protocol it receives. I think you should be able to do this[1] also. It does not really make sense not supporting ssl these days. I will try and have a look at this (ssl?) library and see what this error means.

[1]

[@test2 root]$ curl --proxy https://xxxxxxxx:53245 'https://matrix.org/.well-known/matrix/server' -v
* About to connect() to proxy proxy.dev.marathon.mesos port 53245 (#0)
*   Trying 192.168.122.21...
* Connected to xxxxxxxxx (192.168.122.21) port 53245 (#0)
* Establish HTTP proxy tunnel to matrix.org:443
> CONNECT matrix.org:443 HTTP/1.1
> Host: matrix.org:443
> User-Agent: curl/7.29.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.0 200 Connection established
< Proxy-agent: tinyproxy/1.10.0
<
* Proxy replied OK to CONNECT request
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
*       subject: CN=www.matrix.org
*       start date: Jun 12 02:08:30 2021 GMT
*       expire date: Sep 10 02:08:29 2021 GMT
*       common name: www.matrix.org
*       issuer: CN=R3,O=Let's Encrypt,C=US
> GET /.well-known/matrix/server HTTP/1.1
> User-Agent: curl/7.29.0

[f1-outsourcing]

Do you have a code snippet that I can use to test with? Something that would generate the same error message in my environment?
Maybe it is related to how the connection is made? Here they are writing about ssl records. openssl/openssl#10938 (comment)
In his case he was not reading the first packet correctly

[dklimpel]

I do not have a code snippet.
Extract it from Synapse is not so easy.

I did a request with a proxy that is able to talk https (squid), I can see 2 TLS handshakes.
First for proxy connection and second for connection with matrix.org.

curl --proxy https://localhost:3129 'https://matrix.org/.well-known/matrix/server' -v --proxy-insecure
*   Trying 127.0.0.1:3129...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 3129 (#0)
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Proxy certificate:
*  subject: C=DE; ST=Some-State;
*  start date: Jul 16 12:10:33 2021 GMT
*  expire date: Jul 16 12:10:33 2022 GMT
*  issuer: C=DE; ST=Some-State;
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
* allocate connect buffer!
* Establish HTTP proxy tunnel to matrix.org:443
> CONNECT matrix.org:443 HTTP/1.1
> Host: matrix.org:443
> User-Agent: curl/7.68.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 Connection established
<
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* CONNECT phase completed!
* CONNECT phase completed!
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=www.matrix.org
*  start date: Jun 12 02:08:30 2021 GMT
*  expire date: Sep 10 02:08:29 2021 GMT
*  subjectAltName: host "matrix.org" matched cert's "matrix.org"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x557a37b08860)
> GET /.well-known/matrix/server HTTP/2
> Host: matrix.org
> user-agent: curl/7.68.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
< HTTP/2 200
< date: Thu, 12 Aug 2021 12:13:31 GMT
< content-length: 52
< last-modified: Thu, 12 Aug 2021 12:04:00 GMT
< etag: "34-5c95b87bbec00"
< x-xss-protection: 1; mode=block
< x-content-type-options: nosniff
< strict-transport-security: max-age=31536000; includeSubDomains; preload
< x-frame-options: sameorigin
< referrer-policy: strict-origin-when-cross-origin
< cache-control: max-age=14400
< permissions-policy: interest-cohort=()
< cf-cache-status: HIT
< age: 431
< accept-ranges: bytes
< expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< server: cloudflare
< cf-ray: 67d99e40ba5b60cd-DUB
<
{ "m.server": "matrix-federation.matrix.org:443" }
* Connection #0 to host localhost left intact

[dklimpel]

curl 7.29.0 is not up to date. Release date was Feb 6 2013.

[f1-outsourcing]

curl 7.29.0 is not up to date. Release date was Feb 6 2013.

We should let RedHat worry about such things, the el7 is still a supported distribution ;)

[dklimpel]

Yes it is supported for security bugfixes. But curl 7.29.0 does not support proxy connections with TLS.
https://daniel.haxx.se/blog/2016/11/26/https-proxy-with-curl/
Your curl version ignores the https and does a connection via http whether you tell him to use https or not.

It does not really make sense not supporting ssl these days

But 2013! ;)

[f1-outsourcing]

Genius, genius of you.

I just tried on the docker image with curl 7.78, and whoppa

url: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number

And while I was doing this, I thought lets run the container with
"HTTP_PROXY": "http://192.168.122.xx:53245", "HTTPS_PROXY": "http://192.168.122.xx:53245",

And this seems to work!!!!!! Explore rooms is being filled with external data.

I am embarrassed about wasting your time on this

[f1-outsourcing]

@dklimpel

I just switched back from using a hostname in the proxy config to an ip address, hoping it would resolve the timeout messages that I am getting (in case python is not using the correct ip address). But I keep getting lots of these. Joining a room seems to be slow, however I managed to join 2.

Aug 13 16:02:29 xxxxxx journal: matrix-synapse synapse.http.matrixfederationclient - 635 - INFO - federation_transaction_transmission_loop-1135 - {PUT-O-679} [packet.delivery] Request failed: PUT matrix://packet.delivery/_matrix/federation/v1/send/1628862410582: TimeoutError('Timed out after 60s')
Aug 13 16:02:29 xxxxxx journal: matrix-synapse synapse.http.matrixfederationclient - 635 - INFO - GET-994 - {GET-O-702} [vi.fi] Request failed: GET matrix://vi.fi/_matrix/media/r0/download/vi.fi/rDHTynXjBZZofVeosisEMzKV?allow_remote=false: TimeoutError('')
Aug 13 16:02:29 xxxxxx journal: matrix-synapse synapse.http.matrixfederationclient - 635 - INFO - federation_transaction_transmission_loop-1165 - {PUT-O-685} [furry.lol] Request failed: PUT matrix://furry.lol/_matrix/federation/v1/send/1628862410588: TimeoutError('')
Aug 13 16:02:30 xxxxxx journal: matrix-synapse synapse.http.matrixfederationclient - 635 - INFO - GET-1640 - {GET-O-713} [kapsi.fi] Request failed: GET matrix://kapsi.fi/_matrix/media/r0/download/kapsi.fi/FUJcYeeynfFBUAuPaANxzbcd?allow_remote=false: TimeoutError('')
Aug 13 16:02:30 xxxxxx journal: matrix-synapse synapse.http.matrixfederationclient - 635 - INFO - federation_transaction_transmission_loop-528 - {PUT-O-358} [matrix.onosend.ai] Request failed: PUT matrix://matrix.onosend.ai/_matrix/federation/v1/send/1628862410468: TimeoutError('Timed out after 60s')
Aug 13 16:02:31 xxxxxx journal: matrix-synapse synapse.http.matrixfederationclient - 635 - INFO - GET-1642 - {GET-O-714} [fullcontrol.se] Request failed: GET matrix://fullcontrol.se/_matrix/media/r0/download/fullcontrol.se/gKAdbEJzBMFETgsfCiHaJJvq?allow_remote=false: TimeoutError('Timed out after 60s')
Aug 13 16:02:31 xxxxxx journal: matrix-synapse synapse.http.matrixfederationclient - 635 - INFO - federation_transaction_transmission_loop-235 - {PUT-O-299} [purplewire.xyz] Request failed: PUT matrix://purplewire.xyz/_matrix/federation/v1/send/1628862410409: TimeoutError('')
Aug 13 16:02:31 xxxxxx journal: matrix-synapse synapse.http.matrixfederationclient - 635 - INFO - federation_transaction_transmission_loop-711 - {PUT-O-418} [matrix.splitanatom.com] Request failed: PUT matrix://matrix.splitanatom.com/_matrix/federation/v1/send/1628862410507: TimeoutError('Timed out after 60s')
Aug 13 16:02:31 xxxxxx journal: matrix-synapse synapse.http.matrixfederationclient - 635 - INFO - federation_transaction_transmission_loop-683 - {PUT-O-385} [zgrz.yt] Request failed: PUT matrix://zgrz.yt/_matrix/federation/v1/send/1628862410495: TimeoutError('')
Aug 13 16:02:31 xxxxxx journal: matrix-synapse synapse.http.matrixfederationclient - 635 - INFO - federation_transaction_transmission_loop-582 - {PUT-O-365} [thebeckmeyers.xyz] Request failed: PUT matrix://thebeckmeyers.xyz/_matrix/federation/v1/send/1628862410475: TimeoutError('')
Aug 13 16:02:32 xxxxxx journal: matrix-synapse synapse.http.matrixfederationclient - 635 - INFO - federation_transaction_transmission_loop-313 - {PUT-O-267} [canarymod.net] Request failed: PUT matrix://canarymod.net/_matrix/federation/v1/send/1628862410377: TimeoutError('Timed out after 60s')
Aug 13 16:02:34 xxxxxx journal: matrix-synapse synapse.http.matrixfederationclient - 635 - INFO - federation_transaction_transmission_loop-741 - {PUT-O-618} [exp.farm] Request failed: PUT matrix://exp.farm/_matrix/federation/v1/send/1628862410528: TimeoutError('Timed out after 60s')
Aug 13 16:02:34 xxxxxx journal: matrix-synapse synapse.http.matrixfederationclient - 635 - INFO - federation_transaction_transmission_loop-105 - {PUT-O-227} [matrix.beachcom.org] Request failed: PUT matrix://matrix.beachcom.org/_matrix/federation/v1/send/1628862410337: TimeoutError('')
Aug 13 16:02:35 xxxxxx journal: matrix-synapse synapse.http.matrixfederationclient - 635 - INFO - federation_transaction_transmission_loop-835 - {PUT-O-630} [matrix.privatebit.de] Request failed: PUT matrix://matrix.privatebit.de/_matrix/federation/v1/send/1628862410540: TimeoutError('')
Aug 13 16:02:35 xxxxxx journal: matrix-synapse synapse.http.matrixfederationclient - 635 - INFO - federation_transaction_transmission_loop-758 - {PUT-O-620} [matrix.datenwolke.io] Request failed: PUT matrix://matrix.datenwolke.io/_matrix/federation/v1/send/1628862410530: TimeoutError('Timed out after 60s')
Aug 13 16:02:35 xxxxxx journal: matrix-synapse synapse.http.matrixfederationclient - 635 - INFO - federation_transaction_transmission_loop-782 - {PUT-O-623} [hu-berlin.de] Request failed: PUT matrix://hu-berlin.de/_matrix/federation/v1/send/1628862410533: TimeoutError('Timed out after 60s')
Aug 13 16:02:36 
```xxxxxx journal: matrix-synapse synapse.http.matrixfederationclient - 635 - INFO - federation_tran

[dklimpel]

I think this is not easy to debug.
I would have a look also in proxy server log and perhaps make a tcp dump to have a look if the proxy has to much connections or something else.