Doesn't back off on backfill if remote server returns 404

[kroeckx]

Synapse 1.40.0 does not seem to back off on backfill failures. My log contains a error every 3 seconds, and it's been doing that for at least 30 minutes.

It's trying to talk to Conduit which currently doesn't implement it yet and returns a 404.

[reivilibre]

backoff_on_404=False seems to be the culprit — in other words, Synapse won't currently backoff if the remote server returns 404 (it would have done so otherwise).

To add more complexity:
In this case, backing off would also be undesirable for Conduit — it would mean that we would stop outbound federation etc (assuming that because the backfill failed, the Conduit server is down), when in reality other functionality of Conduit still works.

The log spam and traffic is a nuisance, but it's hard to know what we should do here. If we wanted to support this in the best way, we would probably have to have fine-grained backoffs (in the sense that we can backoff from specific kinds of requests, e.g. backfill only, whilst still doing other requests)...

[reivilibre]

extra thoughts on smarter backoff: #8917
also related: #5442

[callahad]

Also worth linking to Conduit's issue tracker for implementing backfill:

• https://gitlab.com/famedly/conduit/-/issues/44
• https://gitlab.com/famedly/conduit/-/issues/45

@kroeckx Is there any harm here beyond logspam on the Synapse side?

It seems like this would best be tackled as part of #8917, but that's realistically, that's unlikely to get scheduled or completed in the next several months.

[richvdh]

I think this is essentially a duplicate of #5442