Unable to initial sync due to rooms with unknown room versions

[anoadragon453]

When I tried to initial sync on my desktop today, I received the following exception repeatedly, failing all initial sync calls:

2020-09-15 18:54:13,307 - synapse.http.server - 81 - ERROR - GET-394- Failed handle request via 'RoomMemberListRestServlet': <XForwardedForRequest at 0x7fb3f2f50a00 method='GET' uri='/_matrix/client/r0/rooms/!bNAPsppHlSHCDvMRhh%3Aabolivier.bzh/members?not_membership=leave&at=s9916184_217477991_115043_15957241_757516_1059_639556_7802186_145' clientproto='HTTP/1.1' site=8008>
Traceback (most recent call last):
  File "/home/synapse/synapse/env/lib/python3.8/site-packages/synapse/http/server.py", line 230, in _async_render_wrapper
    callback_return = await self._async_render(request)
  File "/home/synapse/synapse/env/lib/python3.8/site-packages/synapse/http/server.py", line 407, in _async_render
    callback_return = await raw_callback_return
  File "/home/synapse/synapse/env/lib/python3.8/site-packages/synapse/rest/client/v1/room.py", line 477, in on_GET
    events = await handler.get_state_events(
  File "/home/synapse/synapse/env/lib/python3.8/site-packages/synapse/handlers/message.py", line 169, in get_state_events
    last_events, _ = await self.store.get_recent_events_for_room(
  File "/home/synapse/synapse/env/lib/python3.8/site-packages/synapse/storage/databases/main/stream.py", line 512, in get_recent_events_for_room
    events = await self.get_events_as_list(
  File "/home/synapse/synapse/env/lib/python3.8/site-packages/synapse/storage/databases/main/events_worker.py", line 297, in get_events_as_list
    event_entry_map = await self._get_events_from_cache_or_db(
  File "/home/synapse/synapse/env/lib/python3.8/site-packages/synapse/storage/databases/main/events_worker.py", line 443, in _get_events_from_cache_or_db
    missing_events = await self._get_events_from_db(
  File "/home/synapse/synapse/env/lib/python3.8/site-packages/synapse/storage/databases/main/events_worker.py", line 631, in _get_events_from_db
    raise Exception(
Exception: Room !bNAPsppHlSHCDvMRhh:abolivier.bzh for event $1NOJvwbfeLigca7tZFsdCOXQO62nG9xZ-4DehGVl__k is unknown

That room leads to one of our broken rooms with an unknown version. Plot twist though is that I deleted all unknown version rooms from rooms forever ago in a vain attempt to get rid of them from my room list (spoiler: it didn't work). They do have entries in room_stats_state and room_stats_current though ftr.

But more importantly, events belonging to these rooms still exist, and these are what Synapse was passing over and validating during the initial sync.

I deleted all events from this room in events and events_json. That then caused me to get the same exception for the other two rooms I have on my homeserver with unknown room versions.

After also deleting the events of those two other rooms, I restarted the server and now initial sync worked without a hitch.

This happened on Synapse v1.19.0, and then continued to fail even after upgrading to Synapse v1.20.0rc3. I've initial sync'd before successfully though, which leads me to believe this Exception being raised is new. Though that particular comment was added back in v1.12.0, and I don't think it's been that long since I've initial sync'd.

So... something changed recently, but this is also an edge case that we're trying to avoid in #7083

So there may not be any action here, but putting here for posterity. It's also useful to note that deleting events from the rooms fix the issue and prevent the broken rooms from appearing in the room list with seemingly no adverse side effects.

[clokep]

@anoadragon453 Would using the delete room API have done everything necessary (I.e. deleting from rooms, room_stats_state and room_stats_current)?

[anoadragon453]

@clokep I wouldn't want to use the delete room API as it looks to try and send events into the room and move users over to another room, all of which will likely fail if the room version is unknown.

However the purge part is perfect, and as such simply using the purge room API should do everything we need.

[richvdh]

I think we can close this. TL;DR seems to be that things stopped working due to incomplete manual manipulation in the db.