Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Fix MultiWriteIdGenerator's handling of restarts. #8374

Merged
merged 7 commits into from
Sep 24, 2020

Commits on Sep 23, 2020

  1. Fix MultiWriteIdGenerator's handling of restarts.

    On startup `MultiWriteIdGenerator` fetches the maximum stream ID for
    each instance from the table and uses that as its initial "current
    position" for each writer. This is problematic as a) it involves either
    a scan of events table or an index (neither of which is ideal), and b)
    if rows are being persisted out of order elsewhere while the process
    restarts then using the maximum stream ID is not correct. This could
    theoretically lead to race conditions where e.g. events that are
    persisted out of order are not sent down sync streams.
    
    We fix this by creating a new table that tracks the current positions of
    each writer to the stream, and update it each time we finish persisting
    a new entry. This is a relatively small overhead when persisting events.
    However for the cache invalidation stream this is a much bigger relative
    overhead, so instead we note that for invalidation we don't actually
    care about reliability over restarts (as there's no caches to
    invalidate) and simply don't bother reading and writing to the new table
    in that particular case.
    erikjohnston committed Sep 23, 2020
    Configuration menu
    Copy the full SHA
    81dac92 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    f5a91ab View commit details
    Browse the repository at this point in the history
  3. Add regression test

    erikjohnston committed Sep 23, 2020
    Configuration menu
    Copy the full SHA
    c08456d View commit details
    Browse the repository at this point in the history
  4. Newfile

    erikjohnston committed Sep 23, 2020
    Configuration menu
    Copy the full SHA
    6e3d562 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    d6f5c72 View commit details
    Browse the repository at this point in the history

Commits on Sep 24, 2020

  1. Update synapse/storage/util/id_generators.py

    Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
    erikjohnston and clokep authored Sep 24, 2020
    Configuration menu
    Copy the full SHA
    e83800f View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    d86f62e View commit details
    Browse the repository at this point in the history