Some media corrupt media decryption once posted into a room

[boogewooge]

Background information

• Dendrite version or git SHA: 0.8.9
• Monolith or Polylith?: Monolith
• SQLite3 or Postgres?: Postgres
• Running in Docker?: No
• go version: 1.18.4
• Client used (if applicable): Element-Desktop and 2 different hosted instances of Element-Web (one of them app.element.io), all 1.11

Description

• What is the problem:

In encrypted rooms on a fresh dendrite instance, a subset of all media in a room randomly can't be decrypted when the room is viewed in Element. Switching the currently viewed room in Element to another one and back rerolls the subset of media for which this is true. So, previously undecryptable media sometimes are displayed just fine afterwards and previously decryptable media becomes undecryptable. (Element shows the Unable to decrypt Image/Video error)

After a cache clear, all media are without decryption errors, without fail. After switching rooms once, the above issue resurfaces.

At the same time, users from a synapse instance in the same room never have this issue, from any Element clients.

Original event sources always stay the same on the dendrite instance, and are identical to the ones synapse users see. The decrypted event source, on the other hand, is the same on both whenever the media event in question is decrypted fine on dendrite side, and differs whenever it isn't. So, the decrypted event source a dendrite user sees is randomly different after switching rooms.

We now had a rare couple rooms where this didn't happen.

No issues of this sort ever for any of the same dendrite users in the same rooms on nheko or fluffychat - media are displayed just fine there.

• Who is affected: At least 4 users (100% asked) from our dendrite instance we tested this with.

• When did this first appear: In the first room we posted media in.

Steps to reproduce

• (Optional?) Fresh dendrite instance.
• Create encrypted room.
• Post (from any client) some media.
• View room from Element Web/Desktop.

I'll gladly provide logs or any of the videos that caused this issue for us.

[boogewooge]

All patterns we thought we had found (apart from it only happening on Element, which remained true) at some point were broken. Closing for now and back to testing.