Skip to content

Latest commit

 

History

History
122 lines (97 loc) · 8.66 KB

TEST_HITLIST.md

File metadata and controls

122 lines (97 loc) · 8.66 KB
Raw

Test Hitlist

This is an attempt to manually enumerate all conceivable failure modes with end-to-end encryption. This hitlist is designed to discern categories of bugs which A) have not been seen in bug reports yet, B) are not known issues yet, C) are potential issues. This testing methodology is responsible for uncovering numerous issues which were previously unknown.

Test permutations

For each test, multiple permutations can be tested:

  • homogenous clients (e.g Alice and Bob are both on JS)
  • heterogeneous clients (e.g Alice is on JS, Bob is using Rust FFI)
  • both clients are on the same server
  • clients are on different servers (testing federation)

This creates a matrix of test permutations, represented via COMPLEMENT_CRYPTO_TEST_CLIENT_MATRIX. There exists a total of 4x4=16 permutations, given:

  • each client can be 1 of 4 types (j,r,J,R)
  • two clients are needed per test

However, in practice there are only 12 permutations because we can ignore duplicate permutations caused by testing on HS1 vs HS2 e.g testing 2x JS clients on HS1 and then re-testing 2x HS clients on HS2 makes no sense:

Alice | Bob | Fed? | Same client?
 j       j     N          Y
 j       r     N          N
 r       j     N          N
 r       r     N          Y

 J       j     Y          Y
 J       r     Y          N
 R       j     Y          N
 R       r     Y          Y
 j       J     Y          Y
 j       R     Y          N
 r       J     Y          N
 r       R     Y          Y

Below permutations can be ignored as they
are duplicates of the first 4

 J       J     N          Y
 J       R     N          N
 R       J     N          N
 R       R     N          Y

The test hitlist will generally not refer to any specific permutation, preferring the terms Alice and Bob. In some cases, federation may be explicitly mentioned if the test makes no sense without federation.

Membership ACLs

  • Happy case Alice and Bob in an encrypted room can send and receive encrypted messages, and decrypt them all.
  • Bob can see messages when he was invited but not joined to the room. Subsequent messages are also decryptable.
  • In a public, shared history visibility room, a new user Bob cannot decrypt earlier messages prior to his join, despite being able to see the events. Subsequent messages are decryptable.
  • Bob leaves the room. Some messages are sent. Bob rejoins and cannot decrypt the messages sent whilst he was gone (ensuring we cycle keys).
  • Bob cannot decrypt older messages when logging in on a new device. When the device is logged out and in again, Bob cannot decrypt messages sent whilst he was logged out.
  • EXPECTED FAIL: Alice invites Bob, Alice sends a message, Bob changes their device, then Bob joins. Bob should be able to see Alice's message.

Key backups

These tests only make sense on a single server, for a single user.

  • New device for Alice cannot decrypt previous messages. Backups can be made on Alice's first device. Alice's new device can download the backup and decrypt the messages. Check backups work cross-platform (e.g create on rust, restore on JS and vice versa).
  • Inputting the wrong recovery key fails to decrypt the backup.

One-time Keys

  • When Alice runs out of OTKs, the fallback key is used.
  • Alice cycles her fallback key when she becomes aware that it has been used.
  • When a fallback key used by multiple sessions, Alice accepts all of them. TODO: when should Alice stop accepting usage of this key?
  • When a OTK is reused, Alice rejects the 2nd+ use of the OTK.

Key Verification: (Short Authentication String)

  • Happy case Alice <-> Bob key verification.
  • Happy case Alice <-> Alice key verification (different devices).
  • A MITMed key fails key verification.
  • Repeat all of the above, but for QR code. (render QR code to png then rescan).
  • Repeat all of the above, but for Emoji representations of SAS.
  • Verification can be cancelled.

Network connectivity

Network connectivity tests are extremely time sensitive as retries are often using timeouts in clients.

  • If a client cannot upload OTKs, it retries.
  • If a client cannot claim OTKs, it retries.
  • If a server cannot send device list updates over federation, it retries. matrix-org/complement#695
  • If a client cannot query device keys for a user, it retries. (TestFailedDeviceKeyDownloadRetries)
  • If a server cannot query device keys on another server, it retries.
  • If a client cannot send a to-device msg, it retries.
  • If a server cannot send a to-device msg to another server, it retries. matrix-org/complement#694

State Synchronisation

This refers to cases where the client has some state and wishes to synchronise it with the server but is interrupted from doing so in a fatal (SIGKILL) manner. Clients MUST persist state they wish to synchronise to avoid state being regenerated and hence getting out-of-sync with server state. All of these tests require persistent storage on clients. These tests will typically intercept responses from the server and then SIGKILL the client. Upon restart, only if the client has persisted the new state prior to uploading AND endpoints are idempotent (since the client will retry the operation) will state remain in sync. These tests aren't limited to clients, as servers also need to synchronise state over federation.

  • If a client is terminated mid-way through uploading OTKs, it re-uploads the same set of OTKs on startup.
  • If a client is terminated mid-way through uploading device keys, it re-uploads the same set of device keys on startup.
  • If a client is terminated mid-way through uploading cross-signing keys, it re-uploads the same set of keys on startup.
  • If a client is terminated mid-way through sending a to-device message, it retries sending the same message on startup.
  • If a client is terminated mid-way through calculating device list changes via /keys/changes, it retries on startup.
  • If a server is terminated mid-way through sending a device list update over federation, it retries on startup.
  • If a server is terminated mid-way through sending a to-device message over federation, it retries on startup.

Room Keys

  • The room key is cycled when a user leaves a room.
  • The room key is cycled when one of a user's devices logs out.
  • The room key is cycled when one of a user's devices is blacklisted.
  • The room key is cycled when history visibility changes to something more restrictive TODO: define precisely.
  • The room key is cycled when the encryption algorithm changes.
  • The room key is cycled when rotation_period_msgs is met (default: 100). (TestRoomKeyIsCycledAfterEnoughMessages)
  • The room key is cycled when rotation_period_ms is exceeded (default: 1 week). (TestRoomKeyIsCycledAfterEnoughTime)
  • The room key is not cycled when one of a user's devices logs in.
  • The room key is not cycled when the client restarts.
  • The room key is not cycled when users change their display name.

Adversarial Attacks

TODO See polyjuice tests, port and add more.

Regression tests

Tests for known failures.