Merge branch 'develop' into rei/slash_report

matrix-org · Sep 16, 2022 · 1962979 · 1962979
2 parents 42aba4b + 74f60ce
commit 1962979
Show file tree

Hide file tree

Showing 126 changed files with 1,776 additions and 338 deletions.
diff --git a/CHANGES.md b/CHANGES.md
@@ -1,3 +1,111 @@
+Synapse 1.67.0 (2022-09-13)
+===========================
+
+This release removes using the deprecated direct TCP replication configuration
+for workers. Server admins should use Redis instead. See the [upgrade
+notes](https://matrix-org.github.io/synapse/v1.67/upgrade.html#upgrading-to-v1670).
+
+The minimum version of `poetry` supported for managing source checkouts is now
+1.2.0.
+
+**Notice:** from the next major release (1.68.0) installing Synapse from a source
+checkout will require a recent Rust compiler. Those using packages or
+`pip install matrix-synapse` will not be affected. See the [upgrade
+notes](https://matrix-org.github.io/synapse/v1.67/upgrade.html#upgrading-to-v1670).
+
+**Notice:** from the next major release (1.68.0), running Synapse with a SQLite
+database will require SQLite version 3.27.0 or higher. (The [current minimum
+ version is SQLite 3.22.0](https://github.com/matrix-org/synapse/blob/release-v1.67/synapse/storage/engines/sqlite.py#L69-L78).)
+See [#12983](https://github.com/matrix-org/synapse/issues/12983) and the [upgrade notes](https://matrix-org.github.io/synapse/v1.67/upgrade.html#upgrading-to-v1670) for more details.
+
+
+No significant changes since 1.67.0rc1.
+
+
+Synapse 1.67.0rc1 (2022-09-06)
+==============================
+
+Features
+--------
+
+- Support setting the registration shared secret in a file, via a new `registration_shared_secret_path` configuration option. ([\#13614](https://github.com/matrix-org/synapse/issues/13614))
+- Change the default startup behaviour so that any missing "additional" configuration files (signing key, etc) are generated automatically. ([\#13615](https://github.com/matrix-org/synapse/issues/13615))
+- Improve performance of sending messages in rooms with thousands of local users. ([\#13634](https://github.com/matrix-org/synapse/issues/13634))
+
+
+Bugfixes
+--------
+
+- Fix a bug introduced in Synapse 1.13 where the [List Rooms admin API](https://matrix-org.github.io/synapse/develop/admin_api/rooms.html#list-room-api) would return integers instead of booleans for the `federatable` and `public` fields when using a Sqlite database. ([\#13509](https://github.com/matrix-org/synapse/issues/13509))
+- Fix bug that user cannot `/forget` rooms after the last member has left the room. ([\#13546](https://github.com/matrix-org/synapse/issues/13546))
+- Faster Room Joins: fix `/make_knock` blocking indefinitely when the room in question is a partial-stated room. ([\#13583](https://github.com/matrix-org/synapse/issues/13583))
+- Fix loading the current stream position behind the actual position. ([\#13585](https://github.com/matrix-org/synapse/issues/13585))
+- Fix a longstanding bug in `register_new_matrix_user` which meant it was always necessary to explicitly give a server URL. ([\#13616](https://github.com/matrix-org/synapse/issues/13616))
+- Fix the running of [MSC1763](https://github.com/matrix-org/matrix-spec-proposals/pull/1763) retention purge_jobs in deployments with background jobs running on a worker by forcing them back onto the main worker. Contributed by Brad @ Beeper. ([\#13632](https://github.com/matrix-org/synapse/issues/13632))
+- Fix a long-standing bug that downloaded media for URL previews was not deleted while database background updates were running. ([\#13657](https://github.com/matrix-org/synapse/issues/13657))
+- Fix [MSC3030](https://github.com/matrix-org/matrix-spec-proposals/pull/3030) `/timestamp_to_event` endpoint to return the correct next event when the events have the same timestamp. ([\#13658](https://github.com/matrix-org/synapse/issues/13658))
+- Fix bug where we wedge media plugins if clients disconnect early. Introduced in v1.22.0. ([\#13660](https://github.com/matrix-org/synapse/issues/13660))
+- Fix a long-standing bug which meant that keys for unwhitelisted servers were not returned by `/_matrix/key/v2/query`. ([\#13683](https://github.com/matrix-org/synapse/issues/13683))
+- Fix a bug introduced in Synapse v1.20.0 that would cause the unstable unread counts from [MSC2654](https://github.com/matrix-org/matrix-spec-proposals/pull/2654) to be calculated even if the feature is disabled. ([\#13694](https://github.com/matrix-org/synapse/issues/13694))
+
+
+Updates to the Docker image
+---------------------------
+
+- Update docker image to use a stable version of poetry. ([\#13688](https://github.com/matrix-org/synapse/issues/13688))
+
+
+Improved Documentation
+----------------------
+
+- Improve the description of the ["chain cover index"](https://matrix-org.github.io/synapse/latest/auth_chain_difference_algorithm.html) used internally by Synapse. ([\#13602](https://github.com/matrix-org/synapse/issues/13602))
+- Document how ["monthly active users"](https://matrix-org.github.io/synapse/latest/usage/administration/monthly_active_users.html) is calculated and used. ([\#13617](https://github.com/matrix-org/synapse/issues/13617))
+- Improve documentation around user registration. ([\#13640](https://github.com/matrix-org/synapse/issues/13640))
+- Remove documentation of legacy `frontend_proxy` worker app. ([\#13645](https://github.com/matrix-org/synapse/issues/13645))
+- Clarify documentation that HTTP replication traffic can be protected with a shared secret. ([\#13656](https://github.com/matrix-org/synapse/issues/13656))
+- Remove unintentional colons from [config manual](https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html) headers. ([\#13665](https://github.com/matrix-org/synapse/issues/13665))
+- Update docs to make enabling metrics more clear. ([\#13678](https://github.com/matrix-org/synapse/issues/13678))
+- Clarify `(room_id, event_id)` global uniqueness and how we should scope our database schemas. ([\#13701](https://github.com/matrix-org/synapse/issues/13701))
+
+
+Deprecations and Removals
+-------------------------
+
+- Drop support for calling `/_matrix/client/v3/rooms/{roomId}/invite` without an `id_access_token`, which was not permitted by the spec. Contributed by @Vetchu. ([\#13241](https://github.com/matrix-org/synapse/issues/13241))
+- Remove redundant `_get_joined_users_from_context` cache. Contributed by Nick @ Beeper (@fizzadar). ([\#13569](https://github.com/matrix-org/synapse/issues/13569))
+- Remove the ability to use direct TCP replication with workers. Direct TCP replication was deprecated in Synapse v1.18.0. Workers now require using Redis. ([\#13647](https://github.com/matrix-org/synapse/issues/13647))
+- Remove support for unstable [private read receipts](https://github.com/matrix-org/matrix-spec-proposals/pull/2285). ([\#13653](https://github.com/matrix-org/synapse/issues/13653), [\#13692](https://github.com/matrix-org/synapse/issues/13692))
+
+
+Internal Changes
+----------------
+
+- Extend the release script to wait for GitHub Actions to finish and to be usable as a guide for the whole process. ([\#13483](https://github.com/matrix-org/synapse/issues/13483))
+- Add experimental configuration option to allow disabling legacy Prometheus metric names. ([\#13540](https://github.com/matrix-org/synapse/issues/13540))
+- Cache user IDs instead of profiles to reduce cache memory usage. Contributed by Nick @ Beeper (@fizzadar). ([\#13573](https://github.com/matrix-org/synapse/issues/13573), [\#13600](https://github.com/matrix-org/synapse/issues/13600))
+- Optimize how Synapse calculates domains to fetch from during backfill. ([\#13575](https://github.com/matrix-org/synapse/issues/13575))
+- Comment about a better future where we can get the state diff between two events. ([\#13586](https://github.com/matrix-org/synapse/issues/13586))
+- Instrument `_check_sigs_and_hash_and_fetch` to trace time spent in child concurrent calls for understandable traces in Jaeger. ([\#13588](https://github.com/matrix-org/synapse/issues/13588))
+- Improve performance of `@cachedList`. ([\#13591](https://github.com/matrix-org/synapse/issues/13591))
+- Minor speed up of fetching large numbers of push rules. ([\#13592](https://github.com/matrix-org/synapse/issues/13592))
+- Optimise push action fetching queries. Contributed by Nick @ Beeper (@fizzadar). ([\#13597](https://github.com/matrix-org/synapse/issues/13597))
+- Rename `event_map` to `unpersisted_events` when computing the auth differences. ([\#13603](https://github.com/matrix-org/synapse/issues/13603))
+- Refactor `get_users_in_room(room_id)` mis-use with dedicated `get_current_hosts_in_room(room_id)` function. ([\#13605](https://github.com/matrix-org/synapse/issues/13605))
+- Use dedicated `get_local_users_in_room(room_id)` function to find local users when calculating `join_authorised_via_users_server` of a `/make_join` request. ([\#13606](https://github.com/matrix-org/synapse/issues/13606))
+- Refactor `get_users_in_room(room_id)` mis-use to lookup single local user with dedicated `check_local_user_in_room(...)` function. ([\#13608](https://github.com/matrix-org/synapse/issues/13608))
+- Drop unused column `application_services_state.last_txn`. ([\#13627](https://github.com/matrix-org/synapse/issues/13627))
+- Improve readability of Complement CI logs by printing failure results last. ([\#13639](https://github.com/matrix-org/synapse/issues/13639))
+- Generalise the `@cancellable` annotation so it can be used on functions other than just servlet methods. ([\#13662](https://github.com/matrix-org/synapse/issues/13662))
+- Introduce a `CommonUsageMetrics` class to share some usage metrics between the Prometheus exporter and the phone home stats. ([\#13671](https://github.com/matrix-org/synapse/issues/13671))
+- Add some logging to help track down #13444. ([\#13679](https://github.com/matrix-org/synapse/issues/13679))
+- Update poetry lock file for v1.2.0. ([\#13689](https://github.com/matrix-org/synapse/issues/13689))
+- Add cache to `is_partial_state_room`. ([\#13693](https://github.com/matrix-org/synapse/issues/13693))
+- Update the Grafana dashboard that is included with Synapse in the `contrib` directory. ([\#13697](https://github.com/matrix-org/synapse/issues/13697))
+- Only run trial CI on all python versions on non-PRs. ([\#13698](https://github.com/matrix-org/synapse/issues/13698))
+- Fix typechecking with latest types-jsonschema. ([\#13712](https://github.com/matrix-org/synapse/issues/13712))
+- Reduce number of CI checks we run for PRs. ([\#13713](https://github.com/matrix-org/synapse/issues/13713))
+
+
 Synapse 1.66.0 (2022-08-31)
 ===========================
 

diff --git a/changelog.d/13162.misc b/changelog.d/13162.misc
@@ -0,0 +1 @@
+Bump the minimum dependency of `matrix_common` to 1.3.0 to make use of the `MXCUri` class. Use `MXCUri` to simplify media retention test code.
diff --git a/changelog.d/13241.removal b/changelog.d/13241.removal
diff --git a/changelog.d/13483.misc b/changelog.d/13483.misc
diff --git a/changelog.d/13509.bugfix b/changelog.d/13509.bugfix
diff --git a/changelog.d/13540.misc b/changelog.d/13540.misc
diff --git a/changelog.d/13546.bugfix b/changelog.d/13546.bugfix
diff --git a/changelog.d/13569.removal b/changelog.d/13569.removal
diff --git a/changelog.d/13573.misc b/changelog.d/13573.misc
diff --git a/changelog.d/13575.misc b/changelog.d/13575.misc
diff --git a/changelog.d/13583.bugfix b/changelog.d/13583.bugfix
diff --git a/changelog.d/13585.bugfix b/changelog.d/13585.bugfix
diff --git a/changelog.d/13586.misc b/changelog.d/13586.misc
diff --git a/changelog.d/13588.misc b/changelog.d/13588.misc
diff --git a/changelog.d/13589.feature b/changelog.d/13589.feature
@@ -0,0 +1 @@
+Keep track when we fail to process a pulled event over federation so we can intelligently back-off in the future.
diff --git a/changelog.d/13591.misc b/changelog.d/13591.misc
diff --git a/changelog.d/13592.misc b/changelog.d/13592.misc
diff --git a/changelog.d/13597.misc b/changelog.d/13597.misc
diff --git a/changelog.d/13600.misc b/changelog.d/13600.misc
diff --git a/changelog.d/13602.doc b/changelog.d/13602.doc
diff --git a/changelog.d/13603.misc b/changelog.d/13603.misc
diff --git a/changelog.d/13605.misc b/changelog.d/13605.misc
diff --git a/changelog.d/13606.misc b/changelog.d/13606.misc
diff --git a/changelog.d/13608.misc b/changelog.d/13608.misc
diff --git a/changelog.d/13614.feature b/changelog.d/13614.feature
diff --git a/changelog.d/13615.feature b/changelog.d/13615.feature
diff --git a/changelog.d/13616.bugfix b/changelog.d/13616.bugfix
diff --git a/changelog.d/13617.doc b/changelog.d/13617.doc
diff --git a/changelog.d/13627.misc b/changelog.d/13627.misc
diff --git a/changelog.d/13632.bugfix b/changelog.d/13632.bugfix
diff --git a/changelog.d/13634.feature b/changelog.d/13634.feature
diff --git a/changelog.d/13639.misc b/changelog.d/13639.misc
diff --git a/changelog.d/13640.doc b/changelog.d/13640.doc
diff --git a/changelog.d/13645.doc b/changelog.d/13645.doc
diff --git a/changelog.d/13647.removal b/changelog.d/13647.removal
diff --git a/changelog.d/13653.removal b/changelog.d/13653.removal
diff --git a/changelog.d/13656.doc b/changelog.d/13656.doc
diff --git a/changelog.d/13657.bugfix b/changelog.d/13657.bugfix
diff --git a/changelog.d/13658.bugfix b/changelog.d/13658.bugfix
diff --git a/changelog.d/13660.bugfix b/changelog.d/13660.bugfix
diff --git a/changelog.d/13662.misc b/changelog.d/13662.misc
diff --git a/changelog.d/13665.doc b/changelog.d/13665.doc
diff --git a/changelog.d/13671.misc b/changelog.d/13671.misc
diff --git a/changelog.d/13678.doc b/changelog.d/13678.doc
diff --git a/changelog.d/13679.misc b/changelog.d/13679.misc
diff --git a/changelog.d/13683.bugfix b/changelog.d/13683.bugfix
diff --git a/changelog.d/13688.docker b/changelog.d/13688.docker
diff --git a/changelog.d/13689.misc b/changelog.d/13689.misc
diff --git a/changelog.d/13692.removal b/changelog.d/13692.removal
diff --git a/changelog.d/13693.misc b/changelog.d/13693.misc
diff --git a/changelog.d/13694.bugfix b/changelog.d/13694.bugfix
diff --git a/changelog.d/13697.misc b/changelog.d/13697.misc
diff --git a/changelog.d/13698.misc b/changelog.d/13698.misc
diff --git a/changelog.d/13701.doc b/changelog.d/13701.doc
diff --git a/changelog.d/13712.misc b/changelog.d/13712.misc
diff --git a/changelog.d/13713.misc b/changelog.d/13713.misc
diff --git a/changelog.d/13723.bugfix b/changelog.d/13723.bugfix
@@ -0,0 +1 @@
+Fix a long-standing bug where previously rejected events could end up in room state because they pass auth checks given the current state of the room.
diff --git a/changelog.d/13736.feature b/changelog.d/13736.feature
@@ -0,0 +1 @@
+Improve validation of request bodies for the following client-server API endpoints: [`/account/3pid/add`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3account3pidadd), [`/account/3pid/bind`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3account3pidbind), [`/account/3pid/delete`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3account3piddelete) and [`/account/3pid/unbind`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3account3pidunbind).
diff --git a/changelog.d/13749.bugfix b/changelog.d/13749.bugfix
@@ -0,0 +1 @@
+Fix a long standing bug where device lists would remain cached when remote users left and rejoined the last room shared with the local homeserver.
diff --git a/changelog.d/13753.misc b/changelog.d/13753.misc
@@ -0,0 +1 @@
+Prepatory work for storing thread IDs for notifications and receipts.
diff --git a/changelog.d/13766.bugfix b/changelog.d/13766.bugfix
@@ -0,0 +1 @@
+Fix a long-standing bug where the `cache_invalidation_stream_seq` sequence would begin at 1 instead of 2.
diff --git a/changelog.d/13770.misc b/changelog.d/13770.misc
@@ -0,0 +1 @@
+Update the script which makes full schema dumps.
diff --git a/changelog.d/13780.misc b/changelog.d/13780.misc
@@ -0,0 +1 @@
+Deduplicate `is_server_notices_room`.
diff --git a/changelog.d/13785.doc b/changelog.d/13785.doc
@@ -0,0 +1 @@
+Add docs for common fix of deleting the `matrix_synapse.egg-info/` directory for fixing Python dependency problems.
diff --git a/changelog.d/13788.misc b/changelog.d/13788.misc
@@ -0,0 +1 @@
+Remove an old, incorrect migration file.
diff --git a/changelog.d/13789.bugfix b/changelog.d/13789.bugfix
@@ -0,0 +1 @@
+Fix a long-standing spec compliance bug where Synapse would accept a trailing slash on the end of `/get_missing_events` federation requests.
diff --git a/changelog.d/13794.doc b/changelog.d/13794.doc
@@ -0,0 +1 @@
+Update request log format documentation to mention the format used when the authenticated user is controlling another user.
diff --git a/changelog.d/13795.misc b/changelog.d/13795.misc
@@ -0,0 +1 @@
+Remove unused method in `synapse.api.auth.Auth`.
diff --git a/changelog.d/13798.misc b/changelog.d/13798.misc
@@ -0,0 +1 @@
+Fix a memory leak when running the unit tests.
diff --git a/changelog.d/13801.feature b/changelog.d/13801.feature
@@ -0,0 +1 @@
+Add `listeners[x].request_id_header` config to specify which request header to extract and use as the request ID in order to correlate requests from a reverse-proxy.
diff --git a/changelog.d/13802.misc b/changelog.d/13802.misc
@@ -0,0 +1 @@
+Use partial indices on SQLite.
diff --git a/changelog.d/13810.feature b/changelog.d/13810.feature
@@ -0,0 +1 @@
+Add an admin API endpoint to find a user based on its external ID in an auth provider.
diff --git a/changelog.d/13814.feature b/changelog.d/13814.feature
@@ -0,0 +1 @@
+Keep track when we fail to process a pulled event over federation so we can intelligently back-off in the future.
diff --git a/changelog.d/13822.misc b/changelog.d/13822.misc
@@ -0,0 +1 @@
+Support providing an index predicate clause when doing upserts.
diff --git a/debian/changelog b/debian/changelog
@@ -1,8 +1,18 @@
-matrix-synapse-py3 (1.66.0ubuntu1) UNRELEASED; urgency=medium
+matrix-synapse-py3 (1.67.0) stable; urgency=medium
 
+  * New Synapse release 1.67.0.
+
+ -- Synapse Packaging team <packages@matrix.org>  Tue, 13 Sep 2022 09:19:56 +0100
+
+matrix-synapse-py3 (1.67.0~rc1) stable; urgency=medium
+
+  [ Erik Johnston ]
   * Use stable poetry 1.2.0 version, rather than a prerelease.
 
- -- Erik Johnston <erik@matrix.org>  Thu, 01 Sep 2022 13:48:31 +0100
+  [ Synapse Packaging team ]
+  * New Synapse release 1.67.0rc1.
+
+ -- Synapse Packaging team <packages@matrix.org>  Tue, 06 Sep 2022 09:01:06 +0100
 
 matrix-synapse-py3 (1.66.0) stable; urgency=medium