From 0bfd0995b76dd2ae57d96f412fe2420ba28130c7 Mon Sep 17 00:00:00 2001 From: "Amber H. Brown" Date: Mon, 29 Jul 2019 22:57:47 +1000 Subject: [PATCH] Squashed commit of the following: commit c2ca163dfc1eaa2715e77a87f1695a3a6ae97a73 Author: Amber H. Brown Date: Fri Jul 26 22:48:20 2019 +1000 don't block commit 8d9a56e0a6fc9fcf2bb9fdcf2ffd02ed06a5004d Merge: b50d8a9dc 4a5fb548b Author: Amber H. Brown Date: Fri Jul 26 21:35:42 2019 +1000 Merge branch 'shhs' of ssh://github.com/matrix-org/synapse into shhs commit 4a5fb548b6a7c0dac7d7aed972b6a6c367287bbd Merge: 95a038657 992333b99 Author: Amber Brown Date: Fri Jul 26 20:59:41 2019 +1000 Merge tag 'v1.2.1' into shhs Synapse 1.2.1 (2019-07-26) ========================== Security update --------------- This release includes *four* security fixes: - Prevent an attack where a federated server could send redactions for arbitrary events in v1 and v2 rooms. ([\#5767](https://github.com/matrix-org/synapse/issues/5767)) - Prevent a denial-of-service attack where cycles of redaction events would make Synapse spin infinitely. Thanks to `@lrizika:matrix.org` for identifying and responsibly disclosing this issue. ([0f2ecb961](https://github.com/matrix-org/synapse/commit/0f2ecb961)) - Prevent an attack where users could be joined or parted from public rooms without their consent. Thanks to @Dylanger for identifying and responsibly disclosing this issue. ([\#5744](https://github.com/matrix-org/synapse/issues/5744)) - Fix a vulnerability where a federated server could spoof read-receipts from users on other servers. Thanks to @Dylanger for identifying this issue too. ([\#5743](https://github.com/matrix-org/synapse/issues/5743)) Additionally, the following fix was in Synapse **1.2.0**, but was not correctly identified during the original release: - It was possible for a room moderator to send a redaction for an `m.room.create` event, which would downgrade the room to version 1. Thanks to `/dev/ponies` for identifying and responsibly disclosing this issue! ([\#5701](https://github.com/matrix-org/synapse/issues/5701)) commit 95a0386579a453afb380897f74e1acc8005cdf19 Author: Amber Brown Date: Fri Jul 26 20:27:31 2019 +1000 don't have a circleci config commit b50d8a9dc1fb218bd43a19fa7e6b5327ca5bf5c3 Author: Amber H. Brown Date: Fri Jul 26 02:26:23 2019 +1000 fix merging forward commit 3edf6e987ebb4ec0bc81a7566758281019b4743e Author: Amber H. Brown Date: Fri Jul 26 02:07:05 2019 +1000 fix this commit f61cdc14e7f6e48b500e83a2b613cefe94cb31a6 Merge: 43cf23475 c0a1301cc Author: Amber H. Brown Date: Fri Jul 26 01:48:50 2019 +1000 Merge tag 'v1.2.0' into shhs No changes since v1.2.0rc2. commit 43cf23475fabb7982bde17899c4cc93de85d15e1 Author: Amber H. Brown Date: Fri Jul 26 01:48:20 2019 +1000 dockerfile update commit b7962f5bfdb8720b44e5eb99a0dcdbab9430c3ed Author: Amber H. Brown Date: Thu Jul 18 23:23:12 2019 +1000 add a wait commit 9bbf2d23c4bdfc4bd3816a03f4e946347fd5bebf Author: Amber H. Brown Date: Wed Jul 17 04:46:20 2019 +1000 fix commit 5daee2eb4a0edacad07357fe11ed8402617046cf Author: Amber H. Brown Date: Wed Jul 17 04:41:00 2019 +1000 fix commit 14c8b036ead1faeec35f2ff99b46488f7ec02aa0 Author: Amber H. Brown Date: Wed Jul 17 04:36:27 2019 +1000 fix commit 7fcd6c1df9df20101f4779f6fda3cd8df2201013 Author: Amber H. Brown Date: Wed Jul 17 04:32:50 2019 +1000 fix commit c43c1adb0c897556179db2a36d3969147408986c Author: Amber H. Brown Date: Wed Jul 17 04:28:11 2019 +1000 fix commit a025abebe8ec6bb35a25299c47aa6ea020eb91cd Author: Amber H. Brown Date: Wed Jul 17 04:02:15 2019 +1000 try now commit c1777f51a961132b377648a519acd33444fe5f60 Author: Amber H. Brown Date: Wed Jul 17 04:00:34 2019 +1000 try now commit 646292cfb197d9967a8e2d4355cca267616767e6 Author: Amber H. Brown Date: Wed Jul 17 03:58:34 2019 +1000 see if we can do a build! commit a175e608e930d0facf37637876b9824b563d16aa Merge: 9b3a63e1c 0e5434264 Author: Amber H. Brown Date: Fri Jul 5 23:49:13 2019 +1000 Merge remote-tracking branch 'origin/develop' into shhs commit 9b3a63e1c8cb22f08473626b8f29df2942ab61e6 Author: Amber H. Brown Date: Fri Jul 5 23:36:41 2019 +1000 linting commit 3d89feb438d7754ba32ec48757814175a8fa9911 Author: Amber H. Brown Date: Fri Jul 5 23:34:24 2019 +1000 linting commit 400bc061caaab8a41f5186b060c68fece544ce2c Author: Amber H. Brown Date: Fri Jul 5 22:44:22 2019 +1000 linting commit a1de642fe7d82efbf6b03c03d7ded1e17f95aca6 Merge: f4343c7d2 54283f3ed Author: Amber H. Brown Date: Fri Jul 5 19:46:11 2019 +1000 Merge tag 'v1.1.0' into shhs Synapse 1.1.0 (2019-07-04) ========================== As of v1.1.0, Synapse no longer supports Python 2, nor Postgres version 9.4. See the [upgrade notes](UPGRADE.rst#upgrading-to-v110) for more details. This release also deprecates the use of environment variables to configure the docker image. See the [docker README](https://github.com/matrix-org/synapse/blob/release-v1.1.0/docker/README.md#legacy-dynamic-configuration-file-support) for more details. No changes since 1.1.0rc2. Synapse 1.1.0rc2 (2019-07-03) ============================= Bugfixes -------- - Fix regression in 1.1rc1 where OPTIONS requests to the media repo would fail. ([\#5593](https://github.com/matrix-org/synapse/issues/5593)) - Removed the `SYNAPSE_SMTP_*` docker container environment variables. Using these environment variables prevented the docker container from starting in Synapse v1.0, even though they didn't actually allow any functionality anyway. ([\#5596](https://github.com/matrix-org/synapse/issues/5596)) - Fix a number of "Starting txn from sentinel context" warnings. ([\#5605](https://github.com/matrix-org/synapse/issues/5605)) Internal Changes ---------------- - Update github templates. ([\#5552](https://github.com/matrix-org/synapse/issues/5552)) Synapse 1.1.0rc1 (2019-07-02) ============================= As of v1.1.0, Synapse no longer supports Python 2, nor Postgres version 9.4. See the [upgrade notes](UPGRADE.rst#upgrading-to-v110) for more details. Features -------- - Added possibilty to disable local password authentication. Contributed by Daniel Hoffend. ([\#5092](https://github.com/matrix-org/synapse/issues/5092)) - Add monthly active users to phonehome stats. ([\#5252](https://github.com/matrix-org/synapse/issues/5252)) - Allow expired user to trigger renewal email sending manually. ([\#5363](https://github.com/matrix-org/synapse/issues/5363)) - Statistics on forward extremities per room are now exposed via Prometheus. ([\#5384](https://github.com/matrix-org/synapse/issues/5384), [\#5458](https://github.com/matrix-org/synapse/issues/5458), [\#5461](https://github.com/matrix-org/synapse/issues/5461)) - Add --no-daemonize option to run synapse in the foreground, per issue #4130. Contributed by Soham Gumaste. ([\#5412](https://github.com/matrix-org/synapse/issues/5412), [\#5587](https://github.com/matrix-org/synapse/issues/5587)) - Fully support SAML2 authentication. Contributed by [Alexander Trost](https://github.com/galexrt) - thank you! ([\#5422](https://github.com/matrix-org/synapse/issues/5422)) - Allow server admins to define implementations of extra rules for allowing or denying incoming events. ([\#5440](https://github.com/matrix-org/synapse/issues/5440), [\#5474](https://github.com/matrix-org/synapse/issues/5474), [\#5477](https://github.com/matrix-org/synapse/issues/5477)) - Add support for handling pagination APIs on client reader worker. ([\#5505](https://github.com/matrix-org/synapse/issues/5505), [\#5513](https://github.com/matrix-org/synapse/issues/5513), [\#5531](https://github.com/matrix-org/synapse/issues/5531)) - Improve help and cmdline option names for --generate-config options. ([\#5512](https://github.com/matrix-org/synapse/issues/5512)) - Allow configuration of the path used for ACME account keys. ([\#5516](https://github.com/matrix-org/synapse/issues/5516), [\#5521](https://github.com/matrix-org/synapse/issues/5521), [\#5522](https://github.com/matrix-org/synapse/issues/5522)) - Add --data-dir and --open-private-ports options. ([\#5524](https://github.com/matrix-org/synapse/issues/5524)) - Split public rooms directory auth config in two settings, in order to manage client auth independently from the federation part of it. Obsoletes the "restrict_public_rooms_to_local_users" configuration setting. If "restrict_public_rooms_to_local_users" is set in the config, Synapse will act as if both new options are enabled, i.e. require authentication through the client API and deny federation requests. ([\#5534](https://github.com/matrix-org/synapse/issues/5534)) - The minimum TLS version used for outgoing federation requests can now be set with `federation_client_minimum_tls_version`. ([\#5550](https://github.com/matrix-org/synapse/issues/5550)) - Optimise devices changed query to not pull unnecessary rows from the database, reducing database load. ([\#5559](https://github.com/matrix-org/synapse/issues/5559)) - Add new metrics for number of forward extremities being persisted and number of state groups involved in resolution. ([\#5476](https://github.com/matrix-org/synapse/issues/5476)) Bugfixes -------- - Fix bug processing incoming events over federation if call to `/get_missing_events` fails. ([\#5042](https://github.com/matrix-org/synapse/issues/5042)) - Prevent more than one room upgrade happening simultaneously on the same room. ([\#5051](https://github.com/matrix-org/synapse/issues/5051)) - Fix a bug where running synapse_port_db would cause the account validity feature to fail because it didn't set the type of the email_sent column to boolean. ([\#5325](https://github.com/matrix-org/synapse/issues/5325)) - Warn about disabling email-based password resets when a reset occurs, and remove warning when someone attempts a phone-based reset. ([\#5387](https://github.com/matrix-org/synapse/issues/5387)) - Fix email notifications for unnamed rooms with multiple people. ([\#5388](https://github.com/matrix-org/synapse/issues/5388)) - Fix exceptions in federation reader worker caused by attempting to renew attestations, which should only happen on master worker. ([\#5389](https://github.com/matrix-org/synapse/issues/5389)) - Fix handling of failures fetching remote content to not log failures as exceptions. ([\#5390](https://github.com/matrix-org/synapse/issues/5390)) - Fix a bug where deactivated users could receive renewal emails if the account validity feature is on. ([\#5394](https://github.com/matrix-org/synapse/issues/5394)) - Fix missing invite state after exchanging 3PID invites over federaton. ([\#5464](https://github.com/matrix-org/synapse/issues/5464)) - Fix intermittent exceptions on Apple hardware. Also fix bug that caused database activity times to be under-reported in log lines. ([\#5498](https://github.com/matrix-org/synapse/issues/5498)) - Fix logging error when a tampered event is detected. ([\#5500](https://github.com/matrix-org/synapse/issues/5500)) - Fix bug where clients could tight loop calling `/sync` for a period. ([\#5507](https://github.com/matrix-org/synapse/issues/5507)) - Fix bug with `jinja2` preventing Synapse from starting. Users who had this problem should now simply need to run `pip install matrix-synapse`. ([\#5514](https://github.com/matrix-org/synapse/issues/5514)) - Fix a regression where homeservers on private IP addresses were incorrectly blacklisted. ([\#5523](https://github.com/matrix-org/synapse/issues/5523)) - Fixed m.login.jwt using unregistred user_id and added pyjwt>=1.6.4 as jwt conditional dependencies. Contributed by Pau Rodriguez-Estivill. ([\#5555](https://github.com/matrix-org/synapse/issues/5555), [\#5586](https://github.com/matrix-org/synapse/issues/5586)) - Fix a bug that would cause invited users to receive several emails for a single 3PID invite in case the inviter is rate limited. ([\#5576](https://github.com/matrix-org/synapse/issues/5576)) Updates to the Docker image --------------------------- - Add ability to change Docker containers [timezone](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) with the `TZ` variable. ([\#5383](https://github.com/matrix-org/synapse/issues/5383)) - Update docker image to use Python 3.7. ([\#5546](https://github.com/matrix-org/synapse/issues/5546)) - Deprecate the use of environment variables for configuration, and make the use of a static configuration the default. ([\#5561](https://github.com/matrix-org/synapse/issues/5561), [\#5562](https://github.com/matrix-org/synapse/issues/5562), [\#5566](https://github.com/matrix-org/synapse/issues/5566), [\#5567](https://github.com/matrix-org/synapse/issues/5567)) - Increase default log level for docker image to INFO. It can still be changed by editing the generated log.config file. ([\#5547](https://github.com/matrix-org/synapse/issues/5547)) - Send synapse logs to the docker logging system, by default. ([\#5565](https://github.com/matrix-org/synapse/issues/5565)) - Open the non-TLS port by default. ([\#5568](https://github.com/matrix-org/synapse/issues/5568)) - Fix failure to start under docker with SAML support enabled. ([\#5490](https://github.com/matrix-org/synapse/issues/5490)) - Use a sensible location for data files when generating a config file. ([\#5563](https://github.com/matrix-org/synapse/issues/5563)) Deprecations and Removals ------------------------- - Python 2.7 is no longer a supported platform. Synapse now requires Python 3.5+ to run. ([\#5425](https://github.com/matrix-org/synapse/issues/5425)) - PostgreSQL 9.4 is no longer supported. Synapse requires Postgres 9.5+ or above for Postgres support. ([\#5448](https://github.com/matrix-org/synapse/issues/5448)) - Remove support for cpu_affinity setting. ([\#5525](https://github.com/matrix-org/synapse/issues/5525)) Improved Documentation ---------------------- - Improve README section on performance troubleshooting. ([\#4276](https://github.com/matrix-org/synapse/issues/4276)) - Add information about how to install and run `black` on the codebase to code_style.rst. ([\#5537](https://github.com/matrix-org/synapse/issues/5537)) - Improve install docs on choosing server_name. ([\#5558](https://github.com/matrix-org/synapse/issues/5558)) Internal Changes ---------------- - Add logging to 3pid invite signature verification. ([\#5015](https://github.com/matrix-org/synapse/issues/5015)) - Update example haproxy config to a more compatible setup. ([\#5313](https://github.com/matrix-org/synapse/issues/5313)) - Track deactivated accounts in the database. ([\#5378](https://github.com/matrix-org/synapse/issues/5378), [\#5465](https://github.com/matrix-org/synapse/issues/5465), [\#5493](https://github.com/matrix-org/synapse/issues/5493)) - Clean up code for sending federation EDUs. ([\#5381](https://github.com/matrix-org/synapse/issues/5381)) - Add a sponsor button to the repo. ([\#5382](https://github.com/matrix-org/synapse/issues/5382), [\#5386](https://github.com/matrix-org/synapse/issues/5386)) - Don't log non-200 responses from federation queries as exceptions. ([\#5383](https://github.com/matrix-org/synapse/issues/5383)) - Update Python syntax in contrib/ to Python 3. ([\#5446](https://github.com/matrix-org/synapse/issues/5446)) - Update federation_client dev script to support `.well-known` and work with python3. ([\#5447](https://github.com/matrix-org/synapse/issues/5447)) - SyTest has been moved to Buildkite. ([\#5459](https://github.com/matrix-org/synapse/issues/5459)) - Demo script now uses python3. ([\#5460](https://github.com/matrix-org/synapse/issues/5460)) - Synapse can now handle RestServlets that return coroutines. ([\#5475](https://github.com/matrix-org/synapse/issues/5475), [\#5585](https://github.com/matrix-org/synapse/issues/5585)) - The demo servers talk to each other again. ([\#5478](https://github.com/matrix-org/synapse/issues/5478)) - Add an EXPERIMENTAL config option to try and periodically clean up extremities by sending dummy events. ([\#5480](https://github.com/matrix-org/synapse/issues/5480)) - Synapse's codebase is now formatted by `black`. ([\#5482](https://github.com/matrix-org/synapse/issues/5482)) - Some cleanups and sanity-checking in the CPU and database metrics. ([\#5499](https://github.com/matrix-org/synapse/issues/5499)) - Improve email notification logging. ([\#5502](https://github.com/matrix-org/synapse/issues/5502)) - Fix "Unexpected entry in 'full_schemas'" log warning. ([\#5509](https://github.com/matrix-org/synapse/issues/5509)) - Improve logging when generating config files. ([\#5510](https://github.com/matrix-org/synapse/issues/5510)) - Refactor and clean up Config parser for maintainability. ([\#5511](https://github.com/matrix-org/synapse/issues/5511)) - Make the config clearer in that email.template_dir is relative to the Synapse's root directory, not the `synapse/` folder within it. ([\#5543](https://github.com/matrix-org/synapse/issues/5543)) - Update v1.0.0 release changelog to include more information about changes to password resets. ([\#5545](https://github.com/matrix-org/synapse/issues/5545)) - Remove non-functioning check_event_hash.py dev script. ([\#5548](https://github.com/matrix-org/synapse/issues/5548)) - Synapse will now only allow TLS v1.2 connections when serving federation, if it terminates TLS. As Synapse's allowed ciphers were only able to be used in TLSv1.2 before, this does not change behaviour. ([\#5550](https://github.com/matrix-org/synapse/issues/5550)) - Logging when running GC collection on generation 0 is now at the DEBUG level, not INFO. ([\#5557](https://github.com/matrix-org/synapse/issues/5557)) - Reduce the amount of stuff we send in the docker context. ([\#5564](https://github.com/matrix-org/synapse/issues/5564)) - Point the reverse links in the Purge History contrib scripts at the intended location. ([\#5570](https://github.com/matrix-org/synapse/issues/5570)) commit f4343c7d2bbde2d4b9674a029867bbe0353616e4 Merge: 4689408a3 463d5a8fd Author: Amber H. Brown Date: Wed Jul 3 22:39:30 2019 +1000 Merge remote-tracking branch 'origin/develop' into shhs commit 4689408a35e6ff9b7b18a5940e79d15e6b846913 Merge: bed45ab20 b4914681a Author: Amber H. Brown Date: Tue Jul 2 18:31:29 2019 +1000 Merge remote-tracking branch 'origin/develop' into shhs commit bed45ab20b985d53fc767a67c77ac21d5996de41 Author: Amber H. Brown Date: Tue Jul 2 18:18:09 2019 +1000 release shhs on tags commit 0993b05ca56198ac8ffe5ad6dea280b8fd99d0cf Author: Amber H. Brown Date: Mon Jul 1 23:13:21 2019 +1000 improve error text when room is too large commit e0011152214ae19283dc043ab355822c0baf7c51 Author: Amber H. Brown Date: Tue Jun 18 21:24:56 2019 +1000 fix commit e60aab14b4203b91bad23417b1664289b4b558b6 Merge: e7c117193 82d9d524b Author: Amber H. Brown Date: Tue Jun 18 21:20:13 2019 +1000 Merge remote-tracking branch 'origin/develop' into shhs commit e7c1171935168eb982fe171660e8f957c870f237 Merge: 8fe26db96 c831748f4 Author: Amber Brown Date: Tue Jun 4 20:41:59 2019 +1000 Merge remote-tracking branch 'origin/master' into shhs commit 8fe26db968dd833976f59c7b122611896a211c30 Merge: c99c10515 4a30e4acb Author: Amber Brown Date: Tue May 21 14:30:47 2019 -0500 Merge remote-tracking branch 'origin/develop' into HEAD commit c99c1051589a8370ae955bc21695343f87d6b8c1 Author: Amber Brown Date: Mon May 20 17:01:50 2019 -0500 SHHS - Room Join Complexity (#5072) commit d142e51f76f8f29b5ce040251803a3fbd3f4a4f4 Merge: d424ba9e5 24b93b9c7 Author: Amber Brown Date: Mon May 20 15:43:08 2019 -0500 Merge remote-tracking branch 'origin/develop' into shhs commit d424ba9e5bbcb63ae3929401a27cb89ac269938f Merge: a1b8767da f1e5b4138 Author: Amber Brown Date: Wed May 15 23:30:22 2019 -0500 Merge remote-tracking branch 'origin/develop' into shhs commit a1b8767da85794eb24e3b9fbce1aed87a04ea328 Merge: faee1e9ba df2ebd75d Author: Amber Brown Date: Mon May 13 15:01:58 2019 -0500 Merge remote-tracking branch 'origin/develop' into shhs commit faee1e9baba74b08fb991d57effd06f87147fc76 Merge: 12875f995 d216a36b3 Author: Amber Brown Date: Wed May 8 16:29:45 2019 -0500 Merge remote-tracking branch 'origin/develop' into shhs commit 12875f995adb7a2d032a8492030a770fbb5c40a3 Merge: ed3814162 c1799b0f8 Author: Amber Brown Date: Wed May 1 10:55:14 2019 -0400 Merge remote-tracking branch 'origin/develop' into shhs commit ed38141620a6137058ed9001037f77230f1a5dcc Author: Amber Brown Date: Mon Apr 29 20:57:42 2019 +1000 target better for the shhs release docker hub, pt 3 commit bd5f62469c6950abc7c8891cf4c9cd115fc9d18a Author: Amber Brown Date: Mon Apr 29 20:43:17 2019 +1000 target better for the shhs release docker hub, pt 2 commit c0f57cab682248099454ada67649972ab3c1cd5f Author: Amber Brown Date: Mon Apr 29 20:36:35 2019 +1000 target better for the shhs release docker hub commit 1d5cf66958245f6b9df3e8ba52ffd3b6cb17eec4 Author: Amber Brown Date: Mon Apr 29 20:33:36 2019 +1000 no media repo == no path checks commit 25256f958bd1c1c4f6e79f070263cc57b71d3c2b Author: Amber Brown Date: Mon Apr 29 20:30:55 2019 +1000 release shhs as a release commit a32aa2ce71f5c208b44fad4343412cd9f5e713f1 Author: Amber Brown Date: Mon Apr 29 20:28:40 2019 +1000 patch up docker commit cbc866a60738a7c24e370b95be4c3bf80ab3885a Author: Amber Brown Date: Fri Apr 26 01:40:01 2019 +1000 Remove Python 2 from the SHHS branch CI (#5099) --- changelog.d/5072.feature | 1 + docs/sample_config.yaml | 11 ++++ synapse/config/server.py | 17 ++++++ synapse/handlers/federation.py | 25 ++++++++ synapse/handlers/room_member.py | 89 +++++++++++++++++++++++++++-- tests/federation/test_complexity.py | 74 +++++++++++++++++++++++- 6 files changed, 212 insertions(+), 5 deletions(-) create mode 100644 changelog.d/5072.feature diff --git a/changelog.d/5072.feature b/changelog.d/5072.feature new file mode 100644 index 000000000000..99fda5616a3a --- /dev/null +++ b/changelog.d/5072.feature @@ -0,0 +1 @@ +Synapse can now be configured to not join remote rooms of a given "complexity" (currently, state events). This option can be used to prevent adverse performance on resource-constrained homeservers. \ No newline at end of file diff --git a/docs/sample_config.yaml b/docs/sample_config.yaml index 7edf15207afc..17523bf50207 100644 --- a/docs/sample_config.yaml +++ b/docs/sample_config.yaml @@ -278,6 +278,17 @@ listeners: # Used by phonehome stats to group together related servers. #server_context: context +# Resource-constrained Homeserver Settings +# +# If limit_large_remote_room_joins is True, the room complexity will be +# checked before a user joins a new remote room. If it is above +# limit_large_remote_room_complexity, it will disallow joining or +# instantly leave. +# +# Uncomment the below lines to enable: +#limit_large_remote_room_joins: True +#limit_large_remote_room_complexity: 1.0 + # Whether to require a user to be in the room to add an alias to it. # Defaults to 'true'. # diff --git a/synapse/config/server.py b/synapse/config/server.py index 00170f139369..d90f1ebb9ccd 100644 --- a/synapse/config/server.py +++ b/synapse/config/server.py @@ -247,6 +247,12 @@ def read_config(self, config, **kwargs): self.gc_thresholds = read_gc_thresholds(config.get("gc_thresholds", None)) + # Resource-constrained Homeserver Configuration + self.limit_large_room_joins = config.get("limit_large_remote_room_joins", False) + self.limit_large_room_complexity = config.get( + "limit_large_remote_room_complexity", 1.0 + ) + bind_port = config.get("bind_port") if bind_port: if config.get("no_tls", False): @@ -617,6 +623,17 @@ def generate_config_section( # Used by phonehome stats to group together related servers. #server_context: context + # Resource-constrained Homeserver Settings + # + # If limit_large_remote_room_joins is True, the room complexity will be + # checked before a user joins a new remote room. If it is above + # limit_large_remote_room_complexity, it will disallow joining or + # instantly leave. + # + # Uncomment the below lines to enable: + #limit_large_remote_room_joins: True + #limit_large_remote_room_complexity: 1.0 + # Whether to require a user to be in the room to add an alias to it. # Defaults to 'true'. # diff --git a/synapse/handlers/federation.py b/synapse/handlers/federation.py index 89b37dbc1c50..10160bfe86b5 100644 --- a/synapse/handlers/federation.py +++ b/synapse/handlers/federation.py @@ -2796,3 +2796,28 @@ def user_joined_room(self, user, room_id): ) else: return user_joined_room(self.distributor, user, room_id) + + @defer.inlineCallbacks + def get_room_complexity(self, remote_room_hosts, room_id): + """ + Fetch the complexity of a remote room over federation. + + Args: + remote_room_hosts (list[str]): The remote servers to ask. + room_id (str): The room ID to ask about. + + Returns: + Deferred[dict] or Deferred[None]: Dict contains the complexity + metric versions, while None means we could not fetch the complexity. + """ + + for host in remote_room_hosts: + res = yield self.federation_client.get_room_complexity(host, room_id) + + # We got a result, return it. + if res: + defer.returnValue(res) + + # We fell off the bottom, couldn't get the complexity from anyone. Oh + # well. + defer.returnValue(None) diff --git a/synapse/handlers/room_member.py b/synapse/handlers/room_member.py index baea08ddd07f..97e26dde1258 100644 --- a/synapse/handlers/room_member.py +++ b/synapse/handlers/room_member.py @@ -26,8 +26,7 @@ from twisted.internet import defer -import synapse.server -import synapse.types +from synapse import types from synapse.api.constants import EventTypes, Membership from synapse.api.errors import AuthError, Codes, HttpResponseException, SynapseError from synapse.types import RoomID, UserID @@ -39,6 +38,11 @@ logger = logging.getLogger(__name__) id_server_scheme = "https://" +ROOM_COMPLEXITY_TOO_GREAT = ( + "Your homeserver is unable to join rooms this large or complex. " + "Please speak to your server administrator, or upgrade your instance " + "to join this room." +) class RoomMemberHandler(object): @@ -543,7 +547,7 @@ def send_membership_event( ), "Sender (%s) must be same as requester (%s)" % (sender, requester.user) assert self.hs.is_mine(sender), "Sender must be our own: %s" % (sender,) else: - requester = synapse.types.create_requester(target_user) + requester = types.create_requester(target_user) prev_event = yield self.event_creation_handler.deduplicate_state_event( event, context @@ -945,6 +949,47 @@ def __init__(self, hs): self.distributor.declare("user_joined_room") self.distributor.declare("user_left_room") + @defer.inlineCallbacks + def _is_remote_room_too_complex(self, room_id, remote_room_hosts): + """ + Check if complexity of a remote room is too great. + + Args: + room_id (str) + remote_room_hosts (list[str]) + + Returns: bool of whether the complexity is too great, or None + if unable to be fetched + """ + max_complexity = self.hs.config.limit_large_room_complexity + complexity = yield self.federation_handler.get_room_complexity( + remote_room_hosts, room_id + ) + + if complexity: + if complexity["v1"] > max_complexity: + return True + return False + return None + + @defer.inlineCallbacks + def _is_local_room_too_complex(self, room_id): + """ + Check if the complexity of a local room is too great. + + Args: + room_id (str) + + Returns: bool + """ + max_complexity = self.hs.config.limit_large_room_complexity + complexity = yield self.store.get_room_complexity(room_id) + + if complexity["v1"] > max_complexity: + return True + + return False + @defer.inlineCallbacks def _remote_join(self, requester, remote_room_hosts, room_id, user, content): """Implements RoomMemberHandler._remote_join @@ -952,7 +997,6 @@ def _remote_join(self, requester, remote_room_hosts, room_id, user, content): # filter ourselves out of remote_room_hosts: do_invite_join ignores it # and if it is the only entry we'd like to return a 404 rather than a # 500. - remote_room_hosts = [ host for host in remote_room_hosts if host != self.hs.hostname ] @@ -960,6 +1004,18 @@ def _remote_join(self, requester, remote_room_hosts, room_id, user, content): if len(remote_room_hosts) == 0: raise SynapseError(404, "No known servers") + if self.hs.config.limit_large_room_joins: + # Fetch the room complexity + too_complex = yield self._is_remote_room_too_complex( + room_id, remote_room_hosts + ) + if too_complex is True: + raise SynapseError( + code=400, + msg=ROOM_COMPLEXITY_TOO_GREAT, + errcode=Codes.RESOURCE_LIMIT_EXCEEDED, + ) + # We don't do an auth check if we are doing an invite # join dance for now, since we're kinda implicitly checking # that we are allowed to join when we decide whether or not we @@ -969,6 +1025,31 @@ def _remote_join(self, requester, remote_room_hosts, room_id, user, content): ) yield self._user_joined_room(user, room_id) + # Check the room we just joined wasn't too large, if we didn't fetch the + # complexity of it before. + if self.hs.config.limit_large_room_joins: + if too_complex is False: + # We checked, and we're under the limit. + return + + # Check again, but with the local state events + too_complex = yield self._is_local_room_too_complex(room_id) + + if too_complex is False: + # We're under the limit. + return + + # The room is too large. Leave. + requester = types.create_requester(user, None, False, None) + yield self.update_membership( + requester=requester, target=user, room_id=room_id, action="leave" + ) + raise SynapseError( + code=400, + msg=ROOM_COMPLEXITY_TOO_GREAT, + errcode=Codes.RESOURCE_LIMIT_EXCEEDED, + ) + @defer.inlineCallbacks def _remote_reject_invite(self, requester, remote_room_hosts, room_id, target): """Implements RoomMemberHandler._remote_reject_invite diff --git a/tests/federation/test_complexity.py b/tests/federation/test_complexity.py index a5b03005d7aa..a03d0cdb0472 100644 --- a/tests/federation/test_complexity.py +++ b/tests/federation/test_complexity.py @@ -13,12 +13,16 @@ # See the License for the specific language governing permissions and # limitations under the License. +from mock import Mock + from twisted.internet import defer +from synapse.api.errors import Codes, SynapseError from synapse.config.ratelimiting import FederationRateLimitConfig from synapse.federation.transport import server from synapse.rest import admin from synapse.rest.client.v1 import login, room +from synapse.types import UserID from synapse.util.ratelimitutils import FederationRateLimiter from tests import unittest @@ -33,7 +37,7 @@ class RoomComplexityTests(unittest.HomeserverTestCase): ] def default_config(self, name="test"): - config = super(RoomComplexityTests, self).default_config(name=name) + config = super().default_config(name=name) config["limit_large_remote_room_joins"] = True config["limit_large_remote_room_complexity"] = 0.05 return config @@ -88,3 +92,71 @@ def test_complexity_simple(self): self.assertEquals(200, channel.code) complexity = channel.json_body["v1"] self.assertEqual(complexity, 1.23) + + def test_join_too_large(self): + + u1 = self.register_user("u1", "pass") + + handler = self.hs.get_room_member_handler() + fed_transport = self.hs.get_federation_transport_client() + + # Mock out some things, because we don't want to test the whole join + fed_transport.client.get_json = Mock(return_value=defer.succeed({"v1": 9999})) + handler.federation_handler.do_invite_join = Mock(return_value=defer.succeed(1)) + + d = handler._remote_join( + None, + ["otherserver.example"], + "roomid", + UserID.from_string(u1), + {"membership": "join"}, + ) + + self.pump() + + # The request failed with a SynapseError saying the resource limit was + # exceeded. + f = self.get_failure(d, SynapseError) + self.assertEqual(f.value.code, 400, f.value) + self.assertEqual(f.value.errcode, Codes.RESOURCE_LIMIT_EXCEEDED) + + def test_join_too_large_once_joined(self): + + u1 = self.register_user("u1", "pass") + u1_token = self.login("u1", "pass") + + # Ok, this might seem a bit weird -- I want to test that we actually + # leave the room, but I don't want to simulate two servers. So, we make + # a local room, which we say we're joining remotely, even if there's no + # remote, because we mock that out. Then, we'll leave the (actually + # local) room, which will be propagated over federation in a real + # scenario. + room_1 = self.helper.create_room_as(u1, tok=u1_token) + + handler = self.hs.get_room_member_handler() + fed_transport = self.hs.get_federation_transport_client() + + # Mock out some things, because we don't want to test the whole join + fed_transport.client.get_json = Mock(return_value=defer.succeed(None)) + handler.federation_handler.do_invite_join = Mock(return_value=defer.succeed(1)) + + # Artificially raise the complexity + self.hs.get_datastore().get_current_state_event_counts = lambda x: defer.succeed( + 600 + ) + + d = handler._remote_join( + None, + ["otherserver.example"], + room_1, + UserID.from_string(u1), + {"membership": "join"}, + ) + + self.pump() + + # The request failed with a SynapseError saying the resource limit was + # exceeded. + f = self.get_failure(d, SynapseError) + self.assertEqual(f.value.code, 400) + self.assertEqual(f.value.errcode, Codes.RESOURCE_LIMIT_EXCEEDED)