Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Add workers settings to configuration manual #14086

Merged
merged 29 commits into from
Oct 27, 2022
Merged
Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
3c0abc6
Add workers settings to configuration manual
dklimpel Oct 6, 2022
167b28e
newsfile
dklimpel Oct 6, 2022
41c7992
Merge branch 'develop' into worker_doc
dklimpel Oct 7, 2022
895fa0d
Update `pusher_instances`
dklimpel Oct 8, 2022
2e6fcbf
Merge remote-tracking branch 'synapse/develop' into worker_doc
dklimpel Oct 13, 2022
2c0652d
Move from `config_documentation.md` to `workers.md`
dklimpel Oct 13, 2022
b6d9285
Merge remote-tracking branch 'synapse/develop' into worker_doc
dklimpel Oct 23, 2022
3b144ef
Move docs back to `config_documentation.md`
dklimpel Oct 24, 2022
1ac4cf6
newsfile and small update
dklimpel Oct 24, 2022
628a448
Apply suggestions from code review
dklimpel Oct 26, 2022
327c81c
update url to python logger
dklimpel Oct 26, 2022
d12e006
update headlines
dklimpel Oct 26, 2022
c106747
update links after headline change
dklimpel Oct 26, 2022
955dab6
remove link from `daemon process`
dklimpel Oct 26, 2022
18d692a
Apply suggestions from code review
dklimpel Oct 26, 2022
1d677f3
Apply suggestions from code review
dklimpel Oct 26, 2022
2c91c5a
update `event persister` doc
dklimpel Oct 26, 2022
b0a20d1
extend example for `federation_sender_instances` and `pusher_instances`
dklimpel Oct 26, 2022
17ec5d3
more infos about stream writers
dklimpel Oct 26, 2022
e0fc9e3
Merge branch 'develop' into worker_doc
dklimpel Oct 26, 2022
76f8b5f
add link to DAG
dklimpel Oct 27, 2022
a3ad96a
small rewording
dklimpel Oct 27, 2022
6c9cea2
Apply suggestions from code review
dklimpel Oct 27, 2022
0f9b29f
Apply suggestions from code review
dklimpel Oct 27, 2022
5045ca0
update `pusher_instances`
dklimpel Oct 27, 2022
a943440
update `worker_listeners`
dklimpel Oct 27, 2022
a214ee8
update `stream_writers`
dklimpel Oct 27, 2022
c0434bd
Update `worker_name`
dklimpel Oct 27, 2022
8394c3e
Fix dmr typo in `stream_writers`!!
Oct 27, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/14086.doc
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Add workers settings to [configuration manual](https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html#worker-configuration-for-workers-configuration).
2 changes: 1 addition & 1 deletion docs/sample_log_config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
# Synapse also supports structured logging for machine readable logs which can
# be ingested by ELK stacks. See [2] for details.
#
# [1]: https://docs.python.org/3.7/library/logging.config.html#configuration-dictionary-schema
# [1]: https://docs.python.org/3.11/library/logging.config.html#configuration-dictionary-schema
dklimpel marked this conversation as resolved.
Show resolved Hide resolved
# [2]: https://matrix-org.github.io/synapse/latest/structured_logging.html

version: 1
Expand Down
200 changes: 172 additions & 28 deletions docs/usage/configuration/config_documentation.md
Original file line number Diff line number Diff line change
Expand Up @@ -159,7 +159,7 @@ including _matrix/...). This is the same URL a user might enter into the
'Custom Homeserver URL' field on their client. If you use Synapse with a
reverse proxy, this should be the URL to reach Synapse via the proxy.
Otherwise, it should be the URL to reach Synapse's client HTTP listener (see
'listeners' below).
['listeners'](#listeners) below).

Defaults to `https://<server_name>/`.

Expand Down Expand Up @@ -1338,7 +1338,8 @@ Config options related to logging.
---
### `log_config`

This option specifies a yaml python logging config file as described [here](https://docs.python.org/3.7/library/logging.config.html#configuration-dictionary-schema).
This option specifies a yaml python logging config file as described
[here](https://docs.python.org/3.11/library/logging.config.html#configuration-dictionary-schema).

Example configuration:
```yaml
Expand Down Expand Up @@ -1766,7 +1767,7 @@ url_preview_ip_range_blacklist:
- 'ff00::/8'
- 'fec0::/10'
```
----
---
### `url_preview_ip_range_whitelist`

This option sets a list of IP address CIDR ranges that the URL preview spider is allowed
Expand Down Expand Up @@ -1860,7 +1861,7 @@ Example configuration:
- 'fr;q=0.8'
- '*;q=0.7'
```
----
---
### `oembed`

oEmbed allows for easier embedding content from a website. It can be
Expand Down Expand Up @@ -1947,7 +1948,7 @@ Example configuration:
```yaml
turn_shared_secret: "YOUR_SHARED_SECRET"
```
----
---
### `turn_username` and `turn_password`

The Username and password if the TURN server needs them and does not use a token.
Expand Down Expand Up @@ -2366,7 +2367,7 @@ Example configuration:
```yaml
session_lifetime: 24h
```
----
---
### `refresh_access_token_lifetime`

Time that an access token remains valid for, if the session is using refresh tokens.
Expand Down Expand Up @@ -2422,7 +2423,7 @@ nonrefreshable_access_token_lifetime: 24h
```

---
## Metrics ###
## Metrics ##
dklimpel marked this conversation as resolved.
Show resolved Hide resolved
Config options related to metrics.

---
Expand Down Expand Up @@ -3670,14 +3671,63 @@ opentracing:
false
```
---
## Workers ##
## Worker configuration for main process
dklimpel marked this conversation as resolved.
Show resolved Hide resolved
Configuration options related to workers.
dklimpel marked this conversation as resolved.
Show resolved Hide resolved
Workers are used to scale horizontally and distribute the load to different processes.
dklimpel marked this conversation as resolved.
Show resolved Hide resolved

The worker configuration is divided into two parts.

1. These switches for the shared configuration to give the main process
the necessary information.
1. And the [switches for the worker configuration file](#worker-configuration-for-workers-configuration)
to configure each worker them self are futher below.
dklimpel marked this conversation as resolved.
Show resolved Hide resolved

The manual how to configure workers is described in detail in the
[worker documentation](../../workers.md)
dklimpel marked this conversation as resolved.
Show resolved Hide resolved

---
### `worker_replication_secret`

A shared secret used by the replication APIs to authenticate HTTP requests
from workers.

By default this is unused and traffic is not authenticated.
dklimpel marked this conversation as resolved.
Show resolved Hide resolved

Example configuration:
```yaml
worker_replication_secret: "secret_secret"
```
---
### `start_pushers`

Controls sending of push notifications on the main process. Set to `false`
if using a [pusher worker](../../workers.md#synapseapppusher). Defaults to `true`.

Example configuration:
```yaml
start_pushers: false
```
---
### `pusher_instances`

It is possible to run multiple [pusher workers](../../workers.md#synapseapppusher),
in which case the work is balanced across them. Use this setting to list the pushers by
[`worker_name`](#worker_name).
DMRobertson marked this conversation as resolved.
Show resolved Hide resolved

If only one pusher worker is configured, this setting is not necessary.
DMRobertson marked this conversation as resolved.
Show resolved Hide resolved

Example configuration:
```yaml
pusher_instances:
- pusher_worker1
- pusher_worker2
DMRobertson marked this conversation as resolved.
Show resolved Hide resolved
```
---
### `send_federation`

Controls sending of outbound federation transactions on the main process.
Set to false if using a federation sender worker. Defaults to true.
Set to false if using a [federation sender worker](../../workers.md#synapseappfederation_sender).
DMRobertson marked this conversation as resolved.
Show resolved Hide resolved
Defaults to `true`.

Example configuration:
```yaml
Expand All @@ -3686,8 +3736,9 @@ send_federation: false
---
### `federation_sender_instances`

It is possible to run multiple federation sender workers, in which case the
work is balanced across them. Use this setting to list the senders.
It is possible to run multiple
[federation sender worker](../../workers.md#synapseappfederation_sender), in which
case the work is balanced across them. Use this setting to list the senders.

This configuration setting must be shared between all federation sender workers, and if
changed all federation sender workers must be stopped at the same time and then
dklimpel marked this conversation as resolved.
Show resolved Hide resolved
Expand All @@ -3702,8 +3753,9 @@ federation_sender_instances:
---
### `instance_map`

When using workers this should be a map from worker name to the
When using workers this should be a map from [`worker_name`](#worker_name) to the
HTTP replication listener of the worker, if configured.
Not every type of worker needs a HTTP replications listener.
DMRobertson marked this conversation as resolved.
Show resolved Hide resolved

Example configuration:
```yaml
Expand All @@ -3717,7 +3769,7 @@ instance_map:

Experimental: When using workers you can define which workers should
handle event persistence and typing notifications. Any worker
DMRobertson marked this conversation as resolved.
Show resolved Hide resolved
specified here must also be in the `instance_map`.
specified here must also be in the [`instance_map`](#instance_map).

Example configuration:
```yaml
Expand All @@ -3728,29 +3780,18 @@ stream_writers:
---
### `run_background_tasks_on`

The worker that is used to run background tasks (e.g. cleaning up expired
data). If not provided this defaults to the main process.
The [worker](../../workers.md#background-tasks) that is used to run
background tasks (e.g. cleaning up expired data). If not provided this
defaults to the main process.

Example configuration:
```yaml
run_background_tasks_on: worker1
```
---
### `worker_replication_secret`
dklimpel marked this conversation as resolved.
Show resolved Hide resolved

A shared secret used by the replication APIs to authenticate HTTP requests
from workers.

By default this is unused and traffic is not authenticated.

Example configuration:
```yaml
worker_replication_secret: "secret_secret"
```
### `redis`

Configuration for Redis when using workers. This *must* be enabled when
using workers (unless using old style direct TCP configuration).
Configuration for Redis when using workers. This *must* be enabled when using workers.
This setting has the following sub-options:
* `enabled`: whether to use Redis support. Defaults to false.
* `host` and `port`: Optional host and port to use to connect to redis. Defaults to
Expand All @@ -3765,6 +3806,109 @@ redis:
port: 6379
password: <secret_password>
```
---
## Worker configuration for workers configuration
dklimpel marked this conversation as resolved.
Show resolved Hide resolved
These switches configure each worker them self with the a worker configuration file.
dklimpel marked this conversation as resolved.
Show resolved Hide resolved

Note also the configuration for the
[main process above](#worker-configuration-for-main-process).
dklimpel marked this conversation as resolved.
Show resolved Hide resolved

The manual how to configure workers is described in detail in the
[worker documentation](../../workers.md)
dklimpel marked this conversation as resolved.
Show resolved Hide resolved

---
### `worker_app`
DMRobertson marked this conversation as resolved.
Show resolved Hide resolved

The type of worker. The currently available worker applications are listed
in [worker documentation](../../workers.md#available-worker-applications).

The most common worker is the
[`synapse.app.generic_worker`](../../workers.md#synapseappgeneric_worker).

Example configuration:
```yaml
worker_app: synapse.app.generic_worker
```
---
### `worker_name`
DMRobertson marked this conversation as resolved.
Show resolved Hide resolved

A unique name for the worker. The worker needs a name to be addressed in
further parameters and identification in log files.

Example configuration:
```yaml
worker_name: generic_worker1
```
---
### `worker_replication_host`

The HTTP replication endpoint that it should talk to on the main Synapse process.
The main Synapse process defines this with a `replication` resource in
[`listeners` option](#listeners).

Example configuration:
```yaml
worker_replication_host: 127.0.0.1
```
---
### `worker_replication_http_port`

The HTTP replication port that it should talk to on the main Synapse process.
The main Synapse process defines this with a `replication` resource in
[`listeners` option](#listeners).

Example configuration:
```yaml
worker_replication_http_port: 9093
```
DMRobertson marked this conversation as resolved.
Show resolved Hide resolved
---
### `worker_listeners`

A worker can handle HTTP requests. If handling HTTP requests, a `worker_listeners`
DMRobertson marked this conversation as resolved.
Show resolved Hide resolved
option with an http listener, in the same way as the [`listeners` option](#listeners)
in the shared config.

Example configuration:
```yaml
worker_listeners:
- type: http
port: 8083
resources:
- names: [client, federation]
```
---
### `worker_daemonize`

Specifies whether the worker should be daemonize. If
[systemd](../../systemd-with-workers/README.md) is used, this must not configured.
Systemd manages daemonization itself. Defaults to `false`.
dklimpel marked this conversation as resolved.
Show resolved Hide resolved

Example configuration:
```yaml
worker_daemonize: true
```
---
DMRobertson marked this conversation as resolved.
Show resolved Hide resolved
### `worker_pid_file`

When running Synapse worker as a daemon, the file to store the pid in. Defaults to none.
This is the same way as the[`pid_file` option](#pid_file) in the shared config.
dklimpel marked this conversation as resolved.
Show resolved Hide resolved

Example configuration:
```yaml
worker_pid_file: DATADIR/generic_worker1.pid
DMRobertson marked this conversation as resolved.
Show resolved Hide resolved
```
---
### `worker_log_config`

This option specifies a yaml python logging config file as described
[here](https://docs.python.org/3.11/library/logging.config.html#configuration-dictionary-schema).
This is the same way as the [`log_config` option](#log_config) in the shared config.
dklimpel marked this conversation as resolved.
Show resolved Hide resolved

Example configuration:
```yaml
worker_log_config: /etc/matrix-synapse/generic-worker-log.yaml
```
---
## Background Updates ##
Configuration settings related to background updates.

Expand Down
Loading