Deprecate the env var way of running the docker image (#5566)

changelog.d/5566.feature

@@ -0,0 +1 @@
+Update Docker image to deprecate the use of environment variables for configuration, and make the use of a static configuration the default.

contrib/docker/README.md

@@ -1,5 +1,9 @@
 # Synapse Docker
 
+FIXME: this is out-of-date as of
+https://github.com/matrix-org/synapse/issues/5518. Contributions to bring it up
+to date would be welcome.
+
 ### Automated configuration
 
 It is recommended that you use Docker Compose to run your containers, including

docker/README.md

@@ -6,39 +6,11 @@ postgres database.
 
 The image also does *not* provide a TURN server.
 
-## Run
-
-### Using docker-compose (easier)
-
-This image is designed to run either with an automatically generated
-configuration file or with a custom configuration that requires manual editing.
-
-An easy way to make use of this image is via docker-compose. See the
-[contrib/docker](https://github.com/matrix-org/synapse/tree/master/contrib/docker) section of the synapse project for
-examples.
-
-### Without Compose (harder)
-
-If you do not wish to use Compose, you may still run this image using plain
-Docker commands. Note that the following is just a guideline and you may need
-to add parameters to the docker run command to account for the network situation
-with your postgres database.
-
-```
-docker run \
-    -d \
-    --name synapse \
-    --mount type=volume,src=synapse-data,dst=/data \
-    -e SYNAPSE_SERVER_NAME=my.matrix.host \
-    -e SYNAPSE_REPORT_STATS=yes \
-    -p 8448:8448 \
-    matrixdotorg/synapse:latest
-```
-
 ## Volumes
 
-The image expects a single volume, located at ``/data``, that will hold:
+By default, the image expects a single volume, located at ``/data``, that will hold:
 
+* configuration files;
 * temporary files during uploads;
 * uploaded media and thumbnails;
 * the SQLite database if you do not configure postgres;
@@ -53,142 +25,90 @@ In order to setup an application service, simply create an ``appservices``
 directory in the data volume and write the application service Yaml
 configuration file there. Multiple application services are supported.
 
-## TLS certificates
-
-Synapse requires a valid TLS certificate. You can do one of the following:
-
- * Provide your own certificate and key (as
-   `${DATA_PATH}/${SYNAPSE_SERVER_NAME}.tls.crt` and
-   `${DATA_PATH}/${SYNAPSE_SERVER_NAME}.tls.key`, or elsewhere by providing an
-   entire config as `${SYNAPSE_CONFIG_PATH}`). In this case, you should forward
-   traffic to port 8448 in the container, for example with `-p 443:8448`.
-
- * Use a reverse proxy to terminate incoming TLS, and forward the plain http
-   traffic to port 8008 in the container. In this case you should set `-e
-   SYNAPSE_NO_TLS=1`.
-
- * Use the ACME (Let's Encrypt) support built into Synapse. This requires
-   `${SYNAPSE_SERVER_NAME}` port 80 to be forwarded to port 8009 in the
-   container, for example with `-p 80:8009`. To enable it in the docker
-   container, set `-e SYNAPSE_ACME=1`.
-
-If you don't do any of these, Synapse will fail to start with an error similar to:
-
-    synapse.config._base.ConfigError: Error accessing file '/data/<server_name>.tls.crt' (config for tls_certificate): No such file or directory
-
-## Environment
-
-Unless you specify a custom path for the configuration file, a very generic
-file will be generated, based on the following environment settings.
-These are a good starting point for setting up your own deployment.
-
-Global settings:
-
-* ``UID``, the user id Synapse will run as [default 991]
-* ``GID``, the group id Synapse will run as [default 991]
-* ``SYNAPSE_CONFIG_PATH``, path to a custom config file
-
-If ``SYNAPSE_CONFIG_PATH`` is set, you should generate a configuration file
-then customize it manually: see [Generating a config
-file](#generating-a-config-file).
-
-Otherwise, a dynamic configuration file will be used.
-
-### Environment variables used to build a dynamic configuration file
-
-The following environment variables are used to build the configuration file
-when ``SYNAPSE_CONFIG_PATH`` is not set.
-
-* ``SYNAPSE_SERVER_NAME`` (mandatory), the server public hostname.
-* ``SYNAPSE_REPORT_STATS``, (mandatory, ``yes`` or ``no``), enable anonymous
-  statistics reporting back to the Matrix project which helps us to get funding.
-* `SYNAPSE_NO_TLS`, (accepts `true`, `false`, `on`, `off`, `1`, `0`, `yes`, `no`]): disable
-  TLS in Synapse (use this if you run your own TLS-capable reverse proxy). Defaults
-  to `false` (ie, TLS is enabled by default).
-* ``SYNAPSE_ENABLE_REGISTRATION``, set this variable to enable registration on
-  the Synapse instance.
-* ``SYNAPSE_ALLOW_GUEST``, set this variable to allow guest joining this server.
-* ``SYNAPSE_EVENT_CACHE_SIZE``, the event cache size [default `10K`].
-* ``SYNAPSE_RECAPTCHA_PUBLIC_KEY``, set this variable to the recaptcha public
-  key in order to enable recaptcha upon registration.
-* ``SYNAPSE_RECAPTCHA_PRIVATE_KEY``, set this variable to the recaptcha private
-  key in order to enable recaptcha upon registration.
-* ``SYNAPSE_TURN_URIS``, set this variable to the coma-separated list of TURN
-  uris to enable TURN for this homeserver.
-* ``SYNAPSE_TURN_SECRET``, set this to the TURN shared secret if required.
-* ``SYNAPSE_MAX_UPLOAD_SIZE``, set this variable to change the max upload size
-  [default `10M`].
-* ``SYNAPSE_ACME``: set this to enable the ACME certificate renewal support.
-
-Shared secrets, that will be initialized to random values if not set:
-
-* ``SYNAPSE_REGISTRATION_SHARED_SECRET``, secret for registrering users if
-  registration is disable.
-* ``SYNAPSE_MACAROON_SECRET_KEY`` secret for signing access tokens
-  to the server.
-
-Database specific values (will use SQLite if not set):
-
-* `POSTGRES_DB` - The database name for the synapse postgres
-  database. [default: `synapse`]
-* `POSTGRES_HOST` - The host of the postgres database if you wish to use
-  postgresql instead of sqlite3. [default: `db` which is useful when using a
-  container on the same docker network in a compose file where the postgres
-  service is called `db`]
-* `POSTGRES_PASSWORD` - The password for the synapse postgres database. **If
-  this is set then postgres will be used instead of sqlite3.** [default: none]
-  **NOTE**: You are highly encouraged to use postgresql! Please use the compose
-  file to make it easier to deploy.
-* `POSTGRES_USER` - The user for the synapse postgres database. [default:
-  `synapse`]
-
-Mail server specific values (will not send emails if not set):
-
-* ``SYNAPSE_SMTP_HOST``, hostname to the mail server.
-* ``SYNAPSE_SMTP_PORT``, TCP port for accessing the mail server [default
-  ``25``].
-* ``SYNAPSE_SMTP_USER``, username for authenticating against the mail server if
-  any.
-* ``SYNAPSE_SMTP_PASSWORD``, password for authenticating against the mail
-  server if any.
-
-### Generating a config file
-
-It is possible to generate a basic configuration file for use with
-`SYNAPSE_CONFIG_PATH` using the `generate` commandline option. You will need to
-specify values for `SYNAPSE_CONFIG_PATH`, `SYNAPSE_SERVER_NAME` and
-`SYNAPSE_REPORT_STATS`, and mount a docker volume to store the data on. For
-example:
+## Generating a configuration file
+
+The first step is to genearte a valid config file. To do this, you can run the
+image with the `generate` commandline option.
+
+You will need to specify values for the `SYNAPSE_SERVER_NAME` and
+`SYNAPSE_REPORT_STATS` environment variable, and mount a docker volume to store
+the configuration on. For example:
 
 ```
 docker run -it --rm \
     --mount type=volume,src=synapse-data,dst=/data \
-    -e SYNAPSE_CONFIG_PATH=/data/homeserver.yaml \
     -e SYNAPSE_SERVER_NAME=my.matrix.host \
     -e SYNAPSE_REPORT_STATS=yes \
     matrixdotorg/synapse:latest generate
 ```
 
-This will generate a `homeserver.yaml` in (typically)
-`/var/lib/docker/volumes/synapse-data/_data`, which you can then customise and
-use with:
+For information on picking a suitable server name, see
+https://github.com/matrix-org/synapse/blob/master/INSTALL.md.
 
-```
-docker run -d --name synapse \
-    --mount type=volume,src=synapse-data,dst=/data \
-    -e SYNAPSE_CONFIG_PATH=/data/homeserver.yaml \
-    matrixdotorg/synapse:latest
-```
+The above command will generate a `homeserver.yaml` in (typically)
+`/var/lib/docker/volumes/synapse-data/_data`. You should check this file, and
+customise it to your needs.
 
-The following environment variables are supported in this mode:
+The following environment variables are supported in `generate` mode:
 
 * `SYNAPSE_SERVER_NAME` (mandatory): the server public hostname.
 * `SYNAPSE_REPORT_STATS` (mandatory, `yes` or `no`): whether to enable
   anonymous statistics reporting.
-* `SYNAPSE_CONFIG_PATH` (mandatory): path to the file to be generated.
 * `SYNAPSE_CONFIG_DIR`: where additional config files (such as the log config
   and event signing key) will be stored. Defaults to `/data`.
+* `SYNAPSE_CONFIG_PATH`: path to the file to be generated. Defaults to
+  `<SYNAPSE_CONFIG_DIR>/homeserver.yaml`.
 * `SYNAPSE_DATA_DIR`: where the generated config will put persistent data
   such as the datatase and media store. Defaults to `/data`.
 * `UID`, `GID`: the user id and group id to use for creating the data
   directories. Defaults to `991`, `991`.
+
+
+## Running synapse
+
+Once you have a valid configuration file, you can start synapse as follows:
+
+```
+docker run -d --name synapse \
+    --mount type=volume,src=synapse-data,dst=/data \
+    -p 8008:8008 \
+    matrixdotorg/synapse:latest
+```
+
+You can then check that it has started correctly with:
+
+```
+docker logs synapse
+```
+
+If all is well, you should now be able to connect to http://localhost:8008 and
+see a confirmation message.
+
+The following environment variables are supported in run mode:
+
+* `SYNAPSE_CONFIG_DIR`: where additional config files are stored. Defaults to
+  `/data`.
+* `SYNAPSE_CONFIG_PATH`: path to the config file. Defaults to
+  `<SYNAPSE_CONFIG_DIR>/homeserver.yaml`.
+* `UID`, `GID`: the user and group id to run Synapse as. Defaults to `991`, `991`.
+
+## TLS support
+
+The default configuration exposes a single HTTP port: http://localhost:8008. It
+is suitable for local testing, but for any practical use, you will either need
+to use a reverse proxy, or configure Synapse to expose an HTTPS port.
+
+For documentation on using a reverse proxy, see
+https://github.com/matrix-org/synapse/blob/master/docs/reverse_proxy.rst.
+
+For more information on enabling TLS support in synapse itself, see
+https://github.com/matrix-org/synapse/blob/master/INSTALL.md#tls-certificates. Of
+course, you will need to expose the TLS port from the container with a `-p`
+argument to `docker run`.
+
+## Legacy dynamic configuration file support
+
+For backwards-compatibility only, the docker image supports creating a dynamic
+configuration file based on environment variables. This is now deprecated, but
+is enabled when the `SYNAPSE_SERVER_NAME` variable is set (and `generate` is
+not given).

docker/start.py

@@ -131,12 +131,13 @@ def run_generate_config(environ, ownership):
 
     Never returns.
     """
-    for v in ("SYNAPSE_SERVER_NAME", "SYNAPSE_REPORT_STATS", "SYNAPSE_CONFIG_PATH"):
+    for v in ("SYNAPSE_SERVER_NAME", "SYNAPSE_REPORT_STATS"):
         if v not in environ:
             error("Environment variable '%s' is mandatory in `generate` mode." % (v,))
 
     server_name = environ["SYNAPSE_SERVER_NAME"]
     config_dir = environ.get("SYNAPSE_CONFIG_DIR", "/data")
+    config_path = environ.get("SYNAPSE_CONFIG_PATH", config_dir + "/homeserver.yaml")
     data_dir = environ.get("SYNAPSE_DATA_DIR", "/data")
 
     # create a suitable log config from our template
@@ -157,7 +158,7 @@ def run_generate_config(environ, ownership):
         "--report-stats",
         environ["SYNAPSE_REPORT_STATS"],
         "--config-path",
-        environ["SYNAPSE_CONFIG_PATH"],
+        config_path,
         "--config-directory",
         config_dir,
         "--data-directory",
@@ -176,11 +177,30 @@ def main(args, environ):
     if mode == "generate":
         return run_generate_config(environ, ownership)
 
-    # In normal mode, generate missing keys if any, then run synapse
-    if "SYNAPSE_CONFIG_PATH" in environ:
-        config_path = environ["SYNAPSE_CONFIG_PATH"]
-    else:
+    if "SYNAPSE_SERVER_NAME" in environ:
+        # backwards-compatibility generate-a-config-on-the-fly mode
+        if "SYNAPSE_CONFIG_PATH" in environ:
+            error(
+                "SYNAPSE_SERVER_NAME and SYNAPSE_CONFIG_PATH are mutually exclusive "
+                "except in `generate` mode."
+            )
+
         config_path = generate_config_from_template(environ, ownership)
+    else:
+        config_dir = environ.get("SYNAPSE_CONFIG_DIR", "/data")
+        config_path = environ.get(
+            "SYNAPSE_CONFIG_PATH", config_dir + "/homeserver.yaml"
+        )
+        if not os.path.exists(config_path):
+            error(
+                "Config file '%s' does not exist. You should either create a new "
+                "config file by running with the `generate` argument (and then edit "
+                "the resulting file before restarting) or specify the path to an "
+                "existing config file with the SYNAPSE_CONFIG_PATH variable."
+                % (config_path,)
+            )
+
+    log("Starting synapse with config file " + config_path)
 
     args = [
         "su-exec",