Make Docker images non-root, by default, and OpenShift compliant (apa…

…che#13376)

Master Issue: apache#11269

### Motivation

In order to increase the overall security of our Pulsar docker images, they should default to run as the non-root user. While updating these permissions, I make sure to comply with the OpenShift spec so the docker image can run on that platform out of the box.

Once we finalize these changes, we will need to update the Apache Pulsar Helm chart to make sure that deployments take advantage of this feature. We'll use the `fsGroup` to make sure that k8s sets the appropriate file system permissions for the zookeeper, bookkeeper, and function pods.

### Modifications

* Default to run as UID 10000. As noted in the `Dockerfile`, this UID is arbitrary. No logic should rely on this id.
* Update filesystem permissions so that the group user has sufficient write permission. The group user is 0 (root).
* Remove unnecessary write access.
    * The `/pulsar/{conf,data,logs}` directories and their members must be writable by the root group. I don't know of any other directories that need to be written to. Note that the `bin/pulsar-admin` too creates a log file in the `/pulsar/logs` directory. Please let me know if there are any additional
    * Note also that the executable file permissions are already set in our git repo. Those permissions are inherited by the docker image when we run the `COPY` directive in the `Dockerfile`.
* There are no changes to the function worker in the k8s runtime. We do not need them because we already merged apache@04b5da0.
* Add note to `conf/bkenv.sh`, as it is a `.sh` script that is not executable (and doesn't need to be).
* Update test docker image and `supervisord` configuration.

Note: it's unclear to me how the OpenShift spec handles restarts. I know that the UID is arbitrary. It's possible that the umask needs to be switched from `022` to `002`. Setting the umask in the docker image does not persist for consumers of the image, so this would need to be set in a helm chart.

### Verifying this change

You can access a test image built with these changes here: `michaelmarshall/pulsar:2.10.0-SNAPSHOT`. I have already run some manual tests like `bin/pulsar standalone` in the container. I still need to deploy an actual cluster to verify that all of the unique components work correctly. Because we already merged apache@04b5da0, the upgrade scenarios are already simplified. If this change is in 2.10.0, that means 2.8 and 2.9 will be compatible for certain function worker upgrade scenarios.

I wrote test criteria in apache#11269. I'll need to follow up on that criteria using my newly build image. I should be able to look closer at this tomorrow.

We'll also need tests to pass, as I modified some tests with this PR.

### References

The following links were useful in understanding how to make these changes:

* https://engineering.bitnami.com/articles/running-non-root-containers-on-openshift.html
* https://cloud.redhat.com/blog/a-guide-to-openshift-and-uids

### Does this pull request potentially affect one of the following parts:

This PR updates our Docker images in a breaking way. It could result in bookkeepers, zookeepers, or functions with insufficient permissions. We will mitigate these permissions by updating the helm chart. These changes are easily overridden by extending the docker image. In k8s, you can use the pod's `securityContext` to override the user or group.

conf/bkenv.sh

@@ -18,6 +18,8 @@
 # under the License.
 #
 
+# NOTE: this script is intentionally not executable. It is only meant to be sourced for environment variables.
+
 # Set JAVA_HOME here to override the environment setting
 # JAVA_HOME=
 

docker/README.md

@@ -0,0 +1,109 @@
+<!--
+
+    Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+    Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+-->
+
+# Apache Pulsar Docker Images
+
+The Apache Pulsar community produces 2 docker images with each official release.
+
+* `apachepulsar/pulsar` - contains the necessary components for a working Pulsar cluster
+* `apachepulsar/pulsar-all` - extends the `apachepulsar/pulsar` image by adding many Pulsar connectors and offloaders
+
+Since the 2.10.0 release, these docker images run as an unnamed, non-root user that is also part of the root group, by
+default. This was done to increase container security. The user is part of the root group to ensure that the container
+image can easily run on OpenShift and to ensure that the Pulsar process can write to configuration files.
+
+## Development
+
+You can build and test these docker images on your own machine by running the `./build.sh` script in this directory.
+Note that you first must build the project in order to have the right dependencies in your local environment.
+
+## Building Derivative Custom Images
+
+If you find the `apachepulsar/pulsar-all` docker image too large, but you want to use a connector or an offloader,
+you can easily build an image with a curated list of connectors or offloaders based on the official Apache Pulsar
+images. You can use the following sample docker image as a guide:
+
+```Dockerfile
+ARG VERSION
+
+# Load the pulsar-all image as a builder image
+FROM apachepulsar/pulsar-all:${VERSION} as pulsar-all
+
+FROM apachepulsar/pulsar:${VERSION}
+
+# Add the cassandra connector
+COPY --from=pulsar-all /pulsar/connectors/pulsar-io-cassandra-*.nar /pulsar/connectors
+
+# Add the jcloud offloader
+COPY --from=pulsar-all /pulsar/connectors/tiered-storage-jcloud-*.nar /pulsar/offloaders
+```
+
+NOTE: the above example uses a wildcard in the `COPY` commands because argument expansion does not work for `COPY`.
+
+Assuming that you have the above `Dockerfile` in your local directory and are running docker on your local host, you can
+run the following command to build a custom image with the cassandra connector and the jcloud offloader.
+
+```shell
+docker build --build-arg VERSION=2.9.1 -t pulsar-custom:2.9.1 .
+```
+
+For reference, here are the sizes of the official 2.9.1 docker images and the custom image built from the above
+`Dockerfile`:
+
+| REPOSITORY              | TAG   | SIZE   |
+| :---------------------- | :---- | :----- |
+| apachepulsar/pulsar     | 2.9.1 | 1.59GB |
+| apachepulsar/pulsar-all | 2.9.1 | 3.44GB |
+| pulsar-custom           | 2.9.1 | 1.6GB  |
+
+
+## Troubleshooting non-root containers
+
+Troubleshooting is harder because the docker image runs as a non-root user. For example, a non-root user won't be able
+to download arbitrary utilities. There are several ways to troubleshoot.
+
+One option is to build a custom docker image that includes your preferred debugging tools. Here is an example of adding
+some tools to an existing docker image.
+
+```Dockerfile
+FROM apachepulsar/pulsar:2.10.0
+
+# Switch to root user to download tools
+USER 0
+
+# Install your preferred utilities
+RUN apt-get update \
+     && apt-get install -y vim net-tools unzip \
+     && apt-get clean \
+     && rm -rf /var/lib/apt/lists/*
+
+# Assuming you still want to run as a non root user by default
+USER 10000
+```
+
+The remaining debug options depend on your environment. For example, if you have access to the host running your
+container, you might be able to use the `docker exec` command to shell into the container. By using the `--user`
+argument, you can run as the root user.
+
+If you're running your container on kubernetes, you can override the container's default user by setting the pod's
+`securityContext`.
+
+Bitnami provides a helpful guide here: https://engineering.bitnami.com/articles/running-non-root-containers-on-openshift.html.

docker/pulsar/Dockerfile

@@ -33,7 +33,13 @@ COPY scripts/pulsar-zookeeper-ruok.sh /pulsar/bin
 COPY scripts/watch-znode.py /pulsar/bin
 COPY scripts/install-pulsar-client.sh /pulsar/bin
 
-RUN mkdir /pulsar/data
+# In order to support running this docker image as a container on OpenShift
+# the final image needs to give the root group sufficient permission.
+# The file permissions are preserved when copying files from this builder image to the target image.
+RUN chmod -R g+w /pulsar/conf
+# Presto writes logs to this directory (at least during tests), so we need to give the process permission
+# to create those log directories. This should be removed when presto is removed.
+RUN chmod g+w /pulsar/lib/presto
 
 ### Create 2nd stage from Ubuntu image
 ### and add OpenJDK and Python dependencies (for Pulsar functions)
@@ -57,6 +63,14 @@ RUN sed -i "s|http://archive\.ubuntu\.com/ubuntu/|${UBUNTU_MIRROR:-mirror://mirr
 
 RUN update-alternatives --install /usr/bin/python python /usr/bin/python3 10
 
+# Pulsar currently writes to the below directories, assuming the default configuration.
+# Note that number 4 is the reason that pulsar components need write access to the /pulsar directory.
+# 1. /pulsar/data - both bookkeepers and zookeepers use this directory
+# 2. /pulsar/logs - function workers write to this directory and pulsar-admin initializes this directory
+# 3. /pulsar/download - functions write to this directory
+# 4. /pulsar - hadoop writes to this directory
+RUN mkdir /pulsar && chmod g+w /pulsar
+
 ENV JAVA_HOME /usr/lib/jvm/java-11-openjdk-amd64
 RUN echo networkaddress.cache.ttl=1 >> /usr/lib/jvm/java-11-openjdk-amd64/conf/security/java.security
 ADD target/python-client/ /pulsar/pulsar-client
@@ -67,4 +81,8 @@ ENV PULSAR_ROOT_LOGGER=INFO,CONSOLE
 COPY --from=pulsar /pulsar /pulsar
 WORKDIR /pulsar
 
+# This script is intentionally run as the root user to make the dependencies available for all UIDs.
 RUN /pulsar/bin/install-pulsar-client.sh
+
+# The UID must be non-zero. Otherwise, it is arbitrary. No logic should rely on its specific value.
+USER 10000

site2/docs/getting-started-docker.md

@@ -20,6 +20,7 @@ A few things to note about this command:
  * The data, metadata, and configuration are persisted on Docker volumes in order to not start "fresh" every 
 time the container is restarted. For details on the volumes you can use `docker volume inspect <sourcename>`
  * For Docker on Windows make sure to configure it to use Linux containers
+ * The docker container will run as UID 10000 and GID 0, by default. You'll need to ensure the mounted volumes give write permission to either UID 10000 or GID 0. Note that UID 10000 is arbitrary, so it is recommended to make these mounts writable for the root group (GID 0).
 
 If you start Pulsar successfully, you will see `INFO`-level log messages like this:
 

tests/docker-images/latest-version-image/Dockerfile

@@ -21,6 +21,7 @@
 
 FROM apachepulsar/pulsar:latest as pulsar-function-go
 
+# Use root for builder
 USER root
 
 RUN rm -rf /var/lib/apt/lists/* && apt-get update
@@ -55,6 +56,16 @@ FROM apachepulsar/pulsar-all:latest as pulsar-all
 ########################################
 FROM apachepulsar/pulsar:latest
 
+# Switch to run as the root user to simplify building container and then running
+# supervisord. Each of the pulsar components are spawned by supervisord and their
+# process configuration files specify that the process will be run with UID 10000.
+# However, any processes exec'ing into the containers will run as root, by default.
+USER root
+
+# We need to define the user in order for supervisord to work correctly
+# We don't need a user defined in the public docker image, though.
+RUN adduser -u 10000 --gid 0 --disabled-login --disabled-password --gecos '' pulsar
+
 RUN rm -rf /var/lib/apt/lists/* && apt update
 
 RUN apt-get clean && apt-get update && apt-get install -y supervisor vim procps curl

tests/docker-images/latest-version-image/conf/bookie.conf

@@ -24,3 +24,4 @@ stdout_logfile=/var/log/pulsar/bookie.log
 directory=/pulsar
 environment=PULSAR_MEM="-Xmx128M -XX:MaxDirectMemorySize=512M",PULSAR_GC="-XX:+UseG1GC"
 command=/pulsar/bin/pulsar bookie
+user=pulsar

tests/docker-images/latest-version-image/conf/broker.conf

@@ -24,4 +24,4 @@ stdout_logfile=/var/log/pulsar/broker.log
 directory=/pulsar
 environment=PULSAR_MEM="-Xmx128M",PULSAR_GC="-XX:+UseG1GC"
 command=/pulsar/bin/pulsar broker
-
+user=pulsar

tests/docker-images/latest-version-image/conf/functions_worker.conf

@@ -24,4 +24,4 @@ stdout_logfile=/var/log/pulsar/functions_worker.log
 directory=/pulsar
 environment=PULSAR_MEM="-Xmx128M",PULSAR_GC="-XX:+UseG1GC"
 command=/pulsar/bin/pulsar functions-worker
-
+user=pulsar

tests/docker-images/latest-version-image/conf/global-zk.conf

@@ -24,4 +24,4 @@ stdout_logfile=/var/log/pulsar/global-zk.log
 directory=/pulsar
 environment=PULSAR_MEM="-Xmx128M",PULSAR_GC="-XX:+UseG1GC"
 command=/pulsar/bin/pulsar configuration-store
-
+user=pulsar

tests/docker-images/latest-version-image/conf/local-zk.conf

@@ -24,4 +24,4 @@ stdout_logfile=/var/log/pulsar/local-zk.log
 directory=/pulsar
 environment=PULSAR_MEM="-Xmx128M",PULSAR_GC="-XX:+UseG1GC"
 command=/pulsar/bin/pulsar zookeeper
-
+user=pulsar

tests/docker-images/latest-version-image/conf/presto_worker.conf

@@ -23,4 +23,5 @@ redirect_stderr=true
 stdout_logfile=/var/log/pulsar/presto_worker.log
 directory=/pulsar
 environment=PULSAR_MEM="-Xmx128M",PULSAR_GC="-XX:+UseG1GC"
-command=/pulsar/bin/pulsar sql-worker start
+command=/pulsar/bin/pulsar sql-worker start
+user=pulsar

tests/docker-images/latest-version-image/conf/proxy.conf

@@ -24,4 +24,4 @@ stdout_logfile=/var/log/pulsar/proxy.log
 directory=/pulsar
 environment=PULSAR_MEM="-Xmx128M",PULSAR_GC="-XX:+UseG1GC"
 command=/pulsar/bin/pulsar proxy
-
+user=pulsar