Add documentation for Apache Airflow Helm Chart 1.9.0 (#767)

docs-archive/helm-chart/1.9.0/.buildinfo

@@ -0,0 +1,4 @@
+# Sphinx build info version 1
+# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done.
+config: 90f0536553127c04cac694e974f721d4
+tags: 645f666f9bcd5a90fca523b33c5a78b7

docs-archive/helm-chart/1.9.0/_sources/adding-connections-and-variables.rst.txt

@@ -0,0 +1,85 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+ ..   http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+
+Adding Connections, Variables and Environment Variables
+=======================================================
+
+You can programmatically add Connections, Variables and arbitrary Environment Variables to your
+Airflow deployment using the Helm chart.
+
+
+Connections and Sensitive Environment Variables
+-----------------------------------------------
+Under the ``secret`` and ``extraSecret`` sections of the ``values.yaml`` you can pass connection strings and sensitive
+environment variables into Airflow using the Helm chart. To illustrate, lets create a yaml file called ``override.yaml``
+to override values under these sections of the ``values.yaml`` file.
+
+.. code-block:: yaml
+
+   # override.yaml
+
+   secret:
+     - envName: "AIRFLOW_CONN_GCP"
+       secretName: "my-airflow-connections"
+       secretKey: "AIRFLOW_CONN_GCP"
+     - envName: "my-env"
+       secretName: "my-secret-name"
+       secretKey: "my-secret-key"
+
+   extraSecrets:
+     my-airflow-connections:
+       data: |
+         AIRFLOW_CONN_GCP: 'base64_encoded_gcp_conn_string'
+     my-secret-name:
+       stringData: |
+         my-secret-key: my-secret
+
+
+Variables
+---------
+Airflow supports Variables which enable users to craft dynamic DAGs. You can set Variables in Airflow in three ways - UI,
+command line, and within your DAG file. See :doc:`apache-airflow:howto/variable` for more.
+
+With the Helm chart, you can also inject environment variables into Airflow. So in the example ``override.yaml`` file,
+we can override values of interest in the ``env`` section of the ``values.yaml`` file.
+
+.. code-block:: yaml
+
+   env:
+     - name: "AIRFLOW_VAR_KEY"
+       value: "value_1"
+     - name: "AIRFLOW_VAR_ANOTHER_KEY"
+       value: "value_2"
+
+
+You can also utilize ``extraEnv`` and ``extraEnvFrom`` if you need the name or value to be templated.
+
+.. code-block:: yaml
+
+   extraEnv: |
+     - name: AIRFLOW_VAR_HELM_RELEASE_NAME
+       value: '{{ .Release.Name }}'
+
+   extraEnvFrom: |
+     - configMapRef:
+         name: '{{ .Release.Name }}-airflow-variables'
+
+   extraConfigMaps:
+     '{{ .Release.Name }}-airflow-variables':
+       data: |
+         AIRFLOW_VAR_HELLO_MESSAGE: "Hi!"

docs-archive/helm-chart/1.9.0/_sources/airflow-configuration.rst.txt

@@ -0,0 +1,50 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+ ..   http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+Configuring Airflow
+-------------------
+
+The chart allows for setting arbitrary Airflow configuration in values under the ``config`` key.
+Some of the defaults in the chart differ from those of core Airflow and can be found in
+`values.yaml <https://github.com/apache/airflow/blob/main/chart/values.yaml>`__.
+
+As an example of setting arbitrary configuration, the following yaml demonstrates how one would
+allow webserver users to view the config from within the UI:
+
+.. code-block:: yaml
+
+   config:
+     webserver:
+       expose_config: 'True'  # by default this is 'False'
+
+Generally speaking, it is useful to familiarize oneself with the Airflow
+configuration prior to installing and deploying the service.
+
+.. note::
+
+  The recommended way to load example DAGs using the official Docker image and chart is to configure the ``AIRFLOW__CORE__LOAD_EXAMPLES`` environment variable
+  in ``extraEnv`` (see :doc:`Parameters reference <parameters-ref>`). The official Docker image has ``AIRFLOW__CORE__LOAD_EXAMPLES=False``
+  set within the image, so you need to override it with an environment variable when deploying the chart in order for the examples to be present.
+
+.. note::
+
+  The  ``AIRFLOW__DATABASE__LOAD_DEFAULT_CONNECTIONS`` variable is not used by the Chart. Airflow Helm Chart is
+  intended to be used as production deployment and loading default connections is not supposed to be handled
+  during Chart installation. The Chart is intended to install and configure the Apache Airflow software
+  and create database structure, but not to fill-in the data which should be managed by the users.
+  The default connections are only meaningful when you want to have a ``quick start`` with Airflow or
+  do some development and adding the data via Helm Chart installation is not a good idea.

docs-archive/helm-chart/1.9.0/_sources/customizing-workers.rst.txt

@@ -0,0 +1,68 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+ ..   http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+Customizing Workers
+===================
+
+Both ``CeleryExecutor`` and ``KubernetesExecutor`` workers can be highly customized with the :ref:`workers parameters <parameters:workers>`.
+For example, to set resources on workers:
+
+.. code-block:: yaml
+
+  workers:
+    resources:
+      requests:
+        cpu: 1
+      limits:
+        cpu: 1
+
+See :ref:`workers parameters <parameters:workers>` for a complete list.
+
+One notable exception for ``KubernetesExecutor`` is that the default anti-affinity applied to ``CeleryExecutor`` workers to spread them across nodes
+is not applied to ``KubernetesExecutor`` workers, as there is no reason to spread out per-task workers.
+
+Custom ``pod_template_file``
+----------------------------
+
+With ``KubernetesExecutor`` or ``CeleryKubernetesExecutor`` you can also provide a complete ``pod_template_file`` to configure Kubernetes workers.
+This may be useful if you need different configuration between worker types for ``CeleryKubernetesExecutor``
+or if you need to customize something not possible with :ref:`workers parameters <parameters:workers>` alone.
+
+As an example, let's say you want to set ``priorityClassName`` on your workers:
+
+.. note::
+
+  The following example is NOT functional, but meant to be illustrative of how you can provide a custom ``pod_template_file``.
+  You're better off starting with the `default pod_template_file`_ instead.
+
+.. _default pod_template_file: https://github.com/apache/airflow/blob/main/chart/files/pod-template-file.kubernetes-helm-yaml
+
+.. code-block:: yaml
+
+  podTemplate: |
+    apiVersion: v1
+    kind: Pod
+    metadata:
+      name: placeholder-name
+      labels:
+        tier: airflow
+        component: worker
+        release: {{ .Release.Name }}
+    spec:
+      priorityClassName: high-priority
+      containers:
+        - name: base

docs-archive/helm-chart/1.9.0/_sources/index.rst.txt

@@ -0,0 +1,157 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+ ..   http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. image:: /img/helm-logo.svg
+    :width: 100
+    :class: no-scaled-link
+
+Helm Chart for Apache Airflow
+=============================
+
+.. toctree::
+    :hidden:
+
+    Home <self>
+    quick-start
+    airflow-configuration
+    adding-connections-and-variables
+    manage-dags-files
+    manage-logs
+    setting-resources-for-containers
+    keda
+    using-additional-containers
+    customizing-workers
+    Installing from sources<installing-helm-chart-from-sources>
+
+.. toctree::
+    :hidden:
+    :caption: Guides
+
+    production-guide
+
+.. toctree::
+    :hidden:
+    :caption: References
+
+    Parameters <parameters-ref>
+    release_notes
+
+
+This chart will bootstrap an `Airflow <https://airflow.apache.org>`__
+deployment on a `Kubernetes <http://kubernetes.io>`__ cluster using the
+`Helm <https://helm.sh>`__ package manager.
+
+Requirements
+------------
+
+-  Kubernetes 1.23+ cluster
+-  Helm 3.0+
+-  PV provisioner support in the underlying infrastructure (optionally)
+
+Features
+--------
+
+* Supported executors: ``LocalExecutor``, ``CeleryExecutor``, ``KubernetesExecutor``, ``LocalKubernetesExecutor``, ``CeleryKubernetesExecutor``
+* Supported Airflow version: ``1.10+``, ``2.0+``
+* Supported database backend: ``PostgresSQL``, ``MySQL``
+* Autoscaling for ``CeleryExecutor`` provided by KEDA
+* PostgreSQL and PgBouncer with a battle-tested configuration
+* Monitoring:
+
+   * StatsD/Prometheus metrics for Airflow
+   * Prometheus metrics for PgBouncer
+   * Flower
+* Automatic database migration after a new deployment
+* Administrator account creation during deployment
+* Kerberos secure configuration
+* One-command deployment for any type of executor. You don't need to provide other services e.g. Redis/Database to test the Airflow.
+
+Installing the Chart
+--------------------
+
+To install this chart using Helm 3, run the following commands:
+
+.. code-block:: bash
+
+    helm repo add apache-airflow https://airflow.apache.org
+    helm upgrade --install airflow apache-airflow/airflow --namespace airflow --create-namespace
+
+The command deploys Airflow on the Kubernetes cluster in the default configuration. The :doc:`parameters-ref`
+section lists the parameters that can be configured during installation.
+
+
+.. tip:: List all releases using ``helm list``.
+
+Upgrading the Chart
+-------------------
+
+To upgrade the chart with the release name ``airflow``:
+
+.. code-block:: bash
+
+    helm upgrade airflow apache-airflow/airflow --namespace airflow
+
+.. note::
+  To upgrade to a new version of the chart, run ``helm repo update`` first.
+
+Uninstalling the Chart
+----------------------
+
+To uninstall/delete the ``airflow`` deployment:
+
+.. code-block:: bash
+
+    helm delete airflow --namespace airflow
+
+The command removes all the Kubernetes components associated with the chart and deletes the release.
+
+.. note::
+  Some kubernetes resources created by the chart `helm hooks <https://helm.sh/docs/topics/charts_hooks/#hook-resources-are-not-managed-with-corresponding-releases>`__ might be left in the namespace after executing ``helm uninstall``, for example, ``brokerUrlSecret`` or ``fernetKeySecret``.
+
+Installing the Chart with Argo CD, Flux, Rancher or Terraform
+-------------------------------------------------------------
+
+When installing the chart using Argo CD, Flux, Rancher or Terraform, you MUST set the four following values, or your application
+will not start as the migrations will not be run:
+
+.. code-block:: yaml
+
+    createUserJob:
+      useHelmHooks: false
+      applyCustomEnv: false
+    migrateDatabaseJob:
+      useHelmHooks: false
+      applyCustomEnv: false
+
+This is so these CI/CD services can perform updates without issues and preserve the immutability of Kubernetes Job manifests.
+
+This also applies if you install the chart using ``--wait`` in your ``helm install`` command.
+
+.. note::
+    While deploying this Helm chart with Argo, you might encounter issues with database migrations not running automatically on upgrade.
+
+To run database migrations with Argo CD automatically, you will need to add:
+
+.. code-block:: yaml
+
+    migrateDatabaseJob:
+        jobAnnotations:
+            "argocd.argoproj.io/hook": Sync
+
+This will run database migrations every time there is a ``Sync`` event in Argo CD. While it is not ideal to run the migrations on every sync, it is a trade-off that allows them to be run automatically.
+
+If you use the Celery(Kubernetes)Executor with the built-in Redis, it is recommended that you set up a static Redis password either by supplying ``redis.passwordSecretName`` and ``redis.data.brokerUrlSecretName`` or ``redis.password``.