Run EA container + managed by fleet + standalone + autodiscovery

docs/en/ingest-management/elastic-agent/configuration/autodiscovery/elastic-agent-kubernetes-autodiscovery.asciidoc

@@ -0,0 +1,10 @@
+[[elastic-agent-kubernetes-autodiscovery]]
+= Kubernetes autodiscovery with {agent}
+
+To use autodiscovery, you will need to modify the manifest file of the {agent}. Refer to <<running-on-kubernetes-standalone>> to learn how to retrieve and configure it.
+
+There are two different ways to use autodiscovery:
+
+* <<conditions-based-autodiscover>>
+
+* <<hints-annotations-autodiscovery>>

docs/en/ingest-management/elastic-agent/configuration/autodiscovery/kubernetes-conditions-autodiscover.asciidoc

@@ -0,0 +1,314 @@
+[[conditions-based-autodiscover]]
+= Conditions based autodiscover
+
+You can define autodiscover conditions in each input to allow {agent} to automatically identify Pods and start monitoring them using predefined integrations. You can find more about these inputs and their configurations <<elastic-agent-input-configuration,here>>.
+
+== Example: Target pods by host and label
+
+To automatically identify a Redis Pod and monitor it with the Redis integration, uncomment the following input configuration inside  the {agent} Standalone manifest [here](https://github.com/elastic/elastic-agent/blob/main/deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml):
+
+
+[source,yaml]
+------------------------------------------------
+- name: redis
+  type: redis/metrics
+  use_output: default
+  meta:
+    package:
+      name: redis
+      version: 0.3.6
+  data_stream:
+    namespace: default
+  streams:
+    - data_stream:
+        dataset: redis.info
+        type: metrics
+      metricsets:
+        - info
+      hosts:
+        - '${kubernetes.pod.ip}:6379'
+      idle_timeout: 20s
+      maxconn: 10
+      network: tcp
+      period: 10s
+      condition: ${kubernetes.labels.app} == 'redis'
+------------------------------------------------
+
+The condition `${kubernetes.labels.app} == 'redis'` will make the {agent} look for a Pod with the `redis` label app within the scope defined in its manifest. You can find the exported fields of the Kubernetes integration {metricbeat-ref}/exported-fields-kubernetes-processor.html[here].
+
+The `redis` input defined in the {agent} manifest only has `info` as metricset. You can find more metricsets in the {metricbeat-ref}/metricbeat-module-redis.html[Redis module page], as well as the configuration options.
+
+To deploy Redis, you can apply the following example manifest:
+
+[source,yaml]
+------------------------------------------------
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: test-redis-config
+data:
+  redis-config: |
+    maxmemory 2mb
+    maxmemory-policy allkeys-lru
+---
+apiVersion: v1
+kind: Pod
+metadata:
+  name: redis
+  labels:
+    k8s-app: redis
+    app: redis
+spec:
+  containers:
+  - name: redis
+    image: redis:5.0.4
+    command:
+      - redis-server
+      - "/redis-master/redis.conf"
+    env:
+    - name: MASTER
+      value: "true"
+    ports:
+    - containerPort: 6379
+    resources:
+      limits:
+        cpu: "0.1"
+    volumeMounts:
+    - mountPath: /redis-master-data
+      name: data
+    - mountPath: /redis-master
+      name: config
+  volumes:
+    - name: data
+      emptyDir: {}
+    - name: config
+      configMap:
+        name: test-redis-config
+        items:
+        - key: redis-config
+          path: redis.conf
+------------------------------------------------
+
+You should now be able to see Redis data flowing in on index `metrics-redis.info-default`. Make sure that the port on your Redis manifest file and the one used in the Redis input match. Note that at this point all the assets (dashboards, ingest pipelines, and so on) related to the Redis integration are not installed. You need to explicitly <<install-uninstall-integration-assets,install them through Kibana>>.
+
+== Example: Target pods by label
+
+WARNING: In some "As a Service" Kubernetes implementations, like GKE, the control plane nodes or even the pods running on them won’t be visible. In these cases, it won’t be possible to use scheduler metricsets, necessary for this example. Refer https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-kubernetes.html#_scheduler_and_controllermanager[here] to find more information.
+
+To set the target host dynamically only for a targeted Pod based on its labels, use a variable in the
+{agent} policy to return path information from the provider:
+
+[source,yaml]
+----
+- data_stream:
+    dataset: kubernetes.scheduler
+    type: metrics
+  metricsets:
+    - scheduler
+  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
+  hosts:
+    - 'https://${kubernetes.pod.ip}:10259'
+  period: 10s
+  ssl.verification_mode: none
+  condition: ${kubernetes.labels.component} == 'kube-scheduler'
+----
+
+If you now deploy another Redis pod with a different port, it should be detected. You can check this by, for example, studying the field `service.address` under `metrics-redis.info-default`. It should now display 2 different services.
+
+To obtain the policy generated by this configuration get inside the {agent} container:
+
+["source", "sh", subs="attributes"]
+------------------------------------------------
+kubectl exec -n kube-system --stdin --tty elastic-agent-standalone-id -- /bin/bash
+------------------------------------------------
+
+Do not forget to change the `elastic-agent-standalone-id` to your {agent} pod's name. Moreover, make sure that your pod is inside `kube-system`. If not, change `-n kube-system` to the right namespace.
+
+Inside the container <<elastic-agent-cmd-options, inspect the output>> of the configuration file you used for the {agent}:
+
+["source", "sh", subs="attributes"]
+------------------------------------------------
+elastic-agent inspect output -o default -c /etc/elastic-agent/agent.yml
+------------------------------------------------
+You should now be able to see the generated policy. If you look for the `scheduler`, it will look similar to this:
+
+[source,yaml]
+----
+- bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
+  hosts:
+    - https://172.19.0.2:10259
+  index: metrics-kubernetes.scheduler-default
+  meta:
+    package:
+      name: kubernetes
+      version: 1.9.0
+  metricsets:
+    - scheduler
+  module: kubernetes
+  name: kubernetes-node-metrics
+  period: 10s
+  processors:
+    - add_fields:
+        fields:
+          labels:
+            component: kube-scheduler
+            tier: control-plane
+          namespace: kube-system
+          namespace_labels:
+            kubernetes_io/metadata_name: kube-system
+          namespace_uid: 03d6fd2f-7279-4db4-9a98-51e50bbe5c62
+          node:
+            hostname: kind-control-plane
+            labels:
+              beta_kubernetes_io/arch: amd64
+              beta_kubernetes_io/os: linux
+              kubernetes_io/arch: amd64
+              kubernetes_io/hostname: kind-control-plane
+              kubernetes_io/os: linux
+              node-role_kubernetes_io/control-plane: ""
+              node_kubernetes_io/exclude-from-external-load-balancers: ""
+            name: kind-control-plane
+            uid: b8d65d6b-61ed-49ef-9770-3b4f40a15a8a
+          pod:
+            ip: 172.19.0.2
+            name: kube-scheduler-kind-control-plane
+            uid: f028ad77-c82a-4f29-ba7e-2504d9b0beef
+        target: kubernetes
+    - add_fields:
+        fields:
+          cluster:
+            name: kind
+            url: kind-control-plane:6443
+        target: orchestrator
+    - add_fields:
+        fields:
+          dataset: kubernetes.scheduler
+          namespace: default
+          type: metrics
+        target: data_stream
+    - add_fields:
+        fields:
+          dataset: kubernetes.scheduler
+          target: event
+    - add_fields:
+        fields:
+          id: ""
+          snapshot: false
+          version: 8.3.0
+        target: elastic_agent
+    - add_fields:
+        fields:
+          id: ""
+        target: agent
+  ssl.verification_mode: none
+----
+
+
+== Example: Dynamic logs path
+
+To set the log path of Pods dynamically in the configuration, use a variable in the
+{agent} policy to return path information from the provider:
+
+[source,yaml]
+----
+- name: container-log
+  id: container-log-${kubernetes.pod.name}-${kubernetes.container.id}
+  type: filestream
+  use_output: default
+  meta:
+    package:
+      name: kubernetes
+      version: 1.9.0
+  data_stream:
+    namespace: default
+  streams:
+    - data_stream:
+      dataset: kubernetes.container_logs
+      type: logs
+      prospector.scanner.symlinks: true
+      parsers:
+        - container: ~
+      paths:
+        - /var/log/containers/*${kubernetes.container.id}.log
+----
+
+The policy generated by this configuration will look similar to this for every pod inside the scope defined in the manifest:
+
+[source,yaml]
+----
+- id: container-log-etcd-kind-control-plane-af311067a62fa5e4d6e5cb4d31e64c1c35d82fe399eb9429cd948d5495496819
+  index: logs-kubernetes.container_logs-default
+  meta:
+    package:
+      name: kubernetes
+      version: 1.9.0
+  name: container-log
+  parsers:
+    - container: null
+  paths:
+    - /var/log/containers/*af311067a62fa5e4d6e5cb4d31e64c1c35d82fe399eb9429cd948d5495496819.log
+  processors:
+    - add_fields:
+        fields:
+          id: af311067a62fa5e4d6e5cb4d31e64c1c35d82fe399eb9429cd948d5495496819
+          image:
+            name: registry.k8s.io/etcd:3.5.4-0
+          runtime: containerd
+        target: container
+    - add_fields:
+        fields:
+          container:
+            name: etcd
+        labels:
+          component: etcd
+          tier: control-plane
+        namespace: kube-system
+        namespace_labels:
+          kubernetes_io/metadata_name: kube-system
+        namespace_uid: 03d6fd2f-7279-4db4-9a98-51e50bbe5c62
+        node:
+          hostname: kind-control-plane
+          labels:
+            beta_kubernetes_io/arch: amd64
+            beta_kubernetes_io/os: linux
+            kubernetes_io/arch: amd64
+            kubernetes_io/hostname: kind-control-plane
+            kubernetes_io/os: linux
+            node-role_kubernetes_io/control-plane: ""
+            node_kubernetes_io/exclude-from-external-load-balancers: ""
+          name: kind-control-plane
+          uid: b8d65d6b-61ed-49ef-9770-3b4f40a15a8a
+        pod:
+          ip: 172.19.0.2
+          name: etcd-kind-control-plane
+          uid: 08970fcf-bb93-487e-b856-02399d81fb29
+      target: kubernetes
+    - add_fields:
+        fields:
+          cluster:
+            name: kind
+            url: kind-control-plane:6443
+        target: orchestrator
+    - add_fields:
+        fields:
+          dataset: kubernetes.container_logs
+          namespace: default
+          type: logs
+        target: data_stream
+    - add_fields:
+        fields:
+          dataset: kubernetes.container_logs
+        target: event
+    - add_fields:
+        fields:
+          id: ""
+          snapshot: false
+          version: 8.3.0
+        target: elastic_agent
+    - add_fields:
+        fields:
+          id: ""
+        target: agent
+  prospector.scanner.symlinks: true
+  type: filestream
+----