Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Elasticsearch] Add dimensions fields for TSDB migration #6623

Merged
merged 8 commits into from
Jun 27, 2023
Merged

[Elasticsearch] Add dimensions fields for TSDB migration #6623

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
cb3db08
Add dimensions fields.
constanca-m Jun 20, 2023
e211c80
Update changelog
constanca-m Jun 20, 2023
5a84c3b
Remove extra field.
constanca-m Jun 21, 2023
76e866a
Remove diemnsions from shard and pending tasks.
constanca-m Jun 21, 2023
4209cf2
Remove diemnsions from shard and pending tasks.
constanca-m Jun 21, 2023
71d154b
Fix changelog.
constanca-m Jun 21, 2023
4a75ad0
Update README
constanca-m Jun 21, 2023
6b0a747
Remove cluster stats package field dimension.
constanca-m Jun 23, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions packages/elasticsearch/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "1.8.0"
changes:
- description: Set fields as dimensions for TSDB migration.
type: enhancement
link: https://github.com/elastic/integrations/pull/6623
- version: "1.7.4"
changes:
- description: Fix elasticsearch.server.tags mapping
Expand Down
6 changes: 6 additions & 0 deletions packages/elasticsearch/data_stream/ccr/fields/ecs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,12 +10,18 @@
external: ecs
- name: host.name
external: ecs
dimension: true
- name: service.address
type: keyword
dimension: true
description: Service address
- name: service.type
external: ecs
- name: service.name
external: ecs
- name: error.message
external: ecs
- name: agent.id
external: ecs
dimension: true

5 changes: 5 additions & 0 deletions packages/elasticsearch/data_stream/cluster_stats/fields/ecs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,12 +10,17 @@
external: ecs
- name: host.name
external: ecs
dimension: true
- name: service.address
type: keyword
dimension: true
description: Service address
- name: service.type
external: ecs
- name: service.name
external: ecs
- name: error.message
external: ecs
- name: agent.id
external: ecs
dimension: true
1 change: 1 addition & 0 deletions packages/elasticsearch/data_stream/cluster_stats/fields/package-fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,7 @@
type: keyword
- name: cluster.name
type: keyword
dimension: true
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wondering if cluster.id will not be a better candidate for the dimension?

Copy link
Contributor Author

@constanca-m constanca-m Jun 23, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did not migrate this one, it is still pending (description).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did not migrate this one, it is still pending (description).

but you are planning to add dimension fields for this data_streams that are blocked by mentioned in description issues, in this PR? or you plan to move those data_streams to another PR?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They will be moved to another PR. I will remove this dimension, but I will leave the ecs ones, just to not cause confusion then.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed the one for cluster stats (as I believe it is also not necessary). I am leaving the enrich dimensions though, even if it is not migrated - I will validate it again when the issue is resolved.

description: |
Elasticsearch cluster name.
- name: cluster.id
Expand Down
5 changes: 5 additions & 0 deletions packages/elasticsearch/data_stream/enrich/fields/ecs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,12 +10,17 @@
external: ecs
- name: host.name
external: ecs
dimension: true
- name: service.address
type: keyword
dimension: true
description: Service address
- name: service.type
external: ecs
- name: service.name
external: ecs
- name: error.message
external: ecs
- name: agent.id
external: ecs
dimension: true
1 change: 1 addition & 0 deletions packages/elasticsearch/data_stream/enrich/fields/fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
fields:
- name: name
type: keyword
dimension: true
- name: task
type: group
fields:
Expand Down
1 change: 1 addition & 0 deletions packages/elasticsearch/data_stream/enrich/fields/package-fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@
fields:
- name: id
type: keyword
dimension: true
description: |
Node ID
- name: name
Expand Down
5 changes: 5 additions & 0 deletions packages/elasticsearch/data_stream/index/fields/ecs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,12 +10,17 @@
external: ecs
- name: host.name
external: ecs
dimension: true
- name: service.address
type: keyword
dimension: true
description: Service address
- name: service.type
external: ecs
- name: service.name
external: ecs
- name: error.message
external: ecs
- name: agent.id
external: ecs
dimension: true
1 change: 1 addition & 0 deletions packages/elasticsearch/data_stream/index/fields/fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
type: keyword
- name: name
type: keyword
dimension: true
description: |
Index name.
- name: primaries
Expand Down
5 changes: 5 additions & 0 deletions packages/elasticsearch/data_stream/index_recovery/fields/ecs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,12 +10,17 @@
external: ecs
- name: host.name
external: ecs
dimension: true
- name: service.address
type: keyword
dimension: true
description: Service address
- name: service.type
external: ecs
- name: service.name
external: ecs
- name: error.message
external: ecs
- name: agent.id
external: ecs
dimension: true
2 changes: 2 additions & 0 deletions packages/elasticsearch/data_stream/index_recovery/fields/fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
type: group
fields:
- name: name
dimension: true
type: keyword
- name: recovery
type: group
Expand Down Expand Up @@ -39,6 +40,7 @@
type: long
- name: id
type: long
dimension: true
description: |
Shard recovery id.
- name: type
Expand Down
5 changes: 5 additions & 0 deletions packages/elasticsearch/data_stream/index_summary/fields/ecs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,12 +10,17 @@
external: ecs
- name: host.name
external: ecs
dimension: true
- name: service.address
type: keyword
dimension: true
description: Service address
- name: service.type
external: ecs
- name: service.name
external: ecs
- name: error.message
external: ecs
- name: agent.id
external: ecs
dimension: true
13 changes: 13 additions & 0 deletions packages/elasticsearch/data_stream/ingest_pipeline/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
---
description: "Pipeline for Elasticsearch Ingest Pipeline"

processors:
- fingerprint:
fields:
- elasticsearch.ingest_pipeline.name
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the type: wildcard is the case as type: object ?

fields:
    - name: name
      type: wildcard
      description: Name / id of the ingest pipeline

can you please share sample of it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you please share sample of it?

Sorry, I don't understand. A sample of the error?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I don't understand. A sample of the error?

sample of the document - part of the document that include this field, there is missing same_event for this data_stream, can't check it there

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is a sample document 

{
  "_index": ".ds-metrics-elasticsearch.ingest_pipeline-default-2023.06.23-000001",
  "_id": "5p2D54gBH7q8D4JF6839",
  "_version": 1,
  "_score": 0,
  "_source": {
    "agent": {
      "name": "kind-control-plane",
      "id": "a781ce37-a210-49d3-8344-6518fb35d4ac",
      "type": "metricbeat",
      "ephemeral_id": "55936ecf-0dfb-474d-9031-284992efdf8a",
      "version": "8.8.0"
    },
    "@timestamp": "2023-06-23T09:09:21.280Z",
    "elasticsearch": {
      "node": {
        "roles": [
          "data_content",
          "data_hot",
          "ingest",
          "master",
          "remote_cluster_client",
          "transform"
        ],
        "name": "instance-0000000000",
        "id": "J_W-dXFXTxuXnGCwbCb6Iw"
      },
      "cluster": {
        "name": "985f2ca8e1a74327aa2c698275330b90",
        "id": "SyM7nU1DRmKd3soposFsXg"
      },
      "ingest_pipeline": {
        "total": {
          "count": 428,
          "failed": 0,
          "time": {
            "total": {
              "ms": 0
            },
            "self": {
              "ms": 0
            }
          }
        },
        "name": "metrics-elasticsearch.stack_monitoring.cluster_stats-1.7.4",
        "name_fingerprint": "LX8WOW8tc72gcK7v5HOrWtDf6v4="
      }
    },
    "ecs": {
      "version": "8.0.0"
    },
    "data_stream": {
      "namespace": "default",
      "type": "metrics",
      "dataset": "elasticsearch.ingest_pipeline"
    },
    "service": {
      "address": "https://test-es-3.es.us-central1.gcp.cloud.es.io:9243",
      "type": "elasticsearch"
    },
    "elastic_agent": {
      "id": "a781ce37-a210-49d3-8344-6518fb35d4ac",
      "version": "8.8.0",
      "snapshot": true
    },
    "host": {
      "hostname": "kind-control-plane",
      "os": {
        "kernel": "5.15.49-linuxkit",
        "codename": "focal",
        "name": "Ubuntu",
        "type": "linux",
        "family": "debian",
        "version": "20.04.6 LTS (Focal Fossa)",
        "platform": "ubuntu"
      },
      "containerized": false,
      "ip": [
        "10.244.0.1",
        "10.244.0.1",
        "10.244.0.1",
        "172.18.0.2",
        "fc00:f853:ccd:e793::2",
        "fe80::42:acff:fe12:2",
        "172.25.0.4"
      ],
      "name": "kind-control-plane",
      "id": "e12fa0193ee24a5cae5f9665f6e7eb8c",
      "mac": [
        "02-42-AC-12-00-02",
        "02-42-AC-19-00-04",
        "22-DE-5A-26-82-AC",
        "3A-AE-FC-E1-7E-8C",
        "7E-91-38-58-97-2B"
      ],
      "architecture": "x86_64"
    },
    "metricset": {
      "period": 10000,
      "name": "ingest_pipeline"
    },
    "event": {
      "duration": 275991722,
      "agent_id_status": "verified",
      "ingested": "2023-06-23T09:09:22Z",
      "module": "elasticsearch",
      "dataset": "elasticsearch.ingest_pipeline"
    }
  },
  "fields": {
    "elastic_agent.version": [
      "8.8.0"
    ],
    "elasticsearch.ingest_pipeline.name_fingerprint": [
      "LX8WOW8tc72gcK7v5HOrWtDf6v4="
    ],
    "host.hostname": [
      "kind-control-plane"
    ],
    "host.mac": [
      "02-42-AC-12-00-02",
      "02-42-AC-19-00-04",
      "22-DE-5A-26-82-AC",
      "3A-AE-FC-E1-7E-8C",
      "7E-91-38-58-97-2B"
    ],
    "service.type": [
      "elasticsearch"
    ],
    "host.ip": [
      "10.244.0.1",
      "10.244.0.1",
      "10.244.0.1",
      "172.18.0.2",
      "fc00:f853:ccd:e793::2",
      "fe80::42:acff:fe12:2",
      "172.25.0.4"
    ],
    "agent.type": [
      "metricbeat"
    ],
    "event.module": [
      "elasticsearch"
    ],
    "host.os.version": [
      "20.04.6 LTS (Focal Fossa)"
    ],
    "elasticsearch.ingest_pipeline.total.time.total.ms": [
      0
    ],
    "host.os.kernel": [
      "5.15.49-linuxkit"
    ],
    "host.os.name": [
      "Ubuntu"
    ],
    "agent.name": [
      "kind-control-plane"
    ],
    "host.name": [
      "kind-control-plane"
    ],
    "elastic_agent.snapshot": [
      true
    ],
    "event.agent_id_status": [
      "verified"
    ],
    "host.id": [
      "e12fa0193ee24a5cae5f9665f6e7eb8c"
    ],
    "elasticsearch.node.roles": [
      "data_content",
      "data_hot",
      "ingest",
      "master",
      "remote_cluster_client",
      "transform"
    ],
    "elasticsearch.node.id": [
      "J_W-dXFXTxuXnGCwbCb6Iw"
    ],
    "elasticsearch.cluster.name": [
      "985f2ca8e1a74327aa2c698275330b90"
    ],
    "elasticsearch.ingest_pipeline.total.failed": [
      0
    ],
    "host.os.type": [
      "linux"
    ],
    "elastic_agent.id": [
      "a781ce37-a210-49d3-8344-6518fb35d4ac"
    ],
    "data_stream.namespace": [
      "default"
    ],
    "elasticsearch.ingest_pipeline.total.time.self.ms": [
      0
    ],
    "metricset.period": [
      10000
    ],
    "host.os.codename": [
      "focal"
    ],
    "elasticsearch.ingest_pipeline.name": [
      "metrics-elasticsearch.stack_monitoring.cluster_stats-1.7.4"
    ],
    "data_stream.type": [
      "metrics"
    ],
    "event.duration": [
      275991722
    ],
    "elasticsearch.cluster.id": [
      "SyM7nU1DRmKd3soposFsXg"
    ],
    "host.architecture": [
      "x86_64"
    ],
    "metricset.name": [
      "ingest_pipeline"
    ],
    "event.ingested": [
      "2023-06-23T09:09:22.000Z"
    ],
    "@timestamp": [
      "2023-06-23T09:09:21.280Z"
    ],
    "elasticsearch.node.name": [
      "instance-0000000000"
    ],
    "agent.id": [
      "a781ce37-a210-49d3-8344-6518fb35d4ac"
    ],
    "elasticsearch.ingest_pipeline.total.count": [
      428
    ],
    "ecs.version": [
      "8.0.0"
    ],
    "host.os.platform": [
      "ubuntu"
    ],
    "host.containerized": [
      false
    ],
    "service.address": [
      "https://test-es-3.es.us-central1.gcp.cloud.es.io:9243"
    ],
    "data_stream.dataset": [
      "elasticsearch.ingest_pipeline"
    ],
    "agent.ephemeral_id": [
      "55936ecf-0dfb-474d-9031-284992efdf8a"
    ],
    "agent.version": [
      "8.8.0"
    ],
    "host.os.family": [
      "debian"
    ],
    "event.dataset": [
      "elasticsearch.ingest_pipeline"
    ]
  }
}

Copy link
Contributor

@tetianakravchenko tetianakravchenko Jun 23, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looking at this sample, "name": "metrics-elasticsearch.stack_monitoring.cluster_stats-1.7.4", seems to be a keyword.
And seems that wildcard belong to the keyword family - https://www.elastic.co/guide/en/elasticsearch/reference/7.17/keyword.html#keyword

does adding a dimension on name field fails?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, the wildcart is not valid to be a dimension.

target_field: elasticsearch.ingest_pipeline.name_fingerprint
ignore_missing: true
on_failure:
- set:
field: 'error.message'
value: '{{ _ingest.on_failure_message }}'
5 changes: 5 additions & 0 deletions packages/elasticsearch/data_stream/ingest_pipeline/fields/ecs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,13 +9,18 @@
- name: event.module
external: ecs
- name: host.name
dimension: true
external: ecs
- name: service.address
type: keyword
dimension: true
description: Service address
- name: service.type
external: ecs
- name: service.name
external: ecs
- name: error.message
external: ecs
- name: agent.id
external: ecs
dimension: true
5 changes: 5 additions & 0 deletions packages/elasticsearch/data_stream/ingest_pipeline/fields/fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,10 @@
- name: name
type: wildcard
description: Name / id of the ingest pipeline
- name: name_fingerprint
ritalwar marked this conversation as resolved.
Show resolved Hide resolved
type: keyword
dimension: true
description: Fingerprint for the ingest pipeline name.
- name: total
type: group
description: Metrics on the total ingest pipeline execution, including all processors.
Expand Down Expand Up @@ -32,6 +36,7 @@
description: The type and the tag for this processor in the format "<type>:<tag>"
- name: order_index
type: long
dimension: true
description: The order this processor appears in the pipeline definition
- name: count
type: long
Expand Down
5 changes: 5 additions & 0 deletions packages/elasticsearch/data_stream/ml_job/fields/ecs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,12 +10,17 @@
external: ecs
- name: host.name
external: ecs
dimension: true
- name: service.address
type: keyword
dimension: true
description: Service address
- name: service.type
external: ecs
- name: service.name
external: ecs
- name: error.message
external: ecs
- name: agent.id
external: ecs
dimension: true
1 change: 1 addition & 0 deletions packages/elasticsearch/data_stream/ml_job/fields/fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
fields:
- name: id
type: keyword
dimension: true
description: |
Unique ml job id.
- name: state
Expand Down
5 changes: 5 additions & 0 deletions packages/elasticsearch/data_stream/node/fields/ecs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,13 +9,18 @@
- name: event.module
external: ecs
- name: host.name
dimension: true
external: ecs
- name: service.address
type: keyword
dimension: true
description: Service address
- name: service.type
external: ecs
- name: service.name
external: ecs
- name: error.message
external: ecs
- name: agent.id
external: ecs
dimension: true
1 change: 1 addition & 0 deletions packages/elasticsearch/data_stream/node/fields/package-fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@
Node ID
- name: name
type: keyword
dimension: true
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wouldn't node.id be a better candidate? it could be not unique for multiple clusters

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Each cluster only has one service.address, so the combination service.address + node.name should be unique

Copy link
Contributor

@tetianakravchenko tetianakravchenko Jun 23, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

service.address is a host address defined in the configuration, so it could be for example localhost:9200 if the agent is running on the same instance with the elasticsearch - that is not unique enough
node.name from my understanding it is a hostname, isnt it? so it can be the same for multiple clusters

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

localhost:9200 is a default value

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the integration to work you need to give the service.address. This way, if you give to the same ES integration the same service.address, you will be receiving metrics from the same clusters as before. I tested with with a local cluster and one on the cloud.

Copy link
Contributor Author

@constanca-m constanca-m Jun 23, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

node.name from my understanding it is a hostname, isnt it? so it can be the same for multiple clusters

The service.address uniquely identifies a cluster for an ES integration, and since node.name is unique per cluster, that combination is enough.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the integration to work you need to give the service.address. This way, if you give to the same ES integration the same service.address, you will be receiving metrics from the same clusters as before.

why? If I set service.address as localhost:9200, install agent on different nodes and use the same policy for those node, I will get correct data

The service.address uniquely identifies a cluster for an ES integration, and since node.name is unique per cluster, that combination is enough.

but there can be the same node.name for 2 different clusters. It is not unique

example: I have 2 different instance: es-test and es-test2 in the same gcp account (it is just for the test, more realistic: have instance with the same name in different accounts/in different cloud providers, just for the test I've changes the hostname of es-test2 to es-test):
Screenshot 2023-06-23 at 23 22 31

service.address the same for both nodes, node.name as well. Since I did not change default value - cluster.name the same as well

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am a bit confused.

To install the integration in some policy you need to set the service.address:
image

This way, the service.address is unique. You cannot connect to two different clusters using the same service.address. So the service.address uniquely identifies a cluster.

If I set service.address as localhost:9200, install agent on different nodes and use the same policy for those node, I will get correct data

So install two different agents? The agent.id is a dimension, so there is no overlapping. If the service.address for the ES is different, there is also no overlap. Otherwise, there is as it should be.

but there can be the same node.name for 2 different clusters. It is not unique

We always have value for service.address. The node.name is unique per cluster, so service.address + node.name is unique.

I tested it it by adding to the policy:

  • 1 local elastic agent
  • 1 cluster with 3 nodes
  • Another cluster with 3 nodes (this one so I could update the version)

I didn't get any overlap.

description: |
Node name.
- name: master
Expand Down
5 changes: 5 additions & 0 deletions packages/elasticsearch/data_stream/node_stats/fields/ecs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,10 @@
- name: event.module
external: ecs
- name: host.name
dimension: true
external: ecs
- name: service.address
dimension: true
type: keyword
description: Service address
- name: service.type
Expand All @@ -19,3 +21,6 @@
external: ecs
- name: error.message
external: ecs
- name: agent.id
external: ecs
dimension: true
1 change: 1 addition & 0 deletions packages/elasticsearch/data_stream/node_stats/fields/package-fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -316,6 +316,7 @@
Node ID
- name: name
type: keyword
dimension: true
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the same as for node

description: |
Node name.
- name: master
Expand Down
9 changes: 9 additions & 0 deletions packages/elasticsearch/docs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -305,6 +305,7 @@ will not collect metrics. A DEBUG log message about this will be emitted in the
| Field | Description | Type |
|---|---|---|
| @timestamp | Date/time when the event originated. This is the date/time extracted from the event, typically representing when the event was generated by the source. If the event source has no original timestamp, this value is typically populated by the first time the event was received by the pipeline. Required field for all events. | date |
| agent.id | Unique identifier of this agent (if one exists). Example: For Beats this would be beat.id. | keyword |
| ccr_auto_follow_stats.follower.failed_read_requests | | alias |
| ccr_auto_follow_stats.number_of_failed_follow_indices | | alias |
| ccr_auto_follow_stats.number_of_failed_remote_cluster_state_requests | | alias |
Expand Down Expand Up @@ -588,6 +589,7 @@ An example event for `cluster_stats` looks as following:
| Field | Description | Type |
|---|---|---|
| @timestamp | Date/time when the event originated. This is the date/time extracted from the event, typically representing when the event was generated by the source. If the event source has no original timestamp, this value is typically populated by the first time the event was received by the pipeline. Required field for all events. | date |
| agent.id | Unique identifier of this agent (if one exists). Example: For Beats this would be beat.id. | keyword |
| cluster_state.master_node | | alias |
| cluster_state.nodes_hash | | alias |
| cluster_state.state_uuid | | alias |
Expand Down Expand Up @@ -767,6 +769,7 @@ An example event for `enrich` looks as following:
| Field | Description | Type |
|---|---|---|
| @timestamp | Date/time when the event originated. This is the date/time extracted from the event, typically representing when the event was generated by the source. If the event source has no original timestamp, this value is typically populated by the first time the event was received by the pipeline. Required field for all events. | date |
| agent.id | Unique identifier of this agent (if one exists). Example: For Beats this would be beat.id. | keyword |
| cluster_uuid | | alias |
| data_stream.dataset | Data stream dataset. | constant_keyword |
| data_stream.namespace | Data stream namespace. | constant_keyword |
Expand Down Expand Up @@ -960,6 +963,7 @@ An example event for `index` looks as following:
| Field | Description | Type |
|---|---|---|
| @timestamp | Date/time when the event originated. This is the date/time extracted from the event, typically representing when the event was generated by the source. If the event source has no original timestamp, this value is typically populated by the first time the event was received by the pipeline. Required field for all events. | date |
| agent.id | Unique identifier of this agent (if one exists). Example: For Beats this would be beat.id. | keyword |
| cluster_uuid | | alias |
| data_stream.dataset | Data stream dataset. | constant_keyword |
| data_stream.namespace | Data stream namespace. | constant_keyword |
Expand Down Expand Up @@ -1224,6 +1228,7 @@ An example event for `index_recovery` looks as following:
| Field | Description | Type |
|---|---|---|
| @timestamp | Date/time when the event originated. This is the date/time extracted from the event, typically representing when the event was generated by the source. If the event source has no original timestamp, this value is typically populated by the first time the event was received by the pipeline. Required field for all events. | date |
| agent.id | Unique identifier of this agent (if one exists). Example: For Beats this would be beat.id. | keyword |
| cluster_uuid | | alias |
| data_stream.dataset | Data stream dataset. | constant_keyword |
| data_stream.namespace | Data stream namespace. | constant_keyword |
Expand Down Expand Up @@ -1457,6 +1462,7 @@ An example event for `index_summary` looks as following:
| Field | Description | Type |
|---|---|---|
| @timestamp | Date/time when the event originated. This is the date/time extracted from the event, typically representing when the event was generated by the source. If the event source has no original timestamp, this value is typically populated by the first time the event was received by the pipeline. Required field for all events. | date |
| agent.id | Unique identifier of this agent (if one exists). Example: For Beats this would be beat.id. | keyword |
| cluster_uuid | | alias |
| data_stream.dataset | Data stream dataset. | constant_keyword |
| data_stream.namespace | Data stream namespace. | constant_keyword |
Expand Down Expand Up @@ -1619,6 +1625,7 @@ An example event for `ml_job` looks as following:
| Field | Description | Type |
|---|---|---|
| @timestamp | Date/time when the event originated. This is the date/time extracted from the event, typically representing when the event was generated by the source. If the event source has no original timestamp, this value is typically populated by the first time the event was received by the pipeline. Required field for all events. | date |
| agent.id | Unique identifier of this agent (if one exists). Example: For Beats this would be beat.id. | keyword |
| cluster_uuid | | alias |
| data_stream.dataset | Data stream dataset. | constant_keyword |
| data_stream.namespace | Data stream namespace. | constant_keyword |
Expand Down Expand Up @@ -1766,6 +1773,7 @@ An example event for `node` looks as following:
| Field | Description | Type |
|---|---|---|
| @timestamp | Date/time when the event originated. This is the date/time extracted from the event, typically representing when the event was generated by the source. If the event source has no original timestamp, this value is typically populated by the first time the event was received by the pipeline. Required field for all events. | date |
| agent.id | Unique identifier of this agent (if one exists). Example: For Beats this would be beat.id. | keyword |
| cluster_uuid | | alias |
| data_stream.dataset | Data stream dataset. | constant_keyword |
| data_stream.namespace | Data stream namespace. | constant_keyword |
Expand Down Expand Up @@ -2197,6 +2205,7 @@ An example event for `node_stats` looks as following:
| Field | Description | Type |
|---|---|---|
| @timestamp | Date/time when the event originated. This is the date/time extracted from the event, typically representing when the event was generated by the source. If the event source has no original timestamp, this value is typically populated by the first time the event was received by the pipeline. Required field for all events. | date |
| agent.id | Unique identifier of this agent (if one exists). Example: For Beats this would be beat.id. | keyword |
| cluster_uuid | | alias |
| data_stream.dataset | Data stream dataset. | constant_keyword |
| data_stream.namespace | Data stream namespace. | constant_keyword |
Expand Down
2 changes: 1 addition & 1 deletion packages/elasticsearch/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
name: elasticsearch
title: Elasticsearch
version: 1.7.4
version: 1.8.0
description: Elasticsearch Integration
type: integration
icons:
Expand Down