Replace read_timestamp with event.created in all remaining places (#1…

…0139) Note that some of the recent module migrations have already started populating `event.created` instead of `read_timestamp`. This PR finishes this work all at once. - Replace `read_timestamp` with `event.created` in remaining Fb modules: - iis.error - kafka - kibana - nginx.access - osquery - redis - traefik.access - No longer excluding `read_timestamp` from integration test results, as it's no longer expected - Finish equivalent migration in Journalbeat by making `read_timestamp` into an alias - Adjust Kibana module's pipeline to use the correct semantics for `event.created`
elastic · Jan 18, 2019 · 31fec4f · 31fec4f
1 parent 24d6d5a
commit 31fec4f
Show file tree

Hide file tree

Showing 16 changed files with 60 additions and 48 deletions.
diff --git a/CHANGELOG.next.asciidoc b/CHANGELOG.next.asciidoc
@@ -30,6 +30,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Fix parsing of GC entries in elasticsearch server log. {issue}9513[9513] {pull}9810[9810]
 - Rename `read_timestamp` to `event.created` for Redis input. {pull}9924[9924]
 - Rename a few `elasticsearch.audit.*` fields to map to ECS. {pull}9293[9293]
+- Rename `read_timestamp` to `event.created` for all Filebeat modules using it. {pull}10139[10139]
 - Rename many `iis.error.*` fields to map to ECS. {pull}9955[9955]
 - Adjust fileset `haproxy.log` to map to ECS. {pull}10143[10143]
 - Rename a few `logstash.*` fields to map to ECS, remove logstash.slowlog.message. {pull}9935[9935]
@@ -39,6 +40,8 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Filesets with multiple ingest pipelines added in {pull}8914[8914] only work with Elasticsearch >= 6.5.0 {pull}10001[10001]
 - Remove service.name from Elastcsearch module. Replace by service.type. {pull}10042[10042]
 - Add grok pattern to support redis 5.0.3 log timestamp. {issue}9819[9819] {pull}10033[10033]
+- Now save the 'first seen' timestamp in `event.created` (previously `read_timestamp`),
+  instead of saving the parsed date. Now aligned with `event.created` semantics elsewhere. {pull}10139[10139]
 
 *Heartbeat*
 
@@ -48,7 +51,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 
 *Journalbeat*
 
-- Rename read_timestamp to event.created to align with ECS. {pull}10043[10043]
+- Rename read_timestamp to event.created to align with ECS. {pull}10043[10043], {pull}10139[10139]
 - Rename host.name to host.hostname to align with ECS. {pull}10043[10043]
 
 *Metricbeat*

diff --git a/dev-tools/ecs-migration.yml b/dev-tools/ecs-migration.yml
@@ -244,10 +244,6 @@
   to: user_agent.original
   alias: true
 
-- from: read_timestamp
-  to: event.created
-  alias: true
-
 # These expand all fields under geoip and user_agent
 - from: apache2.access.geoip.*
   to: source.geo.*
@@ -778,6 +774,7 @@
 - from: read_timestamp
   to: event.created
   alias: false
+  beat: filebeat
 
 # Auditbeat
 
@@ -948,3 +945,4 @@
 - from: read_timestamp
   to: event.created
   alias: true
+  beat: journalbeat
diff --git a/filebeat/_meta/fields.common.yml b/filebeat/_meta/fields.common.yml
@@ -35,12 +35,6 @@
         The input type from which the event was generated. This field is set to the value specified
         for the `type` option in the input section of the Filebeat config file.
 
-    - name: read_timestamp
-      description: >
-        In case the ingest pipeline parses the timestamp from the log contents, it stores
-        the original `@timestamp` (representing the time when the log line was read) in this
-        field.
-
     - name: syslog.facility
       type: long
       required: false
@@ -103,3 +97,8 @@
       type: alias
       path: event.module
       migration: true
+
+    - name: read_timestamp
+      type: alias
+      path: event.created
+      migration: true
diff --git a/filebeat/docs/fields.asciidoc b/filebeat/docs/fields.asciidoc
@@ -5935,14 +5935,6 @@ required: True
 The input type from which the event was generated. This field is set to the value specified for the `type` option in the input section of the Filebeat config file.
 
 
---
-
-*`read_timestamp`*::
-+
---
-In case the ingest pipeline parses the timestamp from the log contents, it stores the original `@timestamp` (representing the time when the log line was read) in this field.
-
-
 --
 
 *`syslog.facility`*::
@@ -6061,6 +6053,15 @@ alias to: event.module
 
 --
 
+*`read_timestamp`*::
++
+--
+type: alias
+
+alias to: event.created
+
+--
+
 [[exported-fields-logstash]]
 == logstash fields
 

diff --git a/filebeat/include/fields.go b/filebeat/include/fields.go
diff --git a/filebeat/module/iis/error/ingest/default.json b/filebeat/module/iis/error/ingest/default.json
@@ -15,7 +15,7 @@
   }, {
     "rename": {
       "field": "@timestamp",
-      "target_field": "read_timestamp"
+      "target_field": "event.created"
     }
   }, {
     "date": {

diff --git a/filebeat/module/kafka/log/ingest/pipeline.json b/filebeat/module/kafka/log/ingest/pipeline.json
@@ -48,7 +48,7 @@
     {
       "rename": {
         "field": "@timestamp",
-        "target_field": "read_timestamp"
+        "target_field": "event.created"
       }
     },
     {

diff --git a/filebeat/module/kibana/log/ingest/pipeline.json b/filebeat/module/kibana/log/ingest/pipeline.json
@@ -9,18 +9,32 @@
         }
     ],
     "processors": [
+        {
+            "rename": {
+                "field": "@timestamp",
+                "target_field": "event.created"
+            }
+        },
+
         {
             "rename": {
                 "field": "json",
                 "target_field": "kibana.log.meta"
             }
         },
         {
-            "rename": {
+            "date": {
                 "field": "kibana.log.meta.@timestamp",
-                "target_field": "read_timestamp"
+                "formats" : ["ISO8601"],
+                "target_field": "@timestamp"
+            }
+        },
+        {
+            "remove": {
+                "field": "kibana.log.meta.@timestamp"
             }
         },
+
         {
             "rename": {
                 "field": "kibana.log.meta.message",
@@ -138,13 +152,6 @@
             }
         },
 
-        {
-            "date": {
-                "field": "read_timestamp",
-                "formats" : ["ISO8601"],
-                "target_field": "@timestamp"
-            }
-        },
         {
             "append": {
                 "field": "service.name",

diff --git a/filebeat/module/nginx/access/ingest/default.json b/filebeat/module/nginx/access/ingest/default.json
@@ -57,7 +57,7 @@
         {
             "rename": {
                 "field": "@timestamp",
-                "target_field": "read_timestamp"
+                "target_field": "event.created"
             }
         },
         {

diff --git a/filebeat/module/osquery/result/ingest/pipeline.json b/filebeat/module/osquery/result/ingest/pipeline.json
@@ -4,7 +4,7 @@
     {
       "rename": {
         "field": "@timestamp",
-        "target_field": "read_timestamp"
+        "target_field": "event.created"
       }
     }, {
       "date": {

diff --git a/filebeat/module/redis/log/ingest/pipeline.json b/filebeat/module/redis/log/ingest/pipeline.json
@@ -51,7 +51,7 @@
         {
             "rename": {
                 "field": "@timestamp",
-                "target_field": "read_timestamp"
+                "target_field": "event.created"
             }
         },
         {

diff --git a/filebeat/module/traefik/access/ingest/pipeline.json b/filebeat/module/traefik/access/ingest/pipeline.json
@@ -25,7 +25,7 @@
         {
             "rename": {
                 "field": "@timestamp",
-                "target_field": "read_timestamp"
+                "target_field": "event.created"
             }
         },
         {

diff --git a/filebeat/tests/system/test_modules.py b/filebeat/tests/system/test_modules.py
@@ -201,7 +201,7 @@ def clean_keys(obj):
     # These keys are host dependent
     host_keys = ["host.name", "agent.hostname", "agent.type", "agent.ephemeral_id", "agent.id"]
     # The create timestamps area always new
-    time_keys = ["read_timestamp", "event.created"]
+    time_keys = ["event.created"]
     # source path and agent.version can be different for each run
     other_keys = ["log.file.path", "agent.version"]
 

diff --git a/journalbeat/_meta/fields.common.yml b/journalbeat/_meta/fields.common.yml
@@ -3,9 +3,6 @@
   description: >
     Contains common fields available in all event types.
   fields:
-    - name: read_timestamp
-      description: >
-          The time when Journalbeat read the journal entry.
     - name: coredump
       type: group
       description: >
@@ -314,3 +311,9 @@
       required: false
       description: >
         Arbitrary fields coming from processes.
+
+    - name: read_timestamp
+      type: alias
+      path: event.created
+      migration: true
+
diff --git a/journalbeat/docs/fields.asciidoc b/journalbeat/docs/fields.asciidoc
@@ -170,14 +170,6 @@ Contains common fields available in all event types.
 
 
 
-*`read_timestamp`*::
-+
---
-The time when Journalbeat read the journal entry.
-
-
---
-
 [float]
 == coredump fields
 
@@ -548,6 +540,15 @@ required: False
 Arbitrary fields coming from processes.
 
 
+--
+
+*`read_timestamp`*::
++
+--
+type: alias
+
+alias to: event.created
+
 --
 
 [[exported-fields-docker-processor]]

diff --git a/journalbeat/include/fields.go b/journalbeat/include/fields.go