Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support MySQL 5.7.19 by mysql/slowlog #6969

Merged
merged 3 commits into from
May 3, 2018

Conversation

kvch
Copy link
Contributor

@kvch kvch commented Apr 27, 2018

  • Make mysql.slowlog.host optional.
  • Rename @timestamp to read_timestamp before transforming logs timestamp.

Output of pipeline tester:

$ ./scripts/tester/execute_pipeline --pipeline "module/mysql/slowlog/ingest/pipeline.json" \
--log '# User@Host: root[root] @  [172.17.0.11]  Id:     5
# Query_time: 0.000100  Lock_time: 0.000033 Rows_sent: 101  Rows_examined: 101
SET timestamp=1524768632;
SELECT intcol1,charcol1 FROM t1;
# Time: 2018-04-26T18:50:32.437295Z' \
--verbose
{
  "docs": [
    {
      "doc": {
        "_id": "id",
        "_index": "index",
        "_ingest": {
          "timestamp": "2018-04-27T20:06:01.986Z"
        },
        "_source": {
          "@timestamp": "2018-04-26T18:50:32.000Z",
          "mysql": {
            "slowlog": {
              "id": "5",
              "ip": "172.17.0.11",
              "lock_time": {
                "sec": "0.000033"
              },
              "query": "SELECT intcol1,charcol1 FROM t1;\n# Time: 2018-04-26T18:50:32.437295Z",
              "query_time": {
                "sec": "0.000100"
              },
              "rows_examined": "101",
              "rows_sent": "101",
              "timestamp": "1524768632",
              "user": "root"
            }
          },
          "read_timestamp": "2018-04-27T20:06:01.974Z"
        },
        "_type": "doc"
      }
    }
  ]
}

Closes #6882
Closes #6492

@kvch kvch changed the title MySQL/slowlog module of Filebeat Support MySQL 5.7.19 by mysql/slowlog Apr 27, 2018
@@ -15,6 +15,11 @@
"remove":{
"field": "message"
}
}, {
"rename": {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this a breaking change?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed.

@kvch kvch force-pushed the fix/filebeat/mysql-slowlog-module branch from 5aae77c to b7682ab Compare May 3, 2018 06:06
@kvch
Copy link
Contributor Author

kvch commented May 3, 2018

@ruflin I have removed the breaking change from the pipeline. It would be nice to add read_timestamp and transform @timestamp in all pipelines. But it should be done in one batch and release it in 7.0.

@ruflin ruflin merged commit a503644 into elastic:master May 3, 2018
@ruflin
Copy link
Member

ruflin commented May 3, 2018

@kvch Agree that we should come up with a convention for read_timestamp/@timestamp across all modules and do it in one go. Perhaps there are also other solutions then only breaking change. Lets discuss this further.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants