Add support to MySQL 8 extra fields in slow log #11417

jsoriano · 2019-03-24T16:44:02Z

Add test files for MySQL 8, Percona 8 and MariaDB 10.3 to filebeat
mysql module.

Add support for MySQL 8 extra fields in slow log.

Update compatibility notes.

jsoriano · 2019-03-24T16:44:52Z

I will open a kind-of-backport for 7.0 updating only compatibility notes and test files, but without support for the extra fields in slow log, this will be only released in 7.1.

ruflin · 2019-03-25T07:15:15Z

filebeat/module/mysql/slowlog/_meta/fields.yml

@@ -108,6 +129,34 @@
      description: >
        Slow log rate limit, a value of 100 means that one in a hundred queries
        or sessions are being logged.
+    - name: read_first


Should we make this read.first etc?

This module is keeping the names of the original fields, I'd continue using these values.

ruflin · 2019-03-25T07:15:23Z

filebeat/module/mysql/slowlog/_meta/fields.yml

@@ -98,6 +103,22 @@
      type: long
      description: >
        Number of merge passes executed for the query.
+    - name: sort_merge_passes


sort.* etc?

Also here I'd keep original names, they are very specific names.

ruflin · 2019-03-25T07:15:30Z

filebeat/module/mysql/slowlog/_meta/fields.yml

@@ -24,7 +24,12 @@
      type: long
      format: bytes
      description: >
-        The size of the query result.
+        The number of bytes sent to client.
+    - name: bytes_received


received.bytes?

Same here, plus having already bytes_sent.

ruflin · 2019-03-25T07:16:38Z

filebeat/module/mysql/slowlog/ingest/pipeline.json

@@ -4,7 +4,7 @@
    "grok": {
      "field": "message",
      "patterns":[
-        "^# User@Host: %{USER:user.name}(\\[%{USER:mysql.slowlog.current_user}\\])? @ %{HOSTNAME:source.domain}? \\[%{IP:source.ip}?\\]%{METRICSPACE}(Id:%{SPACE}%{NUMBER:mysql.thread_id:long}%{METRICSPACE})?(Thread_id:%{SPACE}%{NUMBER:mysql.thread_id}%{METRICSPACE})?(Schema:%{SPACE}%{WORD:mysql.slowlog.schema}?%{METRICSPACE})?(Last_errno: %{NUMBER:mysql.slowlog.last_errno:long}%{METRICSPACE})?(Killed: %{NUMBER:mysql.slowlog.killed:long}%{METRICSPACE})?(QC_hit: %{WORD:mysql.slowlog.query_cache_hit}%{METRICSPACE})?(Query_time: %{NUMBER:temp.duration:float}%{METRICSPACE})?(Lock_time: %{NUMBER:mysql.slowlog.lock_time.sec:float}%{METRICSPACE})?(Rows_sent: %{NUMBER:mysql.slowlog.rows_sent:long}%{METRICSPACE})?(Rows_examined: %{NUMBER:mysql.slowlog.rows_examined:long}%{METRICSPACE})?(Rows_affected: %{NUMBER:mysql.slowlog.rows_affected:long}%{METRICSPACE})?(Bytes_sent: %{NUMBER:mysql.slowlog.bytes_sent:long}%{METRICSPACE})?(Tmp_tables: %{NUMBER:mysql.slowlog.tmp_tables:long}%{METRICSPACE})?(Tmp_disk_tables: %{NUMBER:mysql.slowlog.tmp_disk_tables}%{METRICSPACE})?(Tmp_table_sizes: %{NUMBER:mysql.slowlog.tmp_table_sizes:long}%{METRICSPACE})?(InnoDB_trx_id: %{WORD:mysql.slowlog.innodb.trx_id}%{METRICSPACE})?(QC_Hit: %{WORD:mysql.slowlog.query_cache_hit}%{METRICSPACE})?(Full_scan: %{WORD:mysql.slowlog.full_scan}%{METRICSPACE})?(Full_join: %{WORD:mysql.slowlog.full_join}%{METRICSPACE})?(Tmp_table: %{WORD:mysql.slowlog.tmp_table}%{METRICSPACE})?(Tmp_table_on_disk: %{WORD:mysql.slowlog.tmp_table_on_disk}%{METRICSPACE})?(Filesort: %{WORD:mysql.slowlog.filesort}%{METRICSPACE})?(Filesort_on_disk: %{WORD:mysql.slowlog.filesort_on_disk}%{METRICSPACE})?(Merge_passes: %{NUMBER:mysql.slowlog.merge_passes:long}%{METRICSPACE})?(Priority_queue: %{WORD:mysql.slowlog.priority_queue}%{METRICSPACE})?(No InnoDB statistics available for this query%{METRICSPACE})?(InnoDB_IO_r_ops: %{NUMBER:mysql.slowlog.innodb.io_r_ops:long}%{METRICSPACE})?(InnoDB_IO_r_bytes: %{NUMBER:mysql.slowlog.innodb.io_r_bytes:long}%{METRICSPACE})?(InnoDB_IO_r_wait: %{NUMBER:mysql.slowlog.innodb.io_r_wait.sec:float}%{METRICSPACE})?(InnoDB_rec_lock_wait: %{NUMBER:mysql.slowlog.innodb.rec_lock_wait.sec:float}%{METRICSPACE})?(InnoDB_queue_wait: %{NUMBER:mysql.slowlog.innodb.queue_wait.sec:float}%{METRICSPACE})?(InnoDB_pages_distinct: %{NUMBER:mysql.slowlog.innodb.pages_distinct:long}%{METRICSPACE})?(Log_slow_rate_type: %{WORD:mysql.slowlog.log_slow_rate_type}%{METRICSPACE})?(Log_slow_rate_limit: %{NUMBER:mysql.slowlog.log_slow_rate_limit:long}%{METRICSPACE})?%{EXPLAIN}?(use %{WORD:mysql.slowlog.schema};\n)?SET timestamp=%{NUMBER:mysql.slowlog.timestamp:long};\n%{GREEDYMULTILINE:mysql.slowlog.query}"
+        "^# User@Host: %{USER:user.name}(\\[%{USER:mysql.slowlog.current_user}\\])? @ %{HOSTNAME:source.domain}? \\[%{IP:source.ip}?\\]%{METRICSPACE}(Id:%{SPACE}%{NUMBER:mysql.thread_id:long}%{METRICSPACE})?(Thread_id:%{SPACE}%{NUMBER:mysql.thread_id}%{METRICSPACE})?(Schema:%{SPACE}%{WORD:mysql.slowlog.schema}?%{METRICSPACE})?(Last_errno: %{NUMBER:mysql.slowlog.last_errno:long}%{METRICSPACE})?(Killed: %{NUMBER:mysql.slowlog.killed:long}%{METRICSPACE})?(QC_hit: %{WORD:mysql.slowlog.query_cache_hit}%{METRICSPACE})?(Query_time: %{NUMBER:temp.duration:float}%{METRICSPACE})?(Lock_time: %{NUMBER:mysql.slowlog.lock_time.sec:float}%{METRICSPACE})?(Rows_sent: %{NUMBER:mysql.slowlog.rows_sent:long}%{METRICSPACE})?(Rows_examined: %{NUMBER:mysql.slowlog.rows_examined:long}%{METRICSPACE})?(Rows_affected: %{NUMBER:mysql.slowlog.rows_affected:long}%{METRICSPACE})?(Thread_id: %{NUMBER:mysql.thread_id}%{METRICSPACE})?(Errno: %{NUMBER:mysql.slowlog.last_errno:long}%{METRICSPACE})?(Killed: %{NUMBER:mysql.slowlog.killed:long}%{METRICSPACE})?(Bytes_received: %{NUMBER:mysql.slowlog.bytes_received:long}%{METRICSPACE})?(Bytes_sent: %{NUMBER:mysql.slowlog.bytes_sent:long}%{METRICSPACE})?(Read_first: %{NUMBER:mysql.slowlog.read_first:long}%{METRICSPACE})?(Read_last: %{NUMBER:mysql.slowlog.read_last:long}%{METRICSPACE})?(Read_key: %{NUMBER:mysql.slowlog.read_key:long}%{METRICSPACE})?(Read_next: %{NUMBER:mysql.slowlog.read_next:long}%{METRICSPACE})?(Read_prev: %{NUMBER:mysql.slowlog.read_prev:long}%{METRICSPACE})?(Read_rnd: %{NUMBER:mysql.slowlog.read_rnd:long}%{METRICSPACE})?(Read_rnd_next: %{NUMBER:mysql.slowlog.read_rnd_next:long}%{METRICSPACE})?(Sort_merge_passes: %{NUMBER:mysql.slowlog.sort_merge_passes:long}%{METRICSPACE})?(Sort_range_count: %{NUMBER:mysql.slowlog.sort_range_count:long}%{METRICSPACE})?(Sort_rows: %{NUMBER:mysql.slowlog.sort_rows:long}%{METRICSPACE})?(Sort_scan_count: %{NUMBER:mysql.slowlog.sort_scan_count:long}%{METRICSPACE})?(Created_tmp_disk_tables: %{NUMBER:mysql.slowlog.tmp_disk_tables:long}%{METRICSPACE})?(Created_tmp_tables: %{NUMBER:mysql.slowlog.tmp_tables:long}%{METRICSPACE})?(Tmp_tables: %{NUMBER:mysql.slowlog.tmp_tables:long}%{METRICSPACE})?(Tmp_disk_tables: %{NUMBER:mysql.slowlog.tmp_disk_tables}%{METRICSPACE})?(Tmp_table_sizes: %{NUMBER:mysql.slowlog.tmp_table_sizes:long}%{METRICSPACE})?(Start: %{TIMESTAMP_ISO8601:event.start}%{METRICSPACE})?(End: %{TIMESTAMP_ISO8601:event.end}%{METRICSPACE})?(InnoDB_trx_id: %{WORD:mysql.slowlog.innodb.trx_id}%{METRICSPACE})?(QC_Hit: %{WORD:mysql.slowlog.query_cache_hit}%{METRICSPACE})?(Full_scan: %{WORD:mysql.slowlog.full_scan}%{METRICSPACE})?(Full_join: %{WORD:mysql.slowlog.full_join}%{METRICSPACE})?(Tmp_table: %{WORD:mysql.slowlog.tmp_table}%{METRICSPACE})?(Tmp_table_on_disk: %{WORD:mysql.slowlog.tmp_table_on_disk}%{METRICSPACE})?(Filesort: %{WORD:mysql.slowlog.filesort}%{METRICSPACE})?(Filesort_on_disk: %{WORD:mysql.slowlog.filesort_on_disk}%{METRICSPACE})?(Merge_passes: %{NUMBER:mysql.slowlog.merge_passes:long}%{METRICSPACE})?(Priority_queue: %{WORD:mysql.slowlog.priority_queue}%{METRICSPACE})?(No InnoDB statistics available for this query%{METRICSPACE})?(InnoDB_IO_r_ops: %{NUMBER:mysql.slowlog.innodb.io_r_ops:long}%{METRICSPACE})?(InnoDB_IO_r_bytes: %{NUMBER:mysql.slowlog.innodb.io_r_bytes:long}%{METRICSPACE})?(InnoDB_IO_r_wait: %{NUMBER:mysql.slowlog.innodb.io_r_wait.sec:float}%{METRICSPACE})?(InnoDB_rec_lock_wait: %{NUMBER:mysql.slowlog.innodb.rec_lock_wait.sec:float}%{METRICSPACE})?(InnoDB_queue_wait: %{NUMBER:mysql.slowlog.innodb.queue_wait.sec:float}%{METRICSPACE})?(InnoDB_pages_distinct: %{NUMBER:mysql.slowlog.innodb.pages_distinct:long}%{METRICSPACE})?(Log_slow_rate_type: %{WORD:mysql.slowlog.log_slow_rate_type}%{METRICSPACE})?(Log_slow_rate_limit: %{NUMBER:mysql.slowlog.log_slow_rate_limit:long}%{METRICSPACE})?%{EXPLAIN}?(use %{WORD:mysql.slowlog.schema};\n)?SET timestamp=%{NUMBER:mysql.slowlog.timestamp:long};\n%{GREEDYMULTILINE:mysql.slowlog.query}"


That is a big one :-) I wonder if in the future we could split this up in multiple patterns to make it more readable / reviewable.

Yes, this is quite a monster and painful to modify, not sure what alternatives we have, even with patterns this would be quite ugly.
I think we should parse this with code, not sure if in filebeat side (a processor?) or in painless in the pipeline.

ruflin · 2019-03-25T07:17:12Z

filebeat/module/mysql/slowlog/test/mariadb-10.3.13.log

+# Full_scan: Yes  Full_join: No  Tmp_table: Yes  Tmp_table_on_disk: No
+# Filesort: Yes  Filesort_on_disk: No  Merge_passes: 0  Priority_queue: Yes
+use employees;
+SET timestamp=1553443380;


Would be great to normally have at least 2 log entries to make sure the multiline works as expected.

I think we are covered here, there are several test files, all of them are multiline and most of them generate more than 1 event.

jsoriano · 2019-03-25T16:24:44Z

CI tests failing in metricbeat, not related

Update compatibility notes for MySQL in filebeat module. Add test files for MySQL 8, Percona 8 and MariaDB 10.3. Backports docs and part of test files of #11417 to 7.0.

jsoriano added 4 commits March 24, 2019 16:53

Add support to MySQL 8 extra fields in slow log

09579d3

Add test file for mariadb 10.3

fd1415d

Update docs

196f061

Add test files for percona 8 slowlog

4f160c2

jsoriano added module review Filebeat Filebeat needs_backport PR is waiting to be backported to other branches. v7.0.0 Team:Integrations Label for the Integrations team v7.2.0 labels Mar 24, 2019

jsoriano self-assigned this Mar 24, 2019

jsoriano requested review from a team as code owners March 24, 2019 16:44

Add changelog

4656c3c

jsoriano mentioned this pull request Mar 24, 2019

Add compatibility notes and test files for MySQL 8 in filebeat #11418

Merged

Update

2e3a550

jsoriano added docs and removed v7.0.0 needs_backport PR is waiting to be backported to other branches. labels Mar 24, 2019

ruflin reviewed Mar 25, 2019

View reviewed changes

ruflin added the [zube]: In Review label Mar 25, 2019

ruflin approved these changes Mar 25, 2019

View reviewed changes

jsoriano merged commit 4ec5d13 into elastic:master Mar 25, 2019

jsoriano deleted the filebeat-mysql-8-test-files branch March 25, 2019 16:26

zube bot added [zube]: Done and removed [zube]: In Review labels Mar 25, 2019

exekias removed the [zube]: Done label Apr 8, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support to MySQL 8 extra fields in slow log #11417

Add support to MySQL 8 extra fields in slow log #11417

jsoriano commented Mar 24, 2019 •

edited by ruflin

Loading

jsoriano commented Mar 24, 2019 •

edited

Loading

ruflin Mar 25, 2019

jsoriano Mar 25, 2019

ruflin Mar 25, 2019

jsoriano Mar 25, 2019

ruflin Mar 25, 2019

jsoriano Mar 25, 2019

ruflin Mar 25, 2019

jsoriano Mar 25, 2019

ruflin Mar 25, 2019

jsoriano Mar 25, 2019

jsoriano commented Mar 25, 2019

Add support to MySQL 8 extra fields in slow log #11417

Add support to MySQL 8 extra fields in slow log #11417

Conversation

jsoriano commented Mar 24, 2019 • edited by ruflin Loading

jsoriano commented Mar 24, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsoriano commented Mar 25, 2019

jsoriano commented Mar 24, 2019 •

edited by ruflin

Loading

jsoriano commented Mar 24, 2019 •

edited

Loading