Identify queries associated with scenario/user workflow in log

[gingerwizard]

As discussed with @dmlemeshko we would like to identify the ES queries issued by Kibana when a scenario is executed. We will use these queries for Rally load testing. This is possible using via elasticsearch.logQueries.

However, we also need to identify which queries are associated with which kibana API call and scenario e.g. logging into discover. When loading testing with Rally, we would execute those queries associated with a scenario together - thereby simulating a user action. In order to do this, we need someway of grouping queries in the logs.

One proposal here is for the load gen to attach a unique value to a custom header for each scenario - this could just be a generated uuid and/or the name of the scenario. We need to determine if Kibana can be made to log these headers when logging Kibana queries. A change of this value in the log would in turn indicate a new scenario.

@dliappis

[dmlemeshko]

@gingerwizard,

With the help of security team I may confirm that it is possible to pass a custom header, Kibana will ignore it but will print it out in requests log.

• To see requests logging, it requires adding logging.verbose: true and logging.json: true in your kibana.yml

• After adding custom header x-opaque-id to request, you can find it in logs:

{"type":"response","@timestamp":"2020-10-19T09:26:01-04:00","tags":[],"pid":73579,"method":"get","statusCode":401,"req":{"url":"/api/spaces/_share_saved_object_permissions?type=toa","method":"get","headers":{"x-opaque-id":"foobarbaz","kbn-xsrf":"kibana","user-agent":"PostmanRuntime/7.26.5","accept":"*/*","postman-token":"ced37fd4-c533-4196-a12f-525cfb69c372","host":"black-box.local:5603","accept-encoding":"gzip, deflate, br","connection":"keep-alive"},"remoteAddress":"10.0.0.120","userAgent":"PostmanRuntime/7.26.5"},"res":{"statusCode":401,"responseTime":26,"contentLength":9},"message":"GET /api/spaces/_share_saved_object_permissions?type=toa 401 26ms - 9.0B"}

Security folks are working on enhancing audit logging which be a bit more flexible and might be more useful for this case. Coming in 7.11+ elastic/kibana#52125

[alexfrancoeur]

I stumbled across this issue and I think this capability (originally requested for telemetry purposes) may be related. elastic/kibana#77214