Use stream.dataset instead of event.dataset

[mtojek]

Reported in #313

@mtojek An other thing that just came to my mind is that as part of the migration we should also start thinking about optimising the dashboards. For example the query for event.dataset should become a query for stream.dataset which is a constant keyword and much more efficient. The gent should always add these fields (@michalpristas will know more if this already happens).
I don't think we need to do all these optimisations directly during the migration but could be done later on. But we should start a checklist with these already now so we don't forget.

[michalpristas]

just a small note, agent provides information about dataset to corresponding beats but whether or how they are used or injected into events is up to beat itself. agent does not enrich any event coming from beat.

[ruflin]

@michalpristas @ph Even though events are not sent through the agent, my understanding is that it is up to the agent to ensure that the stream fields and correct index are set: elastic/beats#16562

[michalpristas]

correct index is something we have now. i dont' think we inject these 2 fields separately and that input on the beat side knows to pick them up and inject.
or is the issue meant in a way that agent will generate add_fields processor out of these fields?

[ruflin]

@michalpristas Lets take this to a separate issue.

[mtojek]

@michalpristas Lets take this to a separate issue.

Could you please link it here? I'm not sure if I can proceed with issue or not yet.

/cc @ruflin

[mtojek]

I converted this into general issue as I don't see stream.dataset in collected documents.

[ruflin]

@michalpristas Can you investigate this? ^

[michalpristas]

have this as part of each beat configuration (with according values) will take a look though

processors:
      - add_fields:
          fields:
            stream.type: logs
            stream.dataset: generic
            stream.namespace: default

[michalpristas]

I see them here under fields:

[ruflin]

@michalpristas They must be on the top level and not under fields!

[michalpristas]

let me check what i can do about it, i hope this is not the default behavior of the processor

[michalpristas]

PR here: elastic/beats#17858

[ruflin]

Closing as by now we landed by dataset.name.