Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

processor: new enhancements, API changes and new processor #8563

Merged
merged 12 commits into from
Mar 12, 2024

Conversation

edsiper
Copy link
Member

@edsiper edsiper commented Mar 9, 2024

Log processing example:

pipeline:
  inputs:
    - name: dummy
      dummy: '{"key1": "123.4", "key2": "true", "key3": "delete me", "message": null, "http.url": "https://www.google.com/search?q=example"}'
      rate: 1

      processors:
        logs:
          - name: content_modifier
            action: insert
            key: "new_key"
            value: "new_value"

          - name: content_modifier
            action: upsert
            key: "message"
            value: "this kv should be at the end"

          - name: content_modifier
            action: delete
            key: "message"

          - name: content_modifier
            action: rename
            key: "new_key"
            value: "new_key_renamed"

          - name: content_modifier
            action: hash
            key: "message"

          - name: content_modifier
            action: extract
            key: "http.url"
            pattern: ^(?<http_protocol>https?):\/\/(?<http_domain>[^\/\?]+)(?<http_path>\/[^?]*)?(?:\?(?<http_query_params>.*))?

          - name: content_modifier
            action: convert
            key: "key2"
            converted_type: boolean

  outputs:
    - name : stdout
      match: '*'

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Signed-off-by: Eduardo Silva <eduardo@calyptia.com>
When iterating events when using the log event decoder, specifically in the
function flb_log_event_decoder_next(), the struct flb_log_event 'event' is
zeroed with memset() before the object is decoded in the function
flb_event_decoder_decode_object(), however the first thing that this function
does is to zero the structure content with an extra memset().

This patch simply remove the first memset() since is not necessary.

Signed-off-by: Eduardo Silva <eduardo@calyptia.com>
Signed-off-by: Eduardo Silva <eduardo@calyptia.com>
When handling logs content, plugins needs to perform several encoding
and decoding phases to have the output chunks as desired. Recently,
a new CFL Object interface was added to represent a single log record
as an editable tree structure that handle all the data types
required, however one piece was missing: a representation of the whole
chunk based in CFL Objects.

This patch introduces the new interface called mp_chunk, that with the
help of the log_encoder, log_decoder and CFL Object interfaces can
represent a whole Chunk, so now processors don't need to worry about
serialization in general.

Signed-off-by: Eduardo Silva <eduardo@calyptia.com>
Looking forward to simplify how processors manipulate data from a chunk,
this patch modifies how 'log processors' handles the data when invoking
the plugins API. Instead of passing a Chunk as normal filters or passing
one message at a time, now the whole Chunk is converted to a mp_chunk
for easy processing. Then log processor simply iterate the content as
CFL Objects where data gets modified in place, serialization is only
done once the processing chain is finished.

Signed-off-by: Eduardo Silva <eduardo@calyptia.com>
Signed-off-by: Eduardo Silva <eduardo@calyptia.com>
This new processor, uses the new mp_chunk interface to manipulate logs,
by defining actions and contexts. For log management, the following
contexts are supported (a context defines which component from the Log
will be modified):

  1. metadata: log metadata will be modified
  2. body: log body content will be modified

The following actions for Logs are available:

  - insert : adds a new key value pair into the context
  - upsert : update or insert a new key value pair into the context
  - delete : remove a key value pair from the context
  - rename : rename a key
  - hash   : replace the value of a key with it SHA-256
  - extract: apply a regular expresion to a specific key value and
             appends the matching groups as extra keys.
  - convert: for a given key, convert it value to a string, boolean,
             int or double.

next patches will introduce similar features for traces and metrics
data types.

Signed-off-by: Eduardo Silva <eduardo@calyptia.com>
Signed-off-by: Eduardo Silva <eduardo@calyptia.com>
Signed-off-by: Eduardo Silva <eduardo@calyptia.com>
…ibutes

Signed-off-by: Eduardo Silva <eduardo@calyptia.com>
Signed-off-by: Eduardo Silva <eduardo@calyptia.com>
Signed-off-by: Eduardo Silva <eduardo@calyptia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant