-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
processor: new enhancements, API changes and new processor #8563
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Eduardo Silva <eduardo@calyptia.com>
When iterating events when using the log event decoder, specifically in the function flb_log_event_decoder_next(), the struct flb_log_event 'event' is zeroed with memset() before the object is decoded in the function flb_event_decoder_decode_object(), however the first thing that this function does is to zero the structure content with an extra memset(). This patch simply remove the first memset() since is not necessary. Signed-off-by: Eduardo Silva <eduardo@calyptia.com>
Signed-off-by: Eduardo Silva <eduardo@calyptia.com>
When handling logs content, plugins needs to perform several encoding and decoding phases to have the output chunks as desired. Recently, a new CFL Object interface was added to represent a single log record as an editable tree structure that handle all the data types required, however one piece was missing: a representation of the whole chunk based in CFL Objects. This patch introduces the new interface called mp_chunk, that with the help of the log_encoder, log_decoder and CFL Object interfaces can represent a whole Chunk, so now processors don't need to worry about serialization in general. Signed-off-by: Eduardo Silva <eduardo@calyptia.com>
Looking forward to simplify how processors manipulate data from a chunk, this patch modifies how 'log processors' handles the data when invoking the plugins API. Instead of passing a Chunk as normal filters or passing one message at a time, now the whole Chunk is converted to a mp_chunk for easy processing. Then log processor simply iterate the content as CFL Objects where data gets modified in place, serialization is only done once the processing chain is finished. Signed-off-by: Eduardo Silva <eduardo@calyptia.com>
Signed-off-by: Eduardo Silva <eduardo@calyptia.com>
This new processor, uses the new mp_chunk interface to manipulate logs, by defining actions and contexts. For log management, the following contexts are supported (a context defines which component from the Log will be modified): 1. metadata: log metadata will be modified 2. body: log body content will be modified The following actions for Logs are available: - insert : adds a new key value pair into the context - upsert : update or insert a new key value pair into the context - delete : remove a key value pair from the context - rename : rename a key - hash : replace the value of a key with it SHA-256 - extract: apply a regular expresion to a specific key value and appends the matching groups as extra keys. - convert: for a given key, convert it value to a string, boolean, int or double. next patches will introduce similar features for traces and metrics data types. Signed-off-by: Eduardo Silva <eduardo@calyptia.com>
edsiper
requested review from
niedbalski,
fujimotos,
patrick-stephens,
celalettin1286,
leonardo-albertovich and
koleini
as code owners
March 9, 2024 21:03
Signed-off-by: Eduardo Silva <eduardo@calyptia.com>
Signed-off-by: Eduardo Silva <eduardo@calyptia.com>
…ibutes Signed-off-by: Eduardo Silva <eduardo@calyptia.com>
Signed-off-by: Eduardo Silva <eduardo@calyptia.com>
Signed-off-by: Eduardo Silva <eduardo@calyptia.com>
edsiper
force-pushed
the
processor-log-attributes
branch
from
March 12, 2024 05:02
2cf36e9
to
897510d
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Log processing example:
Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.