core: add log groups support and full OTLP <--> OTLP #8898
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Eduardo Silva <eduardo@calyptia.com>
As of today, log records can support metadata content in addition to the record
content it self, however for some cases this is not enough if we want to share
metadata across a group of records.
Today we serialize log records in the following pseudo-schema:
[ [TIMESTAMP, { METADATA }], { RECORD_CONTENT}]
A common chunk contain records in the following way:
[ [1717104810, {}], {"key": "some value", "number": 12} ]
[ [1717104811, {"color": "blue" }], {"key": "some value", "number": 13} ]
[ [1717104812, {"color": "red" }], {"key": "some value", "number": 14} ]
[ [1717104813, {"color": "green"}], {"key": "some value", "number": 15} ]
[ [1717104814, {}], {"key": "some value", "number": 16} ]
In this patch, we are introducing the concept of `groups`, which are implemented
through a new type of records that marks the beginning and the end of the group. To
avoid breaking changes and overall compatibility, we use the TIMESTAMP field to
mark those as special records.
- start of a group: [ [ -1, { METADATA }, { RECORD_CONTENT} ]
- end of a group : [ [ -2, { METADATA }, { RECORD_CONTENT} ]
Here is an example where the middle records are group together:
[ [1717104810, {}], {"key": "some value", "number": 12} ]
[ [ -1, {"type": "colors"}], {"numbers": true}]
[ [1717104811, {"color": "blue" }], {"key": "some value", "number": 13} ]
[ [1717104812, {"color": "red" }], {"key": "some value", "number": 14} ]
[ [1717104813, {"color": "green"}], {"key": "some value", "number": 15} ]
[ [ -2, {}], {}]
[ [1717104814, {}], {"key": "some value", "number": 16} ]
Iterating the records don't introduce any problems, however is up to the plugins like
inputs, processors, filters and outputs to interpret the group fields or simply skip
them. note: in the next patch the decoder offers a new API to skip groups definitions.
API usage
=========
To manipulate groups, the workflow is as follows:
1. A group gets initialized (the header is opened)
2. the user/dev can optionally add content in the METADATA or RECORD_CONTENT fields.
3. the group header is finalized. Close the header, the group is still open.
4. Append normal records.
5. Group is finalized.
These are the new functions available for groups handling:
int flb_log_event_encoder_group_init(struct flb_log_event_encoder *context);
int flb_log_event_encoder_group_header_end(struct flb_log_event_encoder *context)
int flb_log_event_encoder_group_end(struct flb_log_event_encoder *context)
Signed-off-by: Eduardo Silva <eduardo@calyptia.com>
There are cases where the parser might interpret different the incoming string buffer (e.g: $), by passing a fresh copy of the pattern workaround the problem. There are no performance penalties since this happens on the record accessor context creation, not when doing the lookups. note: the other workaround might be to tweak the parser which is more work. Signed-off-by: Eduardo Silva <eduardo@calyptia.com>
In recent patch, we introduced the concept of groups. This patch implements
the following changes to the log event decoder:
- decoder now has a flag to read or skip groups definitions (default: on)
- new API to retrieve the type of record being read
new functions available:
int flb_log_event_decoder_read_groups(struct flb_log_event_decoder *context,
int read_groups)
int flb_log_event_decoder_get_record_type(struct flb_log_event *event,
int32_t *type)
record types supported:
- FLB_LOG_EVENT_NORMAL
- FLB_LOG_EVENT_GROUP_START
- FLB_LOG_EVENT_GROUP_END
Signed-off-by: Eduardo Silva <eduardo@calyptia.com>
Signed-off-by: Eduardo Silva <eduardo@calyptia.com>
The following patch, enhance the logs handling by using the new Fluent Bit groups support for log events. This provides a smooth communication and translation layer betweeen in_opentelemetry and out_opentelemetry. This patch also adds a new configuration property called `logs_metadata_key` which defines which metadata key holds specific OTLP data per record. The changes are supported for gRPC payloads, upcoming patches will add support for JSON payloads. Signed-off-by: Eduardo Silva <eduardo@calyptia.com>
…port The following patch introduce a big change on how the plugin now process the logs with OTLP format, the high-level features are: 1. Full metadata support: - with the new changes in in_opentelemetry, there is no metadata loss 2. Proper encoding of non-OTLP records as OTLP - The plugin is flexible enough to encode any type of log record. At a low level, we re-architect how resource logs, resource, span_logs, scope and logs are handled. All of this has been implemented on top of the new Fluent Bit groups support for logs. Use cases tested: - OTel Collector (gRPC) --> Fluent Bit --> OTel Collector (gRPC) - dummy, tail, forward --> Fluent Bit --> OTel Collector (gRPC) Signed-off-by: Eduardo Silva <eduardo@calyptia.com>
edsiper
requested review from
leonardo-albertovich,
fujimotos and
koleini
as code owners
4 tasks
Signed-off-by: Eduardo Silva <eduardo@calyptia.com>
Signed-off-by: Eduardo Silva <eduardo@calyptia.com>
edsiper
force-pushed
the
otlp-metadata-end-to-end
branch
from
7dd7d2b
to
c30810e
Compare
|
I tried this on unstable:master, the |
|
@alrz this is supported in otel -> otel, do you want to send otel -> elasticsearch ? the elastic plugin will need to be updated to receive that extra metadata. |
3 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a big patch, every commit has been properly documented. Here is a highlight of the changes in this PR:
groupsconcepts for logs serialization. Now, it is possible to group records that share metadata.These changes will be included in the next v3.1 release which is scheduled for the end of June (or earlier).
Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.