Need to Configure AWS for Fluent Bit to Send Raw Logs to S3 for Spark History Server

[vinodkumar-wagh]

Description:
We are using the "aws-for-fluent-bit" application to send logs from an Amazon EKS cluster to an S3 bucket. Our current setup adds Kubernetes metadata (e.g., pod name, namespace, etc.) and additional fields to the logs, which is suitable for most use cases. However, we have a requirement to send logs in their raw format (as generated by the application) for a spark history server, without any added metadata or fields.

The challenge we are facing is twofold:

1. We need to send raw logs (without enrichment) to the S3 bucket for one application.
2. For all other applications, we want to continue sending logs with the current enriched format, including Kubernetes metadata.

We are seeking guidance on how to configure the "aws-for-fluent-bit" application to:
• Process logs as raw data and send them to S3.
• Continue sending enriched logs for all other applications.

Request:
• How can we configure Fluent Bit to handle logs differently based on the Kubernetes app, ensuring raw logs for one app and enriched logs for others?
• Are there best practices or specific configuration recommendations for this use case?