in_splunk: out_splunk: Propagate ingested splunk_token authentication header

[cosmo0920]

Currently, in_splunk does not have a capability to store and propagate the ingested Splunk token.
This could be needed to handle for simplifying logs pipeline for ingesting payloads of Splunk HEC format.

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• Example configuration file for the change

On terminal:

$ bin/fluent-bit -i splunk -p port=8088 -o splunk -p port=8081 -psplunk_token=db496524-e7e6-4ae9-b3f0-2287d8e65cd4 -p "event_key=\$event" -f 1 -v

And another terminal:

$ bin/fluent-bit -i splunk -p port=8081 -o stdout 

• Debug log output from testing the change

Fluent Bit v3.0.3
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

___________.__                        __    __________.__  __          ________  
\_   _____/|  |  __ __   ____   _____/  |_  \______   \__|/  |_  ___  _\_____  \ 
 |    __)  |  | |  |  \_/ __ \ /    \   __\  |    |  _/  \   __\ \  \/ / _(__  < 
 |     \   |  |_|  |  /\  ___/|   |  \  |    |    |   \  ||  |    \   / /       \
 \___  /   |____/____/  \___  >___|  /__|    |______  /__||__|     \_/ /______  /
     \/                     \/     \/               \/                        \/ 

[2024/04/22 14:03:51] [ info] Configuration:
[2024/04/22 14:03:51] [ info]  flush time     | 1.000000 seconds
[2024/04/22 14:03:51] [ info]  grace          | 5 seconds
[2024/04/22 14:03:51] [ info]  daemon         | 0
[2024/04/22 14:03:51] [ info] ___________
[2024/04/22 14:03:51] [ info]  inputs:
[2024/04/22 14:03:51] [ info]      splunk
[2024/04/22 14:03:51] [ info] ___________
[2024/04/22 14:03:51] [ info]  filters:
[2024/04/22 14:03:51] [ info] ___________
[2024/04/22 14:03:51] [ info]  outputs:
[2024/04/22 14:03:51] [ info]      splunk.0
[2024/04/22 14:03:51] [ info] ___________
[2024/04/22 14:03:51] [ info]  collectors:
[2024/04/22 14:03:51] [ info] [fluent bit] version=3.0.3, commit=b134bf3bed, pid=199949
[2024/04/22 14:03:51] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2024/04/22 14:03:51] [ info] [storage] ver=1.1.6, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/04/22 14:03:51] [ info] [cmetrics] version=0.8.0
[2024/04/22 14:03:51] [ info] [ctraces ] version=0.5.0
[2024/04/22 14:03:51] [ info] [input:splunk:splunk.0] initializing
[2024/04/22 14:03:51] [ info] [output:splunk:splunk.0] worker #1 started
[2024/04/22 14:03:51] [ info] [output:splunk:splunk.0] worker #0 started
[2024/04/22 14:03:51] [ info] [input:splunk:splunk.0] storage_strategy='memory' (memory only)
[2024/04/22 14:03:51] [debug] [splunk:splunk.0] created event channels: read=21 write=22
[2024/04/22 14:03:51] [debug] [downstream] listening on 0.0.0.0:8088
[2024/04/22 14:03:51] [debug] [splunk:splunk.0] created event channels: read=24 write=25
[2024/04/22 14:03:51] [ info] [sp] stream processor started
[2024/04/22 14:04:10] [debug] [task] created task=0x5f8e3b0 id=0 OK
[2024/04/22 14:04:10] [debug] [output:splunk:splunk.0] task_id=0 assigned to thread #0
[2024/04/22 14:04:10] [debug] [upstream] KA connection #49 to 127.0.0.1:8081 is connected
[2024/04/22 14:04:10] [debug] [output:splunk:splunk.0] Could not find a record accessor definition of hec_token
[2024/04/22 14:04:10] [debug] [http_client] not using http_proxy for header
[2024/04/22 14:04:10] [debug] [upstream] KA connection #49 to 127.0.0.1:8081 is now available
[2024/04/22 14:04:10] [debug] [upstream] KA connection #49 to 127.0.0.1:8081 has been disconnected by the remote service
[2024/04/22 14:04:10] [debug] [task] destroy task=0x5f8e3b0 (task_id=0)
[2024/04/22 14:04:10] [debug] [out flush] cb_destroy coro_id=0
[2024/04/22 14:04:11] [debug] [task] created task=0x6002a40 id=0 OK
[2024/04/22 14:04:11] [debug] [output:splunk:splunk.0] task_id=0 assigned to thread #1
[2024/04/22 14:04:11] [debug] [upstream] KA connection #49 to 127.0.0.1:8081 is connected
[2024/04/22 14:04:11] [debug] [output:splunk:splunk.0] Could not find a record accessor definition of hec_token
[2024/04/22 14:04:11] [debug] [http_client] not using http_proxy for header
[2024/04/22 14:04:11] [debug] [upstream] KA connection #49 to 127.0.0.1:8081 is now available
[2024/04/22 14:04:11] [debug] [upstream] KA connection #49 to 127.0.0.1:8081 has been disconnected by the remote service
[2024/04/22 14:04:11] [debug] [out flush] cb_destroy coro_id=0
[2024/04/22 14:04:11] [debug] [task] destroy task=0x6002a40 (task_id=0)
[2024/04/22 14:04:12] [debug] [task] created task=0x60859d0 id=0 OK
[2024/04/22 14:04:12] [debug] [output:splunk:splunk.0] task_id=0 assigned to thread #0
[2024/04/22 14:04:12] [debug] [upstream] KA connection #49 to 127.0.0.1:8081 is connected
[2024/04/22 14:04:12] [debug] [output:splunk:splunk.0] Could not find a record accessor definition of hec_token
[2024/04/22 14:04:12] [debug] [output:splunk:splunk.0] Could not find a record accessor definition of hec_token
[2024/04/22 14:04:12] [debug] [http_client] not using http_proxy for header
[2024/04/22 14:04:12] [debug] [upstream] KA connection #49 to 127.0.0.1:8081 is now available
[2024/04/22 14:04:12] [debug] [upstream] KA connection #49 to 127.0.0.1:8081 has been disconnected by the remote service
[2024/04/22 14:04:12] [debug] [out flush] cb_destroy coro_id=1
[2024/04/22 14:04:12] [debug] [task] destroy task=0x60859d0 (task_id=0)
^C[2024/04/22 14:04:14] [engine] caught signal (SIGINT)
[2024/04/22 14:04:14] [ warn] [engine] service will shutdown in max 5 seconds
[2024/04/22 14:04:14] [ info] [input] pausing splunk.0
[2024/04/22 14:04:15] [ info] [engine] service has stopped (0 pending tasks)
[2024/04/22 14:04:15] [ info] [input] pausing splunk.0
[2024/04/22 14:04:15] [ info] [output:splunk:splunk.0] thread worker #0 stopping...
[2024/04/22 14:04:15] [ info] [output:splunk:splunk.0] thread worker #0 stopped
[2024/04/22 14:04:15] [ info] [output:splunk:splunk.0] thread worker #1 stopping...
[2024/04/22 14:04:15] [ info] [output:splunk:splunk.0] thread worker #1 stopped

Fluent Bit v3.0.3
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

___________.__                        __    __________.__  __          ________  
\_   _____/|  |  __ __   ____   _____/  |_  \______   \__|/  |_  ___  _\_____  \ 
 |    __)  |  | |  |  \_/ __ \ /    \   __\  |    |  _/  \   __\ \  \/ / _(__  < 
 |     \   |  |_|  |  /\  ___/|   |  \  |    |    |   \  ||  |    \   / /       \
 \___  /   |____/____/  \___  >___|  /__|    |______  /__||__|     \_/ /______  /
     \/                     \/     \/               \/                        \/ 

[2024/04/22 14:04:05] [ info] Configuration:
[2024/04/22 14:04:05] [ info]  flush time     | 1.000000 seconds
[2024/04/22 14:04:05] [ info]  grace          | 5 seconds
[2024/04/22 14:04:05] [ info]  daemon         | 0
[2024/04/22 14:04:05] [ info] ___________
[2024/04/22 14:04:05] [ info]  inputs:
[2024/04/22 14:04:05] [ info]      splunk
[2024/04/22 14:04:05] [ info] ___________
[2024/04/22 14:04:05] [ info]  filters:
[2024/04/22 14:04:05] [ info] ___________
[2024/04/22 14:04:05] [ info]  outputs:
[2024/04/22 14:04:05] [ info]      stdout.0
[2024/04/22 14:04:05] [ info] ___________
[2024/04/22 14:04:05] [ info]  collectors:
[2024/04/22 14:04:05] [ info] [fluent bit] version=3.0.3, commit=b134bf3bed, pid=199984
[2024/04/22 14:04:05] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2024/04/22 14:04:05] [ info] [storage] ver=1.1.6, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/04/22 14:04:05] [ info] [cmetrics] version=0.8.0
[2024/04/22 14:04:05] [ info] [ctraces ] version=0.5.0
[2024/04/22 14:04:05] [ info] [input:splunk:splunk.0] initializing
[2024/04/22 14:04:05] [ info] [input:splunk:splunk.0] storage_strategy='memory' (memory only)
[2024/04/22 14:04:05] [debug] [splunk:splunk.0] created event channels: read=21 write=22
[2024/04/22 14:04:05] [debug] [downstream] listening on 0.0.0.0:8081
[2024/04/22 14:04:05] [debug] [stdout:stdout.0] created event channels: read=24 write=25
[2024/04/22 14:04:05] [ info] [sp] stream processor started
[2024/04/22 14:04:05] [ info] [output:stdout:stdout.0] worker #0 started
[2024/04/22 14:04:11] [debug] [task] created task=0x5f303d0 id=0 OK
[2024/04/22 14:04:11] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[0] splunk.0: [[1713762250.731699688, {"hec_token"=>"Splunk db496524-e7e6-4ae9-b3f0-2287d8e65cd4"}], {"time"=>1713762249.733121, "event"=>"{"message":"dummy"}"}]
[2024/04/22 14:04:11] [debug] [out flush] cb_destroy coro_id=0
[2024/04/22 14:04:11] [debug] [task] destroy task=0x5f303d0 (task_id=0)
[0] splunk.0: [[1713762251.613924063, {"hec_token"=>"Splunk db496524-e7e6-4ae9-b3f0-2287d8e65cd4"}], {"time"=>1713762250.637001, "event"=>"{"message":"dummy"}"}]
[2024/04/22 14:04:12] [debug] [task] created task=0x5f94f40 id=0 OK
[2024/04/22 14:04:12] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[2024/04/22 14:04:12] [debug] [out flush] cb_destroy coro_id=1
[2024/04/22 14:04:12] [debug] [task] destroy task=0x5f94f40 (task_id=0)
[2024/04/22 14:04:13] [debug] [task] created task=0x5ffba40 id=0 OK
[2024/04/22 14:04:13] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[0] splunk.0: [[1713762252.610231383, {"hec_token"=>"Splunk db496524-e7e6-4ae9-b3f0-2287d8e65cd4"}], {"time"=>1713762251.610019, "event"=>"{"message":"dummy"}"}]
[1] splunk.0: [[1713762252.610231383, {"hec_token"=>"Splunk db496524-e7e6-4ae9-b3f0-2287d8e65cd4"}], {"time"=>1713762252.038443, "event"=>"{"message":"dummy"}"}]
[2024/04/22 14:04:13] [debug] [out flush] cb_destroy coro_id=2
[2024/04/22 14:04:13] [debug] [task] destroy task=0x5ffba40 (task_id=0)
^C[2024/04/22 14:04:18] [engine] caught signal (SIGINT)
[2024/04/22 14:04:18] [ warn] [engine] service will shutdown in max 5 seconds
[2024/04/22 14:04:18] [ info] [input] pausing splunk.0
[2024/04/22 14:04:18] [ info] [engine] service has stopped (0 pending tasks)
[2024/04/22 14:04:18] [ info] [input] pausing splunk.0
[2024/04/22 14:04:18] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2024/04/22 14:04:18] [ info] [output:stdout:stdout.0] thread worker #0 stopped

• Attached Valgrind output that shows no leaks or memory corruption was found

==199949== 
==199949== HEAP SUMMARY:
==199949==     in use at exit: 0 bytes in 0 blocks
==199949==   total heap usage: 3,699 allocs, 3,699 frees, 2,448,482 bytes allocated
==199949== 
==199949== All heap blocks were freed -- no leaks are possible
==199949== 
==199949== For lists of detected and suppressed errors, rerun with: -s
==199949== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

==199984== 
==199984== HEAP SUMMARY:
==199984==     in use at exit: 0 bytes in 0 blocks
==199984==   total heap usage: 3,174 allocs, 3,174 frees, 1,897,854 bytes allocated
==199984== 
==199984== All heap blocks were freed -- no leaks are possible
==199984== 
==199984== For lists of detected and suppressed errors, rerun with: -s
==199984== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• Run local packaging test showing all targets (including any new ones) build.
• Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

• Documentation required for this feature

Backporting

• Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[edsiper]

thank you