x-pack/filebeat/input/awss3: relax queue_url constraints for non-stan…

…dard endpoints (#35520) Currently, the queue_url option is used to obtain the region name for the SQS receiver. This prevents pointing the input at other sources for testing, so add a region_name option to provide an alternative way to provide the region name for non-AWS endpoints.
elastic · Jun 8, 2023 · 7b1a839 · 7b1a839
1 parent 2248267
commit 7b1a839
Show file tree

Hide file tree

Showing 7 changed files with 435 additions and 260 deletions.
diff --git a/CHANGELOG-developer.next.asciidoc b/CHANGELOG-developer.next.asciidoc
@@ -159,6 +159,7 @@ The list below covers the major changes between 7.0.0-rc2 and main only.
 - Add logging of metric registration in inputmon. {pull}35647[35647]
 - Add Okta API package for entity analytics. {pull}35478[35478]
 - Add benchmarking to HTTPJSON input testing. {pull}35138[35138]
+- Allow non-AWS endpoints for testing Filebeat awss3 input. {issue}35496[35496] {pull}35520[35520]
 
 ==== Deprecated
 

diff --git a/CHANGELOG.next.asciidoc b/CHANGELOG.next.asciidoc
@@ -304,6 +304,7 @@ automatic splitting at root level, if root level element is an array. {pull}3415
 - Mark CEL input as GA. {pull}35559[35559]
 - Add metrics for gcp-pubsub input. {pull}35614[35614]
 - [GCS] Added scheduler debug logs and improved the context passing mechanism by removing them from struct params and passing them as function arguments. {pull}35674[35674]
+- Allow non-AWS endpoints for awss3 input. {issue}35496[35496] {pull}35520[35520]
 
 *Auditbeat*
    - Migration of system/package module storage from gob encoding to flatbuffer encoding in bolt db. {pull}34817[34817]

diff --git a/x-pack/filebeat/docs/inputs/input-aws-s3.asciidoc b/x-pack/filebeat/docs/inputs/input-aws-s3.asciidoc
@@ -257,6 +257,12 @@ configuring multiline options.
 
 URL of the AWS SQS queue that messages will be received from. (Required when `bucket_arn` and `non_aws_bucket_name` are not set).
 
+[float]
+==== `region`
+
+The name of the AWS region of the end point. If this option is given it
+takes precedence over the region name obtained from the `queue_url` value.
+
 [float]
 ==== `visibility_timeout`
 

diff --git a/x-pack/filebeat/input/awss3/config.go b/x-pack/filebeat/input/awss3/config.go
@@ -27,6 +27,7 @@ type config struct {
 	SQSScript           *scriptConfig        `config:"sqs.notification_parsing_script"`
 	MaxNumberOfMessages int                  `config:"max_number_of_messages"`
 	QueueURL            string               `config:"queue_url"`
+	RegionName          string               `config:"region"`
 	BucketARN           string               `config:"bucket_arn"`
 	NonAWSBucketName    string               `config:"non_aws_bucket_name"`
 	BucketListInterval  time.Duration        `config:"bucket_list_interval"`