Populating AWS Lambda Context.clientObject with contents of the X-Amz-Client-Context header

[jimkont]

Describe the bug

According to the AWS Lambda invocation API, a user can pass the X-Amz-Client-Context header and add "Up to 3,583 bytes of base64-encoded data about the invoking client to pass to the function in the context object.ClientContext"

Some OpenTelemetry AWS clients use this header (e.g. here) to populate tracing information.
An example value would be

X-Amz-Client-Context: eyJDdXN0b20iOnsidHJhY2VwYXJlbnQiOiIwMC1kZWQ4ZGUwMzNjYWRlMjY3MThkM2Y5ZjUwMzUwZDg0ZS0xMWU0OTllYTI4NWFkODIwLTAxIn19

which would translate to

{"Custom":{"traceparent":"00-ded8de033cade26718d3f9f50350d84e-11e499ea285ad820-01"}}

and should be populating the Context.clientContext.custom map

Expected behavior

Quarkus should be reading this header and populating the Context object accordingly

Actual behavior

It appears that Quarkus is not reading this header. When populating the Context object, it searches for a different one named "Lambda-Runtime-Client-Context".

Requests that have the header mentioned above result in the Context.clientContext object being null

How to Reproduce?

Using a sample quarkus lambda project, invoke a call by adding a header like

X-Amz-Client-Context: eyJDdXN0b20iOnsidHJhY2VwYXJlbnQiOiIwMC1kZWQ4ZGUwMzNjYWRlMjY3MThkM2Y5ZjUwMzUwZDg0ZS0xMWU0OTllYTI4NWFkODIwLTAxIn19

The context.clientContext object in the handler will be null

Output of uname -a or ver

Darwin K4V32WY17P 21.6.0 Darwin Kernel Version 21.6.0: Mon Dec 19 20:43:09 PST 2022; root:xnu-8020.240.18~2/RELEASE_ARM64_T6000 arm64

Output of java -version

openjdk version "17.0.4.1" 2022-08-12
OpenJDK Runtime Environment Temurin-17.0.4.1+1 (build 17.0.4.1+1)
OpenJDK 64-Bit Server VM Temurin-17.0.4.1+1 (build 17.0.4.1+1, mixed mode)

GraalVM version (if different from Java)

No response

Quarkus version or git rev

2.15.3

Build tool (ie. output of mvnw --version or gradlew --version)

Gradle 7.5.1
Build time: 2022-08-05 21:17:56 UTC
Revision: d1daa0cbf1a0103000b71484e1dbfe096e095918
Kotlin: 1.6.21
Groovy: 3.0.10
Ant: Apache Ant(TM) version 1.10.11 compiled on July 10 2021
JVM: 17.0.4.1 (Eclipse Adoptium 17.0.4.1+1)
OS: Mac OS X 12.6.3 aarch64

Additional information

originated from this discussion: #31552

[quarkus-bot]

/cc @evanchooly (kotlin), @geoand (kotlin), @matejvasek (amazon-lambda), @patriot1burke (amazon-lambda)

[geoand]

@patriot1burke WDYT about this?

Should we be taking X-Amz-Client-Context into account as well?

[patriot1burke]

Looks like it. Should be an easy enough fix.

[jimkont]

Thanks for acknowledging @patriot1burke / @geoand!
I didn't find a lot of documentation around the clientContext object, but this issue (aws/aws-sdk-js#1388 (comment)) describes the expected JSON structure

[gsmet]

@scrocquesel I was wondering if you would be interested in driving this one home?

[scrocquesel]

I was wondering if you would be interested in driving this one home?

Serverless Lambda is out of my expertise, but it seems there might be some connection with the AWS SDK for context propagation from the LambdaClient.

I believe it could be related to the changes made in open-telemetry/opentelemetry-java-instrumentation#11675, which was introduced in version 2.6.0 of opentelemetry-aws-sdk-2.2. However, Quarkus core is still on 2.5.0-alpha for the whole io.opentelemetry.instrumentation group.