Etag in metadata is assumed to be hex string

[sudeeptoroy]

While this is mostly true that the etag is hex string. But RFC define etag as "opaque quoted string"

https://tools.ietf.org/html/rfc7232#page-9

In AmazonS3Client.java getEtag usage can cause issues if metadata.etag is not hex as it's assumed in the entire file.

[shorea]

What problems is this causing? Are you using the S3 client with an alternative S3 implementation?

[sudeeptoroy]

This is used with apm and they tend to attach some metadata for their internal usage.
This breaks this hex assumption.
BinaryUtils.fromHex(s3Object.getObjectMetadata().getETag());

Not sure how easy is will it be for you to refactor this code.

[millems]

1. Do you have an example ETag value that is causing the problematic behavior?
2. What behavior would you expect in this case (obviously not an exception).

[sudeeptoroy]

Have a look at the etag examples below in the webpage.
https://answers.dynatrace.com/spaces/148/uem-open-q-a_2/questions/153939/uem-sensor-deducting-1s-from-last-modified-respons.html
I could not attach the print screen.
However looks like latest version of dynatrace do not add append to the etag so the exception is not coming after upgrade. However this situation should be handled gracefully and better to expect etag as opaque than a hex value.

[millems]

Unfortunately S3 uses the etag for checksum validation, so if it's changed we can't validate the checksum. You can disable GET checksum validation with the com.amazonaws.services.s3.disableGetObjectMD5Validation system property and PUT checksum validation with the com.amazonaws.services.s3.disablePutObjectMD5Validation system property, but that's mostly a workaround.

I'm not sure what we could do if they're changing the etag. Do you have any suggestions as to what you'd expect the SDK to do?

[sudeeptoroy]

Hi, thanks for the information. At this point in time I guess it's fine as they moved out of etag. However if we can do something to handle this gracefully would help. Like make this a key value so that others can modify it and u can still extract ur value from it. Just a suggestion.

[millems]

It's tricky to handle people mutating things in ways we can't predict. They could prepend data, append data, override it altogether, etc. If it was a predictable, standardized mutation, like "data is always appended with a semicolon in front of it", we could probably handle it, but that's not guaranteed here. I'm glad you found a workaround!