AWS CLI uses streaming trailer chunked requests on newer versions

[halkyon]

I upgraded AWS CLI client and started getting MissingContentLength errors. Debugging the request, it seems like it's using STREAMING-UNSIGNED-PAYLOAD-TRAILER or STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER requests, but we don't support these.

PUT
/test-files/testfile.dat

content-encoding:aws-chunked
host:gateway.storjshare.io
transfer-encoding:chunked
x-amz-content-sha256:STREAMING-UNSIGNED-PAYLOAD-TRAILER
x-amz-date:20250121T032638Z
x-amz-decoded-content-length:1048576
x-amz-sdk-checksum-algorithm:CRC64NVME
x-amz-trailer:x-amz-checksum-crc64nvme

An error occurred (MissingContentLength) when calling the PutObject operation: You must provide the Content-Length HTTP header.

AWS CLI version: aws-cli/2.23.2 Python/3.12.6 Linux/6.8.0-51-generic exe/x86_64.ubuntu.24.

I downgraded the client to the previous version (aws-cli/2.15.21 Python/3.11.6 Linux/6.8.0-51-generic exe/x86_64.ubuntu.24 prompt/off) and it worked again. I guess AWS CLI decides to use the streaming payload trailer on newer versions. I couldn't find a workaround to turn it off, short of downgrading the client.

Test command: aws s3 --endpoint https://gateway.storjshare.io cp /tmp/testfile.dat s3://test-files/testfile.dat (1M file), and no config changes in ~/.aws/config. It seems any size file it tries to stream, and also the same for multi-part uploads.

References:

• AWS docs: https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming-trailers.html
• recent minio fix: minio/minio@76913a9
• https://review.dev.storj.io/c/storj/minio/+/6842

[halkyon]

https://raw.githubusercontent.com/aws/aws-cli/v2/CHANGELOG.rst

2.23.0
...
* feature:``s3``: The S3 client attempts to validate response checksums for all S3 API operations that support checksums. However, if the SDK has not implemented the specified checksum algorithm then this validation is skipped. Checksum validation behavior can be configured using the ``when_supported`` and ``when_required`` options - in the shared AWS config file using ``response_checksum_validation``, and as an env variable using ``AWS_RESPONSE_CHECKSUM_VALIDATION``.
...

Maybe this was the change?

[halkyon]

Artur and I discovered it changed in 2.23.0. It works in 2.22.35, which was the last version before it: https://github.com/aws/aws-cli/compare/2.22.35..2.23.0

[halkyon]

Related: aws/aws-cli#9214

[halkyon]

we also need to support the new checksum types, minio fixed this upstream here: minio/minio@827004c