feat: support default checksums #1475
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… jmespath-flatten
… flexible-checksums
|
A new generated diff is ready to view. |
|
A new generated diff is ready to view. |
lauzadis
requested changes
Dec 19, 2024
| import aws.smithy.kotlin.runtime.http.interceptors.FlexibleChecksumsResponseInterceptor | ||
|
|
||
| /** | ||
| * S3 variant of the flexible checksum interceptor where composite checksums are not validated |
There was a problem hiding this comment.
nit/docs: ... of the [FlexibleChecksumsResponseInterceptor]...
There was a problem hiding this comment.
correctness: This file should go in services/s3/common/src/aws/sdk/kotlin/services/s3/internal with all the other S3-specific interceptors
| /** | ||
| * Determines if an operation is set up to send flexible request checksums | ||
| */ | ||
| private fun requestChecksumsConfigured(ctx: ProtocolGenerator.GenerationContext, op: OperationShape): Boolean { |
There was a problem hiding this comment.
naming: "configured" implies that this is dependent on user behavior, but it's just based on the model. I suggest making this an extension function OperationShape.hasRequestAlgorithmMember(...)
Comment on lines
96
to
100
| val interceptor = if (ctx.model.expectShape<ServiceShape>(ctx.settings.service).isS3) { | ||
| AwsRuntimeTypes.Http.Interceptors.S3FlexibleChecksumResponseInterceptor | ||
| } else { | ||
| RuntimeTypes.HttpClient.Interceptors.FlexibleChecksumsResponseInterceptor | ||
| } |
There was a problem hiding this comment.
docs: let's add a comment here about why S3 needs a custom interceptor
There was a problem hiding this comment.
Now that I look at this again, we could replace S3FlexibleChecksumResponseInterceptor with something more generic that could be applied to other services. In case other services start sending composite checksums in the future. What does the team think?
Comment on lines
36
to
37
| private fun usingS3Express(executionContext: ExecutionContext): Boolean = | ||
| executionContext.getOrNull(AttributeKey(S3_EXPRESS_ENDPOINT_PROPERTY_KEY)) != S3_EXPRESS_ENDPOINT_PROPERTY_VALUE |
There was a problem hiding this comment.
style: Could be an extension function off ExecutionContext
Comment on lines
+21
to
+23
| internal class S3ExpressDefaultChecksumAlgorithm( | ||
| private val isS3UploadPart: Boolean, | ||
| ) : HttpInterceptor { |
There was a problem hiding this comment.
style: This works but I think the previous implementation using two different interceptors was cleaner. It's not clear to me why this had to deviate so much from what we already had
Comment on lines
+43
to
+48
| - name: Save Test Reports | ||
| if: failure() | ||
| uses: actions/upload-artifact@v3 | ||
| with: | ||
| name: test-reports | ||
| path: '**/build/reports' |
There was a problem hiding this comment.
What are these test reports used for?
|
A new generated diff is ready to view. |
|
A new generated diff is ready to view. |
This comment has been minimized.
This comment has been minimized.
|
A new generated diff is ready to view. |
This comment has been minimized.
This comment has been minimized.
|
A new generated diff is ready to view. |
This comment has been minimized.
This comment has been minimized.
lauzadis
reviewed
Jan 9, 2025
buildSrc/build.gradle.kts
Outdated
There was a problem hiding this comment.
I'm still a little concerned about this new buildSrc and the fact that it's treated as an include-build by every module. This means every module will have the Kotlin JVM plugin applied to it which shouldn't be necessary everywhere.
https://docs.gradle.org/current/userguide/organizing_gradle_projects.html#sec:build_sources
Can we accomplish the same thing without using buildSrc?
There was a problem hiding this comment.
Discussed offline and we're keeping buildSrc for now
Comment on lines
+44
to
+47
| // FIXME: Enable test | ||
| // Seeing: S3Exception: Checksum Type mismatch occurred, expected checksum Type: null, actual checksum Type: crc32 | ||
| // ListParts has a strange pagination termination condition via [IsTerminated]. Verify it actually works correctly. | ||
| @Ignore |
There was a problem hiding this comment.
When can this test be enabled? Is it still broken?
There was a problem hiding this comment.
Still broken, left a comment about why on slack
|
A new generated diff is ready to view. |
ianbotsf
reviewed
Jan 9, 2025
Comment on lines
212
to
222
| /** | ||
| * Configures request checksum calculation | ||
| */ | ||
| public val AwsRequestChecksumCalculation: EnvironmentSetting<String> = | ||
| strEnvSetting("aws.requestChecksumCalculation", "AWS_REQUEST_CHECKSUM_CALCULATION") | ||
|
|
||
| /** | ||
| * Configures response checksum validation | ||
| */ | ||
| public val AwsResponseChecksumValidation: EnvironmentSetting<String> = | ||
| strEnvSetting("aws.responseChecksumValidation", "AWS_RESPONSE_CHECKSUM_VALIDATION") |
There was a problem hiding this comment.
Question: Why use strEnvSetting for these when they're enum based? Can't we just use enumEnvSetting so you don't have to write custom parsing/matching code?
There was a problem hiding this comment.
Well that's handy. I guess I forgot/couldn't find this type. Let's start using it
This comment has been minimized.
This comment has been minimized.
|
|
A new generated diff is ready to view. |
Affected ArtifactsSignificantly increased in size
Changed in size
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue #
Description of changes
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.