Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Next.JS 14.2.23 - Importing @aws-sdk/crc64-nvme-crt" as required but fails to fetch S3 objects. #6822

Open
3 of 4 tasks
arealesramirez opened this issue Jan 17, 2025 · 9 comments
Open
3 of 4 tasks

Next.JS 14.2.23 - Importing @aws-sdk/crc64-nvme-crt" as required but fails to fetch S3 objects. #6822

arealesramirez opened this issue Jan 17, 2025 · 9 comments
Labels
bug This issue is a bug. investigating Issue is being investigated and/or work is in progress to resolve the issue. p1 This is a high priority issue

Comments

@arealesramirez
Copy link

arealesramirez commented Jan 17, 2025 •
edited by trivikr
Loading

Checkboxes for prior research

Describe the bug

I have a project in Next.JS 14.2.23 facing the following error using NodeJS v20.10.0

Error: Please check whether you have installed the "@aws-sdk/crc64-nvme-crt" package explicitly. 
You must also register the package by calling [require("@aws-sdk/crc64-nvme-crt");] or an ESM equivalent such as [import "@aws-sdk/crc64-nvme-crt";]. 
For more information please go to https://github.com/aws/aws-sdk-js-v3#functionality-requiring-aws-common-runtime-crt
    at selectChecksumAlgorithmFunction (/Users/andres/Repos/my-project/node_modules/@aws-sdk/middleware-flexible-checksums/dist-cjs/index.js:209:15)
    at validateChecksumFromResponse (/Users/andres/Repos/my-project/node_modules/@aws-sdk/middleware-flexible-checksums/dist-cjs/index.js:411:35)
    at /Users/andres/Repos/my-project/node_modules/@aws-sdk/middleware-flexible-checksums/dist-cjs/index.js:466:11
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /Users/andres/Repos/my-project/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:174:20
    at async /Users/andres/Repos/my-project/node_modules/@smithy/middleware-serde/dist-cjs/index.js:33:24
    at async /Users/andres/Repos/my-project/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:483:18
    at async /Users/andres/Repos/my-project/node_modules/@smithy/middleware-retry/dist-cjs/index.js:321:38
    at async /Users/andres/Repos/my-project/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:109:22
    at async /Users/andres/Repos/my-project/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:136:14
    at async /Users/andres/Repos/my-project/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:33:22
    at async getFileStreamFromS3 (webpack-internal:///(rsc)/./services/s3Service.ts:57:22)
    at async GET (webpack-internal:///(rsc)/./app/api/submissions/[submissionId]/file/route.ts:34:28)
    at async /Users/andres/Repos/my-project/node_modules/next/dist/compiled/next-server/app-route.runtime.dev.js:6:57228
    at async eT.execute (/Users/andres/Repos/my-project/node_modules/next/dist/compiled/next-server/app-route.runtime.dev.js:6:46851)
    at async eT.handle (/Users/andres/Repos/my-project/node_modules/next/dist/compiled/next-server/app-route.runtime.dev.js:6:58760)
    at async doRender (/Users/andres/Repos/my-project/node_modules/next/dist/server/base-server.js:1366:42)
    at async cacheEntry.responseCache.get.routeKind (/Users/andres/Repos/my-project/node_modules/next/dist/server/base-server.js:1576:40)
    at async DevServer.renderToResponseWithComponentsImpl (/Users/andres/Repos/my-project/node_modules/next/dist/server/base-server.js:1496:28)
    at async DevServer.renderPageComponent (/Users/andres/Repos/my-project/node_modules/next/dist/server/base-server.js:1924:24)
    at async DevServer.renderToResponseImpl (/Users/andres/Repos/my-project/node_modules/next/dist/server/base-server.js:1962:32)
    at async DevServer.pipeImpl (/Users/andres/Repos/my-project/node_modules/next/dist/server/base-server.js:922:25)
    at async NextNodeServer.handleCatchallRenderRequest (/Users/andres/Repos/my-project/node_modules/next/dist/server/next-server.js:272:17)
    at async DevServer.handleRequestImpl (/Users/andres/Repos/my-project/node_modules/next/dist/server/base-server.js:818:17)
    at async /Users/andres/Repos/my-project/node_modules/next/dist/server/dev/next-dev-server.js:339:20
    at async Span.traceAsyncFn (/Users/andres/Repos/my-project/node_modules/next/dist/trace/trace.js:154:20)
    at async DevServer.handleRequest (/Users/andres/Repos/my-project/node_modules/next/dist/server/dev/next-dev-server.js:336:24)
    at async invokeRender (/Users/andres/Repos/my-project/node_modules/next/dist/server/lib/router-server.js:173:21)
    at async handleRequest (/Users/andres/Repos/my-project/node_modules/next/dist/server/lib/router-server.js:350:24)
    at async requestHandlerImpl (/Users/andres/Repos/my-project/node_modules/next/dist/server/lib/router-server.js:374:13)
    at async Server.requestListener (/Users/andres/Repos/my-project/node_modules/next/dist/server/lib/start-server.js:141:13) {
  '$metadata': { attempts: 1, totalRetryDelay: 0 }
}

This error occurs when I trigger the getFileStreamFromS3 function, which is a straightforward function that fetches an object from s3

import {
  GetObjectCommand,
  PutObjectCommand,
  PutObjectCommandOutput,
  S3Client,
} from "@aws-sdk/client-s3";
import "@ungap/with-resolvers";
import { pdf } from "@/utils/pdf-to-img";
const s3Client = new S3Client({
  region: process.env.AWS_REGION,
  credentials: {
    accessKeyId: process.env.AWS_ACCESS_KEY_ID,
    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
  },
});


export const getFileStreamFromS3 = async (
  key: string
): Promise<ReadableStream> => {
  // await import("@aws-sdk/crc64-nvme-crt");

  const command = new GetObjectCommand({
    Bucket: process.env.AWS_S3_BUCKET_NAME!,
    Key: key,
    ChecksumMode: "ENABLED",
  });

  const response = await s3Client.send(command);

  if (!response.Body) {
    throw new Error("No body returned from S3");
  }

  return response.Body as ReadableStream;
};

// + other functions

According to the error message, I need to install and import @aws-sdk/crc64-nvme-crt, which I did

import "@aws-sdk/crc64-nvme-crt";

However, the error keeps showing up.

Here are the package versions I'm using:

        "@aws-sdk/client-s3": "^3.705.0",
        "@aws-sdk/crc64-nvme-crt": "^3.730.0",

Regression Issue

  • Select this option if this issue appears to be a regression.

SDK version number

@aws-sdk/[email protected], @aws-sdk/crc64-nvme-crt": "^3.730.0

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

v20.10.0

Reproduction Steps

Trigger the following function

export const getFileStreamFromS3 = async (
  key: string
): Promise<ReadableStream> => {
  // await import("@aws-sdk/crc64-nvme-crt");

  const command = new GetObjectCommand({
    Bucket: process.env.AWS_S3_BUCKET_NAME!,
    Key: key,
    ChecksumMode: "ENABLED",
  });

  const response = await s3Client.send(command);

  if (!response.Body) {
    throw new Error("No body returned from S3");
  }

  return response.Body as ReadableStream;
};

And it fails after triggering

  const response = await s3Client.send(command);

Observed Behavior

Here is the error thrown

Error: Please check whether you have installed the "@aws-sdk/crc64-nvme-crt" package explicitly. 
You must also register the package by calling [require("@aws-sdk/crc64-nvme-crt");] or an ESM equivalent such as [import "@aws-sdk/crc64-nvme-crt";]. 
For more information please go to https://github.com/aws/aws-sdk-js-v3#functionality-requiring-aws-common-runtime-crt
    at selectChecksumAlgorithmFunction (/Users/andres/Repos/my-project/node_modules/@aws-sdk/middleware-flexible-checksums/dist-cjs/index.js:209:15)
    at validateChecksumFromResponse (/Users/andres/Repos/my-project/node_modules/@aws-sdk/middleware-flexible-checksums/dist-cjs/index.js:411:35)
    at /Users/andres/Repos/my-project/node_modules/@aws-sdk/middleware-flexible-checksums/dist-cjs/index.js:466:11
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /Users/andres/Repos/my-project/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:174:20
    at async /Users/andres/Repos/my-project/node_modules/@smithy/middleware-serde/dist-cjs/index.js:33:24
    at async /Users/andres/Repos/my-project/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:483:18
    at async /Users/andres/Repos/my-project/node_modules/@smithy/middleware-retry/dist-cjs/index.js:321:38
    at async /Users/andres/Repos/my-project/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:109:22
    at async /Users/andres/Repos/my-project/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:136:14
    at async /Users/andres/Repos/my-project/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:33:22
    at async getFileStreamFromS3 (webpack-internal:///(rsc)/./services/s3Service.ts:57:22)
    at async GET (webpack-internal:///(rsc)/./app/api/submissions/[submissionId]/file/route.ts:34:28)
    at async /Users/andres/Repos/my-project/node_modules/next/dist/compiled/next-server/app-route.runtime.dev.js:6:57228
    at async eT.execute (/Users/andres/Repos/my-project/node_modules/next/dist/compiled/next-server/app-route.runtime.dev.js:6:46851)
    at async eT.handle (/Users/andres/Repos/my-project/node_modules/next/dist/compiled/next-server/app-route.runtime.dev.js:6:58760)
    at async doRender (/Users/andres/Repos/my-project/node_modules/next/dist/server/base-server.js:1366:42)
    at async cacheEntry.responseCache.get.routeKind (/Users/andres/Repos/my-project/node_modules/next/dist/server/base-server.js:1576:40)
    at async DevServer.renderToResponseWithComponentsImpl (/Users/andres/Repos/my-project/node_modules/next/dist/server/base-server.js:1496:28)
    at async DevServer.renderPageComponent (/Users/andres/Repos/my-project/node_modules/next/dist/server/base-server.js:1924:24)
    at async DevServer.renderToResponseImpl (/Users/andres/Repos/my-project/node_modules/next/dist/server/base-server.js:1962:32)
    at async DevServer.pipeImpl (/Users/andres/Repos/my-project/node_modules/next/dist/server/base-server.js:922:25)
    at async NextNodeServer.handleCatchallRenderRequest (/Users/andres/Repos/my-project/node_modules/next/dist/server/next-server.js:272:17)
    at async DevServer.handleRequestImpl (/Users/andres/Repos/my-project/node_modules/next/dist/server/base-server.js:818:17)
    at async /Users/andres/Repos/my-project/node_modules/next/dist/server/dev/next-dev-server.js:339:20
    at async Span.traceAsyncFn (/Users/andres/Repos/my-project/node_modules/next/dist/trace/trace.js:154:20)
    at async DevServer.handleRequest (/Users/andres/Repos/my-project/node_modules/next/dist/server/dev/next-dev-server.js:336:24)
    at async invokeRender (/Users/andres/Repos/my-project/node_modules/next/dist/server/lib/router-server.js:173:21)
    at async handleRequest (/Users/andres/Repos/my-project/node_modules/next/dist/server/lib/router-server.js:350:24)
    at async requestHandlerImpl (/Users/andres/Repos/my-project/node_modules/next/dist/server/lib/router-server.js:374:13)
    at async Server.requestListener (/Users/andres/Repos/my-project/node_modules/next/dist/server/lib/start-server.js:141:13) {
  '$metadata': { attempts: 1, totalRetryDelay: 0 }
}

Expected Behavior

I expected to fetch the file/object from S3

Possible Solution

No response

Additional Information/Context

No response
@arealesramirez arealesramirez added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Jan 17, 2025
@arealesramirez arealesramirez changed the title Next.JS 14.2.23 - Importing @aws-sdk/crc64-nvme-crt") as required but failed to fetch S3 objects. Next.JS 14.2.23 - Importing @aws-sdk/crc64-nvme-crt" as required but fails to fetch S3 objects. Jan 17, 2025
@aBurmeseDev aBurmeseDev added investigating Issue is being investigated and/or work is in progress to resolve the issue. p1 This is a high priority issue and removed needs-triage This issue or PR still needs to be triaged. labels Jan 17, 2025
@trivikr
Copy link
Member

trivikr commented Jan 18, 2025

This issue happens as the S3 objects were uploaded with CRC64NVME checksums - either by another JS SDK application which had the required additional @aws-sdk/crc64-nvme-crt dependency, or using another SDK.

The S3 backend returns the CRC64NVME checksum, and the JS SDK fails since the optional dependency is not available.

@trivikr
Copy link
Member

trivikr commented Jan 18, 2025

We're discussing internally on how the behavior should be.

Ideally, the JS SDK should skip validating the checksum for get operations if the dependency is not available. It's because the application did not explicitly request checksum to be validate using CRC64-NVME algorithm.

The fix is likely to be released on Tuesday Jan 21st or Wednesday Jan 22nd.

For a quick fix, you can pin the SDK version to <3.729.0 in your package.json.
I'll create a minimal repro and provide another workaround which required adding a middleware in the following comment.

@arealesramirez
Copy link
Author

I found out the following:

The project is in a monorepo structure in the following way:
monorepo
-> packages
-> nextjsapp
-> lambda

The two packages, nextjsapp and lambda use @aws-sdk/client-s3 library.

For the NextJS app, the version in package.json was: "@aws-sdk/client-s3": "^3.705.0"
For a Lambda, the version found out in package.json was: "@aws-sdk/client-s3": "^3.723.0",

Now, at the moment of installing the packages, the package installed was "@aws-sdk/client-s3": "3.730.0". Which means that's the version that is used in both packages.

This leads me to the error described above.

After I forced setting the package version to

"@aws-sdk/client-s3": "3.705.0"

in both packages of the monorepo,

deleting the packages (node_modules folder)
deleting the package.lock.json

and reinstalling packages

That forces to install the version "@aws-sdk/client-s3": "3.705.0".

At version 3.705.0, I didn't see any errors

@arealesramirez
Copy link
Author

We're discussing internally on how the behavior should be.

Ideally, the JS SDK should skip validating the checksum for get operations if the dependency is not available. It's because the application did not explicitly request checksum to be validate using CRC64-NVME algorithm.

The fix is likely to be released on Tuesday Jan 21st or Wednesday Jan 22nd.

For a quick fix, you can pin the SDK version to <3.729.0 in your package.json. I'll create a minimal repro and provide another workaround which required adding a middleware in the following comment.

That makes sense, as my solution was to use version 3.705.0

@arealesramirez
Copy link
Author

After the fix is released, will it still require installing and importing "@aws-sdk/crc64-nvme-crt"?

Could it just come as a dependency of the @aws-sdk/client-s3 fix version?

@trivikr
Copy link
Member

trivikr commented Jan 18, 2025

After the fix is released, will it still require installing and importing "@aws-sdk/crc64-nvme-crt"?

We're still discussing internally. As explained in #6822 (comment) I don't think we'll require dependency to be explicitly installed.

Ideally, the JS SDK should skip validating the checksum for get operations if the dependency is not available. It's because the application did not explicitly request checksum to be validate using CRC64-NVME algorithm.

@trivikr
Copy link
Member

trivikr commented Jan 18, 2025

Could it just come as a dependency of the @aws-sdk/client-s3 fix version?

No. The AWS SDK for JavaScript customers are sensitive to install sizes, and the aws-crt dependency is written in C which has a large install size. We cannot install it by default and need to request customers to explicitly install it.

@trivikr
Copy link
Member

trivikr commented Jan 18, 2025 •
edited
Loading

Minimal repro

PutObject

This can be another JS SDK application which has optional dependency, or another AWS SDK which includes CRC64NVME checksum by default. Running this file will register CRC64NVME checksum with S3 backend.

// putObject.mjs
import { S3, ChecksumAlgorithm } from "@aws-sdk/client-s3"; // >=v3.729.0
import "@aws-sdk/crc64-nvme-crt";

const client = new S3();

const Bucket = "test-flexible-checksums-v2"; // Change to your bucket name.
const Key = "foo";
const Body = "bar";

await client.putObject({ Bucket, Key, Body, ChecksumAlgorithm: ChecksumAlgorithm.CRC64NVME });

GetObject

This is a JS SDK application which performs data integrity checks by default.
Since S3 backend returns a CRC64NVME checksum,

// putObject.mjs
import { S3 } from "@aws-sdk/client-s3"; // >=v3.729.0

const client = new S3();

const Bucket = "test-flexible-checksums-v2";
const Key = "foo";

// ChecksumMode not required to be set in >=v3.729.0
await client.getObject({ Bucket, Key, ChecksumMode: "ENABLED" });

This throws error in >=3.729.0

/Users/trivikr/workspace/test/node_modules/@aws-sdk/middleware-flexible-checksums/dist-cjs/index.js:209
        throw new Error(
              ^

Error: Please check whether you have installed the "@aws-sdk/crc64-nvme-crt" package explicitly. 
You must also register the package by calling [require("@aws-sdk/crc64-nvme-crt");] or an ESM equivalent such as [import "@aws-sdk/crc64-nvme-crt";]. 
For more information please go to https://github.com/aws/aws-sdk-js-v3#functionality-requiring-aws-common-runtime-crt
    at selectChecksumAlgorithmFunction (/Users/trivikr/workspace/test/node_modules/@aws-sdk/middleware-flexible-checksums/dist-cjs/index.js:209:15)
    at validateChecksumFromResponse (/Users/trivikr/workspace/test/node_modules/@aws-sdk/middleware-flexible-checksums/dist-cjs/index.js:411:35)
    at /Users/trivikr/workspace/test/node_modules/@aws-sdk/middleware-flexible-checksums/dist-cjs/index.js:466:11
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async /Users/trivikr/workspace/test/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:174:20
    at async /Users/trivikr/workspace/test/node_modules/@smithy/middleware-serde/dist-cjs/index.js:33:24
    at async /Users/trivikr/workspace/test/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:483:18
    at async /Users/trivikr/workspace/test/node_modules/@smithy/middleware-retry/dist-cjs/index.js:321:38
    at async /Users/trivikr/workspace/test/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:109:22
    at async /Users/trivikr/workspace/test/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:136:14 {
  '$metadata': { attempts: 1, totalRetryDelay: 0 }
}

Node.js v22.13.0

Verified that error is not thrown in @aws-sdk/client-s3@<3.729.0

@trivikr
Copy link
Member

trivikr commented Jan 18, 2025 •
edited
Loading

Code workarounds if you can't pin your SDK dependency to <3.729.0

Recommended: Manually install @aws-sdk/crc64-nvme-crt

Manually install and import @aws-sdk/crc64-nvme-crt, if your application and production environment allows it.
This will provide SDK with optional dependency to compute and validate CRC64-NVME checksum returned in getObject response.

// putObject.mjs
import { S3 } from "@aws-sdk/client-s3"; // >=v3.729.0
import "@aws-sdk/crc64-nvme-crt"; // <-- here

const client = new S3();

const Bucket = "test-flexible-checksums-v2";
const Key = "foo";

await client.getObject({ Bucket, Key });

Remove x-amz-checksum-crc64nvme header using a middleware

This will just remove the crc64nvme checksum from the response.
If the GetObject API call responds with any other checksum (say CRC32), then it'll be validated.

// putObject.mjs
import { S3 } from "@aws-sdk/client-s3"; // >=v3.729.0

const client = new S3();

// ToDo: Remove workaround once https://github.com/aws/aws-sdk-js-v3/issues/6822 is fixed.
client.middlewareStack.addRelativeTo(
  (next) => async (args) => {
    const result = await next(args);
    delete result.response.headers["x-amz-checksum-crc64nvme"];
    return result;
  },
  {
    relation: "after",
    toMiddleware: "flexibleChecksumsResponseMiddleware",
  }
);

const Bucket = "test-flexible-checksums-v2";
const Key = "foo";

await client.getObject({ Bucket, Key });

Disable response checksum validation on response

This is not recommended, as it'll disable checksum validation on all responses

// putObject.mjs
import { S3 } from "@aws-sdk/client-s3"; // >=v3.729.0

const client = new S3({
  // ToDo: Remove workaround once https://github.com/aws/aws-sdk-js-v3/issues/6822 is fixed.
  responseChecksumValidation: "WHEN_REQUIRED",
});

const Bucket = "test-flexible-checksums-v2";
const Key = "foo";

await client.getObject({ Bucket, Key });

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug This issue is a bug. investigating Issue is being investigated and/or work is in progress to resolve the issue. p1 This is a high priority issue
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@trivikr @arealesramirez @aBurmeseDev