Instance and container metadata credentials not working with Deno

[soundstep]

Checkboxes for prior research

• I've gone through Developer Guide and API reference
• I've checked AWS Forums and StackOverflow.
• I've searched for previous similar issues and didn't find any solution.

Describe the bug

Automatic credentials fromInstanceMetadata and fromContainerMetadata are not picked up and is hanging (no error message, no timeout) when used with Deno.
This happens on our CI in 2 different Jenkins instances (one EC2, one EKS), it does work locally with environment variables, or on the CI using NodeJS.
Note: I realize the Deno usage is not official, but any help would be greatly appreciated to either resolve the issue or debug it. Deno has been a tremendous improvement in our workflows.

SDK version number

@aws-sdk/[email protected]

Which JavaScript Runtime is this issue in?

Deno

Details of the browser/Node.js/ReactNative version

deno 1.30.2 (release, aarch64-apple-darwin)
v8 10.9.194.5
typescript 4.9.4

Reproduction Steps

Note that the following scripts are working correctly with this port, but this port has been abandoned: https://github.com/christophgysin/aws-sdk-js-v3 (https://deno.land/x/[email protected]/client-s3/mod.ts)

Test no credentials setup

1. Install Deno
2. Create a deno script main.ts:

import { ListObjectsCommand, S3Client } from 'npm:@aws-sdk/[email protected]';

console.log('No AWS credentials setup');

const client = new S3Client({
    region: 'eu-west-1'
});

console.log('S3 client created, executing ListObjectsCommand');

try {
    const res = await client.send(
        new ListObjectsCommand({
            Bucket: 'my-s3-bucket'
        }),
    );

    console.log(res);

    Deno.exit();
} catch (err) {
    console.log(err);
    Deno.exit(1);
}

3. Run the script: deno run -A --reload main.ts

Output:

The script is running and hanging forever:

Test with credentials setup

1. Install Deno
2. Create a deno script main-container.ts:

import { ListObjectsCommand, S3Client } from 'npm:@aws-sdk/[email protected]';
import { fromContainerMetadata } from 'npm:@aws-sdk/[email protected]';

console.log('Using fromContainerMetadata credentials');

const client = new S3Client({
    region: 'eu-west-1',
    credentials = fromContainerMetadata({ timeout: 2000, maxRetries: 1 })
});

console.log('S3 client created, executing ListObjectsCommand');

try {
    const res = await client.send(
        new ListObjectsCommand({
            Bucket: 'my-s3-bucket'
        }),
    );

    console.log(res);

    Deno.exit();
} catch (err) {
    console.log(err);
    Deno.exit(1);
}

3. Run the script: deno run -A --reload main-container.ts

Output:

The script is running and hanging forever:

Test with credentials setup and esm.sh

1. Install Deno
2. Create a deno script main-esmsh.ts:

import { ListObjectsCommand, S3Client } from 'https://esm.sh/@aws-sdk/[email protected]';
// Note: fromContainerMetadata is not available from https://esm.sh/@aws-sdk/[email protected]
import { fromContainerMetadata } from 'https://esm.sh/@aws-sdk/[email protected]';

console.log('Using fromContainerMetadata credentials');

const client = new S3Client({
    region: 'eu-west-1',
    credentials = fromContainerMetadata({ timeout: 2000, maxRetries: 1 })
});

console.log('S3 client created, executing ListObjectsCommand');

try {
    const res = await client.send(
        new ListObjectsCommand({
            Bucket: 'my-s3-bucket'
        }),
    );

    console.log(res);

    Deno.exit();
} catch (err) {
    console.log(err);
    Deno.exit(1);
}

3. Run the script: deno run -A --reload main-esmsh.ts

Output:

The script output an error:

11:20:43  Using fromContainerMetadata credentials
11:20:43  S3 client created, executing ListObjectsCommand
11:20:43  Error: Credential is missing
11:20:43      at re.credentialProvider (https://esm.sh/v106/@aws-sdk/[email protected]/deno/client-s3.js:4:183903)
11:20:43      at re.signRequest (https://esm.sh/v106/@aws-sdk/[email protected]/deno/signature-v4.js:3:522)
11:20:43      at re.sign (https://esm.sh/v106/@aws-sdk/[email protected]/deno/signature-v4.js:2:4709)
11:20:43      at sign (https://esm.sh/v106/@aws-sdk/[email protected]/deno/signature-v4-multi-region.js:2:764)
11:20:43      at https://esm.sh/v106/@aws-sdk/[email protected]/deno/middleware-signing.js:2:2366
11:20:43      at async https://esm.sh/v106/@aws-sdk/[email protected]/deno/middleware-retry.js:2:5465
11:20:43      at async https://esm.sh/v106/@aws-sdk/[email protected]/deno/middleware-logger.js:2:34
11:20:43      at async file:///tmp/jenkins/workspace/Playground/scriptTest/test-ec2-credentials/esmsh/main-container-esmsh.ts:26:17

Observed Behavior

With a script executing a simple ListObjectsCommand on an S3 Bucket (scripts are provided in the reproduction step).

1. The script is hanging forever with either (no error, no output, no timeout):

• no credentials set
• the npm: specifier usage (deno npm specifier)

Output:

2. The script outputs an error if we use an esm.sh URL (see reproduction step).

import { ListObjectsCommand, S3Client } from 'https://esm.sh/@aws-sdk/[email protected]';
import { fromContainerMetadata } from 'https://esm.sh/@aws-sdk/[email protected]';

Note that I had to use @aws-sdk/credential-provider-imds instead of @aws-sdk/credential-providers, as with esm.sh, the fromContainerMetadata export is not available.

Expected Behavior

To be logged in automatically.

Possible Solution

No response

Additional Information/Context

Deno version:

deno 1.30.2 (release, aarch64-apple-darwin)
v8 10.9.194.5
typescript 4.9.4

OS:
Mac OS 12.1, Linux alpine, Linux Centos

[christophgysin]

This was fixed in my fork using a custom implementation for deno: https://github.com/christophgysin/aws-sdk-js-v3/tree/deno/packages/credential-provider-imds-deno

[soundstep]

Oh great, thank you for the info!

[soundstep]

Am I wrong in thinking that the behavior I reported is related to the following package that is not working with Deno as it is using node bindings?
https://github.com/awslabs/aws-crt-nodejs
This is used in the package @aws-sdk/signature-v4-crt, not quite sure how it is related to the rest.
Also mentioned there: christophgysin#38 (comment)
Is there any workaround?

[yenfryherrerafeliz]

Hi @soundstep, thanks for opening this issue. I was able to reproduce the reported behavior, however, I am not very familiar with Deno runtime, and I am currently working on understanding what exactly maybe causing the issue here.

I will get back to you as soon as possible.

Thanks!

[cdeck95]

@soundstep did you find a solution? I am running into the same problem. If so, do you mind showing your updated code or package references?

[yenfryherrerafeliz]

Hi @soundstep, @cdeck95, sorry for the delay answering on this. Basically, this issue is caused by the http library that deno uses, which seems to be different from the one nodejs provides and therefore the request sent is rejected by the imds. Since we do not support Deno runtime, one thing you folks can do is to do a custom implementation to fetch the credentials from the instance metadata service. Actually that is what @christophgysin did. Here is also the documentation.

What I can do now is to mark this issue as feature request to gather community sentiments about this and take further actions based on it.

Thanks!

[soundstep]

Thank you @yenfryherrerafeliz for looking into this.

I think the reason why @christophgysin port has been abandoned is he cannot compile his sdk port anymore because of an underlying crt library, more info there: christophgysin#38 (comment)

We have other EKS containers that run on web identity token, I will try to provide these credentials manually once I get the info I need to see how it goes.

@cdeck95 No solution so far I'm afraid, FYI see the IMDS port from @christophgysin there.

[soundstep]

FYI, this works with web identity token in containers (EKS), the credentials are picked up correctly:

import { ListObjectsCommand, S3Client } from 'npm:@aws-sdk/[email protected]';

const client = new S3Client({
    region: Deno.env.get('AWS_REGION') || 'eu-west-1',
});

const res = await client.send(
    new ListObjectsCommand({
        Bucket: 'my-bucket',
    }),
);

console.log('response length:', res.Contents.length);

I'm not sure I will have the time to look into IMDS (container and instance metadata) for the time being, which is not working.

[andykais]

another error that seems to be deno-specific:

import * as s3 from 'npm:@aws-sdk/client-s3'

const s3_client = new s3.S3({
  region: 'auto',
  endpoint: R2_BASE_URL,
  credentials: {
    accessKeyId: ACCESS_KEY_ID,
    secretAccessKey: ACCESS_KEY_SECRET,
  }
})

const result = await s3_client.createMultipartUpload({
  Bucket: 'MYBUCKET',
  Key: `${Date.now()}`,
})

gives this error:

error: Uncaught (in promise) 400: UnknownError
    at throwDefaultError (file:///home/andrew/node_modules/.deno/@[email protected]/node_modules/@smithy/smithy-client/dist-cjs/default-error-handler.js:8:22)
    at file:///home/andrew/node_modules/.deno/@[email protected]/node_modules/@smithy/smithy-client/dist-cjs/default-error-handler.js:18:39
    at de_CreateMultipartUploadCommandError (file:///home/andrew/node_modules/.deno/@[email protected]/node_modules/@aws-sdk/client-s3/dist-cjs/protocols/Aws_restXml.js:3276:12)
    at Object.runMicrotasks (ext:core/01_core.js:790:30)
    at processTicksAndRejections (ext:deno_node/_next_tick.ts:53:10)
    at runNextTicks (ext:deno_node/_next_tick.ts:71:3)
    at eventLoopTick (ext:core/01_core.js:184:21)
    at async file:///home/andrew/node_modules/.deno/@[email protected]/node_modules/@smithy/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24
    at async file:///home/andrew/node_modules/.deno/@[email protected]/node_modules/@aws-sdk/middleware-signing/dist-cjs/awsAuthMiddleware.js:30:20
    at async file:///home/andrew/node_modules/.deno/@[email protected]/node_modules/@smithy/middleware-retry/dist-cjs/retryMiddleware.js:27:46

running the same snippet in node (with a few changes like removing the npm: specifier and wrapping the top-level await) will succeed so I am confident this is a deno specific issue. I have also been able to run S3::listObjects in deno, so it seems to be some specific logic that is hit by this method.

This is using R2 so dont crucify me 😅. This is how they would normally call your api from node https://developers.cloudflare.com/r2/reference/data-location/

[edit] one more datapoint: I can still successfully use the old sdk.

// @deno-types="npm:aws-sdk/clients/s3.d.ts"
import aws_sdk_s3 from 'npm:aws-sdk/clients/s3.js';


const s3_client = new aws_sdk_s3({
  accessKeyId: ACCESS_KEY_ID,
  secretAccessKey: ACCESS_KEY_SECRET,
  endpoint: R2_BASE_URL,
})

const result = await new Promise((resolve, reject) => {
  s3_client.createMultipartUpload({
    Bucket: 'MYBUCKET',
    Key: `${Date.now()}`,
  }, (error, result) => {
    if (error) reject(error)
    else resolve(result)
  })
})

[aBurmeseDev]

Hey all - checking in here.

This issue seems to be third-party related which we do not support as previously mentioned by my colleague here: #4405 (comment)

Please feel free to refer to the workarounds mentioned in the thread and we'll be closing this as not planned.

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.