CDK deploy seems to be deleting CloudFormationExecutionRole and other bootstrapped items

[Sheharyar566]

Describe the bug

I'm new to CDK and started working on an existing project. It's already deployed on an account, and I'm tasked with setting up a dev environment (on a different account). But for some reason, cdk deploy is failing right at the end.
By looking at the logs, it seems like when I run cdk bootstrap stack-name it creates a few roles, like execution role, file publishing role, two or three other roles, along with a repository. The bootstrap succeeds. After this, when I run cdk deploy it uploads all of the lambdas, dynamo tables and all of the other stuff.

But once it is done, it seems like it is trying to delete the above created roles and the repository. But the repository deletion fails saying the repository still has images and can't be deleted. The process fails. If I try to run cdk deploy again, it says the roles are not found or invalid (which of course don't exist now since cdk rollback for some reason deleted them).

Of course, bootstrapping again fails as well, because the repository exists (as it couldn't be deleted).

For reference, I have tried with [email protected], also I tried with [email protected] (I don't know about this version but I saw it mentioned somewhere - so I thought why not)

Upon looking at the CDK Diff 's output. Seems like cdk deploy is removing a bunch of stuff including the items created during cdk bootstrap. (I've omitted the items it's adding - Please do let me know if I need to add any other logs or stuff).

Parameters
[-] Parameter TrustedAccounts: {"Description":"List of AWS accounts that are trusted to publish assets and deploy stacks to this environment","Default":"","Type":"CommaDelimitedList"}
[-] Parameter TrustedAccountsForLookup: {"Description":"List of AWS accounts that are trusted to look up values in this environment","Default":"","Type":"CommaDelimitedList"}
[-] Parameter CloudFormationExecutionPolicies: {"Description":"List of the ManagedPolicy ARN(s) to attach to the CloudFormation deployment role","Default":"","Type":"CommaDelimitedList"}
[-] Parameter FileAssetsBucketName: {"Description":"The name of the S3 bucket used for file assets","Default":"","Type":"String"}
[-] Parameter FileAssetsBucketKmsKeyId: {"Description":"Empty to create a new key (default), 'AWS_MANAGED_KEY' to use a managed S3 key, or the ID/ARN of an existing key.","Default":"","Type":"String"}
[-] Parameter ContainerAssetsRepositoryName: {"Description":"A user-provided custom name to use for the container assets ECR repository","Default":"","Type":"String"}
[-] Parameter Qualifier: {"Description":"An identifier to distinguish multiple bootstrap stacks in the same environment","Default":"hnb659fds","Type":"String","AllowedPattern":"[A-Za-z0-9_-]{1,10}","ConstraintDescription":"Qualifier must be an alphanumeric identifier of at most 10 characters"}
[-] Parameter PublicAccessBlockConfiguration: {"Description":"Whether or not to enable S3 Staging Bucket Public Access Block Configuration","Default":"true","Type":"String","AllowedValues":["true","false"]}
[-] Parameter InputPermissionsBoundary: {"Description":"Whether or not to use either the CDK supplied or custom permissions boundary","Default":"","Type":"String"}
[-] Parameter UseExamplePermissionsBoundary: {"Default":"false","AllowedValues":["true","false"],"Type":"String"}
[-] Parameter BootstrapVariant: {"Type":"String","Default":"AWS CDK: Default Resources","Description":"Describe the provenance of the resources in this bootstrap stack. Change this when you customize the template. To prevent accidents, the CDK CLI will not overwrite bootstrap stacks with a different variant."}

Conditions
[-] Condition HasTrustedAccounts: {"Fn::Not":[{"Fn::Equals":["",{"Fn::Join":["",{"Ref":"TrustedAccounts"}]}]}]}
[-] Condition HasTrustedAccountsForLookup: {"Fn::Not":[{"Fn::Equals":["",{"Fn::Join":["",{"Ref":"TrustedAccountsForLookup"}]}]}]}
[-] Condition HasCloudFormationExecutionPolicies: {"Fn::Not":[{"Fn::Equals":["",{"Fn::Join":["",{"Ref":"CloudFormationExecutionPolicies"}]}]}]}
[-] Condition HasCustomFileAssetsBucketName: {"Fn::Not":[{"Fn::Equals":["",{"Ref":"FileAssetsBucketName"}]}]}
[-] Condition CreateNewKey: {"Fn::Equals":["",{"Ref":"FileAssetsBucketKmsKeyId"}]}
[-] Condition UseAwsManagedKey: {"Fn::Equals":["AWS_MANAGED_KEY",{"Ref":"FileAssetsBucketKmsKeyId"}]}
[-] Condition ShouldCreatePermissionsBoundary: {"Fn::Equals":["true",{"Ref":"UseExamplePermissionsBoundary"}]}
[-] Condition PermissionsBoundarySet: {"Fn::Not":[{"Fn::Equals":["",{"Ref":"InputPermissionsBoundary"}]}]}
[-] Condition HasCustomContainerAssetsRepositoryName: {"Fn::Not":[{"Fn::Equals":["",{"Ref":"ContainerAssetsRepositoryName"}]}]}
[-] Condition UsePublicAccessBlockConfiguration: {"Fn::Equals":["true",{"Ref":"PublicAccessBlockConfiguration"}]}

Resources
[-] AWS::KMS::Key FileAssetsBucketEncryptionKey destroy
[-] AWS::KMS::Alias FileAssetsBucketEncryptionKeyAlias destroy
[-] AWS::S3::Bucket StagingBucket orphan
[-] AWS::S3::BucketPolicy StagingBucketPolicy destroy
[-] AWS::ECR::Repository ContainerAssetsRepository destroy
[-] AWS::IAM::Role FilePublishingRole destroy
[-] AWS::IAM::Role ImagePublishingRole destroy
[-] AWS::IAM::Role LookupRole destroy
[-] AWS::IAM::Policy FilePublishingRoleDefaultPolicy destroy
[-] AWS::IAM::Policy ImagePublishingRoleDefaultPolicy destroy
[-] AWS::IAM::Role DeploymentActionRole destroy
[-] AWS::IAM::Role CloudFormationExecutionRole destroy
[-] AWS::IAM::ManagedPolicy CdkBoostrapPermissionsBoundaryPolicy destroy
[-] AWS::SSM::Parameter CdkBootstrapVersion destroy

Outputs
[-] Output BucketName: {"Description":"The name of the S3 bucket owned by the CDK toolkit stack","Value":{"Fn::Sub":"${StagingBucket}"}}
[-] Output BucketDomainName: {"Description":"The domain name of the S3 bucket owned by the CDK toolkit stack","Value":{"Fn::Sub":"${StagingBucket.RegionalDomainName}"}}
[-] Output FileAssetKeyArn: {"Description":"The ARN of the KMS key used to encrypt the asset bucket (deprecated)","Value":{"Fn::If":["CreateNewKey",{"Fn::Sub":"${FileAssetsBucketEncryptionKey.Arn}"},{"Fn::Sub":"${FileAssetsBucketKmsKeyId}"}]},"Export":{"Name":{"Fn::Sub":"CdkBootstrap-${Qualifier}-FileAssetKeyArn"}}}
[-] Output ImageRepositoryName: {"Description":"The name of the ECR repository which hosts docker image assets","Value":{"Fn::Sub":"${ContainerAssetsRepository}"}}
[-] Output BootstrapVersion BootstrapVersion: {"Description":"The version of the bootstrap resources that are currently mastered in this stack","Value":{"Fn::GetAtt":["CdkBootstrapVersion","Value"]}}

Regression Issue

• Select this option if this issue appears to be a regression.

Last Known Working CDK Version

No response

Expected Behavior

The deployment succeeds without issue

Current Behavior

Deployment seems to be deleting bootstrapped items

Reproduction Steps

I need help in figuring out how to reproduce it.

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.174.1 (build f353fc7)

Framework Version

No response

Node.js Version

20.11.0

OS

Mac OS

Language

Python

Language Version

3.13.0

Other information

No response

[ashishdhingra]

@Sheharyar566 Good afternoon. Thanks for opening the issue. The normal command line for bootstrapping a CDK environment is specified at Bootstrap your environment for use with the AWS CDK. When we execute command like cdk bootstrap aws://123456789012/us-east-1, you are explicitly specifying the AWS account and region. In this case, it uses the default --toolkit-stack-name with value CDKToolkit as specified at cdk bootstrap.

In your use case, when you are executing cdk bootstrap stack-name, you are relying on default AWS profile on your machine to determine AWS account and region to use. By passing stack-name, you might be inadvertently setting the --toolkit-stack-name as the name of your CDK stack. Hence, when you try to deploy actual CDK application stack with name stack-name, it notices the diff and tries to replace the CDK bootstrap stack with the same name.

Please share the following:

• What is the output of cdk bootstrap stack-name --debug in your case? (also are you using the stack-name same as that of your CDK application stack?)
• Could you try just using cdk bootstrap? This should create CDK bootstrap stack with name CDKToolkit.

Mentioning here just for reference, CDK bootstrap logic.

Thanks,
Ashish

[Sheharyar566]

Thanks for the response @ashishdhingra
Here's the exact command I'm trying to run when bootstrapping.
cdk bootstrap --toolkit-stack-name spine-stack-eu --profile dev
I'm explicitly referrencing the profile. The only reason I'm explicitly setting the stack name using --toolkit-stack-name is because we seem to be using the same name in the code:
spine_stack = SpineStack(app, "spine-stack-eu", env=env_EU)

Here's the bootstrap's log:

[WARNING] aws-cdk-lib.CfnResource#addDependsOn is deprecated.
  use addDependency
  This API will be removed in the next major release.
[WARNING] aws-cdk-lib.aws_stepfunctions.StateMachineProps#definition is deprecated.
  use definitionBody: DefinitionBody.fromChainable()
  This API will be removed in the next major release.
[WARNING] aws-cdk-lib.aws_stepfunctions.StateMachineProps#definition is deprecated.
  use definitionBody: DefinitionBody.fromChainable()
  This API will be removed in the next major release.
 ⏳  Bootstrapping environment aws://123412341234/eu-central-1...
Trusted accounts for deployment: (none)
Trusted accounts for lookup: (none)
Using default execution policy of 'arn:aws:iam::aws:policy/AdministratorAccess'. Pass '--cloudformation-execution-policies' to customize.
spine-stack-eu: creating CloudFormation changeset...
spine-stack-eu |  0/12 | 9:26:20 AM | REVIEW_IN_PROGRESS   | AWS::CloudFormation::Stack | spine-stack-eu User Initiated
spine-stack-eu |  0/12 | 9:26:28 AM | CREATE_IN_PROGRESS   | AWS::CloudFormation::Stack | spine-stack-eu User Initiated
spine-stack-eu |  0/12 | 9:26:31 AM | CREATE_IN_PROGRESS   | AWS::IAM::Role          | LookupRole 
spine-stack-eu |  0/12 | 9:26:31 AM | CREATE_IN_PROGRESS   | AWS::S3::Bucket         | StagingBucket 
spine-stack-eu |  0/12 | 9:26:31 AM | CREATE_IN_PROGRESS   | AWS::IAM::Role          | ImagePublishingRole 
spine-stack-eu |  0/12 | 9:26:31 AM | CREATE_IN_PROGRESS   | AWS::IAM::Role          | FilePublishingRole 
spine-stack-eu |  0/12 | 9:26:31 AM | CREATE_IN_PROGRESS   | AWS::SSM::Parameter     | CdkBootstrapVersion 
spine-stack-eu |  0/12 | 9:26:31 AM | CREATE_IN_PROGRESS   | AWS::IAM::Role          | CloudFormationExecutionRole 
spine-stack-eu |  0/12 | 9:26:31 AM | CREATE_IN_PROGRESS   | AWS::ECR::Repository    | ContainerAssetsRepository 
spine-stack-eu |  0/12 | 9:26:31 AM | CREATE_IN_PROGRESS   | AWS::SSM::Parameter     | CdkBootstrapVersion Resource creation Initiated
spine-stack-eu |  0/12 | 9:26:31 AM | CREATE_IN_PROGRESS   | AWS::IAM::Role          | CloudFormationExecutionRole Resource creation Initiated
spine-stack-eu |  0/12 | 9:26:31 AM | CREATE_IN_PROGRESS   | AWS::IAM::Role          | ImagePublishingRole Resource creation Initiated
spine-stack-eu |  0/12 | 9:26:31 AM | CREATE_IN_PROGRESS   | AWS::IAM::Role          | FilePublishingRole Resource creation Initiated
spine-stack-eu |  0/12 | 9:26:31 AM | CREATE_IN_PROGRESS   | AWS::ECR::Repository    | ContainerAssetsRepository Resource creation Initiated
spine-stack-eu |  0/12 | 9:26:32 AM | CREATE_IN_PROGRESS   | AWS::S3::Bucket         | StagingBucket Resource creation Initiated
spine-stack-eu |  0/12 | 9:26:32 AM | CREATE_IN_PROGRESS   | AWS::IAM::Role          | LookupRole Resource creation Initiated
spine-stack-eu |  1/12 | 9:26:32 AM | CREATE_COMPLETE      | AWS::SSM::Parameter     | CdkBootstrapVersion 
spine-stack-eu |  2/12 | 9:26:32 AM | CREATE_COMPLETE      | AWS::ECR::Repository    | ContainerAssetsRepository 
spine-stack-eu |  3/12 | 9:26:46 AM | CREATE_COMPLETE      | AWS::S3::Bucket         | StagingBucket 
spine-stack-eu |  3/12 | 9:26:47 AM | CREATE_IN_PROGRESS   | AWS::S3::BucketPolicy   | StagingBucketPolicy 
spine-stack-eu |  3/12 | 9:26:48 AM | CREATE_IN_PROGRESS   | AWS::S3::BucketPolicy   | StagingBucketPolicy Resource creation Initiated
spine-stack-eu |  4/12 | 9:26:48 AM | CREATE_COMPLETE      | AWS::S3::BucketPolicy   | StagingBucketPolicy 
spine-stack-eu |  5/12 | 9:26:49 AM | CREATE_COMPLETE      | AWS::IAM::Role          | FilePublishingRole 
spine-stack-eu |  6/12 | 9:26:49 AM | CREATE_COMPLETE      | AWS::IAM::Role          | ImagePublishingRole 
spine-stack-eu |  7/12 | 9:26:50 AM | CREATE_COMPLETE      | AWS::IAM::Role          | CloudFormationExecutionRole 
spine-stack-eu |  7/12 | 9:26:50 AM | CREATE_IN_PROGRESS   | AWS::IAM::Policy        | FilePublishingRoleDefaultPolicy 
spine-stack-eu |  7/12 | 9:26:50 AM | CREATE_IN_PROGRESS   | AWS::IAM::Policy        | ImagePublishingRoleDefaultPolicy 
spine-stack-eu |  8/12 | 9:26:50 AM | CREATE_COMPLETE      | AWS::IAM::Role          | LookupRole 
spine-stack-eu |  8/12 | 9:26:51 AM | CREATE_IN_PROGRESS   | AWS::IAM::Role          | DeploymentActionRole 
spine-stack-eu |  8/12 | 9:26:51 AM | CREATE_IN_PROGRESS   | AWS::IAM::Policy        | FilePublishingRoleDefaultPolicy Resource creation Initiated
spine-stack-eu |  8/12 | 9:26:51 AM | CREATE_IN_PROGRESS   | AWS::IAM::Policy        | ImagePublishingRoleDefaultPolicy Resource creation Initiated
spine-stack-eu |  8/12 | 9:26:52 AM | CREATE_IN_PROGRESS   | AWS::IAM::Role          | DeploymentActionRole Resource creation Initiated
spine-stack-eu |  9/12 | 9:27:07 AM | CREATE_COMPLETE      | AWS::IAM::Policy        | FilePublishingRoleDefaultPolicy 
spine-stack-eu | 10/12 | 9:27:07 AM | CREATE_COMPLETE      | AWS::IAM::Policy        | ImagePublishingRoleDefaultPolicy 
spine-stack-eu | 11/12 | 9:27:10 AM | CREATE_COMPLETE      | AWS::IAM::Role          | DeploymentActionRole 
spine-stack-eu | 12/12 | 9:27:11 AM | CREATE_COMPLETE      | AWS::CloudFormation::Stack | spine-stack-eu 
 ✅  Environment aws://123412341234/eu-central-1 bootstrapped.

[ashishdhingra]

@Sheharyar566 Thanks for your response. Please refer to following resources in CDK developer guide:

• AWS CDK bootstrapping.
• When to bootstrap your environment
• Default resources created during bootstrapping
• cdk bootstrap

The bootstrap stack is a CloudFormation stack that provisions an Amazon S3 bucket and Amazon ECR repository in the AWS environment. The AWS CDK CLI uses these resources to store synthesized templates and related assets during deployment.

In other words, bootstrap stack is supposed to be only by CDK CLI, and is separate from your application stack. CDK CLI uses bootstrap stack to deploy number of application stacks. In your case, spine-stack-eu is your application stack, NOT bootstrap stack. So while you are allowed to use any name for --toolkit-stack-name (other than default CDKToolkit), this stack is used only by CDK CLI.

Hope it helps. I understand you are new to CDK; so I would recommend going through AWS CDK Immersion Day Workshop.

Thanks,
Ashish