aws-codepipeline-actions: Problem Cloning the AWS CDK Repository on Windows

[azarboon]

Describe the bug

When cloning the AWS CDK repository on a Windows system, I encountered issues where certain files appear in the Git index but are not present on the filesystem. This causes git status to show these files as deleted, and attempts to restore or interact with them fail. The issue might be related to Windows' filesystem limitations, but the root cause remains unclear.

To fix the issue, I cloned the repository using Windows Subsystem for Linux (WSL) in VS Code, the Bash terminal in VS Code, and Git Desktop's Bash, but none of these helped. I also changed case sensitivity (through core.ignorecase), enabled long file paths (through core.longpaths), and modified the Windows registry, but this did not resolve the problem.

Regression Issue

• Select this option if this issue appears to be a regression.

Last Known Working CDK Version

No response

Expected Behavior

The repository should clone successfully on Windows, with all files checked out to the filesystem as expected. Running git status and git diff after merging updates from upstream/main should not show any unexpected deletions or errors related to file handling. Cross-platform compatibility issues, if present, should be handled gracefully to allow contributors to work seamlessly regardless of their operating system.

Current Behavior

git status
On branch main
Your branch is up to date with 'origin/main'.

Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        deleted:    packages/@aws-cdk-testing/framework-integ/test/aws-codepipeline-actions/test/integ.codepipeline-with-nested-stack.js.snapshot/cross-region-stack-123456789012:us-west-2.assets.json
        deleted:    packages/@aws-cdk-testing/framework-integ/test/aws-codepipeline-actions/test/integ.codepipeline-with-nested-stack.js.snapshot/cross-region-stack-123456789012:us-west-2.template.json

no changes added to commit (use "git add" and/or "git commit -a")

Also,

git diff
diff --git a/packages/@aws-cdk-testing/framework-integ/test/aws-codepipeline-actions/test/integ.codepipeline-with-nested-stack.js.snapshot/cross-region-stack-123456789012:us-west-2.assets.json b/packages/@aws-cdk-testing/framework-integ/test/aws-codepipeline-actions/test/integ.codepipeline-with-nested-stack.js.snapshot/cross-region-stack-123456789012:us-west-2.assets.json
deleted file mode 100644
index 8352392771..0000000000
--- a/packages/@aws-cdk-testing/framework-integ/test/aws-codepipeline-actions/test/integ.codepipeline-with-nested-stack.js.snapshot/cross-region-stack-123456789012:us-west-2.assets.json
+++ /dev/null
@@ -1,20 +0,0 @@
-{
-  "version": "38.0.1",
-  "files": {
-    "6ead11ac2a258f5956aa4d821020d99d8460173de308c7fd28a388dc243e3f89": {
-      "source": {
-        "path": "cross-region-stack-123456789012:us-west-2.template.json",
-        "packaging": "file"
-      },
-      "destinations": {
-        "123456789012-us-west-2": {
-          "bucketName": "cdk-hnb659fds-assets-123456789012-us-west-2",
-          "objectKey": "6ead11ac2a258f5956aa4d821020d99d8460173de308c7fd28a388dc243e3f89.json",
-          "region": "us-west-2",
-          "assumeRoleArn": "arn:${AWS::Partition}:iam::123456789012:role/cdk-hnb659fds-file-publishing-role-123456789012-us-west-2"
-        }
-      }
-    }
-  },
-  "dockerImages": {}
-}
\ No newline at end of file
diff --git a/packages/@aws-cdk-testing/framework-integ/test/aws-codepipeline-actions/test/integ.codepipeline-with-nested-stack.js.snapshot/cross-region-stack-123456789012:us-west-2.template.json b/packages/@aws-cdk-testing/framework-integ/test/aws-codepipeline-actions/test/integ.codepipeline-with-nested-stack.js.snapshot/cross-region-stack-123456789012:us-west-2.template.json
deleted file mode 100644
index d37e16bccf..0000000000
--- a/packages/@aws-cdk-testing/framework-integ/test/aws-codepipeline-actions/test/integ.codepipeline-with-nested-stack.js.snapshot/cross-region-stack-123456789012:us-west-2.template.json
+++ /dev/null
@@ -1,167 +0,0 @@
-{
- "Resources": {
-  "CrossRegionCodePipelineReplicationBucketEncryptionKey70216490": {
-   "Type": "AWS::KMS::Key",
-   "Properties": {
-    "KeyPolicy": {
-     "Statement": [
-      {
-       "Action": "kms:*",
-       "Effect": "Allow",
-       "Principal": {
-        "AWS": "arn:aws:iam::123456789012:root"
-       },
-       "Resource": "*"
-      },
-      {
-       "Action": [
-        "kms:Decrypt",
-        "kms:DescribeKey",
-        "kms:Encrypt",
-        "kms:GenerateDataKey*",
-        "kms:ReEncrypt*"
-       ],
-       "Effect": "Allow",
-       "Principal": {
-        "AWS": "arn:aws:iam::123456789012:role/MyPipelineRoleName"
-       },
-       "Resource": "*"
-      }
-     ],
-     "Version": "2012-10-17"
-    }
-   },
-   "UpdateReplacePolicy": "Delete",
-   "DeletionPolicy": "Delete"
-  },
-  "CrossRegionCodePipelineReplicationBucketEncryptionAliasF1A0F37D": {
-   "Type": "AWS::KMS::Alias",
-   "Properties": {
-    "AliasName": "alias/ort-us-wtencryptionalias8f9701ce6a32f909886f",
-    "TargetKeyId": {
-     "Fn::GetAtt": [
-      "CrossRegionCodePipelineReplicationBucketEncryptionKey70216490",
-      "Arn"
-     ]
-    }
-   },
-   "UpdateReplacePolicy": "Delete",
-   "DeletionPolicy": "Delete"
-  },
-  "CrossRegionCodePipelineReplicationBucketFC3227F2": {
-   "Type": "AWS::S3::Bucket",
-   "Properties": {
-    "BucketEncryption": {
-     "ServerSideEncryptionConfiguration": [
-      {
-       "ServerSideEncryptionByDefault": {
-        "KMSMasterKeyID": {
-         "Fn::Join": [
-          "",
-          [
-           "arn:aws:kms:us-west-2:123456789012:",
-           {
-            "Ref": "CrossRegionCodePipelineReplicationBucketEncryptionAliasF1A0F37D"  
-           }
-          ]
-         ]
-        },
-        "SSEAlgorithm": "aws:kms"
-       }
-      }
-     ]
-    },
-    "BucketName": "cross-region-support-us-weplicationbucket8a287d3945436008ebfd",    
-    "PublicAccessBlockConfiguration": {
-     "BlockPublicAcls": true,
-     "BlockPublicPolicy": true,
-     "IgnorePublicAcls": true,
-     "RestrictPublicBuckets": true
-    }
-   },
-   "UpdateReplacePolicy": "Retain",
-   "DeletionPolicy": "Retain"
-  },
-  "CrossRegionCodePipelineReplicationBucketPolicyB7BA2BCA": {
-   "Type": "AWS::S3::BucketPolicy",
-   "Properties": {
-    "Bucket": {
-     "Ref": "CrossRegionCodePipelineReplicationBucketFC3227F2"
-    },
-    "PolicyDocument": {
-     "Statement": [
-      {
-       "Action": "s3:*",
-       "Condition": {
-        "Bool": {
-         "aws:SecureTransport": "false"
-        }
-       },
-       "Effect": "Deny",
-       "Principal": {
-        "AWS": "*"
-       },
-       "Resource": [
-        {
-         "Fn::GetAtt": [
-          "CrossRegionCodePipelineReplicationBucketFC3227F2",
-          "Arn"
-         ]
-        },
-        {
-         "Fn::Join": [
-          "",
-          [
-           {
-            "Fn::GetAtt": [
-             "CrossRegionCodePipelineReplicationBucketFC3227F2",
-             "Arn"
-            ]
-           },
-           "/*"
-          ]
-         ]
-        }
-       ]
-      }
-     ],
-     "Version": "2012-10-17"
-    }
-   }
-  }
- },
- "Parameters": {
-  "BootstrapVersion": {
-   "Type": "AWS::SSM::Parameter::Value<String>",
-   "Default": "/cdk-bootstrap/hnb659fds/version",
-   "Description": "Version of the CDK Bootstrap resources in this environment, automatically retrieved from SSM Parameter Store. [cdk:skip]"
-  }
- },
- "Rules": {
-  "CheckBootstrapVersion": {
-   "Assertions": [
-    {
-     "Assert": {
-      "Fn::Not": [
-       {
-        "Fn::Contains": [
-         [
-          "1",
-          "2",
-          "3",
-          "4",
-          "5"
-         ],
-         {
-          "Ref": "BootstrapVersion"
-         }
-        ]
-       }
-      ]
-     },
-     "AssertDescription": "CDK bootstrap stack version 6 required. Please run 'cdk bootstrap' with a recent version of the CDK CLI."
-    }
-   ]
-  }
- }
-}
\ No newline at end of file

Also,

git restore "packages/@aws-cdk-testing/framework-integ/test/aws-codepipeline-actions/test/integ.codepipeline-with-nested-stack.js.snapshot/cross-region-stack-123456789012:us-west-2.assets.json"
git restore "packages/@aws-cdk-testing/framework-integ/test/aws-codepipeline-actions/test/integ.codepipeline-with-nested-stack.js.snapshot/cross-region-stack-123456789012:us-west-2.template.json"

error: unable to stat just-written file packages/@aws-cdk-testing/framework-integ/test/aws-codepipeline-actions/test/integ.codepipeline-with-nested-stack.js.snapshot/cross-region-stack-123456789012:us-west-2.assets.json: No such file or directory
error: unable to stat just-written file packages/@aws-cdk-testing/framework-integ/test/aws-codepipeline-actions/test/integ.codepipeline-with-nested-stack.js.snapshot/cross-region-stack-123456789012:us-west-2.template.json: No such file or directory

Reproduction Steps

# Fork the AWS CDK repository on GitHub
# Clone the forked repository
git clone https://github.com/<your-username>/aws-cdk.git
cd aws-cdk

# Add the upstream repository to fetch updates
git remote add upstream https://github.com/aws/aws-cdk.git

# Fetch the latest changes from upstream and merge them
git fetch upstream
git merge upstream/main

# Check the status to see any changes
git status

# View differences in the repository
git diff

# Attempt to restore problematic files
git restore <file>

Possible Solution

1. Investigate and address why the files cannot be created on Windows systems.
2. Consider renaming files with incompatible characters (e.g., replacing colons with _ or -).
3. Provide a script or process to handle these files for Windows users specifically.

Additional Information/Context

No response

CDK CLI Version

2.173.2 (build f8e6207)

Framework Version

No response

Node.js Version

v22.9.0

OS

Windows 11 Home Single Language - version 23H2 - OS build 22631.4602

Language

TypeScript
but seems this issue is agnostic of language.

Language Version

No response

Other information

Git: Version 2.42
Cloning method: Git Bash / WSL / GitHub Desktop
Repository: AWS CDK (main branch)

[azarboon]

For anyone who encounter this issue, here is a temporarily:

git clone https://github.com/your-username/aws-cdk3.git 
cd aws-cdk3 
git remote add upstream https://github.com/aws/aws-cdk.git 
git fetch upstream 
git checkout main 
git merge upstream/main 
echo "packages/@aws-cdk-testing/framework-integ/test/aws-codepipeline-actions/test/integ.codepipeline-with-nested-stack.js.snapshot/" >> .git/info/exclude 
git rm --cached packages/@aws-cdk-testing/framework-integ/test/aws-codepipeline-actions/test/integ.codepipeline-with-nested-stack.js.snapshot/cross-region-stack-123456789012:us-west-2.assets.json && git rm --cached packages/@aws-cdk-testing/framework-integ/test/aws-codepipeline-actions/test/integ.codepipeline-with-nested-stack.js.snapshot/cross-region-stack-123456789012:us-west-2.template.json 
git commit -m "Remove problematic files from git index" 
git push origin main 
git checkout -b your-branch-name

[ashishdhingra]

Got below error while cloning on Windows:

Cloning into 'D:\source\GitHub\aws-cdk'...
remote: Enumerating objects: 396735, done.        
remote: Counting objects: 100% (111/111), done.        
remote: Compressing objects: 100% (72/72), done.        
remote: Total 396735 (delta 59), reused 41 (delta 39), pack-reused 396624 (from 2)        
Receiving objects: 100% (396735/396735), 1.06 GiB | 18.14 MiB/s, done.
Resolving deltas: 100% (322693/322693), done.
error: invalid path 'packages/@aws-cdk-testing/framework-integ/test/aws-codepipeline-actions/test/integ.codepipeline-with-nested-stack.js.snapshot/cross-region-stack-123456789012:us-west-2.assets.json'
fatal: unable to checkout working tree
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry with 'git restore --source=HEAD :/'


Would you like to retry cloning ?

Character : is not a valid path for Windows.
Need to reach out to CDK team for inputs. CC @khushail

[khushail]

Seeking Team's insights on this issue for further analysis.

[samson-keung]

I looked into why the repo has files with : in the file name. Here is my findings:

The codepipeline.Pipeline construct contains code that creates a nested stack. When it creates the nested stack, it assign a stack id to the stack, the id has the : character:

aws-cdk/packages/aws-cdk-lib/aws-codepipeline/lib/pipeline.ts

Line 826 in a09ccdb

const supportStackId = `cross-region-stack-${this.reuseCrossRegionSupportStacks ? pipelineAccount : pipelineStack.stackName}:${actionRegion}`;

Then when the CDK app get synthesized, the generate template file name is based on the stack id, hence, inherited the : char.

---

Looking at the code, I do not see a prop on codepipeline.Pipeline to override the id for the nested stack. And knowing that : in file name is not supported in Windows, I think it was a mistake to use : in the stack id. I think the solution is to introduce a feature flag which brach the code to use a different id format.

[azarboon]

I looked into why the repo has files with : in the file name. Here is my findings:

The codepipeline.Pipeline construct contains code that creates a nested stack. When it creates the nested stack, it assign a stack id to the stack, the id has the : character:

aws-cdk/packages/aws-cdk-lib/aws-codepipeline/lib/pipeline.ts

Line 826 in a09ccdb

const supportStackId = cross-region-stack-${this.reuseCrossRegionSupportStacks ? pipelineAccount : pipelineStack.stackName}:${actionRegion};
Then when the CDK app get synthesized, the generate template file name is based on the stack id, hence, inherited the : char.

Looking at the code, I do not see a prop on codepipeline.Pipeline to override the id for the nested stack. And knowing that : in file name is not supported in Windows, I think it was a mistake to use : in the stack id. I think the solution is to introduce a feature flag which brach the code to use a different id format.

@samson-keung Thanks for this. I'm wondering why are you suggesting feature flag. May I make a PR and just replace : with -?

[samson-keung]

@azarboon It has to be behind feature flag because changing the id of the nested stack can cause behavioural changes to existing CDK apps. E.g. the nested stack being replaced or a new stack being created. There may be other cases where CDK users are relying on reading the stack id.

A reference in the Contribution Guide about breaking changes: https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md#breaking-changes

Please feel free to open a PR! Contributions are always welcome.

[azarboon]

@azarboon It has to be behind feature flag because changing the id of the nested stack can cause behavioural changes to existing CDK apps. E.g. the nested stack being replaced or a new stack being created. There may be other cases where CDK users are relying on reading the stack id.

A reference in the Contribution Guide about breaking changes: https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md#breaking-changes

Please feel free to open a PR! Contributions are always welcome.

@samson-keung Thank you for your response. However, I am currently facing some challenges with creating pull requests. I have detailed the issue in #32910. Could you please assist me with resolving this? Once the issue is addressed, I will proceed with the task.