Remove ami pinning from scale-config.yml files

[jeanschmidt]

At news year eve, we had an small CI outage. Queue started to grow due to lack of capacity to create new linux instances. After investigating the issue we noticed that this is due to the pinning of labels in the ami tag of scale-config.yml and its variants. This is due to the fact that for security reasons, Amazon removed the tag on its services, so we could not resolve the AMI ID from the tag search.

The goal of this tag is to enable to migrate to a newer AMI type runner-by-runner, so we can troubleshoot problems and avoid the issue of being stuck in the migration because of a particular job that runs in a particular instance. This was included with the concept of variants.

Now that the migration is complete, the correct approach is to REMOVE these labels and rely on the labels that are pinned at release/deploy time. They are safer, for many reasons, somo of them:

• The AMI id is pinned at release time, so if the label is not available anymore, the instances are still able to be created;
• At the release time, we are immediately notified that the label is not available anymore and we need to upgrade, plus it prevents moving forward and deploying a broken state.
• We are able to test the changes (honestly, many times we don't, but can and should)
• Minor version upgrades, that are potentially problematic, can be rolled back faster;
• We have "hands on controls" and are aware of the releases, so we know when a release is triggered and can monitor. Over have the change take immediate effect when Amazon releases a newer minor version;
• Rolls forward is faster to a newer release and is guaranteed to be complete;

So, to avoid outages similar to what we had, this action should be taken.

This is on top of the following changes that correctly reflected the pinning we're using to the release:

• https://github.com/pytorch-labs/pytorch-gha-infra/commit/ea8466e37083f3d450095b8eb671dcb09596d994 - released on: https://github.com/pytorch-labs/pytorch-gha-infra/actions/runs/12752938298
• pytorch/ci-infra@f564bcf - released on: https://github.com/pytorch/ci-infra/actions/runs/12753299999

cc @zxiiro @malfet @atalman @seemethere @ZainRizvi

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment

Name	Status	Preview	Updated (UTC)
torchci	⬜️ Ignored (Inspect)		Jan 13, 2025 6:13pm

[huydhn]

LGTM! We also need to clean up https://github.com/pytorch-labs/test-infra/blob/main/.github/scale-config.yml I think