Layring for cross image and /scripts directory

[am11]

Arcade is cloned in the layer where llvm is built src/azurelinux/3.0/net10.0/crossdeps-builder/amd64/Dockerfile. If we make a change for one platform in rootfs script which resides in arcade repo, then we need to rebuild multiple layers in order to pick up that change:

• src/azurelinux/3.0/net10.0/crossdeps-llvm/amd64/Dockerfile
• src/azurelinux/3.0/net10.0/crossdeps-builder/amd64/Dockerfile
• src/azurelinux/3.0/net10.0/cross/<that platform>

this way we end up rebuilding a lot of unrelated stuff and waste many hours. We should move git clone step to cross dockerfiles because that's where we use the scripts.

cc @sbomer

[sbomer]

I left a related comment in #1313 (comment).

It's expected that we would rebuild the /cross images when we flow a change to arcade.

The Dockerfile layers are also set up to avoid having to rebuild llvm when touching the arcade clone step - but unfortunately our ci doesn't reuse cached layers in this way. I'd love to see that limitation addressed if possible, but lacking that we may be able to fix it by moving the arcade clone to a separate image.

I wouldn't move it to the cross Dockerfiles, because they should ideally all share the same clone of arcade.

[mthalman]

but lacking that we may be able to fix it by moving the arcade clone to a separate image.

If the Dockerfile of this image was to be left to clone as it does today, we would never rebuild the image since the Dockerfile won't have been changed. So the image would get stale over time. I think we'd want to check out a specific commit of the arcade repo instead. But of course then we would want to have a system in place for updating that commit.

[sbomer]

Agreed - to be clear, there are two separate problems:

1. The arcade clone doesn't get updated until we touch that file or do a non-cached build, related to Define a repo policy for referencing specific software versions #1265.
2. When we touch the Dockerfile that clones arcade to force an update, not all of the dependent images get updated. I filed Dockerfiles aren't rebuilt when there are changes to dependent Dockerfiles docker-tools#1525 to track this, and the current issue can track adding a workaround if we really need it.

[am11]

My observation is:

• Touching crossdeps-llvm rebuilds all cross images.
• Touching crossdeps-builder does not rebuild crossdeps-llvm.

If you've observed the same then there is more to it.

[sbomer]

The crossdeps-llvm image is a base image of the cross images. Thes crossdeps-builder image is a base image of an intermediate stage, but not of the final stage of the crossdeps-llvm image. I suspect that's what's causing the difference.