ld-linux.so missing or at the wrong path in aarch64 SDKs generated from containers
#147
Assignees
Labels
bug
Something isn't working
Comments
euanh
added a commit
to euanh/swift-sdk-generator
that referenced
this issue
Nov 21, 2024
This commit extends the end-to-end tests to cover all currently-supported combinations of Swift version, Linux distribution and CPU architecture. For Ubuntu, there are also tests of SDKs built directly from packages and from container images. These tests run slowly (about 30 minutes on my machine), so they are disabled unless the SWIFT_SDK_GENERATOR_RUN_SLOW_TESTS environment variable is set. There is further scope for speeding up the generation of SDKs from container images, which might allow more tests to be included by default. Quite a lot of time, on macOS, is also spent waiting for the toolchain embedded in the SDK to be verified - this is more difficult to mitigate. Splitting the tests up makes it easier to see problems which affect particular configurations. Each integration test case tries to build two different programs; ideally these would be reported as separate test cases, but for now some manual re-testing is needed to separate the causes of test failures. Currently there are two problems on main which cause tests to fail: * Issue swiftlang#147 causes ld-linux-aarch64.so.1 not to be copied into SDKs built from containers. Attempted builds using these SDKs fails at the final link stage. * Issue swiftlang#152 causes new shims not to be copied into all Swift 6.0 SDKs. Attempted builds with these SDKs fail during compilation stage, if Foundation is used. In some cases issue swiftlang#152 masks issue swiftlang#147, because it occurs earlier in the build process. For instance, using ubuntu_aarch64_6.0.2-RELEASE_with-docker to build the 'Foundation' test case currently fails because of swiftlang#152, but if that were to be fixed it would then still fail because of swiftlang#147. | SDK | Hello World | Foundation | | ----------------------------------------- | ----------- | ---------- | | ubuntu_aarch64_5.9.2-RELEASE | ok | ok | | ubuntu_aarch64_5.9.2-RELEASE_with-docker | FAIL1 | FAIL1 | | ubuntu_aarch64_5.10.1-RELEASE | ok | ok | | ubuntu_aarch64_5.10.1-RELEASE_with-docker | FAIL1 | FAIL1 | | ubuntu_aarch64_6.0.2-RELEASE | ok | FAIL2 | | ubuntu_aarch64_6.0.2-RELEASE_with-docker | FAIL1 | FAIL2 | | | | | | ubuntu_x86_64_5.9.2-RELEASE | ok | ok | | ubuntu_x86_64_5.9.2-RELEASE_with-docker | ok | ok | | ubuntu_x86_64_5.10.1-RELEASE | ok | ok | | ubuntu_x86_64_5.10.1-RELEASE_with-docker | ok | ok | | ubuntu_x86_64_6.0.2-RELEASE | ok | FAIL2 | | ubuntu_x86_64_6.0.2-RELEASE_with-docker | ok | FAIL2 | | | | | | rhel_aarch64_5.9.2-RELEASE_with-docker | FAIL1 | FAIL1 | | rhel_aarch64_5.10.1-RELEASE_with-docker | FAIL1 | FAIL1 | | rhel_aarch64_6.0.2-RELEASE_with-docker | FAIL1 | FAIL2 | | | | | | rhel_x86_64_5.9.2-RELEASE_with-docker | ok | ok | | rhel_x86_64_5.10.1-RELEASE_with-docker | ok | ok | | rhel_x86_64_6.0.2-RELEASE_with-docker | ok | FAIL2 | FAIL1: cannot find /lib/ld-linux-aarch64.so.1 (Issue swiftlang#147) FAIL2: missing required module '_FoundationCShims' (Issue swiftlang#152)
euanh
added a commit
to euanh/swift-sdk-generator
that referenced
this issue
Nov 22, 2024
…tions | SDK | Hello World | Foundation | | ----------------------------------------- | ----------- | ---------- | | rhel_aarch64_5.9.2-RELEASE_with-docker | ok | ok | | rhel_aarch64_5.10.1-RELEASE_with-docker | ok | ok | | rhel_aarch64_6.0.2-RELEASE_with-docker | ok | FAIL2 | | | | | | rhel_x86_64_5.9.2-RELEASE_with-docker | ok | ok | | rhel_x86_64_5.10.1-RELEASE_with-docker | ok | ok | | rhel_x86_64_6.0.2-RELEASE_with-docker | ok | FAIL2 | | | | | | ubuntu_aarch64_5.9.2-RELEASE | ok | ok | | ubuntu_aarch64_5.9.2-RELEASE_with-docker | ok | ok | | ubuntu_aarch64_5.10.1-RELEASE | ok | ok | | ubuntu_aarch64_5.10.1-RELEASE_with-docker | ok | ok | | ubuntu_aarch64_6.0.2-RELEASE | ok | FAIL2 | | ubuntu_aarch64_6.0.2-RELEASE_with-docker | ok | FAIL2 | | | | | | ubuntu_x86_64_5.9.2-RELEASE | ok | ok | | ubuntu_x86_64_5.9.2-RELEASE_with-docker | ok | ok | | ubuntu_x86_64_5.10.1-RELEASE | ok | ok | | ubuntu_x86_64_5.10.1-RELEASE_with-docker | ok | ok | | ubuntu_x86_64_6.0.2-RELEASE | ok | FAIL2 | | ubuntu_x86_64_6.0.2-RELEASE_with-docker | ok | FAIL2 | FAIL1: cannot find /lib/ld-linux-aarch64.so.1 FAIL2: missing required module '_FoundationCShims' Fixes: swiftlang#147
euanh
added a commit
to euanh/swift-sdk-generator
that referenced
this issue
Nov 22, 2024
…tions Issue swiftlang#147 occurs because the ELF interpreter is found at different paths, and with different names, on different architectures. We happen to pick up the x86_64 interpreter because it is stored in /lib64, which we aready copy, but we miss the aarch64 interpreter. An ELF binary contains an interpreter path which can point anywhere, but there are defacto standard default locations. * According to https://refspecs.linuxbase.org/elf/x86_64-abi-0.99.pdf) the x86_64 / amd64 ABI specifies that the interpreter should be at /lib/ld64.so.1, but Linux overrides this and uses /lib64/ld-linux-x86-64.so.2. * I couldn't find a similar document for ARM64 / aarch64 - https://github.com/ARM-software/abi-aa/tree/main/aaelf64 is silent on the interpreter path - but most sources I could find place it at /lib/ld-linux-aarch64.so.1 - for example zulu-openjdk/zulu-openjdk#11 There is a pattern to the paths and filenames but it seems risky to try to construct them; instead this PR adds a new field to `Triple` which returns the default path for each architecture. For now we only support x86_64 and aarch64 - if new architectures are added in future we will need to add new entries for them. This change fixes issue swiftlang#147. Compared to swiftlang#153, all the basic 'hello world' tests now pass. The tests which import Foundation still fail on all Swift 6.0 SDKs. | SDK | Hello World | Foundation | | ----------------------------------------- | ----------- | ---------- | | rhel_aarch64_5.9.2-RELEASE_with-docker | ok | ok | | rhel_aarch64_5.10.1-RELEASE_with-docker | ok | ok | | rhel_aarch64_6.0.2-RELEASE_with-docker | ok | FAIL2 | | | | | | rhel_x86_64_5.9.2-RELEASE_with-docker | ok | ok | | rhel_x86_64_5.10.1-RELEASE_with-docker | ok | ok | | rhel_x86_64_6.0.2-RELEASE_with-docker | ok | FAIL2 | | | | | | ubuntu_aarch64_5.9.2-RELEASE | ok | ok | | ubuntu_aarch64_5.9.2-RELEASE_with-docker | ok | ok | | ubuntu_aarch64_5.10.1-RELEASE | ok | ok | | ubuntu_aarch64_5.10.1-RELEASE_with-docker | ok | ok | | ubuntu_aarch64_6.0.2-RELEASE | ok | FAIL2 | | ubuntu_aarch64_6.0.2-RELEASE_with-docker | ok | FAIL2 | | | | | | ubuntu_x86_64_5.9.2-RELEASE | ok | ok | | ubuntu_x86_64_5.9.2-RELEASE_with-docker | ok | ok | | ubuntu_x86_64_5.10.1-RELEASE | ok | ok | | ubuntu_x86_64_5.10.1-RELEASE_with-docker | ok | ok | | ubuntu_x86_64_6.0.2-RELEASE | ok | FAIL2 | | ubuntu_x86_64_6.0.2-RELEASE_with-docker | ok | FAIL2 | FAIL1: cannot find /lib/ld-linux-aarch64.so.1 FAIL2: missing required module '_FoundationCShims' Fixes: swiftlang#147
euanh
added a commit
to euanh/swift-sdk-generator
that referenced
this issue
Nov 25, 2024
This commit extends the end-to-end tests to cover all currently-supported combinations of Swift version, Linux distribution and CPU architecture. For Ubuntu, there are also tests of SDKs built directly from packages and from container images. These tests run slowly (about 30 minutes on my machine), so they are disabled unless the SWIFT_SDK_GENERATOR_RUN_SLOW_TESTS environment variable is set. There is further scope for speeding up the generation of SDKs from container images, which might allow more tests to be included by default. Quite a lot of time, on macOS, is also spent waiting for the toolchain embedded in the SDK to be verified - this is more difficult to mitigate. Splitting the tests up makes it easier to see problems which affect particular configurations. Each integration test case tries to build two different programs; ideally these would be reported as separate test cases, but for now some manual re-testing is needed to separate the causes of test failures. Currently there are two problems on main which cause tests to fail: * Issue swiftlang#147 causes ld-linux-aarch64.so.1 not to be copied into SDKs built from containers. Attempted builds using these SDKs fails at the final link stage. * Issue swiftlang#152 causes new shims not to be copied into all Swift 6.0 SDKs. Attempted builds with these SDKs fail during compilation stage, if Foundation is used. In some cases issue swiftlang#152 masks issue swiftlang#147, because it occurs earlier in the build process. For instance, using ubuntu_aarch64_6.0.2-RELEASE_with-docker to build the 'Foundation' test case currently fails because of swiftlang#152, but if that were to be fixed it would then still fail because of swiftlang#147. | SDK | Hello World | Foundation | | ----------------------------------------- | ----------- | ---------- | | ubuntu_aarch64_5.9.2-RELEASE | ok | ok | | ubuntu_aarch64_5.9.2-RELEASE_with-docker | FAIL1 | FAIL1 | | ubuntu_aarch64_5.10.1-RELEASE | ok | ok | | ubuntu_aarch64_5.10.1-RELEASE_with-docker | FAIL1 | FAIL1 | | ubuntu_aarch64_6.0.2-RELEASE | ok | FAIL2 | | ubuntu_aarch64_6.0.2-RELEASE_with-docker | FAIL1 | FAIL2 | | | | | | ubuntu_x86_64_5.9.2-RELEASE | ok | ok | | ubuntu_x86_64_5.9.2-RELEASE_with-docker | ok | ok | | ubuntu_x86_64_5.10.1-RELEASE | ok | ok | | ubuntu_x86_64_5.10.1-RELEASE_with-docker | ok | ok | | ubuntu_x86_64_6.0.2-RELEASE | ok | FAIL2 | | ubuntu_x86_64_6.0.2-RELEASE_with-docker | ok | FAIL2 | | | | | | rhel_aarch64_5.9.2-RELEASE_with-docker | FAIL1 | FAIL1 | | rhel_aarch64_5.10.1-RELEASE_with-docker | FAIL1 | FAIL1 | | rhel_aarch64_6.0.2-RELEASE_with-docker | FAIL1 | FAIL2 | | | | | | rhel_x86_64_5.9.2-RELEASE_with-docker | ok | ok | | rhel_x86_64_5.10.1-RELEASE_with-docker | ok | ok | | rhel_x86_64_6.0.2-RELEASE_with-docker | ok | FAIL2 | FAIL1: cannot find /lib/ld-linux-aarch64.so.1 (Issue swiftlang#147) FAIL2: missing required module '_FoundationCShims' (Issue swiftlang#152)
euanh
added a commit
to euanh/swift-sdk-generator
that referenced
this issue
Nov 25, 2024
…tions Issue swiftlang#147 occurs because the ELF interpreter is found at different paths, and with different names, on different architectures. We happen to pick up the x86_64 interpreter because it is stored in /lib64, which we aready copy, but we miss the aarch64 interpreter. An ELF binary contains an interpreter path which can point anywhere, but there are defacto standard default locations. * According to https://refspecs.linuxbase.org/elf/x86_64-abi-0.99.pdf) the x86_64 / amd64 ABI specifies that the interpreter should be at /lib/ld64.so.1, but Linux overrides this and uses /lib64/ld-linux-x86-64.so.2. * I couldn't find a similar document for ARM64 / aarch64 - https://github.com/ARM-software/abi-aa/tree/main/aaelf64 is silent on the interpreter path - but most sources I could find place it at /lib/ld-linux-aarch64.so.1 - for example zulu-openjdk/zulu-openjdk#11 There is a pattern to the paths and filenames but it seems risky to try to construct them; instead this PR adds a new field to `Triple` which returns the default path for each architecture. For now we only support x86_64 and aarch64 - if new architectures are added in future we will need to add new entries for them. This change fixes issue swiftlang#147. Compared to swiftlang#153, all the basic 'hello world' tests now pass. The tests which import Foundation still fail on all Swift 6.0 SDKs. | SDK | Hello World | Foundation | | ----------------------------------------- | ----------- | ---------- | | ubuntu_aarch64_5.9.2-RELEASE | ok | ok | | ubuntu_aarch64_5.9.2-RELEASE_with-docker | ok | ok | | ubuntu_aarch64_5.10.1-RELEASE | ok | ok | | ubuntu_aarch64_5.10.1-RELEASE_with-docker | ok | ok | | ubuntu_aarch64_6.0.2-RELEASE | ok | FAIL2 | | ubuntu_aarch64_6.0.2-RELEASE_with-docker | ok | FAIL2 | | | | | | ubuntu_x86_64_5.9.2-RELEASE | ok | ok | | ubuntu_x86_64_5.9.2-RELEASE_with-docker | ok | ok | | ubuntu_x86_64_5.10.1-RELEASE | ok | ok | | ubuntu_x86_64_5.10.1-RELEASE_with-docker | ok | ok | | ubuntu_x86_64_6.0.2-RELEASE | ok | FAIL2 | | ubuntu_x86_64_6.0.2-RELEASE_with-docker | ok | FAIL2 | | | | | | rhel_aarch64_5.9.2-RELEASE_with-docker | ok | ok | | rhel_aarch64_5.10.1-RELEASE_with-docker | ok | ok | | rhel_aarch64_6.0.2-RELEASE_with-docker | ok | FAIL2 | | | | | | rhel_x86_64_5.9.2-RELEASE_with-docker | ok | ok | | rhel_x86_64_5.10.1-RELEASE_with-docker | ok | ok | | rhel_x86_64_6.0.2-RELEASE_with-docker | ok | FAIL2 | FAIL1: cannot find /lib/ld-linux-aarch64.so.1 FAIL2: missing required module '_FoundationCShims' Fixes: swiftlang#147
euanh
added a commit
to euanh/swift-sdk-generator
that referenced
this issue
Nov 25, 2024
…tions Issue swiftlang#147 occurs because the ELF interpreter is found at different paths, and with different names, on different architectures. We happen to pick up the x86_64 interpreter because it is stored in /lib64, which we aready copy, but we miss the aarch64 interpreter. An ELF binary contains an interpreter path which can point anywhere, but there are defacto standard default locations. * According to https://refspecs.linuxbase.org/elf/x86_64-abi-0.99.pdf) the x86_64 / amd64 ABI specifies that the interpreter should be at /lib/ld64.so.1, but Linux overrides this and uses /lib64/ld-linux-x86-64.so.2. * I couldn't find a similar document for ARM64 / aarch64 - https://github.com/ARM-software/abi-aa/tree/main/aaelf64 is silent on the interpreter path - but most sources I could find place it at /lib/ld-linux-aarch64.so.1 - for example zulu-openjdk/zulu-openjdk#11 There is a pattern to the paths and filenames but it seems risky to try to construct them; instead this PR adds a new field to `Triple` which returns the default path for each architecture. For now we only support x86_64 and aarch64 - if new architectures are added in future we will need to add new entries for them. This change fixes issue swiftlang#147. Compared to swiftlang#153, all the basic 'hello world' tests now pass. The tests which import Foundation still fail on all Swift 6.0 SDKs. | SDK | Hello World | Foundation | | ----------------------------------------- | ----------- | ---------- | | ubuntu_aarch64_5.9.2-RELEASE | ok | ok | | ubuntu_aarch64_5.9.2-RELEASE_with-docker | ok | ok | | ubuntu_aarch64_5.10.1-RELEASE | ok | ok | | ubuntu_aarch64_5.10.1-RELEASE_with-docker | ok | ok | | ubuntu_aarch64_6.0.2-RELEASE | ok | FAIL2 | | ubuntu_aarch64_6.0.2-RELEASE_with-docker | ok | FAIL2 | | | | | | ubuntu_x86_64_5.9.2-RELEASE | ok | ok | | ubuntu_x86_64_5.9.2-RELEASE_with-docker | ok | ok | | ubuntu_x86_64_5.10.1-RELEASE | ok | ok | | ubuntu_x86_64_5.10.1-RELEASE_with-docker | ok | ok | | ubuntu_x86_64_6.0.2-RELEASE | ok | FAIL2 | | ubuntu_x86_64_6.0.2-RELEASE_with-docker | ok | FAIL2 | | | | | | rhel_aarch64_5.9.2-RELEASE_with-docker | ok | ok | | rhel_aarch64_5.10.1-RELEASE_with-docker | ok | ok | | rhel_aarch64_6.0.2-RELEASE_with-docker | ok | FAIL2 | | | | | | rhel_x86_64_5.9.2-RELEASE_with-docker | ok | ok | | rhel_x86_64_5.10.1-RELEASE_with-docker | ok | ok | | rhel_x86_64_6.0.2-RELEASE_with-docker | ok | FAIL2 | FAIL1: cannot find /lib/ld-linux-aarch64.so.1 FAIL2: missing required module '_FoundationCShims' Fixes: swiftlang#147
euanh
added a commit
to euanh/swift-sdk-generator
that referenced
this issue
Nov 26, 2024
Building with a Swift 6.0 SDK fails because the new `_FoundationCShims`
framework headers can't be found:
<unknown>:0: error: missing required module '_FoundationCShims'
The necessary files are present in `$PLATFORM.sdk/usr/lib/swift`,
but are not part of the list which is copied to
`$PLATFORM.sdk/usr/include`. The Static Linux SDK and WASM SDK
generator don't copy these files into /usr/include; instead they
set the `swiftResourcesPath` and `swiftStaticResourcesPath` fields
in `swift-sdk.json`, which causes the build system to look in those
paths for framework headers.
We can do the same for Linux SDKs, however we must stop copying the
files to `/usr/include`, otherwise the build will fail because the
framework header definitions are duplicated:
error: redefinition of module 'DispatchIntrospection'
6 | }
7 |
8 | module DispatchIntrospection [system] [extern_c] {
| `- error: redefinition of module 'DispatchIntrospection'
9 | header "introspection.h"
10 | export *
This change fixes issue swiftlang#152. Compared to swiftlang#153, the end-to-end
tests now pass for all supported Swift versions, distributions and
architectures.
| SDK | Hello World | Foundation |
| ----------------------------------------- | ----------- | ---------- |
| ubuntu_aarch64_5.9.2-RELEASE | ok | ok |
| ubuntu_aarch64_5.9.2-RELEASE_with-docker | ok | ok |
| ubuntu_aarch64_5.10.1-RELEASE | ok | ok |
| ubuntu_aarch64_5.10.1-RELEASE_with-docker | ok | ok |
| ubuntu_aarch64_6.0.2-RELEASE | ok | ok |
| ubuntu_aarch64_6.0.2-RELEASE_with-docker | ok | ok |
| | | |
| ubuntu_x86_64_5.9.2-RELEASE | ok | ok |
| ubuntu_x86_64_5.9.2-RELEASE_with-docker | ok | ok |
| ubuntu_x86_64_5.10.1-RELEASE | ok | ok |
| ubuntu_x86_64_5.10.1-RELEASE_with-docker | ok | ok |
| ubuntu_x86_64_6.0.2-RELEASE | ok | ok |
| ubuntu_x86_64_6.0.2-RELEASE_with-docker | ok | ok |
| | | |
| rhel_aarch64_5.9.2-RELEASE_with-docker | ok | ok |
| rhel_aarch64_5.10.1-RELEASE_with-docker | ok | ok |
| rhel_aarch64_6.0.2-RELEASE_with-docker | ok | ok |
| | | |
| rhel_x86_64_5.9.2-RELEASE_with-docker | ok | ok |
| rhel_x86_64_5.10.1-RELEASE_with-docker | ok | ok |
| rhel_x86_64_6.0.2-RELEASE_with-docker | ok | ok |
FAIL1: cannot find /lib/ld-linux-aarch64.so.1 (swiftlang#147)
FAIL2: missing required module '_FoundationCShims' (swiftlang#152)
Fixes: swiftlang#152
Depends on: swiftlang#153, swiftlang#154
euanh
added a commit
to euanh/swift-sdk-generator
that referenced
this issue
Nov 27, 2024
…tions Issue swiftlang#147 occurs because the ELF interpreter is found at different paths, and with different names, on different architectures. We happen to pick up the x86_64 interpreter because it is stored in /lib64, which we aready copy, but we miss the aarch64 interpreter. An ELF binary contains an interpreter path which can point anywhere, but there are defacto standard default locations. * According to https://refspecs.linuxbase.org/elf/x86_64-abi-0.99.pdf) the x86_64 / amd64 ABI specifies that the interpreter should be at /lib/ld64.so.1, but Linux overrides this and uses /lib64/ld-linux-x86-64.so.2. * I couldn't find a similar document for ARM64 / aarch64 - https://github.com/ARM-software/abi-aa/tree/main/aaelf64 is silent on the interpreter path - but most sources I could find place it at /lib/ld-linux-aarch64.so.1 - for example zulu-openjdk/zulu-openjdk#11 There is a pattern to the paths and filenames but it seems risky to try to construct them; instead this PR adds a new field to `Triple` which returns the default path for each architecture. For now we only support x86_64 and aarch64 - if new architectures are added in future we will need to add new entries for them. This change fixes issue swiftlang#147. Compared to swiftlang#153, all the basic 'hello world' tests now pass. The tests which import Foundation still fail on all Swift 6.0 SDKs. | SDK | Hello World | Foundation | | ----------------------------------------- | ----------- | ---------- | | ubuntu_aarch64_5.9.2-RELEASE | ok | ok | | ubuntu_aarch64_5.9.2-RELEASE_with-docker | ok | ok | | ubuntu_aarch64_5.10.1-RELEASE | ok | ok | | ubuntu_aarch64_5.10.1-RELEASE_with-docker | ok | ok | | ubuntu_aarch64_6.0.2-RELEASE | ok | FAIL2 | | ubuntu_aarch64_6.0.2-RELEASE_with-docker | ok | FAIL2 | | | | | | ubuntu_x86_64_5.9.2-RELEASE | ok | ok | | ubuntu_x86_64_5.9.2-RELEASE_with-docker | ok | ok | | ubuntu_x86_64_5.10.1-RELEASE | ok | ok | | ubuntu_x86_64_5.10.1-RELEASE_with-docker | ok | ok | | ubuntu_x86_64_6.0.2-RELEASE | ok | FAIL2 | | ubuntu_x86_64_6.0.2-RELEASE_with-docker | ok | FAIL2 | | | | | | rhel_aarch64_5.9.2-RELEASE_with-docker | ok | ok | | rhel_aarch64_5.10.1-RELEASE_with-docker | ok | ok | | rhel_aarch64_6.0.2-RELEASE_with-docker | ok | FAIL2 | | | | | | rhel_x86_64_5.9.2-RELEASE_with-docker | ok | ok | | rhel_x86_64_5.10.1-RELEASE_with-docker | ok | ok | | rhel_x86_64_6.0.2-RELEASE_with-docker | ok | FAIL2 | FAIL1: cannot find /lib/ld-linux-aarch64.so.1 FAIL2: missing required module '_FoundationCShims' Fixes: swiftlang#147
euanh
added a commit
to euanh/swift-sdk-generator
that referenced
this issue
Nov 27, 2024
Building with a Swift 6.0 SDK fails because the new `_FoundationCShims`
framework headers can't be found:
<unknown>:0: error: missing required module '_FoundationCShims'
The necessary files are present in `$PLATFORM.sdk/usr/lib/swift`,
but are not part of the list which is copied to
`$PLATFORM.sdk/usr/include`. The Static Linux SDK and WASM SDK
generator don't copy these files into /usr/include; instead they
set the `swiftResourcesPath` and `swiftStaticResourcesPath` fields
in `swift-sdk.json`, which causes the build system to look in those
paths for framework headers.
We can do the same for Linux SDKs, however we must stop copying the
files to `/usr/include`, otherwise the build will fail because the
framework header definitions are duplicated:
error: redefinition of module 'DispatchIntrospection'
6 | }
7 |
8 | module DispatchIntrospection [system] [extern_c] {
| `- error: redefinition of module 'DispatchIntrospection'
9 | header "introspection.h"
10 | export *
This change fixes issue swiftlang#152. Compared to swiftlang#153, the end-to-end
tests now pass for all supported Swift versions, distributions and
architectures.
| SDK | Hello World | Foundation |
| ----------------------------------------- | ----------- | ---------- |
| ubuntu_aarch64_5.9.2-RELEASE | ok | ok |
| ubuntu_aarch64_5.9.2-RELEASE_with-docker | ok | ok |
| ubuntu_aarch64_5.10.1-RELEASE | ok | ok |
| ubuntu_aarch64_5.10.1-RELEASE_with-docker | ok | ok |
| ubuntu_aarch64_6.0.2-RELEASE | ok | ok |
| ubuntu_aarch64_6.0.2-RELEASE_with-docker | ok | ok |
| | | |
| ubuntu_x86_64_5.9.2-RELEASE | ok | ok |
| ubuntu_x86_64_5.9.2-RELEASE_with-docker | ok | ok |
| ubuntu_x86_64_5.10.1-RELEASE | ok | ok |
| ubuntu_x86_64_5.10.1-RELEASE_with-docker | ok | ok |
| ubuntu_x86_64_6.0.2-RELEASE | ok | ok |
| ubuntu_x86_64_6.0.2-RELEASE_with-docker | ok | ok |
| | | |
| rhel_aarch64_5.9.2-RELEASE_with-docker | ok | ok |
| rhel_aarch64_5.10.1-RELEASE_with-docker | ok | ok |
| rhel_aarch64_6.0.2-RELEASE_with-docker | ok | ok |
| | | |
| rhel_x86_64_5.9.2-RELEASE_with-docker | ok | ok |
| rhel_x86_64_5.10.1-RELEASE_with-docker | ok | ok |
| rhel_x86_64_6.0.2-RELEASE_with-docker | ok | ok |
FAIL1: cannot find /lib/ld-linux-aarch64.so.1 (swiftlang#147)
FAIL2: missing required module '_FoundationCShims' (swiftlang#152)
Fixes: swiftlang#152
Depends on: swiftlang#153, swiftlang#154
euanh
added a commit
that referenced
this issue
Nov 27, 2024
…ult (#153) This commit extends the end-to-end tests to cover all currently-supported combinations of Swift version, Linux distribution and CPU architecture. For Ubuntu, there are also tests of SDKs built directly from packages and from container images. These tests run slowly (more than 30 minutes on my machine), so they are disabled unless the SWIFT_SDK_GENERATOR_RUN_SLOW_TESTS environment variable is set. There is further scope for speeding up the generation of SDKs from container images, which might allow more tests to be included by default. Quite a lot of time, on macOS, is also spent waiting for the toolchain embedded in the SDK to be verified - this is more difficult to mitigate. Splitting the tests up makes it easier to see problems which affect particular configurations. Each integration test case tries to build two different programs; ideally these would be reported as separate test cases, but for now some manual re-testing is needed to separate the causes of test failures. Currently there are two pre-existing problems on main which cause tests to fail: * Issue #147 causes ld-linux-aarch64.so.1 not to be copied into SDKs built from containers. Attempted builds using these SDKs fails at the final link stage. * Issue #152 causes new shims not to be copied into all Swift 6.0 SDKs. Attempted builds with these SDKs fail during compilation stage, if Foundation is used. In some cases issue #152 masks issue #147, because it occurs earlier in the build process. For instance, using ubuntu_aarch64_6.0.2-RELEASE_with-docker to build the 'Foundation' test case currently fails because of #152, but if that were to be fixed it would then still fail because of #147. | SDK | Hello World | Foundation | | ----------------------------------------- | ----------- | ---------- | | ubuntu_aarch64_5.9.2-RELEASE | ok | ok | | ubuntu_aarch64_5.9.2-RELEASE_with-docker | FAIL1 | FAIL1 | | ubuntu_aarch64_5.10.1-RELEASE | ok | ok | | ubuntu_aarch64_5.10.1-RELEASE_with-docker | FAIL1 | FAIL1 | | ubuntu_aarch64_6.0.2-RELEASE | ok | FAIL2 | | ubuntu_aarch64_6.0.2-RELEASE_with-docker | FAIL1 | FAIL2 | | | | | | ubuntu_x86_64_5.9.2-RELEASE | ok | ok | | ubuntu_x86_64_5.9.2-RELEASE_with-docker | ok | ok | | ubuntu_x86_64_5.10.1-RELEASE | ok | ok | | ubuntu_x86_64_5.10.1-RELEASE_with-docker | ok | ok | | ubuntu_x86_64_6.0.2-RELEASE | ok | FAIL2 | | ubuntu_x86_64_6.0.2-RELEASE_with-docker | ok | FAIL2 | | | | | | rhel_aarch64_5.9.2-RELEASE_with-docker | FAIL1 | FAIL1 | | rhel_aarch64_5.10.1-RELEASE_with-docker | FAIL1 | FAIL1 | | rhel_aarch64_6.0.2-RELEASE_with-docker | FAIL1 | FAIL2 | | | | | | rhel_x86_64_5.9.2-RELEASE_with-docker | ok | ok | | rhel_x86_64_5.10.1-RELEASE_with-docker | ok | ok | | rhel_x86_64_6.0.2-RELEASE_with-docker | ok | FAIL2 | FAIL1: cannot find /lib/ld-linux-aarch64.so.1 (Issue #147) FAIL2: missing required module '_FoundationCShims' (Issue #152)
euanh
added a commit
to euanh/swift-sdk-generator
that referenced
this issue
Nov 27, 2024
…tions Issue swiftlang#147 occurs because the ELF interpreter is found at different paths, and with different names, on different architectures. We happen to pick up the x86_64 interpreter because it is stored in /lib64, which we aready copy, but we miss the aarch64 interpreter. An ELF binary contains an interpreter path which can point anywhere, but there are defacto standard default locations. * According to https://refspecs.linuxbase.org/elf/x86_64-abi-0.99.pdf) the x86_64 / amd64 ABI specifies that the interpreter should be at /lib/ld64.so.1, but Linux overrides this and uses /lib64/ld-linux-x86-64.so.2. * I couldn't find a similar document for ARM64 / aarch64 - https://github.com/ARM-software/abi-aa/tree/main/aaelf64 is silent on the interpreter path - but most sources I could find place it at /lib/ld-linux-aarch64.so.1 - for example zulu-openjdk/zulu-openjdk#11 There is a pattern to the paths and filenames but it seems risky to try to construct them; instead this PR adds a new field to `Triple` which returns the default path for each architecture. For now we only support x86_64 and aarch64 - if new architectures are added in future we will need to add new entries for them. This change fixes issue swiftlang#147. Compared to swiftlang#153, all the basic 'hello world' tests now pass. The tests which import Foundation still fail on all Swift 6.0 SDKs. | SDK | Hello World | Foundation | | ----------------------------------------- | ----------- | ---------- | | ubuntu_aarch64_5.9.2-RELEASE | ok | ok | | ubuntu_aarch64_5.9.2-RELEASE_with-docker | ok | ok | | ubuntu_aarch64_5.10.1-RELEASE | ok | ok | | ubuntu_aarch64_5.10.1-RELEASE_with-docker | ok | ok | | ubuntu_aarch64_6.0.2-RELEASE | ok | FAIL2 | | ubuntu_aarch64_6.0.2-RELEASE_with-docker | ok | FAIL2 | | | | | | ubuntu_x86_64_5.9.2-RELEASE | ok | ok | | ubuntu_x86_64_5.9.2-RELEASE_with-docker | ok | ok | | ubuntu_x86_64_5.10.1-RELEASE | ok | ok | | ubuntu_x86_64_5.10.1-RELEASE_with-docker | ok | ok | | ubuntu_x86_64_6.0.2-RELEASE | ok | FAIL2 | | ubuntu_x86_64_6.0.2-RELEASE_with-docker | ok | FAIL2 | | | | | | rhel_aarch64_5.9.2-RELEASE_with-docker | ok | ok | | rhel_aarch64_5.10.1-RELEASE_with-docker | ok | ok | | rhel_aarch64_6.0.2-RELEASE_with-docker | ok | FAIL2 | | | | | | rhel_x86_64_5.9.2-RELEASE_with-docker | ok | ok | | rhel_x86_64_5.10.1-RELEASE_with-docker | ok | ok | | rhel_x86_64_6.0.2-RELEASE_with-docker | ok | FAIL2 | FAIL1: cannot find /lib/ld-linux-aarch64.so.1 FAIL2: missing required module '_FoundationCShims' Fixes: swiftlang#147
euanh
added a commit
that referenced
this issue
Nov 27, 2024
…tions (#154) Issue #147 occurs because the ELF interpreter is found at different paths, and with different names, on different architectures. We happen to pick up the x86_64 interpreter because it is stored in /lib64, which we aready copy, but we miss the aarch64 interpreter. An ELF binary contains an interpreter path which can point anywhere, but there are defacto standard default locations. * According to https://refspecs.linuxbase.org/elf/x86_64-abi-0.99.pdf) the x86_64 / amd64 ABI specifies that the interpreter should be at /lib/ld64.so.1, but Linux overrides this and uses /lib64/ld-linux-x86-64.so.2. * I couldn't find a similar document for ARM64 / aarch64 - https://github.com/ARM-software/abi-aa/tree/main/aaelf64 is silent on the interpreter path - but most sources I could find place it at /lib/ld-linux-aarch64.so.1 - for example zulu-openjdk/zulu-openjdk#11 There is a pattern to the paths and filenames but it seems risky to try to construct them; instead this PR adds a new field to `Triple` which returns the default path for each architecture. For now we only support x86_64 and aarch64 - if new architectures are added in future we will need to add new entries for them. This change fixes issue #147. Compared to #153, all the basic 'hello world' tests now pass. The tests which import Foundation still fail on all Swift 6.0 SDKs. | SDK | Hello World | Foundation | | ----------------------------------------- | ----------- | ---------- | | ubuntu_aarch64_5.9.2-RELEASE | ok | ok | | ubuntu_aarch64_5.9.2-RELEASE_with-docker | ok | ok | | ubuntu_aarch64_5.10.1-RELEASE | ok | ok | | ubuntu_aarch64_5.10.1-RELEASE_with-docker | ok | ok | | ubuntu_aarch64_6.0.2-RELEASE | ok | FAIL2 | | ubuntu_aarch64_6.0.2-RELEASE_with-docker | ok | FAIL2 | | | | | | ubuntu_x86_64_5.9.2-RELEASE | ok | ok | | ubuntu_x86_64_5.9.2-RELEASE_with-docker | ok | ok | | ubuntu_x86_64_5.10.1-RELEASE | ok | ok | | ubuntu_x86_64_5.10.1-RELEASE_with-docker | ok | ok | | ubuntu_x86_64_6.0.2-RELEASE | ok | FAIL2 | | ubuntu_x86_64_6.0.2-RELEASE_with-docker | ok | FAIL2 | | | | | | rhel_aarch64_5.9.2-RELEASE_with-docker | ok | ok | | rhel_aarch64_5.10.1-RELEASE_with-docker | ok | ok | | rhel_aarch64_6.0.2-RELEASE_with-docker | ok | FAIL2 | | | | | | rhel_x86_64_5.9.2-RELEASE_with-docker | ok | ok | | rhel_x86_64_5.10.1-RELEASE_with-docker | ok | ok | | rhel_x86_64_6.0.2-RELEASE_with-docker | ok | FAIL2 | FAIL1: cannot find /lib/ld-linux-aarch64.so.1 FAIL2: missing required module '_FoundationCShims' Fixes: #147
euanh
added a commit
to euanh/swift-sdk-generator
that referenced
this issue
Nov 27, 2024
Building with a Swift 6.0 SDK fails because the new `_FoundationCShims`
framework headers can't be found:
<unknown>:0: error: missing required module '_FoundationCShims'
The necessary files are present in `$PLATFORM.sdk/usr/lib/swift`,
but are not part of the list which is copied to
`$PLATFORM.sdk/usr/include`. The Static Linux SDK and WASM SDK
generator don't copy these files into /usr/include; instead they
set the `swiftResourcesPath` and `swiftStaticResourcesPath` fields
in `swift-sdk.json`, which causes the build system to look in those
paths for framework headers.
We can do the same for Linux SDKs, however we must stop copying the
files to `/usr/include`, otherwise the build will fail because the
framework header definitions are duplicated:
error: redefinition of module 'DispatchIntrospection'
6 | }
7 |
8 | module DispatchIntrospection [system] [extern_c] {
| `- error: redefinition of module 'DispatchIntrospection'
9 | header "introspection.h"
10 | export *
This change fixes issue swiftlang#152. Compared to swiftlang#153, the end-to-end
tests now pass for all supported Swift versions, distributions and
architectures.
| SDK | Hello World | Foundation |
| ----------------------------------------- | ----------- | ---------- |
| ubuntu_aarch64_5.9.2-RELEASE | ok | ok |
| ubuntu_aarch64_5.9.2-RELEASE_with-docker | ok | ok |
| ubuntu_aarch64_5.10.1-RELEASE | ok | ok |
| ubuntu_aarch64_5.10.1-RELEASE_with-docker | ok | ok |
| ubuntu_aarch64_6.0.2-RELEASE | ok | ok |
| ubuntu_aarch64_6.0.2-RELEASE_with-docker | ok | ok |
| | | |
| ubuntu_x86_64_5.9.2-RELEASE | ok | ok |
| ubuntu_x86_64_5.9.2-RELEASE_with-docker | ok | ok |
| ubuntu_x86_64_5.10.1-RELEASE | ok | ok |
| ubuntu_x86_64_5.10.1-RELEASE_with-docker | ok | ok |
| ubuntu_x86_64_6.0.2-RELEASE | ok | ok |
| ubuntu_x86_64_6.0.2-RELEASE_with-docker | ok | ok |
| | | |
| rhel_aarch64_5.9.2-RELEASE_with-docker | ok | ok |
| rhel_aarch64_5.10.1-RELEASE_with-docker | ok | ok |
| rhel_aarch64_6.0.2-RELEASE_with-docker | ok | ok |
| | | |
| rhel_x86_64_5.9.2-RELEASE_with-docker | ok | ok |
| rhel_x86_64_5.10.1-RELEASE_with-docker | ok | ok |
| rhel_x86_64_6.0.2-RELEASE_with-docker | ok | ok |
FAIL1: cannot find /lib/ld-linux-aarch64.so.1 (swiftlang#147)
FAIL2: missing required module '_FoundationCShims' (swiftlang#152)
Fixes: swiftlang#152
Depends on: swiftlang#153, swiftlang#154
euanh
added a commit
to euanh/swift-sdk-generator
that referenced
this issue
Nov 28, 2024
Building with a Swift 6.0 SDK fails because the new `_FoundationCShims`
framework headers can't be found:
<unknown>:0: error: missing required module '_FoundationCShims'
The necessary files are present in `$PLATFORM.sdk/usr/lib/swift`,
but are not part of the list which is copied to
`$PLATFORM.sdk/usr/include`. The Static Linux SDK and WASM SDK
generator don't copy these files into /usr/include; instead they
set the `swiftResourcesPath` and `swiftStaticResourcesPath` fields
in `swift-sdk.json`, which causes the build system to look in those
paths for framework headers.
We can do the same for Linux SDKs, however we must stop copying the
files to `/usr/include`, otherwise the build will fail because the
framework header definitions are duplicated:
error: redefinition of module 'DispatchIntrospection'
6 | }
7 |
8 | module DispatchIntrospection [system] [extern_c] {
| `- error: redefinition of module 'DispatchIntrospection'
9 | header "introspection.h"
10 | export *
This change fixes issue swiftlang#152. Compared to swiftlang#153, the end-to-end
tests now pass for all supported Swift versions, distributions and
architectures.
| SDK | Hello World | Foundation |
| ----------------------------------------- | ----------- | ---------- |
| ubuntu_aarch64_5.9.2-RELEASE | ok | ok |
| ubuntu_aarch64_5.9.2-RELEASE_with-docker | ok | ok |
| ubuntu_aarch64_5.10.1-RELEASE | ok | ok |
| ubuntu_aarch64_5.10.1-RELEASE_with-docker | ok | ok |
| ubuntu_aarch64_6.0.2-RELEASE | ok | ok |
| ubuntu_aarch64_6.0.2-RELEASE_with-docker | ok | ok |
| | | |
| ubuntu_x86_64_5.9.2-RELEASE | ok | ok |
| ubuntu_x86_64_5.9.2-RELEASE_with-docker | ok | ok |
| ubuntu_x86_64_5.10.1-RELEASE | ok | ok |
| ubuntu_x86_64_5.10.1-RELEASE_with-docker | ok | ok |
| ubuntu_x86_64_6.0.2-RELEASE | ok | ok |
| ubuntu_x86_64_6.0.2-RELEASE_with-docker | ok | ok |
| | | |
| rhel_aarch64_5.9.2-RELEASE_with-docker | ok | ok |
| rhel_aarch64_5.10.1-RELEASE_with-docker | ok | ok |
| rhel_aarch64_6.0.2-RELEASE_with-docker | ok | ok |
| | | |
| rhel_x86_64_5.9.2-RELEASE_with-docker | ok | ok |
| rhel_x86_64_5.10.1-RELEASE_with-docker | ok | ok |
| rhel_x86_64_6.0.2-RELEASE_with-docker | ok | ok |
FAIL1: cannot find /lib/ld-linux-aarch64.so.1 (swiftlang#147)
FAIL2: missing required module '_FoundationCShims' (swiftlang#152)
Fixes: swiftlang#152
Depends on: swiftlang#153, swiftlang#154
euanh
added a commit
that referenced
this issue
Nov 28, 2024
Building with a Swift 6.0 SDK fails because the new `_FoundationCShims`
framework headers can't be found:
<unknown>:0: error: missing required module '_FoundationCShims'
The necessary files are present in `$PLATFORM.sdk/usr/lib/swift`,
but are not part of the list which is copied to
`$PLATFORM.sdk/usr/include`. The Static Linux SDK and WASM SDK
generator don't copy these files into /usr/include; instead they
set the `swiftResourcesPath` and `swiftStaticResourcesPath` fields
in `swift-sdk.json`, which causes the build system to look in those
paths for framework headers.
We can do the same for Linux SDKs, however we must stop copying the
files to `/usr/include`, otherwise the build will fail because the
framework header definitions are duplicated:
error: redefinition of module 'DispatchIntrospection'
6 | }
7 |
8 | module DispatchIntrospection [system] [extern_c] {
| `- error: redefinition of module 'DispatchIntrospection'
9 | header "introspection.h"
10 | export *
This change fixes issue #152. Compared to #153, the end-to-end
tests now pass for all supported Swift versions, distributions and
architectures.
| SDK | Hello World | Foundation |
| ----------------------------------------- | ----------- | ---------- |
| ubuntu_aarch64_5.9.2-RELEASE | ok | ok |
| ubuntu_aarch64_5.9.2-RELEASE_with-docker | ok | ok |
| ubuntu_aarch64_5.10.1-RELEASE | ok | ok |
| ubuntu_aarch64_5.10.1-RELEASE_with-docker | ok | ok |
| ubuntu_aarch64_6.0.2-RELEASE | ok | ok |
| ubuntu_aarch64_6.0.2-RELEASE_with-docker | ok | ok |
| | | |
| ubuntu_x86_64_5.9.2-RELEASE | ok | ok |
| ubuntu_x86_64_5.9.2-RELEASE_with-docker | ok | ok |
| ubuntu_x86_64_5.10.1-RELEASE | ok | ok |
| ubuntu_x86_64_5.10.1-RELEASE_with-docker | ok | ok |
| ubuntu_x86_64_6.0.2-RELEASE | ok | ok |
| ubuntu_x86_64_6.0.2-RELEASE_with-docker | ok | ok |
| | | |
| rhel_aarch64_5.9.2-RELEASE_with-docker | ok | ok |
| rhel_aarch64_5.10.1-RELEASE_with-docker | ok | ok |
| rhel_aarch64_6.0.2-RELEASE_with-docker | ok | ok |
| | | |
| rhel_x86_64_5.9.2-RELEASE_with-docker | ok | ok |
| rhel_x86_64_5.10.1-RELEASE_with-docker | ok | ok |
| rhel_x86_64_6.0.2-RELEASE_with-docker | ok | ok |
FAIL1: cannot find /lib/ld-linux-aarch64.so.1 (#147)
FAIL2: missing required module '_FoundationCShims' (#152)
Fixes: #152
Depends on: #153, #154
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
A test binary fails to link when using an
aarch64SDK generated from a container:This is because
/lib/ld-linux-aarch64.so.1cannot be found in the SDK.Symptoms
Symptoms vary depending on how the SDK was built.
SDKs built directly from Ubuntu packages are not affected
x86_64 SDKs are not affected
Ubuntu-derived SDKs built from containers contain the correct file, but not at the expected path:
RHEL-derived SDKs built from containers do not contain the file at all:
The expected name and path of
ld-linux.sovaries depending on the architecture and some distributions store the file elsewhere and symlink it into the expected location. The generator is not copying all these files into the SDK.Related to #138
The text was updated successfully, but these errors were encountered: