Skip to content

Commit

Permalink
tests: Refactor and expand end-to-end test coverage, disabled by default
Browse files Browse the repository at this point in the history
This commit extends the end-to-end tests to cover all currently-supported
combinations of Swift version, Linux distribution and CPU architecture.
For Ubuntu, there are also tests of SDKs built directly from packages
and from container images.

These tests run slowly (about 30 minutes on my machine), so they
are disabled unless the SWIFT_SDK_GENERATOR_RUN_SLOW_TESTS environment
variable is set.

There is further scope for speeding up the generation of SDKs from
container images, which might allow more tests to be included by default.
Quite a lot of time, on macOS, is also spent waiting for the toolchain
embedded in the SDK to be verified - this is more difficult to mitigate.

Splitting the tests up makes it easier to see problems which affect
particular configurations.   Each integration test case tries to build
two different programs;  ideally these would be reported as separate
test cases, but for now some manual re-testing is needed to separate
the causes of test failures.

Currently there are two problems on main which cause tests to fail:
 * Issue swiftlang#147 causes ld-linux-aarch64.so.1 not to be copied into
   SDKs built from containers.  Attempted builds using these SDKs
   fails at the final link stage.
 * Issue swiftlang#152 causes new shims not to be copied into all Swift 6.0
   SDKs.   Attempted builds with these SDKs fail during compilation
   stage, if Foundation is used.

In some cases issue swiftlang#152 masks issue swiftlang#147, because it occurs earlier
in the build process.   For instance, using ubuntu_aarch64_6.0.2-RELEASE_with-docker
to build the 'Foundation' test case currently fails because of swiftlang#152,
but if that were to be fixed it would then still fail because of swiftlang#147.

| SDK                                       | Hello World | Foundation |
| ----------------------------------------- | ----------- | ---------- |
| ubuntu_aarch64_5.9.2-RELEASE              | ok          | ok         |
| ubuntu_aarch64_5.9.2-RELEASE_with-docker  | FAIL1       | FAIL1      |
| ubuntu_aarch64_5.10.1-RELEASE             | ok          | ok         |
| ubuntu_aarch64_5.10.1-RELEASE_with-docker | FAIL1       | FAIL1      |
| ubuntu_aarch64_6.0.2-RELEASE              | ok          | FAIL2      |
| ubuntu_aarch64_6.0.2-RELEASE_with-docker  | FAIL1       | FAIL2      |
|                                           |             |            |
| ubuntu_x86_64_5.9.2-RELEASE               | ok          | ok         |
| ubuntu_x86_64_5.9.2-RELEASE_with-docker   | ok          | ok         |
| ubuntu_x86_64_5.10.1-RELEASE              | ok          | ok         |
| ubuntu_x86_64_5.10.1-RELEASE_with-docker  | ok          | ok         |
| ubuntu_x86_64_6.0.2-RELEASE               | ok          | FAIL2      |
| ubuntu_x86_64_6.0.2-RELEASE_with-docker   | ok          | FAIL2      |
|                                           |             |            |
| rhel_aarch64_5.9.2-RELEASE_with-docker    | FAIL1       | FAIL1      |
| rhel_aarch64_5.10.1-RELEASE_with-docker   | FAIL1       | FAIL1      |
| rhel_aarch64_6.0.2-RELEASE_with-docker    | FAIL1       | FAIL2      |
|                                           |             |            |
| rhel_x86_64_5.9.2-RELEASE_with-docker     | ok          | ok         |
| rhel_x86_64_5.10.1-RELEASE_with-docker    | ok          | ok         |
| rhel_x86_64_6.0.2-RELEASE_with-docker     | ok          | FAIL2      |

FAIL1: cannot find /lib/ld-linux-aarch64.so.1       (Issue swiftlang#147)
FAIL2: missing required module '_FoundationCShims'  (Issue swiftlang#152)
  • Loading branch information
euanh committed Nov 21, 2024
1 parent f6da543 commit 4725774
Showing 1 changed file with 301 additions and 92 deletions.
Loading

0 comments on commit 4725774

Please sign in to comment.