Implement a basic Roc test runner

[ageron]

This PR implements a basic test runner for Roc:

• Dockerfile is based on debian:latest, it installs Roc and runs download-basic-cli.roc to ensure the basic-cli platform is in Roc's cache (since the container will run without network access)
• bin/run.sh runs roc test ... and captures the output. It sanitises it (i.e., normalises) by removing timing output (e.g., "in 123 ms.").
• If the tests succeed, then result.json will contain {version: 1, status: "pass"}. If the tests fail, then if the sanitised output contains -- EXPECT FAILED in or -- EXPECT PANICKED in , then result.json will contain status="fail", or else it will contain status="error" (which means that none of the tests could run).
• tests/{slug}/{slug}-test.roc for slug in all-fail, empty-file, partial-fail, success, and syntax-error, implement a very basic test for each of these basic scenarios. The corresponding expected output is in tests/{slug}/expected_results.json.

[ageron]

Mmmh, the tests pass on my machine (using podman on MacOS), but they seem to fail here with a weird DlOpen error (failed to map segment from shared object). Investigating...

[BNAndras]

Can we pin this to a specific Debian version? If feasible, it might be better to use an Alpine image to reduce the overall image size. That'll save Exercism money since we're deploying 71+ different Docker images to each test runner machine in the cloud.

[ageron]

I can definitely pin this to debian:bookworm instead. However, I tried using alpine, but Roc failed, probably because I need to install some libraries. I'll start with debian:bookworm, and I'll switch to alpine asap.

[BNAndras]

If Alpine doesn't work out, that's fine. It's just a nice starting point for reducing file size, but I don't believe all tracks can use Alpine images due to some limitations in what's available.

[ageron]

Trying to solve the DlOpen error (see above), I've switched to Ubuntu 22.24. Still works fine on my machine but fails in Github CI. I think it might be a permission issue on the /tmp folder (see this SO answer).

[BNAndras]

What I know about Docker or Github CI can fit in a thimble, but Erik can probably help you out further. He's pretty handy. :)

[ageron]

I looked into migrating to Alpine, but Roc is currently built on glibc whereas Alpine uses musl. There's an ongoing effort to make Roc support musl, so we should be able to migrate to Alpine once that work is completed.

[ageron]

The DlOpen error (failed to map segment from shared object) is probably due to a permission issue with the /tmp folder (see this SO answer). Have you see this before? It might be possible to fix it by creating our own tmp folder and pointing TMPDIR to it, since Github action's /tmp folder does not seem to be writable (not sure about this).

[ageron]

I'm really stumped by the DlOpen error. I've tried many things now, and it still fails on Github Actions, while the same Docker container works just fine on my machine. 😢 I've asked the Roc community for some help. Please let me know if you have any idea of what this could be. Perhaps AppArmor is activated on the host and is blocking something?

[ErikSchierboom]

This is an impressive PR, well done! I've just pushed a commit to fix the tmp folder issue. CI still fails though

[ageron]

Thanks @ErikSchierboom !
I've tried compiling Roc from the latest source code, and it actually solved the problem, hurray, I was going nuts.
The Roc devs are planning to make a new release very shortly, so we won't have to use my version more than a week or so.

[ageron]

Great, the tests finally pass! 👍
Note that the Dockerfile currently downloads the latest release from the official github repo, but then it replaces the roc executable with the version I compiled from the latest source (which I uploaded to my ageron/prototype-roc-test-runner repo). When the Roc devs publish the next release, I'll remove this part of the Dockerfile.

[ageron]

Thanks for your help @ErikSchierboom