Hspec as an alternative test framework?

[schoettl]

@simonmichael does anything speak against hspec as alternative test framework?

Downsides:

• three more dependencies (hspec, hspec-core, hspec-contrib)
• a bit more complex argument handling or a different way to pass arguments to hspec

Benefits:

• a --fail-fast option, see Question: stop tests on first failure (--fail-fast) haskell/test-framework#55 (I have many long running tests that I'd like to stop on first failure).
• print cpu time
• colored diffs (?)
• dry-run, to see what test files are loaded
• re-run failed tests only
• read options from .hspec config file
• some more, e.g. html output

Sources:

• Compare shelltest -h and http://hspec.github.io/options.html
• https://hackage.haskell.org/package/hspec-core-2.7.2/docs/Test-Hspec-Core-Runner.html#g:2
• Question: stop tests on first failure (--fail-fast) haskell/test-framework#55

[simonmichael]

Thanks! I agree that we should move off test-framework, if only because it's no longer maintained.

Of the current big two, tasty and hspec, I think tasty is the most successful and the most future-proof, and I'd favour choosing that. But I'll review this when I can.

[simonmichael]

Hi @schoettl, when you have time could you rebase this against latest master.

[simonmichael]

I had a look at this. I see both runners are still supported for now, I guess there's no harm in keeping both until we have tested a bit more. Merging, thank you!

[simonmichael]

CI failed with stackage nightly/GHC 9.4, so we probably need some bounds or code changes for newer hspec.
https://github.com/simonmichael/shelltestrunner/actions/runs/4199072857

[simonmichael]

Fixed.

[simonmichael]

Belatedly testing this; at least in the shelltestrunner and hledger repos, it seems to run a few tests but very slowly/hangingly. To reproduce in this repo, add --hspec in Makefile and run make test. @schoettl would you be able to look at this ?

[simonmichael]

Repro:

<
a
$ cat
a

$ shelltest a.test --hspec

It works with echo but not cat. Perhaps the fix is in shelltest.hs:runCommandWithInput.

[schoettl]

Hi @simonmichael, thanks for testing and fixes. I plan to look at it this weekend.