[SWT-NNNN] Exit tests

[grynspan]

One of the first enhancement requests we received for swift-testing was the ability to test for precondition failures and other critical failures that terminate the current process when they occur. This feature is also frequently requested for XCTest. With swift-testing, we have the opportunity to build such a feature in an ergonomic way.

Read the full proposal here.

[grynspan]

Rebased.

[dabrahams]

Nice. Three notes:

1. You can probably support exit tests on platforms that don't let you spawn, because there is always(?) a test driver program running on a host system like a Mac. I outlined how this could work for googletest and CMake here: instead of looping over all the tests in a process on the target platform, you loop over the expected-to-fail ones on the host and launch a separate target process for each one.
2. I'm not familiar enough with what you're doing to know whether you've covered this case already, but it's sometimes important to be able to verify stdout/stderr contents in addition to detecting abnormal exit. I hope your design allows that.
3. With GTest I found it very inconvenient that platforms weren't uniform w.r.t. what I could detect in an exit test. I had to make uniformity macros that allowed me to write one test for all platforms and verify as much as possible. They are described (and linked) here. A better library would provide these facilities OOTB.

[grynspan]

@dabrahams Please read the draft pitch. :) 1. and 2. are covered in it. It sounds like 3. is sufficiently covered I think? .signal() is not supported on Windows because Windows itself only has emulated signal support. I know how we can fix that (by replacing the default signal handlers on Windows that call exit(3) with something that communicates across the "back channel") but I've punted that from the initial proposal for the sake of complexity.

[dabrahams]

I skimmed the pitch; I didn't see coverage for 1; what I saw was

The need for exit tests on other platforms is just as strong as it is on the supported platforms (macOS, Linux, and Windows). These platforms do not support spawning new processes, so a different mechanism for running exit tests would be needed.

emphasis mine. I read that as meaning “we don't have an answer for these platforms.”

What I see about 2) is an explicit statement that it isn't supported.

It sounds like 3 is not sufficiently covered, since the Windows signal detection case is one of the ones for which I had to build a workaround manually, and you say you're punting it 'till later.

Maybe by “covered” you just mean “discussed.” In that case I don't see much discussion of strategies for handling these things, so I thought my comment would be useful. If it wasn't, feel free to ignore it.

[grynspan]

I skimmed the pitch; I didn't see coverage for 1; what I saw was

The need for exit tests on other platforms is just as strong as it is on the supported platforms (macOS, Linux, and Windows). These platforms do not support spawning new processes, so a different mechanism for running exit tests would be needed.

emphasis mine. I read that as meaning “we don't have an answer for these platforms.”

I'm sure that, as a former Apple employee, you can appreciate that my ability to discuss iOS et al. is limited. 🙂

What I see about 2) is an explicit statement that it isn't supported.

It's listed as a future direction. That doesn't mean it can't be done, just that it's not part of this pitch.

It sounds like 3 is not sufficiently covered, since the Windows signal detection case is one of the ones for which I had to build a workaround manually, and you say you're punting it 'till later.

If you want to detect specifically that a Windows process terminated with a specific signal, that's something that needs additional work beyond the scope of this pitch. If you want to detect that a Windows process terminated abnormally, which would include termination due to an unhandled signal, that's supported and covered by the general .failure exit condition.

[grynspan]

To be clear (I'm now reading my initial reply as snarky, which wasn't intentional), I appreciate the feedback @dabrahams. I can certainly amend the pitch to make it clearer we're thinking about these things.

Edit: I added a "future directions" section about Windows signals.

[grynspan]

Windows signals are now handled! #766

[grynspan]

And stdout/stderr will be soon too. #773

[dabrahams]

Windows signals are now handled! #766

Nice; That puts you at least one step ahead of googletest, IIRC.

[grynspan]

@swift-ci test

[maartene]

Is there any news on this PR?

[grynspan]

I'm still actively maintaining this branch/PR. We are not planning to formally propose this feature until the testing workgroup is set up, at the earliest.