Speed up hypothesis tests

[dstansby]

• tests/test_store/test_stateful.py::test_zarr_hierarchy[memory]
• tests/test_store/test_stateful.py::test_zarr_store[memory]

currently take up around 70% of our total test time (e.g., see this recent test run).

Since these tests are not comparing two different store implementations, but the same one (comparing MemoryStore to MemoryStore), I think it makes sense to cut the number of examples right down here to save a lot of testing time.

[dcherian]

5 examples is too small to be useful, and this time will just keep going up as we start testing the other stores.

What do you think of making a separate nightly action that runs the stateful tests, and auto-opens an issue on failure similar to this action I wrote for Xarray

[dstansby]

What's the minimum number of examples to be useful?

My thinking was checking MemoryStore against MemoryStore was useful to make sure the test itself works, but it's just comparing instances of identical classes, so there's no benefit to running lots of tests, especially as it's taking up 70% of the total test time. Maybe I'm misunderstanding the tests though?

I'd be 👍 to doing a nightly action if there is good reason to run lots of examples for these tests.

[dcherian]

I don't think 5 examples is useful for testing all the actions because it generates random sequences of actions and longer examples tend to come later in the run. Run pytest -s --hypothesis-verbosity=verbose to see how it tries things.

I agree that testing MemoryStore against MemoryStore seems silly, but this is on par for the property and stateful testing literature. Very simple and seemingly-silly tests (e.g. "get the same key twice", "set the same key twice, and get it") tend to be very effective particularly when you have some source of complexity (like async in Zarr's case).

My goal was to remove MemoryStore in favour of testing LocalStore against MemoryStore but LocalStore had more inconsistencies than I cared to fix at that point. Are you interested in trying this?

[dstansby]

How about cuting down to 50 examples then? Even that would give us a big and quick win in terms of test times. Then in the medium term we could look at 1) a nighly run of the stateful tests, and 2) testing LocalStore against MemoryStore.

[dcherian]

OK sounds good.

[dcherian]

Sorry I had this all backwards. I have already done the nightly action: https://github.com/zarr-developers/zarr-python/actions/workflows/hypothesis.yaml

I think the error here is in running the hypothesis tests in all test commands instead of just that action. Are you able to take a look at that.

For example on this PR we run the tests separately too.