Improve stepwise to not forget failed tests

[nicoddemus]

Now --stepwise will remember the last failed test, even if the previous pytest invocations did not receive --stepwise.

Previously it would always clear the cache if not active, which hinders certain interactive workflows, which is the prime use case for the flag.

[The-Compiler]

while handy to have it remembered within a "working session", would it be unexpected to have it remembered across such sessions?

I'm worried about this too. Imagine someone:

• Reads about --stepwise being a thing and gives it a try
• Uses pytest normally for the next couple of months
• Now actually wants to use --stepwise after a new Python version came out and broke something

If you've long forgotten about your previous session, that'll be quite surprising...

I don't really have a solution unfortunately. A couple of ideas:

• Add an argument (or --stepwise=keep or whatever) to opt into keeping it, either when originally using --stepwise, or in the subsequent call without it
• Add some sort of option for a stepwise max age (say, a week by default?) after which it's discarded
• Invalidate the cache as soon as new tests are added (not sure what happens currently)
• Keep the behavior, but make it clearer in the UI: Store a timestamp, and instead of stepwise: skipping 1 already passed items., display something like stepwise: skipping 1 already passed items (cache from 2025-01-10, use --stepwise-reset to discard)

I don't really like the first three as they all come with their own downsides, but maybe the fourth one would at least partially help?

[nicoddemus]

@The-Compiler thanks for the consideration, indeed now that I think about it more it might be surprising unless the user is aware of the change.

Invalidate the cache as soon as new tests are added (not sure what happens currently)
Keep the behavior, but make it clearer in the UI

I think those two are actually reasonable, and they even make sense to be implemented together. I will take a stab at them.

Thanks!

[The-Compiler]

Amazing work, thanks! Left a couple comments, mainly small nitpicks.

[nicoddemus]

@The-Compiler ready for another review. 👍

[The-Compiler]

Sorry for the delay! Wanted to play around with this a bit rather than just looking at the diff, and didn't get around to it yet.

One last thing I noticed regarding the output, and then I think this is good to go!

[The-Compiler]

Suggested change

	f"skipping {failed_index} already passed items (cache from {self.cached_info.last_cache_date},"
	f"skipping {failed_index} already passed items (cache from {self.cached_info.last_cache_date:%c},"

maybe, where %c is "Locale’s appropriate date and time representation."?

Without this, it gets displayed as e.g. cache from 2025-01-15 14:25:45.458813 which seems a bit verbose, and the fractional seconds don't really add any useful information.

[nicoddemus]

I did try that initially, but to me it showed up as the default locale (C): IIUC we need to call locale.setlocale at some point explicitly for %c to work properly, otherwise it will always use the C locale (and we cannot do that because this is a global setting, which pytest definitely should not touch IMO).

But I might be wrong, can you test that in your machine?

If %c does not work, we might explicitly use %YY-%M-%d %h:%m:%s to get rid of the fractional seconds, which I agree is not useful.

[The-Compiler]

Oh wow, looks like it! My locale setup is a bit weird at the moment anyways (experimented with en_DK to get proper dates, but turns out it's not supported everywhere so now I get a mess of formats...). Because of that, I expected that to be an issue on my end.

What do you think about making a timedelta to now and then just formatting that in instead? That would then show up as 0:00:10 ago or 42 days, 1:23:45 ago which seems a much nicer format for the information we actually want to convey here.

edit: Whoops, I completely neglected that this would show microseconds too... I suppose that could be fixed with a datetime.timedelta(seconds=int(td.total_seconds())) (with td being the original delta).

[nicoddemus]

Good idea about timedelta, will do.

[nicoddemus]

Done.

I considered only showing that information if the cache was "old", but then what could be considered an "old" cache? 1 hr? 1 day? In the ended I kept it simple and always show the cache age for now.

[The-Compiler]

Hehe, I have been thinking about that as well, and came to the same conclusion as you did 😅

[The-Compiler]

Thanks for sticking with this, looks great! Will merge in a day or two if nobody else wants to take a look.

[The-Compiler]

Thanks again! ✨

[nicoddemus]

Thanks @The-Compiler for merging! However that would be better as a squash probably. 👍

[The-Compiler]

I did consider that and decide against it, given that there were significant changes/amendments in the history of this PR.