IfFileExists incorrectly caches results

[spakin]

Brief outline of the bug

Once \IfFileExists determines that a file exists, it remembers this fact and subsequently neglects to check if the file has ceased to exist.

Minimal example showing the bug

Run with LuaLaTeX:

\RequirePackage{latexbug}
\documentclass{article}

\begin{document}

\IfFileExists{\jobname.zzz}{YES}{NO}

\begin{filecontents}{\jobname.zzz}
This file will be deleted.
\end{filecontents}

\IfFileExists{\jobname.zzz}{YES}{NO}

\directlua{os.remove('\jobname.zzz')}

\IfFileExists{\jobname.zzz}{YES}{NO}

\end{document}

(Derived from @davidcarlisle's answer to https://tex.stackexchange.com/questions/611391/delete-a-file-using-luacode.)

• The first \IfFileExists should typeset NO (assuming you don't have the .zzz file it's looking for).
• The second \IfFileExists should typeset YES because the code used a filecontents environment to create the .zzz file.
• The third \IfFileExists should typeset NO because the code deleted the .zzz file using Lua's os.remove function. However, because \IfFileExists incorrectly remembers the previous true path it takes that path again and erroneously outputs YES.

Log file (required) and possibly PDF file

iffileexists.log
iffileexists.pdf

Possible solution

I'm no LaTeX3 expert, but it looks to me like the caching may originate in \__file_full_name_aux:n. Perhaps retaining only the false path will solve the problem?

\cs_new:Npn \__file_full_name_aux:n #1
  {
    \exp_args:Ne \__file_full_name_auxi:nn { \__file_size:n {#1} } {#1}
  }

[josephwright]

Caching here is by-design - file operations are slow, and thus looking up a file's existence multiple times is not desirable. The working assumption for a (La)TeX run is that as a batch process, if files exist at the start of the run, they will be available throughout.

[josephwright]

Code comment in l3file covers this

%   To avoid repeated reading of files we need to cache the loading:
%   this is important as the code here is used by \emph{all} file checks.
%   The same marker is used in the \LaTeXe{} kernel, meaning that we get a
%   double-saving with for example \cs{IfFileExists}. As this is all about
%   performance, we use the low-level approach for the conditionals. For
%   a file already seen, the size is reported as $-1$ so it's distinct from
%   any non-cached ones.

[muzimuzhi]

BTW, it's a relatively new feature, introduced in the 2023-06-01 release. See texdoc ltnews37, sec. "Performance in checking file existence".

latex2e/base/doc/ltnews37.tex

Lines 416 to 422 in 051cfdb

	\subsection{Performance in checking file existence}
	The addition of hooks, etc., to file operations had a side effect of making
	multiple checks that the file existed. In larger documents using
	many files, these file system operations caused non-trivial performance
	impact. We now cache the existence of files, such that these repeated filesystem
	calls are avoided.

[spakin]

I see. Would it be a nontrivial change to support cache bypass as an option to \IfFileExists—defaulting to file existence being cached—for use in cases where files come and go (as mimicked by my MWE)?

If that would be difficult to add to LaTeX3, I'll copy over (and rename) the simpler, more straightforward LaTeX2ε implementation of \IfFileExists and use that in my code.

[josephwright]

@spakin I've just added latex3/latex3#1635 to the expl3 side of things, and we will sort an interface for this for the June 2025 LaTeX release - the expl3 code iwll likely be released mid-January to support this.

[josephwright]

@spakin If you need a temporary hack, use

\ExplSyntaxOn
\cs_gset_protected:Npn \file_forget:n #1
  { \cs_undefine:c { __file_seen_ \file_full_name:n {#1} : } }
\ExplSyntaxOff

in the preamble then

\ExplSyntaxOn
\file_forget:n {<name>}
\ExplSyntaxOff

at the point you delete the file.

[spakin]

@josephwright: Thanks for writing latex3/latex3#1635 and for providing a temporary hack. Much appreciated.

[FrankMittelbach]

Scott, the team has decided to offer that interface only on the L3 layer level so \file_forget:n {<name>} will be available with the next upload of the l3kernel code but the 2e side will not get any CamelCase name for it in June. The rationale is that 2e doesn't have any interfaces for deleting files so any code doing this needs low-level coding.

[josephwright]

If we want \file_forget:n in the next release, I should do an extra dev release - can sort today

[FrankMittelbach]

I thought that would happen rather automatically any time soon, eg when you prepare a new upload to ctan. I meant the one you hinted at for January. I don't think it really needs to be earlier just for that (only means extra work for the CTAN maintainers).