Tagging gh762 (spacing after math displays) (#1573)

* WIP, just some tests * aproaches to solve tagging 762 * WIP * one test * and a bit more doc (WIP) * WIP * drop the debugging * test + doc * another test + changelog * one more * and another six * placeholder in ltnews * Update required/latex-lab/latex-lab-math.dtx Co-authored-by: Joseph Wright <[email protected]> * a real ltnews entry --------- Co-authored-by: Joseph Wright <[email protected]>
latex3 · Dec 10, 2024 · a2cb14c · a2cb14c
1 parent 6037915
commit a2cb14c
Show file tree

Hide file tree

Showing 20 changed files with 905 additions and 38 deletions.
diff --git a/base/doc/ltnews41.tex b/base/doc/ltnews41.tex
@@ -73,12 +73,19 @@
           	    {github issue#1 #2}%
            )}%
      \par\smallskip}
-%% But Chris has to mostly disable \href for his TEXPAD app:
-%% \def\href #1#2{#2} % Only For Chris' deficient TeX engine
 
 % simple solution right now (just link to the first issue if there are more)
 \def\getfirstgithubissue#1 #2\relax{#1}
 
+
+\providecommand\taggingissue[2][]{\ifhmode\unskip\fi
+     \quad\penalty500\strut\nobreak\hfill
+     \mbox{\small\slshape(%
+       \href{https://github.com/latex3/tagging-project/issues/\getfirstgithubissue#2 \relax}%
+          	    {tagging-project issue#1 #2}%
+           )}%
+     \par\smallskip}
+
 \providecommand\sxissue[1]{\ifhmode\unskip
      \else
        % githubissue preceding
@@ -180,6 +187,44 @@ \section{Replacement for the legacy mark mechanism}
 
 
 
+\section{News from Tagged PDF project}
+
+\emph{to write}
+
+\subsection{Fixing the spacing after display math}
+
+When \LaTeX{} produces an accessible (tagged) PDF it has to add
+structure data into the PDF to mark (i.e., tag) individual
+elements. If the \pdfTeX{} engine is used this has to be done with the
+help of \cs{pdfliteral}s which are whatsit nodes like
+\cs{special} or \cs{write}. This means that they should be added only in
+places, where these extra nodes are not affecting the spacing\Dash \TeX{}
+can't, for example, look backwards past such a whatsit node so
+consecutive spaces that are normally collapsed into one, suddenly
+appear both, if such a node separates them.
+
+The situation is especially complicated with math displays, because
+there \TeX{} adds penalties and spaces with low-level procedures, that
+are not directly accessible from the macro level, and the tagging
+structures have to appear somewhere in the middle of that to ensure
+that the formula and the PDF structures are not separated by page
+break. Because of this it is necessary to use some fairly complex
+methods (essentially disable \TeX's mechanisms and reprogram them on
+the macro level) to get the structure data in the right places.
+
+Our first attempt in doing that was slightly faulty and resulted in
+some cases in an extra space (an additional \cs{parskip} space when
+there shouldn't be one). This has now corrected and the gymnastics to
+achieve this are rather an \enquote{interesting} study in obfuscated
+\TeX{} coding.
+
+In \LuaTeX{} the situation is much better because there the structures can
+be added later when the formula processing has already finished.
+%
+\taggingissue{762}
+
+
+
 \section{New or improved commands}
 
 \section{Code improvements}
@@ -249,8 +294,6 @@ \subsection{Fix the use of  \texttt{localmathalpabets}}
 \githubissue[s]{1101 1028}
 
 
-
-
 %\section{Changes to packages in the \pkg{amsmath} category}
 
 %\section{Changes to packages in the \pkg{graphics} category}

diff --git a/required/latex-lab/changes.txt b/required/latex-lab/changes.txt
@@ -1,7 +1,15 @@
+2024-12-01  Frank Mittelbach  <[email protected]>
+
+	* latex-lab-block.dtx (subsection{Handling \tn{par} after the end of the list}):
+	Handle \parskip after $$display correctly (tagging/762)
+
+	* latex-lab-math.dtx (subsection{Content grabbing}):
+	Handle \parskip after $$display correctly (tagging/762)
+
 2024-11-26 Ulrike Fischer <[email protected]>
-	* latex-lab-math.dtx: changed handling of math/mathml/structelem to handle 
+	* latex-lab-math.dtx: changed handling of math/mathml/structelem to handle
 	fakemath better(tagging/764)
-	* latex-lab-math.dtx: made all main math sockets tagging sockets and 
+	* latex-lab-math.dtx: made all main math sockets tagging sockets and
 	assign the plugs directly.
 	* disable para tagging also in $$-display (tagging/765)
 

diff --git a/required/latex-lab/latex-lab-block.dtx b/required/latex-lab/latex-lab-block.dtx
@@ -9,8 +9,8 @@
 %
 %    https://www.latex-project.org/lppl.txt
 %
-\def\ltlabblockdate{2024-11-25}
-\def\ltlabblockversion{0.8w}
+\def\ltlabblockdate{2024-12-01}
+\def\ltlabblockversion{0.8x}
 %<*driver>
 \documentclass[kernel]{l3doc}
 \usepackage{amstext}
@@ -765,6 +765,25 @@
 \def\@doendpe{\@endpetrue
   \def\par
     {
+%    \end{macrocode}
+%    If we are processing a \verb=$$= math display and we encounter a real
+%    \cs{par} after it, we need to add a \cs{parskip} when tagging is
+%    done, because the one
+%    added by \TeX{} is always canceled by the processing in
+%    \cs{__math_tag_dollardollar_display_end:} in that case. This is
+%    signaled by the global legacy switch \texttt{@domathendpe} which is set
+%    to true in that case. Once the skip is applied we set it to
+%    false. If there is no \cs{par} at all, it will be reset in
+%    \cs{everypar} when the next paragraph starts.
+%  \changes{v0.8x}{2024/12/01}{Handle \cs{parskip} after \texttt{\$\$}
+%    display (tagging/762)}
+%    \begin{macrocode}
+      \if@domathendpe
+%        \typeout{------->~ add~ another~ parskip~ that~ is~
+%                           not~ canceled:~ \the\parskip }
+        \skip_vertical:n { \tex_parskip:D }
+        \@domathendpefalse
+      \fi
       \@restorepar
       \clubpenalty\@clubpenalty
 %    \end{macrocode}
@@ -787,6 +806,14 @@
   \everypar{{\setbox\z@\lastbox}
             \everypar{}
             \@endpefalse
+%    \end{macrocode}
+%    Not sure what is faster: testing for the status of the switch or
+%    setting it unconditionally to false (globally), probably roughly the same,
+%    so we set it always:
+%    \begin{macrocode}
+%           \if@domathendpe
+            \@domathendpefalse
+%           \fi
   }
 }
 %    \end{macrocode}
@@ -800,6 +827,21 @@
 % \end{macro}
 %
 %
+%  \begin{macro}{\if@domathendpe,\@domathendpefalse,\@domathendpetrue}
+%    Signal that special paragraph handling after a math display is required.
+%  \changes{v0.8x}{2024/12/01}{Handle \cs{parskip} after \texttt{\$\$}
+%    display (tagging/762)}
+%    \begin{macrocode}
+\newif\if@domathendpe
+\def\@domathendpefalse{\global\let\if@domathendpe\iffalse}
+\def\@domathendpetrue {\global\let\if@domathendpe\iftrue}
+%    \end{macrocode}
+%  \end{macro}
+%
+%
+%
+%
+%
 % \subsection{Object and template interfaces}
 %
 %

diff --git a/required/latex-lab/latex-lab-math.dtx b/required/latex-lab/latex-lab-math.dtx
@@ -19,8 +19,8 @@
 % for those people who are interested or want to report an issue.
 %
 %
-\def\ltlabmathdate{2024-11-26}
-\def\ltlabmathversion{0.6j}
+\def\ltlabmathdate{2024-12-01}
+\def\ltlabmathversion{0.6k}
 %
 %<*driver>
 \documentclass{l3doc}
@@ -1991,30 +1991,54 @@
     %  \ShowTagging{struct-stack}
     \para_raw_end:
 %    \end{macrocode}
-%    The \cs{postdisplaypenalty} was temporarily set to 10000 inside
-%    the display and the \cs{belowdisplayskip} and the
+%    The \cs{postdisplaypenalty} was temporarily set
+%    to 10000 inside the display and the \cs{belowdisplayskip} and the
 %    \cs{belowdisplayshortskip} was negated, so whatever was inserted
-%    it should have been a negative skip. Whatever skip was added we
-%    pick it ups value up here, so that we can correct the spacing
-%    after the tagging code was inserted.
+%    it should have been a negative skip. Whatever 
+%    was added, we pick up the value up, so that we can correct
+%    the spacing after the tagging code was inserted.
 %    \begin{macrocode}
     \l_@@_tmpa_skip \lastskip
     \tag_socket_use:n{math/display/formula/end}
 %    \end{macrocode}
-%    Now we add a skip without indroducing a page break possibility,
+%    Now we add a skip without introducing a page break possibility,
 %    that should bring the current vertical position back to the point
 %    where \TeX{} would add the penalty and the \enquote{below skip}.
 % \changes{v0.6f}{2024-09-30}{Correct logic for inserting below skips
 %                             after displays (tagging/721)}
 %    \begin{macrocode}
     \nobreak
-    \skip_vertical:n { -\l_@@_tmpa_skip  } % remove the negative belowdisplayskip
+    \skip_vertical:n { -\l_@@_tmpa_skip  }
 %    \end{macrocode}
-%    Then we finally add the real stuff:
+%    Then we finally add the real stuff: the true \cs{postdisplaypenalty}
+%    and then negated value of skip we saved from above. It may look
+%    strange that we have two identical negated skips next to each
+%    other, but if you think about it, that is correct: the first
+%    cancels the \enquote{below skip} that \TeX{} had added and the
+%    second puts the same amount after the penalty (which is where it
+%    should be).
 %    \begin{macrocode}
     \penalty \postdisplaypenalty
     \skip_vertical:n { -\l_@@_tmpa_skip  } % insert the correct skip
-  \@doendpe             % this has no \end{...} to take care of it
+%    \end{macrocode}
+%    As we are now in vertical mode the situation is different from
+%    the way \TeX{} would handle things after a display: \TeX{} would
+%    internally switch to horizontal mode without adding a
+%    \cs{parskip}. But this is not possible to do on the macro
+%    level. Therefore we have to neutralize the upcoming \cs{parskip}
+%    since we can't  prevent it from being set.
+%  \changes{v0.6k}{2024/12/01}{Handle \cs{parskip} after \texttt{\$\$}
+%    display correctly (tagging/762)}
+%    \begin{macrocode}
+%    \typeout{------->~ add~ negative~ parskip~ (to~ cancel~ the~
+%                       one~ that~ TeX~ will~ add)}
+    \skip_vertical:n { -\tex_parskip:D }
+%    \end{macrocode}
+%    
+%    We also set the \texttt{@domathendpetrue} flag to signal that the    
+%    \begin{macrocode}
+    \@domathendpetrue
+    \@doendpe             % this has no \end{...} to take care of it
 }
 
 %    \end{macrocode}

diff --git a/required/latex-lab/testfiles-LM/LM-3-3.tlg b/required/latex-lab/testfiles-LM/LM-3-3.tlg
@@ -515,6 +515,11 @@ Package tagpdf Info: Parent-Child 'P' --> 'MC'.
 (tagpdf)             Rolemapped from: 'P' --> 'MC' on line ...
 Package tagpdf Info: closing structure 79 tagged /text
 Package tagpdf Info: closing structure 78 tagged /text-unit
+The sequence \g__tag_mc_main_marks_seq contains the items (without outer braces):
+>  {e+}
+>  {5}
+>  {9}.
+[1]
 Package tagpdf Info: closing structure 75 tagged /Sect
 Package tagpdf Info: Parent-Child 'Sect' --> 'Sect'.
 (tagpdf)             Relation is 1 (='0..n')
@@ -556,11 +561,6 @@ Package tagpdf Info: Parent-Child 'P' --> 'MC'.
 (tagpdf)             Rolemapped from: 'P' --> 'MC' on line ...
 Package tagpdf Info: closing structure 85 tagged /text
 Package tagpdf Info: closing structure 84 tagged /text-unit
-The sequence \g__tag_mc_main_marks_seq contains the items (without outer braces):
->  {e+}
->  {5}
->  {9}.
-[1]
 Package tagpdf Info: closing structure 81 tagged /Sect
 Package tagpdf Info: Parent-Child 'Sect' --> 'Sect'.
 (tagpdf)             Relation is 1 (='0..n')

diff --git a/required/latex-lab/testfiles-block/blocks-description-01b.luatex.tlg b/required/latex-lab/testfiles-block/blocks-description-01b.luatex.tlg
@@ -399,7 +399,7 @@ The sequence \g__tag_struct_tag_stack_seq contains the items (without outer brac
 l. ...  \typeout{-->A}\ShowTagging{struct-stack}
                                              \show\par
 > \par=macro:
-->\@restorepar \clubpenalty \@clubpenalty \__kernel_displayblock_doendpe: \@endpefalse \everypar {}\par .
+->\if@domathendpe \skip_vertical:n {\tex_parskip:D }\@domathendpefalse \fi \@restorepar \clubpenalty \@clubpenalty \__kernel_displayblock_doendpe: \@endpefalse \everypar {}\par .
 l. ......out{-->A}\ShowTagging{struct-stack}\show\par
 ===> @endpe: true 
 ==> blockenv common ending on input line ...
@@ -421,7 +421,7 @@ The sequence \g__tag_struct_tag_stack_seq contains the items (without outer brac
 l. ...  \typeout{-->B}\ShowTagging{struct-stack}
                                              \show\par
 > \par=macro:
-->\@restorepar \clubpenalty \@clubpenalty \__kernel_displayblock_doendpe: \@endpefalse \everypar {}\par .
+->\if@domathendpe \skip_vertical:n {\tex_parskip:D }\@domathendpefalse \fi \@restorepar \clubpenalty \@clubpenalty \__kernel_displayblock_doendpe: \@endpefalse \everypar {}\par .
 l. ......out{-->B}\ShowTagging{struct-stack}\show\par
 ===> @endpe: true 
 Package tagpdf Info: closing structure 22 tagged /text-unit
@@ -467,7 +467,7 @@ The sequence \g__tag_struct_tag_stack_seq contains the items (without outer brac
 l. ...  \typeout{-->C}\ShowTagging{struct-stack}
                                              \show\par
 > \par=macro:
-->\@restorepar \clubpenalty \@clubpenalty \__kernel_displayblock_doendpe: \@endpefalse \everypar {}\par .
+->\if@domathendpe \skip_vertical:n {\tex_parskip:D }\@domathendpefalse \fi \@restorepar \clubpenalty \@clubpenalty \__kernel_displayblock_doendpe: \@endpefalse \everypar {}\par .
 l. ......out{-->C}\ShowTagging{struct-stack}\show\par
 ===> @endpe: false
 ==> blockenv common ending on input line ...
@@ -485,7 +485,7 @@ The sequence \g__tag_struct_tag_stack_seq contains the items (without outer brac
 l. ...  \typeout{-->D}\ShowTagging{struct-stack}
                                              \show\par
 > \par=macro:
-->\@restorepar \clubpenalty \@clubpenalty \__kernel_displayblock_doendpe: \@endpefalse \everypar {}\par .
+->\if@domathendpe \skip_vertical:n {\tex_parskip:D }\@domathendpefalse \fi \@restorepar \clubpenalty \@clubpenalty \__kernel_displayblock_doendpe: \@endpefalse \everypar {}\par .
 l. ......out{-->D}\ShowTagging{struct-stack}\show\par
 ===> @endpe: true 
 ==> flattened=false on input line ...

diff --git a/required/latex-lab/testfiles-block/blocks-description-01b.tlg b/required/latex-lab/testfiles-block/blocks-description-01b.tlg
@@ -399,7 +399,7 @@ The sequence \g__tag_struct_tag_stack_seq contains the items (without outer brac
 l. ...  \typeout{-->A}\ShowTagging{struct-stack}
                                                \show\par
 > \par=macro:
-->\@restorepar \clubpenalty \@clubpenalty \__kernel_displayblock_doendpe: \@endpefalse \everypar {}\par .
+->\if@domathendpe \skip_vertical:n {\tex_parskip:D }\@domathendpefalse \fi \@restorepar \clubpenalty \@clubpenalty \__kernel_displayblock_doendpe: \@endpefalse \everypar {}\par .
 l. ......t{-->A}\ShowTagging{struct-stack}\show\par
 ===> @endpe: true 
 ==> blockenv common ending on input line ...
@@ -421,7 +421,7 @@ The sequence \g__tag_struct_tag_stack_seq contains the items (without outer brac
 l. ...  \typeout{-->B}\ShowTagging{struct-stack}
                                                \show\par
 > \par=macro:
-->\@restorepar \clubpenalty \@clubpenalty \__kernel_displayblock_doendpe: \@endpefalse \everypar {}\par .
+->\if@domathendpe \skip_vertical:n {\tex_parskip:D }\@domathendpefalse \fi \@restorepar \clubpenalty \@clubpenalty \__kernel_displayblock_doendpe: \@endpefalse \everypar {}\par .
 l. ......t{-->B}\ShowTagging{struct-stack}\show\par
 ===> @endpe: true 
 Package tagpdf Info: closing structure 22 tagged /text-unit
@@ -467,7 +467,7 @@ The sequence \g__tag_struct_tag_stack_seq contains the items (without outer brac
 l. ...  \typeout{-->C}\ShowTagging{struct-stack}
                                                \show\par
 > \par=macro:
-->\@restorepar \clubpenalty \@clubpenalty \__kernel_displayblock_doendpe: \@endpefalse \everypar {}\par .
+->\if@domathendpe \skip_vertical:n {\tex_parskip:D }\@domathendpefalse \fi \@restorepar \clubpenalty \@clubpenalty \__kernel_displayblock_doendpe: \@endpefalse \everypar {}\par .
 l. ......t{-->C}\ShowTagging{struct-stack}\show\par
 ===> @endpe: false
 ==> blockenv common ending on input line ...
@@ -485,7 +485,7 @@ The sequence \g__tag_struct_tag_stack_seq contains the items (without outer brac
 l. ...  \typeout{-->D}\ShowTagging{struct-stack}
                                                \show\par
 > \par=macro:
-->\@restorepar \clubpenalty \@clubpenalty \__kernel_displayblock_doendpe: \@endpefalse \everypar {}\par .
+->\if@domathendpe \skip_vertical:n {\tex_parskip:D }\@domathendpefalse \fi \@restorepar \clubpenalty \@clubpenalty \__kernel_displayblock_doendpe: \@endpefalse \everypar {}\par .
 l. ......t{-->D}\ShowTagging{struct-stack}\show\par
 ===> @endpe: true 
 ==> flattened=false on input line ...

diff --git a/required/latex-lab/testfiles-block/blocks-description-03.luatex.tlg b/required/latex-lab/testfiles-block/blocks-description-03.luatex.tlg
@@ -107,7 +107,7 @@ The sequence \g__tag_struct_tag_stack_seq contains the items (without outer brac
 l. ...  \typeout{-->B}\ShowTagging{struct-stack}
                                              \show\par
 > \par=macro:
-->\@restorepar \clubpenalty \@clubpenalty \__kernel_displayblock_doendpe: \@endpefalse \everypar {}\par .
+->\if@domathendpe \skip_vertical:n {\tex_parskip:D }\@domathendpefalse \fi \@restorepar \clubpenalty \@clubpenalty \__kernel_displayblock_doendpe: \@endpefalse \everypar {}\par .
 l. ......out{-->B}\ShowTagging{struct-stack}\show\par
 ==> flattened=false on input line ...
 ==> Structure-end text-unit after displayblock on input line ...

diff --git a/required/latex-lab/testfiles-block/blocks-description-03.tlg b/required/latex-lab/testfiles-block/blocks-description-03.tlg
@@ -107,7 +107,7 @@ The sequence \g__tag_struct_tag_stack_seq contains the items (without outer brac
 l. ...  \typeout{-->B}\ShowTagging{struct-stack}
                                                \show\par
 > \par=macro:
-->\@restorepar \clubpenalty \@clubpenalty \__kernel_displayblock_doendpe: \@endpefalse \everypar {}\par .
+->\if@domathendpe \skip_vertical:n {\tex_parskip:D }\@domathendpefalse \fi \@restorepar \clubpenalty \@clubpenalty \__kernel_displayblock_doendpe: \@endpefalse \everypar {}\par .
 l. ......t{-->B}\ShowTagging{struct-stack}\show\par
 ==> flattened=false on input line ...
 ==> Structure-end text-unit after displayblock on input line ...

diff --git a/required/latex-lab/testfiles-block/blocks-itemize-02.luatex.tlg b/required/latex-lab/testfiles-block/blocks-itemize-02.luatex.tlg
@@ -63,7 +63,7 @@ The sequence \g__tag_struct_tag_stack_seq contains the items (without outer brac
 l. ...\typeout{-->A}\ShowTagging{struct-stack}
                                            \show\par\makeatletter\typeout{==...
 > \par=macro:
-->\@restorepar \clubpenalty \@clubpenalty \__kernel_displayblock_doendpe: \@endpefalse \everypar {}\par .
+->\if@domathendpe \skip_vertical:n {\tex_parskip:D }\@domathendpefalse \fi \@restorepar \clubpenalty \@clubpenalty \__kernel_displayblock_doendpe: \@endpefalse \everypar {}\par .
 l. ......out{-->A}\ShowTagging{struct-stack}\show\par
                                                   \makeatletter\typeout{==>@...
 ==>@endpe true

diff --git a/required/latex-lab/testfiles-block/blocks-itemize-02.tlg b/required/latex-lab/testfiles-block/blocks-itemize-02.tlg
@@ -63,7 +63,7 @@ The sequence \g__tag_struct_tag_stack_seq contains the items (without outer brac
 l. ...\typeout{-->A}\ShowTagging{struct-stack}
                                              \show\par\makeatletter\typeout{...
 > \par=macro:
-->\@restorepar \clubpenalty \@clubpenalty \__kernel_displayblock_doendpe: \@endpefalse \everypar {}\par .
+->\if@domathendpe \skip_vertical:n {\tex_parskip:D }\@domathendpefalse \fi \@restorepar \clubpenalty \@clubpenalty \__kernel_displayblock_doendpe: \@endpefalse \everypar {}\par .
 l. ......t{-->A}\ShowTagging{struct-stack}\show\par
                                                   \makeatletter\typeout{==>@...
 ==>@endpe true

diff --git a/required/latex-lab/testfiles-math-luatex/math-suspended-gh661.tlg b/required/latex-lab/testfiles-math-luatex/math-suspended-gh661.tlg
@@ -141,6 +141,7 @@ Completed box being shipped out [1]
 ...\glue 6.0 plus 3.0 minus 3.0
 ...\penalty 0
 ...\glue 6.0 plus 3.0 minus 3.0
+...\glue 0.0 plus -1.0
 ...\glue(\parskip) 0.0 plus 1.0
 ...\glue(\parskip) 0.0
 ...\glue(\baselineskip) 7.69446

diff --git a/required/latex-lab/testfiles-math/math-suspended-gh661.tlg b/required/latex-lab/testfiles-math/math-suspended-gh661.tlg
@@ -151,6 +151,7 @@ Completed box being shipped out [1]
 ...\glue 6.0 plus 3.0 minus 3.0
 ...\penalty 0
 ...\glue 6.0 plus 3.0 minus 3.0
+...\glue 0.0 plus -1.0
 ...\glue(\parskip) 0.0 plus 1.0
 ...\glue(\parskip) 0.0
 ...\glue(\baselineskip) 7.6955

diff --git a/required/latex-lab/testfiles-math/mathcapture-016.tlg b/required/latex-lab/testfiles-math/mathcapture-016.tlg
@@ -100,6 +100,7 @@ Completed box being shipped out [1]
 ...\glue 10.0 plus 2.0 minus 5.0
 ...\penalty 0
 ...\glue 10.0 plus 2.0 minus 5.0
+...\glue 0.0 plus -1.0
 ...\glue(\parskip) 0.0 plus 1.0
 ...\glue(\parskip) 0.0
 ...\glue(\baselineskip) 4.27946