sile-texmode could be merged, if you want it

[ctrlcctrlv]

Ostensibly, this is a tracking issue for the merger of my sile-texmode, aka LaTeXmode for SILE, into packages/.

It's around 300-500 lines of code depending on how you count the .sil files.

LaTeXmode for SILE

evil?.sil

\begin[papersize=250mm x 20mm]{document}\script[src=packages/texmode]\texmode[_="% ]
\script[src=packages/font-fallback]
\font:add-fallback[family=Noto Serif CJK JP]
% This is exactly how Japanese people talk. Anime and nichanneru, extremely reliable sources, taught me that.
\font[size=20pt]
おっす、世界！元気のか！俺ハイハイ！さって〜行くぞ！じゃね‼‼

The above is known as a ``quote''.
"]\end{document}

evil?.pdf

But...

But I'm more opening it to spur discussion. I don't really expect this to be merged upstream. I worked very hard on it, yes, but my effort is not the barometer by which upstream-worthiness is measured :-)

In #324 (comment), @simoncozens, our BDFL, wrote...

You know what, right now, I'm thinking that all user-contributed packages and classes should just go in the SILE distribution. Avoids all the package manager madness, and we're talking about a bunch of tiny Lua files. Right now the packages and classes directories take up 300 kilobytes. If that grew by a factor of 100, I still wouldn't be worried.

@alerque made some good points there, but here's perhaps another one. You guys aren't going to like everything users will want. I'd say, my SILE TeXmode is an excellent example.

Users also are sometimes willing to put up with more bugginess to get a feature they need ASAP.

I would really like some way to get my work on sile-project.org without needing it to be fully upstreamed. Something like weechat.org's plugins database.

[simoncozens]

Hmm, I'm torn. On the one hand, we now have package manager madness. On the other, this is very cool indeed. On the third hand, it encourages more use of the TeXlike mode (boo hiss). On the fourth hand, people are gonna use that anyway.

Heck, let's put it in the distribution. If we have boustrophedon support in there, this can certainly go in.

It will need autodoc first though. return { documentation = [[...]]} - see e.g. packages/features.lua

[alerque]

This is ... like the playwright that cooked up Romeo and Juliet collaborated with the art director for Mad Max and they wrote a poem together.

1. Mentioning and linking on the website as a showcase of things (crazy or otherwise) done with SILE is not a problem either way. Whether it's an external package or something we upstream, we can help make in known what can be done.

2. I don't think encouraging the TeXlike mode is as much of an issue as Simon does. I like it and use it a lot and I'm against making it a second class citizen. XML is great for machine generated stuff if the source format is structured similarly, but it is terrible for human editing. At this point we have close to full parity between what input formats can accomplish. I'd like to push on towards 100% parity, not move away from it.

   Hence one of the things I'd like to see before considering upstreaming this is how to get it to play nicely with XML input mode. We already have a way of embedding Lua or TeXlike stuff in XML, TeXlike or XML stuff in Lua, Lua or XML in TeXlike mode. This command shouldn't be an exception to that.

3. We need tests. Some of this will probably play nicer with the Busted test system than the home-grown layout regression testing, but some of both is probably in order.

I'm not at all sure this is conceptually the best way to go about this, and I don't think we should upstream it until we get that hashed out. I've been working for a while on being able to embed other languages ... we already do this 100% for Lua code in TeXlike inputs. However I've also been setting up similar things for Fluent (#665) and Lilypond (#435). This case is a little bit different being a hybrid between regular TeXlike language inputs and raw string replacements, but it has some of the same issues. Another similar case is Mathjax (#220 or several other math related ideas and WIP).

I don't really want 4 or 5 different ways to embed other languages, I think we need to have first class support for letting packages do this. Right now \script is hard coded into a list of formats to pass through to some other interpreter. We need to expose this in a way that packages can extend as well without modifying the core.

Another thought to throw out is that for many of the replacements, using string matching and replacements (ala input filter) is going to be more brittle that just defining full blown commands and letting SILE do the parsing. This doesn't handle all the things (i.e. not combining characters, but \RightArrow), but it might be worth exploring the ability to scope some commands to only function inside a namespace (i.e. only when nested in texmode).

I suggest carrying on a little bit as an external package because that will add a lot of flexibility to iterate faster and cover your own needs, but as things progress I would definitely keep an eye on how to address the above issues so it can be upstreamed.

[ctrlcctrlv]

@alerque

1. Cool!
2. Agreed. I'll probably never personally use XML mode, but I don't mind supporting it if it can be supported. :-) Supporting it should actually be easier as I don't need a mini-parser as many exist.
3. We definitely do need tests.

In re: conceptual issues, see also my latest comment on #940.

@simoncozens Thanks for your support!

[alerque]

Cross-linking #652 which has some related discussions.

[alerque]

Migrating comments from 940 since this that PR is closed and this issue is open and probably a better home for ongoing discussion:

@ctrlcctrlv wroteE:

I wish you would have used Rusty strings instead to escape. Perhaps it's not too late?

I explain why at § Known issues.

@alerque wrote:

I used a backslash escape because that's how we've done all the other escapes in TeXlike syntax. It did cross my mind that a Lua-like [[...]] syntax would be an interesting alternative. I didn't implement it though because I thought further thought & discussion would be necessary.

Ultimately though I don't think the answer is to change the escaping so much as change where you're trying to pass input in the first place. Rather than trying to pass CONTENT in \cmd[_="CONTENT"]{} you should just be able to pass it as content \cmd{CONTENT}. This is MASSIVELY more flexible, as commands also have an environment format \begin{cmd}CONTENT\end{cmd}. This is already used by \script for Lua code and you can see another example in my #499 package PR. With this syntax the end delimiter becomes "\end{your_cmd}", which is pretty hefty and unlikely to be a problem for most embedded languages. I think your \texmode should be re-implemented to operate on pass-through content as content instead of a big option string.

[ctrlcctrlv]

@alerque An earlier version did, in fact, operate on content and used inputfilter for everything. This is the reason for the macro-like names of e.g. TEXLIKECC, they used to be calling loadstring, creating actual commands.

But there's a few reasons this cannot work anymore/did not work well.

• In Escape texlike strings #940, you've made it so \" is no longer a permissible command. So, no AST can be generated.
• TeX commands can take multiple arguments. I think this should be implemented in our TeX mode, I very badly want to work on a tabular.lua, but so long as \begin{tabular}{ l | c | r } cannot be a thing it's not very appetizing. (Meh, I guess actually can parse content and pull out the first subenviron, but still, a bit dubious.)
• Even when it was "working", it still wasn't working well. I could only replace ligatures like --- in strings via inputfilter. If I tried to replace in a content and return back the table, it would ignore my change.

I ask that in registerCommand, you add an options table, and an option, takesstring, meaning, I suppose, "I will parse content myself". Or fix inputfilter.

[alerque]

@alerque An earlier version did, in fact, operate on content and used inputfilter for everything. This is the reason for the macro-like names of e.g. TEXLIKECC, they used to be calling loadstring, creating actual commands.

I don't think inputfilter is the way to do this...that's not how I would handle {content}.

* In #940, you've made it so `\"` is no longer a permissible command. So, no AST can be generated.

We can probably get that back, but I'm not convinced you need it. If you're not using registered SILE commands and doing your own substitutions, you don't need it. See the end...

* TeX commands can take multiple arguments. I think this should be implemented in our TeX mode, I very badly want to work on a tabular.lua, but so long as `\begin{tabular}{ l | c | r }` cannot be a thing it's not very appetizing. (Meh, I guess actually can parse content and pull out the first subenviron, but still, a bit dubious.)

SILE environments can take options too, as many as you please. Something like this should work:

\begin[alignment="{ l | c | r }"]{tabular}...

I don't really have any interest replicating in LaTeX's way of handling multiple arguments. I can't think of a single thing it's good for other than TeX didn't have any better way of parsing them and it got stuck with them.

If you want to make a \texmode to make it easy to type some Unicode characters most people don't know Compose sequences for that's one thing, but trying to replicate TeX's entire way of thinking about grammar and things like tabular is ... not very appealing. If you like it that much, use LuaLaTeX. I don't see what the benefit of SILE is if you try to clone TeX with it.

* Even when it was "working", it still wasn't working well. I could only replace ligatures like `---` in strings via inputfilter. If I tried to replace in a `content` and return back the table, it would ignore my change.

I ask that in registerCommand, you add an options table, and an option, takesstring, meaning, I suppose, "I will parse content myself". Or fix inputfilter.

But RegisterCommand() does already have an options table.

I think you've missed the point of what I'm suggesting. There already is a mechanism for what you are calling "takesstring": It's called passthrough, and you have to tell the TeXlike parser ahead of time what commands will operate in that mode so they don't get parsed into the AST.

I don't think inputfilter is broken, it just isn't meant to introduce a whole new grammar than happens to conflict with TeXlike syntax commands. Hence the passthrough commands & environments like we user for \script and I started for \lilypond.

[ctrlcctrlv]

How would I tell the parser to ignore the texmode environment?

[alerque]

Add your command name here:

https://github.com/sile-typesetter/sile/blob/master/core/inputs-texlike.lua#L9-L11

Note there is no way for a 3rd party package to change this because by the time any package you load is loaded the parser has already parsed the document! I haven't figured out a way around that yet except to hard code this list of potential passthrough commands.

[simoncozens]

Totally agree with @alerque. Making things easier for TeX users, fine. Trying to replicate TeX, less so...

[ctrlcctrlv]

@alerque This is quite a problem as I don't want users to need to patch SILE. If I PR a change to that file you'll accept it?

[alerque]

I'm leery of adding commands there out of the box until there is a clear use case, at least in large part because it blocks being able to register that as a command for use any other way. At the same timet I don't want to hamper 3rd party hacking! One of the reasons SILE is in Lua in the first place was to make it possible to tamper with the internals. Right now the passthroughCommands flag is an internal that is not accessable to tampering from a package or document.

How about we make it more hackable by adding a way to add items to that array from the command line invocation before the parser initializes, then wait and see what the use case ends up looking like.

[ctrlcctrlv]

Yes to command line :-)

Let me know when it's merged and I'll update sile-texmode to use it.

[alerque]

Actually come to think of you don't need anything from this end, this is already hackable. Example:

\begin{document}
\begin{script}
SILE.registerCommand("foo", function (options, content)
	SU.debug("foo", content[1])
end)
\end{script}
\begin{foo}
bar\baz{\"} quiz
\end{foo}
\end{document}

Run with:

$ sile -e 'SILE.inputs.TeXlike.passthroughCommands.foo = true' -d foo test.sil

The -e evaluates Lua code before loading the document, so you can modify the parser there. Adding foo as a command that will be passthrough mode prepares the way for the \foo command we then register. Note that it's contents are not parsed as SILE comands, the \baz{} and \" both just come through as part of the same literal string. You can do whatever you want with them at that point. The -d foo is just to enable the debug tag I used to show where the content was ending up.

[ctrlcctrlv]

Ooh :-) shiny 🤩

[alerque]

I'm pretty sure at this point we don't plan on bundling texmode in SILE releases, but I do want to improve the interface so 3rd party packages aren't at the disadvantage they are now when it comes to being able to do their own parsing of content. I've opened #1092 to track that specifically.

If there is anything else I've missed in here than needs tracking feel free to remind me ;-)