Handling Escape characters

[esatterwhite]

I'm looking for some suggestions around the best way to handle an escape character. I understand there are parser modes, but that requires a start and stop character. Which would effectively be like quoting - which is what we do now.

But and example would be # is a part of my syntax, but there us a use case where CSS hex codes show up #CCCCCC which we don't want parsed. Now we tell people to quote them "#CCCCCC" but it would be nice to have an explicit escape char to bail out of the parsing.

\#CCCCCC

Javascript does something similar with quoted strings '\"literal quoted' and it doesn't necessarily have to have a matching close character

[msujew]

Hey @esatterwhite,

you might want to give a bit more context to the problem. Handling escape characters without clear delimiters isn't really easy and quite context dependent. Since they're escaped, I assume these escaped color tokens should be lexed as something else? Let's say you have a color token and a string token, which also accepts escaped colors. This should look something like this:

const color = createToken({
  name: "color",
  pattern: /#[0-9a-zA-F]{6}/
});
const text = createToken({
  name: "string",
  pattern:  /"[^"]*"|'[^']*'|\\#[0-9a-zA-F]{6}/ // A normal string or an escaped color
});

Is this what you're looking for?

[msujew]

parable tokens

Is that something like an operator? Google doesn't give me any good results when searching for parable tokens.

Anyway, so from my understanding your main issue is that some color codes - something like #ceaded - are parsed as # and ceaded instead of its intended color code. And in these cases it should be escaped like \#ceaded so that it's parsed as a single color code, right?

In that case it should be fairly straightforward:

const color = createToken({
  name: "color",
  pattern: /#[0-9][0-9a-fA-F]{5}|\\#[0-9a-fA-F]{6}/
});
const numSign = createToken({
  name: "numSign",
  pattern: "#",
  longer_alt: color // the color token is a potentially longer version of the num sign token
});
const ID = createToken({
  name: "ID",
  pattern:  /[a-zA-Z]\w*/
});

[esatterwhite]

Parsable

[esatterwhite]

Well no I don't think It is that simple. The color code was an edge case we ran into. We use # to trigger some logic and I want to give users a way to explicitly opt out of that.

I may want #whizbang to follow normal logic and be parsed in some cases. And #whizbang to just be treated as a literal value when the user indicates that is the case.

Maybe this is helpful
https://github.com/logdna/setup-chain-node#lookupinput

[esatterwhite]

Right now, the behavior handles quoted strings, so if you want a thing to not be parsed, you would wrap it. "#whizbang" . But it isn't very obvious and is a bit clunky

I would rather have an escape char \#whizbang

[esatterwhite]

This issue is the original request logdna/setup-chain-node#14

[myndzi]

I also (had) a question along these lines. My use case is that I would like to parse a query syntax that may contain quoted strings. For example:
foo = "bar"

Whenever you want to accept arbitrary strings, you also need to have a way to escape quotes within them:
foo = "b\"ar"

It seemed like multi-mode lexing was a great fit for this: when parsing at a position where I can accept a string, a literal " can put it in a "double-quoted" mode. This mode can quite efficiently accept "everything but a quote" and also "an escaped quote", which the parser can unescape and concatenate.

I initially implemented like this:

  const QDouble = createToken({ name: "QDouble", pattern: /"/, label: '"', push_mode: 'double_quoted' });
  const NotQDouble = createToken({ name: "NotQDouble", pattern: /[^"]+/, label: "str" });
  const QDoubleEnd = createToken({ name: "QDoubleEnd", pattern: /"/, label: '"', pop_mode: true });

  const ESC = createToken({ name: "ESC", pattern: /\\./, label: '\\' });

  const CalculatorLexer = new Lexer({
    modes: {
      "query": allTokens,
      "single_quoted": [ESC, NotQSingle, QSingleEnd],
      "double_quoted": [ESC, NotQDouble, QDoubleEnd],
    },
    defaultMode: "query"
  });

This wasn't working as expected, even with longer_alt. However, as I was writing this question, I realized the answer:

"NotQDouble" accepts backslashes, so the "ESC" token never has a chance if it's not the first character in the list, no matter what order I tell the lexer to process the tokens in.

The solution is simple:

  const NotQDouble = createToken({ name: "NotQDouble", pattern: /[^"\\]+/, label: "str" });

(note the addition of \\ to the negated character class)

[bd82]

Hello @myndzi

I normally handle quoted strings and escapes with a single regexp: e.g:

• https://stackoverflow.com/a/249937

Would this work in your case?

[esatterwhite]

Not really. I don't want to escape a quote. I want to escape the current parser behavior and return the. Next thing as a single token. Right now this is being done by quoting.

'"#ccc"' vs '\#ccc' or '\#CCC,#fff'