Allow lexer to choose shorter token ?

[dhowe]

I have a language in which spaces are allowed, as well as '<' and '>', but I want the lexer to treat the regex '<\d+>' as a single token. Is there any way to make the lexer recognize the (shorter) WEIGHT token below?

const WEIGHT = createToken({ name: "WEIGHT", pattern: /\s*<\d+>\s*/ });
const TEXT = createToken({ name: "TEXT", pattern: /[\w\d\s<>]+/ });

Examples (-> expected)

1. b <3> -> [ TEXT, WEIGHT ]
2. a <3> b -> [ TEXT, WEIGHT, TEXT ]
3. a <2> <em>b</em> <3> [ TEXT, WEIGHT, TEXT, WEIGHT ]

Code test:

const { Lexer, createToken } = require("chevrotain");

const WEIGHT = createToken({ name: "WEIGHT", pattern: /\s*<\d+>\s*/ });
const TEXT = createToken({ name: "TEXT", pattern: /[\w\d\s/<>]+/ });

const lexer = new Lexer([WEIGHT, TEXT]);

const tests = [
  'b <3>', // [ TEXT, WEIGHT ]
  'a <3> b', // [ TEXT, WEIGHT, TEXT ]
  'a <2> <em>b</em> <3>', //[ TEXT, WEIGHT, TEXT, WEIGHT ]
]

tests.forEach((input, i) => {
  let result = lexer.tokenize(input);
  if (result.errors.length) throw Error(result.errors[0].message);
  console.log(i, result.tokens.map(t => t.tokenType.name + ': "' + t.image + '"'));
})

Output:

0 [ 'TEXT: "b <3>"' ]
1 [ 'TEXT: "a <3> b"' ]
2 [ 'TEXT: "a <2> <em>b</em> <3>"' ]

[msujew]

Hey @dhowe,

Trying to directly answer your question: You can manipulate your TEXT pattern so it doesn't lex the WEIGHT pattern. The issue in your lexer isn't that the lexer chooses the longest token - it never does, it chooses the first token to match - but rather that your TEXT token never terminates in your examples. It will parse the full text and doesn't stop, so the lexer never has the opportunity to even try lexing the WEIGHT token. For that you would need to use negative lookahead in your regex, so that it stops if it encounters a pattern that could match the WEIGHT token.

On a more general note, I have a feeling like you're trying to accomplish too much using just a lexer. Your lexer should be as atomic as possible. You should treat <3> as three separate tokens, something like [LESS_THAN, TEXT, MORE_THAN] instead of trying to parse it as a single one. You can then later write a WEIGHT parser rule that parses these tokens in the exact same order that they should appear. This also makes your language way more resilient against lexer errors - compared to parser errors, from which your parser can recover from. Similarly, the TEXT token should probably be split in <, > and word/numbers tokens, as it is too eager right now. Lastly, is your language whitespace aware? If not, you should consider creating a whitespace token that the lexer skips.

[dhowe]

Interesting suggestions, thank you. Is there an example somewhere that uses negative lookahead ? Meantime, I will try the 2nd approach and report back... (and yes, the language is whitespace aware)