Custom Token Pattern's matched result not being consumed

[reaperdtme]

Hey, I'm pretty new to this, so bear with me.
What I'm doing seems relatively straightforward, and it looks like it's working as it should, but the behavior is incorrect.
I have simple grammar syntax:

A tab delimited description block initiated by a "`d" followed by at least one identifier as so:

`d A mover, activator, watcher
observer {}

In this case, we're not even running into tab delineation.

My rule in pseudo code is

rule concept
  option () => consume(Description)
  at_least_one( identifier )

My custom token pattern for description correctly returns:

Exec Result [
  '`d A mover, activator, watcher\n',
  undefined,
  undefined,
  index: 0,
 ]

The parser acknowledges this but then fails with an EarlyExitException on the at_least_one(identifier) trying to match against the same description string but on line two:

EarlyExitException: Expecting: expecting at least one iteration which starts with one of these possible Token sequences::
    <[Identifier]>
  but found: '`d A mover, activator, watcher
  '

{
    token: {
      image: '`d A mover, activator, watcher\n',
      startOffset: 31,
      endOffset: 61,
      startLine: 2,
      endLine: 2,
      startColumn: 1,
      endColumn: 31,
      tokenTypeIdx: 19,
      tokenType: [Object],
      payload: 'A mover, activator, watcher'
    },
    resyncedTokens: [],
    previousToken: {
      image: '`d A mover, activator, watcher\n',
      startOffset: 0,
      endOffset: 30,
      startLine: 1,
      endLine: 1,
      startColumn: 1,
      endColumn: 31,
      tokenTypeIdx: 19,
      tokenType: [Object],
      payload: 'A mover, activator, watcher'
    },
    context: { ruleStack: [Array], ruleOccurrenceStack: [Array] }
  }
]

Offset 31 in the sample text is the 'o' in the observer, but the error message says the image is '`d'
Why is the description line on line 1 being replayed as line two against my identifier match

My identifier token is a simple /[a-zA-Z]\w*/

[msujew]

Hey @lightchaser411,

Offset 31 in the sample text is the 'o' in the observer, but the error message says the image is '`d'
Why is the description line on line 1 being replayed as line two against my identifier match

Good question. It seems like your custom token pattern is a bit too eager in what it accepts. It'd be great if you were to share your code that implements the pattern as the error is likely somewhere in there. What I assume happens here is that your custom pattern matches the first description token correctly. Afterwards the lexer continues and tries to run your pattern again, which erroneously returns the same token, instead of returning null. This would allow the lexer to continue to your identifier token. Can you look at the output of your lexer result? It probably contains [Description, Description] instead of [Description, identifier].

Note that you can likely replace your custom pattern definition with a regex like /`d[^\n\r]*/

[bd82]

A good way to debug such issues is to track the output of the Lexer (array of tokens) and completely ignore the parsing step.