Prevent white spaces in certain rules

[rzcoder]

I would like to preserve the whitespace ignoring for most rules, but at the same time, I want to prohibit having spaces in certain rules.

For example, I have variable declaration like this $var byte[] = [1,2,3], where rules for var name and type looks like this:

this.RULE("variableName", () => {
	this.CONSUME(t.$);
	this.CONSUME(t.Identifier);
});

this.RULE("type", () => {
	this.CONSUME(t.Identifier);
	this.OPTION(() => {
		this.CONSUME(t.LSquare);
		this.CONSUME(t.RSquare);
	})
});

and I want to prevent "correct" parsing of something like $ var byte [ ] = [1,2,3].
I understand that I can use a single token for a variable name, but this approach doesn't seem right to me.
I also wouldn't want to manually place whitespace tokens for most high-level rules.

[bd82]

Hello @rzcoder

Note that the syntax you want to prevent is allowed TypeScript and Java, so it is not that "strange".

• https://www.typescriptlang.org/play?#code/MYewdgzgLgBAHgLjAVwLYCMCmAnA2jAXRgF4ZcBGAGgCZKBmAoA
• http://tpcg.io/_RDKDDT

Whitespace decides between alternative example

Even though whitespace is ignored in the rest of the grammar.

chevrotain/examples/grammars/css/css.js

Lines 426 to 447 in 4a68802

	$.OR([
	{
	GATE: () => {
	const prevToken = $.LA(0);
	const nextToken = $.LA(1);
	// This is the only place in CSS where the grammar is whitespace sensitive.
	return nextToken.startOffset > prevToken.endOffset;
	},
	ALT: () => {
	$.OPTION2(() => {
	$.SUBRULE($.combinator);
	});
	$.SUBRULE($.selector);
	},
	},
	{
	ALT: () => {
	$.SUBRULE2($.combinator);
	$.SUBRULE2($.selector);
	},
	},
	]);

Recommended approach: Treat this as a linting/formatting warning, not a syntax error.

Not all errors should be syntax (grammar) errors, it is often easier to parse a super-set of the allowed grammar
and perform additional analysis in a post-parsing steps, for example:

• Formatting warning.
• Semantic errors (assigning a string value to a number val)
• Operator precedence
  • can be defined using a grammar pattern.
  • But it is much faster (parsing wise) to handle it in a post processing step.

  • chevrotain/examples/grammars/ecma5/ecma5_parser.js

    Lines 41 to 48 in 4a68802

    	// A.3 Expressions
    	// Note that the binary expression operators are treated as a flat list
    	// instead of using a new rule for each precedence level.
    	// This is both faster and less verbose but it means additional logic must be used to re-order the flat list
    	// into a precedence tree.
    	// This approach was used in the swift compiler.
    	// https://developer.apple.com/library/content/documentation/Swift/Conceptual/Swift_Programming_Language/Expressions.html#//apple_ref/doc/uid/TP40014097-CH32-ID383
    	// (scroll down to the note on binary expressions)

  • chevrotain/examples/grammars/ecma5/ecma5_parser.js

    Lines 257 to 283 in 4a68802

    	$.RULE("BinaryExpression", () => {
    	$.SUBRULE($.UnaryExpression);
    	$.MANY(() => {
    	$.OR(
    	$.c3 ||
    	($.c3 = [
    	// flat list of binary operators
    	{ ALT: () => $.CONSUME(t.AbsAssignmentOperator) },
    	{ ALT: () => $.CONSUME(t.VerticalBarVerticalBar) },
    	{ ALT: () => $.CONSUME(t.AmpersandAmpersand) },
    	{ ALT: () => $.CONSUME(t.VerticalBar) },
    	{ ALT: () => $.CONSUME(t.Circumflex) },
    	{ ALT: () => $.CONSUME(t.Ampersand) },
    	{ ALT: () => $.CONSUME(t.AbsEqualityOperator) },
    	{ ALT: () => $.CONSUME(t.AbsRelationalOperator) },
    	{ ALT: () => $.CONSUME(t.InstanceOfTok) },
    	{ ALT: () => $.CONSUME(t.InTok) },
    	{ ALT: () => $.CONSUME(t.AbsShiftOperator) },
    	{
    	ALT: () => $.CONSUME(t.AbsMultiplicativeOperator),
    	},
    	{ ALT: () => $.CONSUME(t.AbsAdditiveOperator) },
    	]),
    	);
    	$.SUBRULE2($.UnaryExpression);
    	});
    	});

[rzcoder]

Hey @bd82!

Great thanks for answering! Checking LookAhead works for me so far.

Can I also ask, if code like this is common way to use simple CONSUME's with GATES?

this.RULE("variableName", () => {
  this.OR([{
    GATE: () => {
      const prevToken = this.LA(1);
      const nextToken = this.LA(2);
      return prevToken && nextToken && prevToken.endOffset + 1 === nextToken!.startOffset;
    },
    ALT: () => {
      this.CONSUME(t.$);
      this.CONSUME(t.Identifier);
    },
  }]);
});

And I also want to note that whitespace characters are a rather important part of the language syntax. Even for something like TypeScript, there's a significant difference in how a newline character is handled depending on the context. For example https://www.typescriptlang.org/play?ssl=5&ssc=14&pln=1&pc=1#code/MYewdgzgLgBAHgLjAVwLYCMCmAnGMBQMAvDANoCMANAEyUDMAuvoaJLAJ5Jpa4FkPEyVWoyA

It would be nice to have the ability to generate whitespace characters in the main token stream (like special group) and have the option for automatic ignoring or, conversely, explicit indication of the absence/presence of such a character within the rules.

[bd82]

if code like this is common way to use simple CONSUME's with GATES?

I don't think this is a common way, it is more of a workaround.
For example: I don't believe the error recovery logic would be aware of these GATES.

And I also want to note that whitespace characters are a rather important part of the language syntax

Depends on the language, (e.g Python + whitespace). But the question is if such edge cases of significant whitespace are by design or workarounds in the grammar due to bugs / ambiguities in the design.

If you are designing your own (new) language I do believe there is value in the simplicity of being as whitespace insensitive as possible.

It would be nice to have the ability to generate whitespace characters in the main token stream

I believe Antlr has such a feature, called Token Channels:

• https://medium.com/pragmatic-programmers/broadcasting-tokens-on-different-channels-9adfd1292763

There was some work in the past to generalize the Lexer APIs, which could part of the pre-requisite for such a feature.

• Custom Lexer Adapters #528

Perhaps you could try overriding some of Chevrotain internals to implement it in a limited manner for your use case,
Something like:

// Three Channels:
// 1. REAL (ID/$/[/])
// 2. ALL (everything including whitespace)
//     - All channel may be "virtual" as a "view" on the other two channels.
// 3. WS (only WS)
this.RULE("variableName", () => {
        // CONSUME from REAL channel would skip whitespace as normal.
	this.CONSUME(t.$, channels.REAL);
	this.CONSUME(t.Identifier, channels.REAL);
});

this.RULE("type", () => {
	this.CONSUME(t.Identifier, channels.REAL);
	this.OPTION(() => {
                // by `CONSUME` from ALL channel, we no longer skip whitespace here.
		this.CONSUME(t.LSquare, channels.ALL);
                // if we add this.CONSUME(t.SINGLE_SPACE, channels.WS) 
                // we will ensure the brackets always have a single space between them.
		this.CONSUME(t.RSquare, channels.ALL);
	})
});

This does not sound complicated if you only consider CONSUME
However, when other features are intermixed it could become much more complex, e.g:

• Lookahead calculation combined with channels.
• Error recovery
• Grammar Recording during initialization (?)

[matthew-dean]

@rzcoder What I did with my parser was capture all whitespace during tokenization, but then I extended the Chevrotain parser class such that when the tokens are assigned, they get split into a separate data structure and are discarded from the super class. That way, they can be checked / referenced in gates, yet they are not considered during Chevrotain's recording / parsing phase.