Incremental Parsing in Nelua

[stefanos82]

I watched "Tree-sitter - a new parsing system for programming tools" by Max Brunsfeld and what fascinated me the most is Incremental Parsing.

That is exactly what I have had in mind for Nelua while I shared my thoughts on Discord, but didn't know the proper terminology to use.

Does LPegRex or LPeg / LPegLabel support such mechanism?

It could improve the coding experience exponentially, because the parsing would be validated during typing in real-time.

[stefanos82]

Seems like it's possible and just got published 😄 https://github.com/zyedidia/gpeg#publications

I have just downloaded the publication and thesis of this user to study it carefully; they based their research on LPeg which is quite fascinating!

[edubart]

Does LPegRex or LPeg / LPegLabel support such a mechanism?

The way I typically use LPeg or LPegLabel usually has to parse everything again on every change, and I don't think incremental parsing is feasible with it.

On the other hand, LPegLabel supports error recovery, one feature mentioned in that talk. But in Nelua error recovery is not used, the Nelua parser will usually stop on first syntax error and then ignore the renaming code to parse, but this is just a design decision, the only benefit of recovering from errors would be to give extra syntax errors. I've chosen to fail on the first error for the Nelua compiler, to make it simple.

That is exactly what I have had in mind for Nelua while I shared my thoughts on Discord, but didn't know the proper terminology to use.

Probably the terminology you are looking for is incremental compiling or incremental parsing.

It could improve the coding experience exponentially, because the parsing would be validated during typing in real-time.

In the Nelua compiler not much, the compiler does not spend much time parsing, it spends more time analyzing. But incremental parsing makes sense in text editors, because you usually want your syntax highlighted instantaneously.

I watched "Tree-sitter - a new parsing system for programming tools" by Max Brunsfeld and what fascinated me the most is Incremental Parsing.

Worth mentioning that there is a tree sitter parser for Nelua in https://github.com/codehz/tree-sitter-nelua

[stefanos82]

The way I typically use LPeg or LPegLabel usually has to parse everything again on every change, and I don't think incremental parsing is feasible with it.

Originally I planned to write

Wait..what? You go through the entire parsing procedure on every run? Why?!

but I decided to run a simple test first and realized the reasoning behind your decision to re-parse on every code modification or whenever possible:

This simple code

print("Hello")

produces the following AST output:

Block {
  Call {
    {
      String {
        "Hello"
      }
    },
    Id {
      "print"
    }
  }
}

whereas this updated version:

local msg = "hello"
print(msg)

would produce the following AST output:

Block {
  VarDecl {
    "local",
    {
      IdDecl {
        "msg",
        false
      }
    },
    {
      String {
        "hello"
      }
    }
  },
  Call {
    {
      Id {
        "msg"
      }
    },
    Id {
      "print"
    }
  }
}

therefore the re-parsing is mandatory anyway.

On the other hand, LPegLabel supports error recovery, one feature mentioned in that talk. But in Nelua error recovery is not used, the Nelua parser will usually stop on first syntax error and then ignore the renaming code to parse, but this is just a design decision, the only benefit of recovering from errors would be to give extra syntax errors. I've chosen to fail on the first error for the Nelua compiler, to make it simple.

Fair enough.

Probably the terminology you are looking for is incremental compiling or incremental parsing.

As I said above, when I shared my thoughts on Discord I didn't know Incremental Parsing existed as terminology or as algorithm; thus the current title in this thread topic.

In the Nelua compiler not much, the compiler does not spend much time parsing, it spends more time analyzing. But incremental parsing makes sense in text editors, because you usually want your syntax highlighted instantaneously.

I wrote down the spec files you suggested to study so I can understand how Nelua internals work during execution time:

1. syntaxdefs_spec
2. typechecker_spec
3. luagenerator_spec
4. cgenerator_spec
5. preprocessor_spec
6. stdlib_spec
7. runner_spec

Which files from spec are responsible for analyzing the code?

Worth mentioning that there is a tree sitter parser for Nelua in https://github.com/codehz/tree-sitter-nelua

Nice, it's good to know 👍

[edubart]

Which files from spec are responsible for analyzing the code?

It was typechecker_spec.lua, but I've renamed it to analyzer_spec.lua in commit 22e8882

[stefanos82]

That's why I couldn't find it? 🤣

[stefanos82]

Question: which file is or files are responsible for startup?

[stefanos82]

By the way,

nelua-lang/lualib/nelua/runner.lua

Line 42 in b6b67e6

console.info('Copyright (C) 2019-2021 Eduardo Bart (https://nelua.io/)')

could be replaced with console.infof("Copyright (C) 2019-%s Eduardo Bart (https://nelua.io/)", os.date("%Y")) and be done with it.