feat: introduce precompiled-grammars

[antfu]

resolves #878

Relies on:

• feat: introduce @shikijs/themes and @shikijs/langs packages #879
• feat: support bypassing RegExp literals vscode-textmate#3
• feat: expose raw options on EmulatedRegExp slevithan/oniguruma-to-es#15

This PR introduces a new @shikijs/langs-precompiled package that inlines the compiled RegExp literals inside the grammar objects. And a new @shikijs/engine-javascript/raw that only takes precompiled grammar to further optimize the performance and bundle size

[netlify]

✅ Deploy Preview for shiki-next ready!

Name	Link
🔨 Latest commit	1c20551
🔍 Latest deploy log	https://app.netlify.com/sites/shiki-next/deploys/6776656148b58f0008f35dbd
😎 Deploy Preview	https://deploy-preview-880--shiki-next.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[netlify]

✅ Deploy Preview for shiki-matsu ready!

Name	Link
🔨 Latest commit	1c20551
🔍 Latest deploy log	https://app.netlify.com/sites/shiki-matsu/deploys/67766561f13883000981a82b
😎 Deploy Preview	https://deploy-preview-880--shiki-matsu.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[codecov]

Codecov Report

Attention: Patch coverage is 88.53503% with 18 lines in your changes missing coverage. Please review.

Project coverage is 95.17%. Comparing base (d28d797) to head (1c20551).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
packages/engine-javascript/src/scanner.ts	83.33%	16 Missing ⚠️
packages/engine-javascript/src/engine-raw.ts	89.47%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #880      +/-   ##
==========================================
- Coverage   95.21%   95.17%   -0.04%     
==========================================
  Files          83       86       +3     
  Lines        7063     7095      +32     
  Branches     1461     1468       +7     
==========================================
+ Hits         6725     6753      +28     
- Misses        330      334       +4     
  Partials        8        8              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[slevithan]

oniguruma-to-es v0.10.0 has been published. It includes a rawArgs property on EmulatedRegExp instances so they can be accurately serialized.

rawArgs type:

{
  pattern: string;
  flags: string;
  options: {
    strategy?: string;
    useEmulationGroups?: boolean;
  };
}

[slevithan]

Although you're not currently using toDetails, heads up that v0.10.0 also renamed toDetails's subclass property as options, for consistency.

Also, if you bump to v0.10.0 as part of this PR, it's worth subsequently bumping tm-grammars and running report-engine-js. Although v0.10.0's improvements don't change support for existing grammars, a couple of grammars have now landed upstream fixes.

[antfu]

Did a rough benchmarking. The performance still varies with different languages, but it's indeed faster than the pure JS version.

[slevithan]

If this code could somehow know whether the regex it's evaluating will have its captures used by the grammar, it could get a perf boost by avoiding the use of EmulatedRegExp when capture values are not used (also described in #878).

  if (value instanceof EmulatedRegExp) {
    if (capturesUsedInGrammarForThisRegex || value.rawArgs.options.strategy) {
      return `new EmulatedRegExp(${JSON.stringify(value.rawArgs.pattern)},${JSON.stringify(value.rawArgs.flags)},${JSON.stringify(value.rawArgs.options)})`
    }
    // For perf, avoid extra handling from `EmulatedRegExp` since it isn't needed in this case
    return value.toString()
  }

[antfu]

Would you like to send a PR for that? Thanks!