Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix 'tree-sitter generate' memory explosion on Hack grammar #365

Open
mjambon opened this issue Nov 7, 2022 · 0 comments
Open

Fix 'tree-sitter generate' memory explosion on Hack grammar #365

mjambon opened this issue Nov 7, 2022 · 0 comments
Labels
blocking A task in some other project depends on this bug Something isn't working help wanted Extra attention is needed priority:high blocks a user r2c user not originally reported by an external user

Comments

@mjambon
Copy link
Member

mjambon commented Nov 7, 2022

This is preventing us from merging and using semgrep/ocaml-tree-sitter-core#48.

What we know:

  • Processing the tree-sitter-hack grammar after rewriting by ocaml-tree-sitter has always consumed a lot of memory. It now requires over 16 GB, which is more than a reasonable host should have to support.
  • ocaml-tree-sitter unhides all the rules by removing the leading underscore from the rule name. Re-hiding all these rules except the entry point leads to high CPU usage which times out after 50 min (on @mjambon's old laptop).

We need to investigate tree-sitter, which is a Rust program. The first step would be to come up with a minimal test case and file a bug with the tree-sitter project. Right now, we know that the modified Hack grammar is problematic but other grammars of similar size don't show excessive memory or CPU consumption.

@mjambon mjambon added bug Something isn't working help wanted Extra attention is needed blocking A task in some other project depends on this priority:high blocks a user r2c user not originally reported by an external user labels Nov 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
blocking A task in some other project depends on this bug Something isn't working help wanted Extra attention is needed priority:high blocks a user r2c user not originally reported by an external user
Development

No branches or pull requests

1 participant