-
Notifications
You must be signed in to change notification settings - Fork 400
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
dune build -w
memory usage grows without bounds when using OCaml syntax in dune files
#6900
Comments
Thanks for the great report and repro case. I was able to reproduce and bisect this down to d3296a6 (#5877). I'm not familiar enough with the |
I naively tried to replace |
Thanks for taking a look. Changing equality doesn't help it's only for hash collisions. I think the way to go about it is just to change the memoization store to stop storing old values. We're currently using a hash table, but I suppose we only need the last value for every dune file path we evaluate. |
Do you mean using |
Yes. The idea is to use a |
Before trying that I tried to print the sizes of the tables used by the two call sites of |
That would be expected. The hashing function for this function is poor, so all the entries would go into a single bucket. |
I don't think I understand how My assumption with this issue is that every time the file is modified, a new binding is added to the table, and the table is never cleared. This is what I wanted to check with |
Would we ever want to trim our memo table to fit some constraints? We could experiment with a cache replacement policy. |
Never mind, it's the total number of bindings.
Yes, that confirms it. |
We discussed this and yes there must be a growing table but we don't know which one.
|
I actually used another technique: I added the sizes of the memo tables to the trace-file output. It shows several growing tables:
(for comparison, the same test on dune itself does not show any growing table) I'll have a look at the input/outputs of these tables. |
For some of these tables, the input is compared using physical equality For rules (inputs of |
You can try switching the store for libraries and see if that helps. We don't really need old libs and they're quick enough to construct.
That's not necessarily bad for rules. It's often the case that people go back and forth between a few states. |
CHANGES: - Validate file extension for `$ dune ocaml top-module`. (ocaml/dune#8005, fixes ocaml/dune#8004, @3Rafal) - Include the time it takes to read/write state files when `--trace-file` is enabled (ocaml/dune#7960, @rgrinberg) - Add `dune show` command group which is an alias of `dune describe`. (ocaml/dune#7946, @Alizter) - Include source tree scans in the traces produced by `--trace-file` (ocaml/dune#7937, @rgrinberg) - Cinaps: The promotion rules for cinaps would only offer one file at a time no matter how many promotions were available. Now we offer all the promotions at once (ocaml/dune#7901, @rgrinberg) - Do not re-run OCaml syntax files on every iteration of the watch mode. This is too memory consuming. (ocaml/dune#7894, fix ocaml/dune#6900, @rgrinberg) - Remove some compatibility code for old version of dune that generated `.merlin` files. Now dune will never remove `.merlin` files automatically (ocaml/dune#7562) - Add `dune show env` command and make `dune printenv` an alias of it. (ocaml/dune#7985, @Alizter) - Add additional metadata to the traces provided by `--trace-file` whenever `--trace-extended` is passed (ocaml/dune#7778, @rleshchinskiy) - Extensions used in `(dialect)` can contain periods (e.g., `cppo.ml`). (ocaml/dune#7782, fixes ocaml/dune#7777, @nojb) - Allow `(include_subdirs qualified)` to be used when libraries define a `(modules ...)` field (ocaml/dune#7797, fixes ocaml/dune#7597, @anmonteiro) - `$ dune describe` is now a command group, so arguments to subcommands must be passed after subcommand itself. (ocaml/dune#7919, @Alizter) - The `interface` and `implementation` fields of a `(dialect)` are now optional (ocaml/dune#7757, @gpetiot) - Add commands `dune show targets` and `dune show aliases` that display all the available targets and aliases in a given directory respectively. (ocaml/dune#7770, grants ocaml/dune#265, @Alizter) - Allow multiple globs in library's `(stdlib (internal_modules ..))` (@anmonteiro, ocaml/dune#7878) - Attach melange rules to the default alias (ocaml/dune#7926, @haochenx) - In opam constraints, reject `(and)` and `(or)` with no arguments at parse time (ocaml/dune#7730, @emillon) - Compute digests and manage sandboxes in background threads (ocaml/dune#7947, @rgrinberg) - Add `(build_if)` to the `(test)` stanza. When it evaluates to false, the executable is not built. (ocaml/dune#7899, fixes ocaml/dune#6938, @emillon) - Add necessary parentheses in generated opam constraints (ocaml/dune#7682, fixes ocaml/dune#3431, @Lucccyo)
CHANGES: - Validate file extension for `$ dune ocaml top-module`. (ocaml/dune#8005, fixes ocaml/dune#8004, @3Rafal) - Include the time it takes to read/write state files when `--trace-file` is enabled (ocaml/dune#7960, @rgrinberg) - Add `dune show` command group which is an alias of `dune describe`. (ocaml/dune#7946, @Alizter) - Include source tree scans in the traces produced by `--trace-file` (ocaml/dune#7937, @rgrinberg) - Cinaps: The promotion rules for cinaps would only offer one file at a time no matter how many promotions were available. Now we offer all the promotions at once (ocaml/dune#7901, @rgrinberg) - Do not re-run OCaml syntax files on every iteration of the watch mode. This is too memory consuming. (ocaml/dune#7894, fix ocaml/dune#6900, @rgrinberg) - Add `--all` option to `dune rpc status` to show all Dune RPC servers running. (ocaml/dune#8011, fix ocaml/dune#7902, @Alizter) - Remove some compatibility code for old version of dune that generated `.merlin` files. Now dune will never remove `.merlin` files automatically (ocaml/dune#7562) - Add `dune show env` command and make `dune printenv` an alias of it. (ocaml/dune#7985, @Alizter) - Add additional metadata to the traces provided by `--trace-file` whenever `--trace-extended` is passed (ocaml/dune#7778, @rleshchinskiy) - Extensions used in `(dialect)` can contain periods (e.g., `cppo.ml`). (ocaml/dune#7782, fixes ocaml/dune#7777, @nojb) - Allow `(include_subdirs qualified)` to be used when libraries define a `(modules ...)` field (ocaml/dune#7797, fixes ocaml/dune#7597, @anmonteiro) - `$ dune describe` is now a command group, so arguments to subcommands must be passed after subcommand itself. (ocaml/dune#7919, @Alizter) - The `interface` and `implementation` fields of a `(dialect)` are now optional (ocaml/dune#7757, @gpetiot) - Add commands `dune show targets` and `dune show aliases` that display all the available targets and aliases in a given directory respectively. (ocaml/dune#7770, grants ocaml/dune#265, @Alizter) - Allow multiple globs in library's `(stdlib (internal_modules ..))` (@anmonteiro, ocaml/dune#7878) - Attach melange rules to the default alias (ocaml/dune#7926, @haochenx) - In opam constraints, reject `(and)` and `(or)` with no arguments at parse time (ocaml/dune#7730, @emillon) - Compute digests and manage sandboxes in background threads (ocaml/dune#7947, @rgrinberg) - Add `(build_if)` to the `(test)` stanza. When it evaluates to false, the executable is not built. (ocaml/dune#7899, fixes ocaml/dune#6938, @emillon) - Add necessary parentheses in generated opam constraints (ocaml/dune#7682, fixes ocaml/dune#3431, @Lucccyo)
Expected Behavior
dune build -w
memory usage should stay below some upper bound over a longer period of time.Actual Behavior
dune build -w
grows without bounds.Reproduction
Minimal example:
In another terminal:
More realistic example:
git clone https://github.com/ocaml/stdlib-shims.git && git clone https://github.com/mirage/bigarray-compat.git
(I believe the presence of dune files containing OCaml syntax is sufficient to trigger their execution and this memory leak)dune build -w
and make some normal editsSpecifications
dune
(output ofdune --version
): 3.6.2ocaml
(output ofocamlc --version
): 4.14.0 or 5.0Additional information
dune --trace-file
output:Memtrace output (added to dune 3.6.2 in
bin/main.ml
):The text was updated successfully, but these errors were encountered: