Make it possible to import packages on container with lower than 2 GB of memory

[schlichtanders]

This is actually a really crucial problem I am facing currently. I already pushed it at the specific package which I was not able to import, SciMLSensitivity, but @ChrisRackauckas mentioned this is rather a generic Julia issue.

What happens? julia is killed as soon as I merely import a package...

This happens, because the docker container is restricted to 2 GB of RAM, while loading this package needs more than 2 GB of RAM (or at least in this default setup without any additional configuration it needs more).

Hence I am unable to use binder for SciML, and also anyone else is unable to use mybinder for SciML. A real pity.
It would be great if loading a package does not kill a moderate machine like a binder instance with 2 GB of RAM.

[gbaraldi]

1.8.2 might make this a bit better because it makes julia more aware of memory constraints. It should release in a couple days if no issues are found

[PallHaraldsson]

New in 1.9, can you try something like:

julia --heap-size-hint=2G

I think compilation of packages makes new julia processes, then they have their own memory and GC, and the limitation is likely not inherited (but should it?). You can also try limiting parallel compilation. Maybe just disable it with:

ENV["JULIA_NUM_PRECOMPILE_TASKS"] = 1

[maleadt]

This is essentially a dup of #24617.

New in 1.9, can you try something like:

julia --heap-size-hint=2G

No need for that, it will be auto-detected. Only use --heap-size if you want (or need) to restrict further.

[schlichtanders]

It would also be great to have a way to change julia's import behaviour from within the running julia kernel

[schlichtanders]

sorry, wrong button

[PallHaraldsson]

What do you mean "import behavior" (and Julia kernel, if Jypyter, then an issue on the needs to be opened elsewhere).

I think you might be able to do:

ENV["JULIA_NUM_PRECOMPILE_TASKS"] = 1

from within Jupyter (and Pluto etc.), might not be too late, or start it that way from the shell, similar to you would do for Julia (at least on Linux):

$ JULIA_NUM_PRECOMPILE_TASKS = 1 julia

[PallHaraldsson]

This might be fixed already with: #46796

I would check with master/nightly at the download page. It looks like it made it into 1.8.2 which should appear soon (it was merged recently). If I misread, then 1.8.3 seems coming too (but I see no proof of an open backport PR for it yet): #46796 (comment)

When #46844 lands it's also likely to help (with memory use). See also: #46831

[maleadt]

1.8.2 is imminent, and should fix this.

[schlichtanders]

This is unfortunately not fixed in julia 1.8.2

the kernel still dies when importing

EDIT: here the plain terminal test on mybinder with 2GB memory:

[maleadt]

What distro are you trying this on? Does it use cgroupv1 or v2? #46796 (comment) may be relevant.