fix(gatsby): lower memory pressure in SSR

[vladar]

Description

1. Lower memory pressure

This PR lowers peak memory consumption of gatsby builds up to 2x.

Before this PR we put massive pressure on memory and FS while running SSR in workers. This PR restricts concurrency of html-file writes as well as page-data file reads which greatly reduces this pressure.

The improvement is proportional to the number of CPU cores and the size of your page-data files and HTML. On my test site, memory consumption per SSR worker dropped from 500MB+ to ~200MB.

For a machine with 4 physical cores, it means ~1200MB reduction of peak memory.

2. WorkerPool unification

Also, I've noticed that we create SSR worker pool twice. One worker pool was effectively used for develop and the other - for builds but they both were created for gatsby build.

This PR unifies the build and develop behavior - now both use the same Worker pool and we don't create unnecessary processes.

Note that those two changes are independant.

[KyleAMathews]

wouldn't we want the concurrency to match the size of the jest worker pool?

[vladar]

This is concurrency within one worker. So this effectively sets total concurrency across all workers to workerPoolSize * 2. I was hesitating if we should set { concurrency: 1 } or { concurrency: 2 } here but 2 seems to keep memory usage as low as 1, so settled on value 2.

To clarify your question - do you mean changing it to 1 here? Because setting it to workerPoolSize will effectively make it workerPoolSize * workerPoolSize which doesn't make much sense to me unless I am missing something.

[KyleAMathews]

oh my bad. I didn't read this closely and thought this was in the main node process. Yeah 2 makes sense then. SSR is pure CPU so reading in any more page-data.json files wouldn't speed up anything and 1 would very slightly slow it down as the next SSR has to wait for the read to finish.

[KyleAMathews]

Great find!