TypeScript 2.0 related garbage collection issue affecting ts-loader

[johnnyreilly]

Hi,

I'm helping out on ts-loader and trying to get a new version shipped. As part of this I've extended our integration test pack, which already runs against TypeScript 1.6, 1.7 and 1.8 to run against 2.0 as well. Here's the rub....

Our integration test pack rolls over and dies once we start using TS 2.0. See an example here: https://travis-ci.org/TypeStrong/ts-loader/jobs/165652081

<--- Last few GCs --->
   73846 ms: Mark-sweep 1352.6 (1434.9) -> 1351.5 (1434.9) MB, 1483.5 / 0 ms [allocation failure] [GC in old space requested].
   75391 ms: Mark-sweep 1351.5 (1434.9) -> 1349.5 (1434.9) MB, 1545.6 / 0 ms [allocation failure] [GC in old space requested].
   76931 ms: Mark-sweep 1349.5 (1434.9) -> 1349.4 (1434.9) MB, 1539.9 / 0 ms [last resort gc].
   78363 ms: Mark-sweep 1349.4 (1434.9) -> 1349.4 (1434.9) MB, 1432.1 / 0 ms [last resort gc].
<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 0x1617664c9fa9 <JS Object>
    1: createMap [/home/travis/build/TypeStrong/ts-loader/node_modules/typescript/lib/typescript.js:1032] [pc=0x3e0e4f288c0f] (this=0x1c6fdd4cea61 <an Object with map 0xf18d45bcb89>,template=0x161766404189 <undefined>)
    2: arguments adaptor frame: 0->1
    3: createSymbolTable [/home/travis/build/TypeStrong/ts-loader/node_modules/typescript/lib/typescript.js:~20130] [pc=0x3e0e4f95f87c] (this...
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
Aborted (core dumped)
npm ERR! Test failed.  See above for more details.

However other versions of TypeScript with the exact same codebase do not face this issue; see this example running against TS 1.8. (Yes there are failing tests here; but those are things that need addressing; the test pack isn't actually failing to run.)

It may be a a node problem (well maybe) - I can see perhaps related issues:
webpack/webpack#1875
nodejs/node#1741
karma-runner/karma#1868

To be clear; ts-loader works with TS 2.0. But it's clearly causing strain that the previous versions of TypeScript didn't.

Do you have any idea of why TS 2.0 might seem to be triggering this issue? I'm well aware that our integration test pack may well be less than ideal but I'm surprised that this issue is occurring. Any suggestions gratefully received!

Expected behavior:

TypeScript 2.0 doesn't experience a garbage collection problem in our test pack.

Actual behavior:

TypeScript 2.0 does experience a garbage collection problem in our test pack.

[mhegazy]

this is not a GC issue. this is out of memory issue. have you checked what are you holding on to while running these tests? do you clean the state after every test? it is possible that TS 2.0 is using more memory than TS 1.8 so tests used to pass barely in 1.8, but starts to break in TS 2.0. I would start looking at memory profiles with running the tests, and see what is held in memory after each test. if you see TS holding to objects that it should not hold on to, then it is a TS issue, if your tests are holding to objects that it should not hold on to, then it is a test cleanup issue.

[johnnyreilly]

Thanks for the response @mhegazy. I've just picked up looking into the ts-loader test pack and don't know the code very well - it's a little unclear. I'll do some investigation. I thought it was worth asking the question though since it's not been a problem before.

[johnnyreilly]

Was able to work around the issue.

[HerringtonDarkholme]

I'm quite sure ts-loader's problem is exactly what @mhegazy said. ts-loader barely passed tests under 1.8. And 2.0's flow checking just uses more memory and ts-loader blows up. The correct fix is to clean up compiler instance after each test.

[johnnyreilly]

Yup - that's effectively what we're doing now

[johnnyreilly]

For what it's worth we now have a new test pack in ts-loader that's less brittle than the existing one. (We run both packs in our continuous integration tests.) We've also taken steps to make the existing pack more resilient.