Need a big optimization for Windows

[OshotOkill]

Do you want to request a feature or report a bug?

Feature

What is the current behavior?

I've tested a lot about the installation speed between MacOS and Windows. According to the results it seems that yarn has far less optimizations for Windows. e.g. Here are the comparisons of installing react-native :

Test Machines:

• ThinkPad X1 Carbon 4, 1TB PCI-E SSD, 16GB Memory
• MacBook Air 2014, 256GB SSD, 4GB Memory

No cache & same network environment

MacOS

npm@3.10.9: 1m 31s

yarn@0.15.1: 39s

Windows

npm@3.10.9: 2m 24s

yarn@0.15.1: 2m 19s

So, it seems yarn has no advantage over npm on Windows. Did anyone face with this appearance?

Please mention your node.js, yarn and operating system version.
nodejs: 6.8.0
yarn: 0.15.1
OS: Windows 10 14393.321 & MacOS 10.12

[atkawa7]

+1

[Daniel15]

Hi @OshotOkill! Thanks for trying Yarn. Are you using Cygwin or WSL ("Bash on Ubuntu on Windows")? Both are known to have pretty bad disk IO performance.

Also, React Native has a huge number of files so copying them into node_modules is pretty slow, and disk IO for lots of little files on Windows is generally slower than Mac OS (which itself is slower than Linux ext4). We do have a task to experiment with hardlinks (#499) which should improve perf in this scenario.

No cache & same network environment

The main improvement with Yarn is when you have a warm cache (ie. after you've installed a package at least once), but with React Native, the huge number of files will also be a cause of some of the slowness.

[OshotOkill]

@Daniel15 Nope, I'm not using Cygwin/MinGW/MSYS2 or WSL (the latter fails due to a knotty bug).

According to your description I can assume the problem is caused by the file system (NTFS) right ? Even if a warm cache exists, the copying process still runs much slower than MacOS.

Hope the dev teams can come up with a solution ASAP. Thanks.

[fooey]

I'm seeing the same.

Do install, wipe node_modules, install

MacBookPro takes 17 seconds, my Windows machine takes 122 seconds.

[cpojer]

Somebody pointed out this might be related to anti-virus software scanning node_modules and the global yarn cache continuously. Can you try disabling it for those folders?

[OshotOkill]

@cpojer I guess they are right . I don't have any anti-virus software on my machine except the pre-installed Windows Defender, so I banned the scanning of global cache folder & my project folder and did some test:

Default: 128.08s

No scanning of cache folder: 104.43s

No scanning of project folder: 78.28s

No scanning of cache folder & project folder: 53.48s

Though it's slower than Mac for 10+s, it has a significant boost.

This should be informed from the official docs I think.

[Daniel15]

Somebody pointed out this might be related to anti-virus software scanning node_modules and the global yarn cache continuously.

Good catch! I totally forgot about this as I already have c:\src whitelisted on my computers.

@OshotOkill - Would you like to submit a pull request adding a note about antivirus apps to the website, in the Windows installation instructions? Here's the file you'd need to edit: https://github.com/yarnpkg/website/blob/master/en/docs/_installations/windows.md (you can edit it directly on Github). It'd be appreciated 😄

[fooey]

I wasn't as meticulous as @OshotOkill, but I added exceptions for my source and my node install folder, and then specifically exempted the yarn, npm and node binaries and now my fresh install time on Windows is down to 50 seconds from 122 seconds.

[OshotOkill]

@Daniel15 PR is ready. Apologize for my poor English.

[OshotOkill]

PR has been merged. Close this issue.

[voliva]

This is still painfully slow on windows, even deactivating anti-virus and Windows defender. I don't think it's just an environment issue (like this anti-virus solution) but it looks like yarn tries to copy all the files, 1-by-1 even if you install some unrelated dependency.

Why not just delete/copy the files that need to change? If I had webpack installed and is not modified when I installed rimraf, It shouldn't have to be copied again from the cache to the local node_modules folder.

[amcsi]

I have created a StackOverflow article about this too: http://stackoverflow.com/questions/40566222/yarn-5x-slower-on-windows

By the way in my (dual-booted) Ubuntu benchmarks I was using the same NTFS drive as the one Windows normally runs on; and it's still fast there.

[bestander]

Adding node.exe to Windows Defender exclusions gave me a huge performance boost http://126kr.com/article/1884rsed7l

[amcsi]

I'll definitely try this out!

[amcsi]

It did seem to improve the speed a bit 212 -> 170 seconds
So it seems to help, but it could still be improved, because it's still more than 3x slower than in Linux

[bestander]

Another issue I have noticed - Indexing service on Windows tries to index every file in node_modules.
I don't really need it at all, so I disabled it http://www.softwareok.com/?seite=faq-Windows-10&faq=53 and gained another performance boost.

[amcsi]

My windows isn't set to index the path in question, so that still doesn't solve the issue.

[Altiano]

So to sum up there are 4 ways to improve performance:

• Whitelist project folder from AV
• Whiteilst the Yarn cache directory ((%LocalAppData%\Yarn)) from AV
• Adding node.exe to Windows Defender exclusions
• Disabling Indexing service on Windows on node_modules folder

[amcsi]

@Altiano yes, but it's still not enough to get performance even close to Mac/Linux

[michaellashinsky]

Seems kinda sketchy that you'd have to disable AV or indexing on directories to make yarn as fast or faster than npm. After all, you don't have to do this for npm. I decided to give yarn a shot because it stated it was fast and the offline installs made coding without a network connection a plausible thing. Is there no way to optimize the linking?

[OshotOkill]

According to some issues that relate here and the comments above, I'd like to reopen this issue in order to gather some other solutions.

Personally I suggest to list the hardware configurations about your test machine and upload some related pics. There could be many other irrelevant elements that make a big difference between platforms rather than Yarn itself, i.e. the benchmark performance of the SSD on a MacBook is usually much better than a Windows machine.

[bestander]

I think you have badly hoisted node_modules, this bug is going to be fixed in #2676

[ghost]

With @bestander's introduction of hardlinking in #2620 (Which works fine in Windows 7 without administrator privileges), my overall installation times, and node_modules/ size, dropped.

Without hardlinking:

Done in 167.76s.

real    2m49.633s
user    0m0.229s
sys     0m1.368s

du -sh node_modules/
216M    node_modules/

With hardlinking:

Done in 58.07s.

real    0m59.967s
user    0m0.183s
sys     0m1.369s

du -sh node_modules/
189M    node_modules/

[bestander]

Wait for 0.21.1, it will have @kittens' fix to hoisting. Should be even faster

…

On Wed, 15 Feb 2017 at 20:04, Hutson Betts ***@***.***> wrote: With @bestander <https://github.com/bestander>'s introduction of hardlinking in #2620 <#2620> (Which works fine in Windows 7 without administrator privileges), my overall installation times, and node_modules/ size, dropped. Without hardlinking: Done in 167.76s. real 2m49.633s user 0m0.229s sys 0m1.368s du -sh node_modules/ 216M node_modules/ With hardlinking: Done in 58.07s. real 0m59.967s user 0m0.183s sys 0m1.369s du -sh node_modules/ 189M node_modules/ — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#990 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACBdWAEghXfPo4bX9mN0hV8l8YaH2rmlks5rc1pigaJpZM4KVwpA> .

[kuncevic]

I am on Win7 /w Yarn v0.21.3

[3/4] Linking dependencies...
Done in 947.71s. 

Had to wait this amount of time adding any new package with yarn add ...
Defender off
Indexing disabled

Have some other AV running so just following these steps as above mentioned by @Altiano

Whitelist project folder from AV
Whiteilst the Yarn cache directory ((%LocalAppData%\Yarn)) from AV

Will update on this one

[bestander]

@kuncevic, what would be a clean install time for the project?
What is the size of node_modules folder in files?
How does it compare to npm install?

[kumarharsh]

@bestander this is the same problem with me. Any yarn add or yarn remove takes equal time - about , even after @kittens' hoisting fixes.

Everytime this happens:

1. First, the Fetching packages runs (takes about 30s):
   
2. Then, linking dependencies pauses for about 1 or 2 minutes.
   
3. Then, it resumes with next step, and installs(?) 63k files again.
   

As I said, this happens every single time I run yarn add or yarn remove. It does not matter if the dependency I'm installing depends on any other dependency, A simple npm install for installing a new dependency or upgrading an existing one takes a fraction of this time. Things did improve by 2x with @kittens' hoisting fixes, but still the time taken is too much.

@bestander if you want a reproducible case, please clone this repo: https://github.com/kumarharsh/yarn-bug, and run yarn install, and then yarn add react-helmet.

[bestander]

Yarn is preserving determinism every time it runs add/remove, so it needs to check if any dependency got hoisted to the root of node_modules when dependencies change.
That is why it runs full linking phase.

Fetching dependencies - download, you can't optimise it.
First linking phase (1561 operations) - it creates all the folders for all the dependencies.
Second linking phase (63K operations) - it copies the files from cache to node_modues.

Yarn optimises file copy operations by checking if the files are the same before doing the copy.
We might want to profile this area better and see if we can decrease number of unnecessary IO.
Maybe on Windows copying would be faster then checking?

What about npm, how fast it does clean install?

[kumarharsh]

A clean install for npm (npm install) takes 552301.1944ms.
Installing an additional dependency (npm install weird) takes 57023.7593ms. (Most of this time is wasted in paperjs trying to install canvas as a dep - but this time would be common for both npm & yarn)

A clean install for yarn (yarn install) takes 612698.4915ms.
Installing an additional dependency (yarn add weird) takes 495633.0307ms.

npm version 3.10.9
yarn version 0.21.3

[vbfox]

@bestander @kumarharsh Yarn doesn't optimises file copy operations on windows due to a libuv/nodejs bug (See #2958 for a potential fix in yarn code) that isn't present on node 7.1+ so you can get your second command (yarn add) to be a lot faster just by upgrading node.

Using windows file copy operation is a little bit faster than using node API to copy files too (See #2960 for a potential PR) and would optimize yarn install a little bit but I don't know if it would egalize with npm (didn't test)

[kuncevic]

Just updated to 7.8.0

nvm install 7.8.0
npm install npm -g (came with 4.4.4)
nvm use 7.8.0
`git clone https://github.com/angular/material2`
cd material2
yarn install - Done in 210.22s.
rimraf node_modules
yarn install - Done in 180.66s.
rimraf node_modules
yarn install - Done in 181.11s.

However by doing yarn add rimraf got it done in 20.52s. but why yarn install after removing node_modules taking so long?

p.s.

rimraf node_modules
npm install - Done in 332.4s
rimraf node_modules
npm install - Done in 402s
rimraf node_modules
npm install - Done in 489.6s

[vbfox]

@kuncevic Nice to see that upgrading node works for yarn add :)

Regarding empty node_modules a good thing to do is to measure how much is due to yarn and how much is due to FileSystem, Hard drive & Anti virus.
What I did to test that was to copy the full node_module (As generated by yarn, not npm) of material2 somewhere in yarn cache :

for /f "delims=" %i in ('yarn cache dir') do set yarncachedir=%i
xcopy /E /Y /I /Q node_modules %yarncachedir%\x-temp

And then for each test I cleaned node_modules & ran either yarn install, npm install or an xcopy from the previously created folder :

rd node_modules /s /q
powershell -Command "Measure-Command { xcopy /E /Y /I /Q %yarncachedir%\x-temp node_modules}"

And took the total seconds.

Results

Here are the results on 3 PCs

• 🏠 Home PC: Samsung 950 Pro NVMe, ESET Nod32
• 🏢 Work PC: Samsung 850 EVO SATA, TrendMicro OfficeScan that I can't disable
• 🍎 MacBook pro: 2015 version, on macos, no anti virus

	yarn 🏠	npm 🏠	xcopy 🏠	yarn 🏢	npm 🏢	xcopy 🏢	yarn 🍎	npm 🍎
AV Disabled	34s	90s	23s	-	-	-	32s	92s
AV Exclude cache & code	38s	104s	29s	-	-	-	-	-
Av Exclude cache only	43s	-	31s	-	-	-	-	-
Av full	48s	122s	32s	100s	274s	236s	-	-

Each time AV was enabled it was toping the CPU chart during yarn install or xcopy (On my home PC 30% cpu total was taken at the max but on my work PC it fill one core for xcopy & all my cores for yarn)
xcopy is slower on my work PC than yarn, I suspect because it doesn't copy files in parallel while yarn does (That shouldn't matter for IO bound operations but AV are making it a CPU bound operation & xcopy wasn't written to fight so much stupidity 😄 )

In conclusion

• yarn is faster than npm and can even be faster than xcopy when AV make file copy CPU bound
• Windows on a good SSD isn't really slower than a MacBook Pro 2015 (That already has a good SSD) even if it's hard to compare as not exactly the same packages install, & not all post-install scripts do the same thing
• Some changes could be done in yarn to sidestep that (symlinking files ?) but essentially coping lot of small files is slow
• On windows AV can make it slower, mine add 30% when enabled in both source & dest folders 😞
• Corporate AVs can be a magnitude slower than home AVs & kill performance enough for any copy operation to be painful (when it make the naturally IO bound operation CPU bound) 😡

[silverqx]

Adding npm, yarn cache folders and node.exe to defender's exclusion list would be enough, of course all this can't be in indexed folders. Now yarn add / rm takes 7 secs

[bestander]

Thanks everyone, a significant optimization for Windows landed in 0.24 #3234 (comment)

[cowwoc]

@vbfox Can you please add version numbers for npm and yarn in your benchmark?

[nebulr]

this is still a piece of shit for MacOSX

[thecjharries]

I'm still experiencing some crazy install times. yarn add seems to install and link everything (all items in package.json, ~30k dependencies) all the time.

Linux versions:

$ yarn -v
1.3.2
$ node -v
v8.9.3

Windows versions:

> conemu-cyg-64.exe --version
ConEmu cygwin/msys connector version 1.2.2
> wslbridge.exe --version
wslbridge 0.2.3
> Get-ItemProperty 'HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion' | Select-Object ProductName, CurrentMajorVersionNumber, CurrentMinorVersionNumber, ReleaseId, CurrentBuild, CurrentBuildNumber, BuildLabEx


ProductName               : Windows 10 Pro Insider Preview
CurrentMajorVersionNumber : 10
CurrentMinorVersionNumber : 0
ReleaseId                 : 1709
CurrentBuild              : 17025
CurrentBuildNumber        : 17025
BuildLabEx                : 17025.1000.amd64fre.rs_prerelease.171020-1626

I've got two (and a half) questions:

1. What's the accepted solution to the issue in this thread? Was it use close instead of finish event for file copy completion #3234 or tweaking Windows Defender?

   • If the solution was tweaking Windows Defender, is there a complete writeup of what to do somewhere?

2. Is my issue actually related to this thread, or should I create a new one?

[bestander]

Thanks for raising a new issue, I'll respond there

[mkamranhamid]

It's been almost an hour now and I'm waiting for this command to finish the process. I've followed the above points mentioned by @Altiano but nothing works

[mkamranhamid]

do we have any alternative for this? like can i use npm i -g . will it act in the same way or I'll have to make some changes because this code uses yarn workspace

[mkamranhamid]

So finally after struggling for 2-3 hours, I had to use npm i -g . instead of yarn global add file:.. npm worked like a charm

[robbykrlos]

Not sure how this is helping, but it is. Initially one change triggered a compilation that took ~1 minute. After running the steps below, it took 3-10 seconds. Not sure if it's still related to Windows Defender, or how fast the file access is improved after the complete delete and re-install (maybe it's like a defragmentation that helps on reading speeds?)

Install RimRaf:
npm install rimraf -g
And in the project folder delete the node_modules folder with:
rimraf node_modules
Install again all node_modules:

yarn install
yarn serve

Build and Serve are now working normally 👍