Archive uploads instead of uploading individual files

[jry-anchor]

I have a 900MB node_modules directory with 86k files. Using actions/upload-artifact as-is takes about an hour and a half -- the logs appear to indicate that files are uploaded individually.

If I tar/gzip the directory first, the tar takes about 30 seconds, reducing it to 175MB, then the upload takes about 25 seconds.

Example of how actions/cache does it: https://github.com/actions/cache/blob/master/src/save.ts

[g105b]

You should not use actions/upload-artifact for this, instead you should use actions/cache - see https://github.com/actions/cache

Edit Dec 2020: I was wrong. actions/cache can cache the process of running npm install, but doesn't make sense to share the node_modules directory between steps.

[jry-anchor]

@g105b in that case the Passing data between jobs in a workflow doc needs an update, since that page doesn't mention limitations or actions/cache at all. Also Comparing artifacts and dependency caching. How many files and/or which size of directory warrants using caching instead of archiving?

Also calling it a cache for some "artifacts" but not others is a bit of a misnomer, since I need the exact contents of the directory in all the jobs in the workflow.

[g105b]

I understand and share your frustration. I hope this is made a bit simpler soon.

[Clemens-E]

Hi, I'm trying to have a build process spit out a fully working artifact that includes everything it needs to run (node_modules, dist, etc).
But I'm facing the same problem: uploading 16k files takes longer than actually building the whole application.

If I understand this correct, actions/cache would only help if you need to share the node_modules between workflow runs?

Is there any workaround to force actions/upload-artifact to somehow tar the files instead of uploading them all individually?
I also tried using actions/create-zip-file, which speeds everything up, but when downloading the artifact there now is a zip inside a zip, and I feel like that's unnecessary.
I'm guessing #69 would fix this problem, but it doesn't seem like that will be a thing soon

[MCOfficer]

You can get a massive boost by taring the folder before uploading it (Zip also works, but then you're wasting time on compression). The resulting file being zipped again is unfortunate, but the speedup is worth it.
Edit: see also #39

[g105b]

Any movement on this?

[Th3S4mur41]

Facing the same issue... The current solution to the problem is to use upload-tartifact and download-tartifact which do exactly that.

It would be nicer to have an option in the original action to tar before upload / untar after download instead

[olalonde]

Bump. This should probably be done by default.

[raimille1]

Bump!

[g105b]

Any movement on this?