pytorch channel very slow in many countries - should it be on the CDN?

[rgommers]

I'm opening this discussion here on request of @jezdez, after discussion on Twitter about the possible cause of large download speed variations depending on location: https://twitter.com/jezdez/status/1546442755538296832. In that thread, @jph00 says that downloading the pytorch package from the pytorch channel is extremely slow compared to in the US:

"If you're in Asia or Australasia you've probably noticed downloading PyTorch from conda is really slow. Here in Aus I get 32kB/s. :( Here's the trick: connect to a US VPN. When I do that it jumps to 6MB/s! I've seen this for >12 months for many places & PyTorch versions."
(clarification: he means conda install pytorch -c pytorch, not from defaults or conda-forge).

There are a large number of reports on pytorch/pytorch#17023 that confirm this.

On the same thread, @kkraus14 suggests this may be because the pytorch channel is not on the conda CDN: "This almost sounds like the pytorch channel isn't on the CDN that other channels like conda-forge and rapidsai are on. I vaguely remember some download issues with the rapidsai channel before it got added to the CDN"

There is very little info on what is and isn't on the CDN, and how this gets determined. This blog post is the only thing I could find: https://www.anaconda.com/blog/how-we-made-conda-faster-4-7. Relevant part:

"We’re working on rolling out the new file format to all of the channels that we host on our CDN. This takes time because the actual conversion of a .tar.bz2 that is uploaded to the channel may take quite a while to convert to the new format. MKL, for example, can take 30 minutes per package that we need to convert. We use a very high compression setting for zstd, 22, and that takes a while to do its thing. Unfortunately, we won’t be able to offer the new file format to arbitrary anaconda.org channels for the time being, as the application behind anaconda.org does not support the new format. CDN-hosted channels are special, because they mirror content from anaconda.org, but are then free to do more things, like patching repodata or offering the new format."

If there is more info on this somewhere, I'd love to see it. And absent that, should this be added somewhere? If the answer is "that requires talking to Anaconda and it requires $$ changing hands", that's a perfectly valid answer - it'd still be great to have that stated somewhere.

Also, is there any effort to measure, or data about, download speeds in various regions? I'm traveling around Europe and have noticed very large differences - for example in the Netherlands it's always very fast, Spain was pretty fast too, while in Croatia I'm getting <100kb/s regularly also for conda-forge and defaults.

[jakirkham]

Would recommend that, if this is pursued, it should be done with coordination between the Anaconda & PyTorch teams. Recall there were a few issues we went through when pursuing this for RAPIDS where we had to fix some things. Real time response in particular is quite important when addressing and iterating on issues. It may be worth just blocking out some time and iterating on a live call.