Packages take a long time to become visible to conda search after upload

[duncanmmacleod]

I regularly see that packages that get uploaded to https://anaconda.org/conda-forge take ~1 hour or more (as timed by the Uploaded timestamp on the /files page for that package) to become visible with conda search. Is there an way to more regularly update the package list (repodata?) for a channel to have packages become usable with a lower latency?

My current example is osx-64/python-lalburst-1.5.9-py310h93fb10b_2.tar.bz2 which was Uploaded '1 hour and 6 minutes ago' by cf-staging, but isn't available from conda search

$ date && conda search python-lalburst=1.5.9=py310h93fb10b_2
Fri Dec 10 07:19:40 PST 2021
Loading channels: done
No match found for: python-lalburst==1.5.9=py310h93fb10b_2. Search: *python-lalburst*==1.5.9=py310h93fb10b_2

PackagesNotFoundError: The following packages are not available from current channels:

  - python-lalburst==1.5.9=py310h93fb10b_2

Current channels:

  - https://conda.anaconda.org/conda-forge/osx-64
  - https://conda.anaconda.org/conda-forge/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.

Is there anything that I ('the user') can do to speed things up?

[duncanmmacleod]

Update: this package became available around about 0729 PST (December 10), so approximately 75 minutes after being uploaded.

[jakirkham]

Packages that are uploaded to conda-forge are mirrored to a CDN. As it takes some time to do the mirroring, they don't show up immediately.

That said, it is typically closer to ~30mins. 75mins does sound a bit long. @chenghlee do you have any thoughts here?

[chenghlee]

Digging through the logs on the sync worker, since 2021-09, each CDN sync run now takes ~47 minutes; about 60% of that time (~28 min) is spent running conda index.

These numbers seem to be mostly independent of the number of packages being synced, so there's definitely something a little to a lot weird going on. I've opened a ticket with Anaconda's Infrastructure team to take a look at what's going on, though given the time of year, we may not have a real answer until Q1 of next year.

[duncanmmacleod]

As far as I can recall it's been pretty slow for a while, so waiting O(weeks) for an update is fine, happy to hear there may be some potential for improvement.

[isuruf]

@chenghlee, any updates on this?