Infrastructure update for Channel Cloning/CDN sync #1022
Comments
|
@conda/conda-maintainers FYI! |
|
Hey everyone, I'd very much like your input on this, since we don't want to run this migration at a random point in time. The current time window we have closes next Wednesday and then reopens after October 21st. Ideally we can run the migration before next Wednesday. |
|
I think this is probably okay, the conda-canary channel is used among other things, for releases based on merges into the main branch of a number of repos under the conda organization, and every PR can also build "review" packages of conda on request (by maintainers). That way we can iterate quickly and test the work on branches and the mainline easily end-to-end. My main question is whether this change would lead to a slower than currently process to have the packages available for installation once they are uploaded? @kenodegard Do you have any concerns related to the review or canary builds? |
|
I don't have any concerns at this time (after all several other much larger channels have already been successfully migrated). It would be great if we could do the migration before the end of the week as next week through the end of the month gets into our release window. |
|
@jezdez The process won’t change - if at all it’s easier to increase the frequency after the migration. I can easily run both migrations tomorrow if that’s fine with you, and update about the progress here. |
|
Sounds good to me then. |
|
Both channels have been sucessfully migrated. For conda-test, all work is now done. For conda-canary, we were so far unable to enable the clean-up feature of the clone, which deletes unneeded files from the underlying S3 bucket, due to some packages being recognized as unneeded. I will analyze these and post a list here to discuss with all of you before we continue, to ensure that these changes/deletions are correct. |
|
Thank you, @morremeyer ! |
|
I checked on the affected package archives and added a list to this comment. We are currently investigating an issue with the anaconda.org backend in relation to this, so I'm not going to investigate this further at this point and circle back to it when that issue is resolved. In the meantime, we will leave the clone running with the cleanup feature disabled so that these archives do not get deleted. Affected files |
Hey everyone,
we (Anaconda) have been working on upgrades for the channel cloning/CDN sync infrastructure of anaconda.org, which is is used for two channels for conda, too:
What's changing?
last-updated.jsonfile at the channel root, which contains the last update time both as UNIX and ISO8601 timestamps. We are using this directly to monitor the updates end-to-end. Check it out for the anaconda channel if you're interested!We have already made this change for multiple channels: anaconda, r, bioconda, and conda-forge have already been migrated successfully, proving that both the maintenance process and the new infrastructure work as desired.
There will not be any downtime during the migration.
Next steps
I have prepared this change on the new infrastructure already, it is now running in a dry-run mode to keep its caches up to date.
The actual migration can be performed any time now, preferably (since the people running it are in the CEST time zone) between 08:00 and 14:00 UTC.
Here's three proposals for a time frame that would work well for us:
The switch to the new infrastructure will take about 30 minutes, with some follow up work afterwards.
We will keep you up to date about progress here.
Looking forward to any questions you might have and when you think we should make the change!
The text was updated successfully, but these errors were encountered: