Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Taskcluster cache for local Mercurial clones #2193

Open
marco-c opened this issue May 29, 2024 · 5 comments
Open

Add Taskcluster cache for local Mercurial clones #2193

marco-c opened this issue May 29, 2024 · 5 comments
Assignees

Comments

@marco-c
Copy link
Collaborator

marco-c commented May 29, 2024

In order to actually use the work done in #1517, we need to enable local cloning.
In order to enable local cloning, we need to make sure we don't clone from scratch in each task, but we rely on a Taskcluster cache.

@La0
Copy link
Collaborator

La0 commented Jun 25, 2024

I'm looking into this and have a few questions:

  • there already exists firefox source code checkouts in Taskcluster cache, but they are restricted to tasks ran by run-task: would you consider moving the code-review bot execution to run-task ?
  • if using custom caches for the code review bot, we would need one cache per repository.
    • when the cache is empty, some code-review bot tasks would need to populate it (initial clone), and will take longer: is that ok ?

@La0
Copy link
Collaborator

La0 commented Jun 27, 2024

I noticed that we already have cache enabled for the Integration hook.

Unfortunately the hook on Community TC does not successfully run anymore due to a missing secret.

We could restore that functionality, and test the clone & re-use from cache functionalities from that hook before updating runtime tasks

WDYT @marco-c ?

@La0
Copy link
Collaborator

La0 commented Jul 1, 2024

I spent a bit of time today on this topic and noticed a few things:

  • the integration test is broken because of invalid Taskcluster proxy url and automated conversion from docker-worker to generic-worker. As I do not know what's the long term plan regarding this hook & its viability, I stopped looking into it.
  • I made a first PR to control directly the mercurial cache behaviour from the CLI args, removing Taskcluster "production" check, and simplifying the overall code. In the end, the taskcluster cache is usable because I removed the tempdir path creation. As robust_checkout is already used, the cache will be automatically purged & setup at the correct version on each run
  • The mercurial version tools extension is not at the latest version (but just lack 1 whitespace changeset)
  • The mercurial version is outdated (6.4.5 in code vs 6.7.4 on pypi)
  • I have issues running locally related to sentry-sdk (probably an invalid version locally that has changed some modules)

Next steps:

@marco-c
Copy link
Collaborator Author

marco-c commented Jul 8, 2024

I'm looking into this and have a few questions:

* there already exists [firefox source code checkouts](https://firefox-source-docs.mozilla.org/taskcluster/caches.html#version-control-caches) in Taskcluster cache, but they are restricted to tasks ran by run-task: would you consider moving the code-review bot execution to run-task ?

Yes, but I'm not sure what would be needed to do that.
Long term, we might even consider moving the review bot into mozilla-central itself, which would reduce the maintenance a lot.

* if using custom caches for the code review bot, we would need one cache per repository.
  
  * when the cache is empty, some code-review bot tasks would need to populate it (initial clone), and will take longer: is that ok ?

Yeah I guess there's not way around that.

@marco-c
Copy link
Collaborator Author

marco-c commented Jul 8, 2024

Regarding the integration hook, I think it has been broken for a while. It would be useful to have it working again, though it's not a huge priority.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants