File Cache is valid if checkout or contents hasn't changed

[Eh2406]

#10482 allow the registry to have cashe keys at a per file level.
On master if the currently checked out commit changes all cached files are regenerated.
After this commit we also check if GITs hash of the particular file has changed.

To avoid thrashing cached files this PR only checks for the presence of both hashes, but does not write them both. This means that nightly after this branch will still be able to share file caches with older versions, specifically current stable.

When sufficient versions of stable know how to read the new format, a one line change can be made to start writing the new format.

[rust-highfive]

r? @ehuss

(rust-highfive has picked a reviewer for you, use r? to override)

[Eh2406]

cc @arlosi. Thanks for the set up work.

[weihanglo]

Hi. I've read through #10064 and #10482 but cannot find the discussion about the new index format <file>:<remote>. I might miss something 😞 Can I have a link to where it was discussed in the PR description?

[Eh2406]

Hi. I've read through #10064 and #10482 but cannot find the discussion about the new index format <file>:<remote>. I might miss something 😞 Can I have a link to where it was discussed in the PR description?

Did not miss any discussion. This was an independent thought I had when I looked at this code for the first time after reviewing #10482. I'm not sure how much context to explain. If I end up with two little, I'm sorry, I can add more.

Before #10482 the entire git checkout had one version number. We used for that version number the hash of the commit at HEAD. Once we read the data out of git we saved it too disc in a slightly optimized format. To make sure we didn't read an optimized file that was now out of date, the file started with the version number. If we fetched a new HEAD commit then we would overwrite all cache files.

The big change from #10482 is to think of each file as having its own version number. This will be critical for HTTP based indexes, but was not intended to make a change for git based indexes. To maintain the same behavior, no matter what file is being looked at it's version number is "hash of the commit at HEAD". Thus it has basically the same performance, and maintains the same state on disk.

The idea of this PR is that the file will be considered up to date if either it ends with "hash of the commit at HEAD" or starts with a "hash of the files contents". When enough time has passed we can start writing files with both hashes, and get cash hits more often. But we will continue to use files on disk that were created by older Cargos (like stable) that only have the commit hash.

[weihanglo]

Wow, thanks! What I guessed is very close to your detailed explanation 😆

There are some places I believe we can improve:

• We should pass index_version, which is per-file level cache key, into load_helper.
  

  cargo/src/cargo/sources/registry/remote.rs

  Line 200 in b1603eb

  match load_helper(&self, path, current_version.as_deref()) {

• Since the RemoteRegistry::load is an ad-hoc implementation, could we have a doc comment to explain the new format of index_version? That would help other contributors a lot. (I know we are not writing any at this time, so feel free to skip this suggestion)

[Eh2406]

Both excellent points! The code has been corrected, and a comment has been added.

[weihanglo]

That looks really nice! Thank you!

[weihanglo]

@bors r+

[bors]

📌 Commit 8058c3d has been approved by weihanglo

[bors]

⌛ Testing commit 8058c3d with merge 1a052ae...

[bors]

☀️ Test successful - checks-actions
Approved by: weihanglo
Pushing 1a052ae to master...