Index performance tracking issue

[casey]

Full sync times on ordinals.com from block 0 to 758607:

date	sync time	change	commit
2022-10-14	28 hours 11 minutes	initial benchmark
2022-10-15	27 hours 03 minutes	switched to two-phase commit
2022-10-26	17 hours 30 minutes	added caching updater, faster rarity check
2022-10-28	12 hours 15 minutes	fetch blocks in background
2022-10-31	11 hours 33 minutes	speed up epoch construction
2022-11-21	15 hours 50 minutes	performance regression	3916d4b
2022-11-22	15 hours 23 minutes	track commit times	c25c380
2022-11-23	13 hours 58 minutes	non-zeroing redb	c3351fe

• https://github.com/casey/ord/issues/457
• https://github.com/casey/ord/issues/309
• https://github.com/casey/ord/issues/715
• persist cache across updates
• Speed up rarity check in index #724
• extend_from_slice is slow
• chunks_exact is slow
• build rarity.txt table as final step?

[casey]

Just tried the loop version of the rarity check, and it was 11 hours 55 minutes, so a little faster, but not a ton. I'm trying again with the fully unrolled version.

[casey]

Fully unrolled version was 11 hours 54 minutes, so not a huge improvement, and probably not worth the loss in clarity.

[casey]

This version was slower, 12 hours 50 minutes:

  fn from(ordinal: Ordinal) -> Self {
    for (epoch, starting_ordinal) in Self::STARTING_ORDINALS.into_iter().enumerate().skip(1) {
      if starting_ordinal > ordinal {
        return Epoch(epoch as u64 - 1);
      }
    }
    Epoch(Self::STARTING_ORDINALS.len() as u64 - 1)
  }

[casey]

Moved sync times to the wiki: https://github.com/casey/ord/wiki

Closing, since we don't really need to worry about full index sync time for now.