Optimize Redb temporary state cleanup

[eserilev]

Description

#4718 Introduces Redb as another database implementation for the beacon node. According to most metrics we've seen so far, its performance is on-par with leveldb in most cases. One case in which it is drastically underperforming is at temp state cleanup during a node restart. The latest restart on holesky took roughly 1.5 hours!

The issue can be found in do_atomically

lighthouse/beacon_node/store/src/database/redb_impl.rs

Lines 166 to 203 in 06490d4

	pub fn do_atomically(&self, ops_batch: Vec<KeyValueStoreOp>) -> Result<(), Error> {
	let open_db = self.db.read();
	let mut tx = open_db.begin_write()?;
	tx.set_durability(self.write_options().into());
	for op in ops_batch {
	match op {
	KeyValueStoreOp::PutKeyValue(column, key, value) => {
	let _timer = metrics::start_timer(&metrics::DISK_DB_WRITE_TIMES);
	metrics::inc_counter_vec_by(
	&metrics::DISK_DB_WRITE_BYTES,
	&[&column],
	value.len() as u64,
	);
	metrics::inc_counter_vec(&metrics::DISK_DB_WRITE_COUNT, &[&column]);
	let table_definition: TableDefinition<'_, &[u8], &[u8]> =
	TableDefinition::new(&column);
	let mut table = tx.open_table(table_definition)?;
	table.insert(key.as_slice(), value.as_slice())?;
	drop(table);
	}
	KeyValueStoreOp::DeleteKey(column, key) => {
	metrics::inc_counter_vec(&metrics::DISK_DB_DELETE_COUNT, &[&column]);
	let _timer = metrics::start_timer(&metrics::DISK_DB_DELETE_TIMES);
	let table_definition: TableDefinition<'_, &[u8], &[u8]> =
	TableDefinition::new(&column);
	let mut table = tx.open_table(table_definition)?;
	table.remove(key.as_slice())?;
	drop(table);
	}
	}
	}
	tx.commit()?;
	Ok(())
	}

The way this function is structured, we are constantly opening and closing write transactions during each iteration. Since Redb only allows for one open write transaction at a time, and write transactions can only be opened against individual tables, we will need to refactor our garbage collection logic to conform to Redb functionality.

My best idea so far is to make the garbage collection logic atomic on a per table basis. Instead of passing in an ops vec that contains transactions across multiple tables, we create a vec for each table. We can then pass each vec into a new function that keeps a single write transaction open across the full vec of ops. As long as we're ok with garbage collection only being atomic across individual tables, this should help us get to the performance were looking for.

I think Michael also mentioned that tree-states will help us reduce the amount of temp states being stored in general.