Optimize IOPS for opening an index

[wjones127]

When we open a vector index, we start by getting the tail of the file to load the version of the index.

lance/rust/lance/src/index.rs

Lines 619 to 622 in c0e1f15

	let reader: Arc<dyn Reader> = self.object_store.open(&index_file).await?.into();
	let tailing_bytes = read_last_block(reader.as_ref()).await?;
	let (major_version, minor_version) = read_version(&tailing_bytes)?;

This requires two IOPS:

1. Getting the size of the index file
2. A range request for the end

We could optimize this by writing the size of the index file in the manifest, so we don't have to make the HEAD/metadata IOP to get the size. We could then pass the trailing_bytes down to the next step, in case it could be re-used. For example, in the V2 index path, we read the Lance file metadata next. It's possible this could be fully contained in those trailing bytes, and we could save another IOP.

Implementing this would reduce cold start latency for search queries.

These kinds of optimizations were already implemented for reading manifests: #2396

See also: #2420