[Bug]: server v2 - memIterator/ InitGenesis

[alpe]

Is there an existing issue for this?

• I have searched the existing issues

What happened?

tlDr:

• fix memiterator by copying the btree
• fix InitGenesis workflow

Background:

I have some issues with the memIterator.Next. In some rare cases this panic happens on sims run:

https://github.com/cosmos/cosmos-sdk/blob/main/server/v2/stf/branch/changeset.go#L190        	
the tests have moved the prev iterator to a position out of the data slice. Any thoughts?
	/opt/homebrew/opt/go/libexec/src/runtime/panic.go:770 +0x124
[github.com/tidwall/btree.(*IterG[...]).Prev(0x106253f60)](http://github.com/tidwall/btree.(*IterG[...]).Prev(0x106253f60))
	/Users/alex/go/pkg/mod/github.com/tidwall/btree@v1.7.0/btreeg.go:1353 +0x204
[cosmossdk.io/server/v2/stf/branch.(*memIterator).Next(0x14000ee4750)](http://cosmossdk.io/server/v2/stf/branch.(*memIterator).Next(0x14000ee4750))
	/Users/alex/workspace/cosmos/cosmos-sdk/server/v2/stf/branch/changeset.go:190 +0x74
[cosmossdk.io/server/v2/stf/branch.(*mergedIterator).Next(0x14009760c30)](http://cosmossdk.io/server/v2/stf/branch.(*mergedIterator).Next(0x14009760c30))
	/Users/alex/workspace/cosmos/cosmos-sdk/server/v2/stf/branch/mergeiter.go:69 +0xf8
[cosmossdk.io/x/staking/keeper.Keeper.ApplyAndReturnValidatorSetUpdates({{{0x10fa530c8](http://cosmossdk.io/x/staking/keeper.Keeper.ApplyAndReturnValidatorSetUpdates(%7B%7B%7B0x10fa530c8), 0x140010e36c0}, {0x1061dfc00, 0x107f19a40}, {0x1061cdb80, 0x107f19a40}, {0x1061eee80, 0x107f19a40}, {0x1061cdb20, 0x107f19a40}, ...}, ...}, ...)
	/Users/alex/workspace/cosmos/cosmos-sdk/x/staking/keeper/val_state_change.go:166 +0x408
[cosmossdk.io/x/staking/keeper.Keeper.BlockValidatorUpdates({{{0x10fa530c8](http://cosmossdk.io/x/staking/keeper.Keeper.BlockValidatorUpdates(%7B%7B%7B0x10fa530c8), 0x140010e36c0}, {0x1061dfc00, 0x107f19a40}, {0x1061cdb80, 0x107f19a40}, {0x1061eee80, 0x107f19a40}, {0x1061cdb20, 0x107f19a40}, ...}, ...}, ...)
	/Users/alex/workspace/cosmos/cosmos-sdk/x/staking/keeper/val_state_change.go:36 +0x54

The issue can be replicated with this test:

func TestBreakBtree(t *testing.T) {
	myPrefix, otherPrefix := byte(0x01), byte(0x02)
	parent := newChangeSet()
	for i := byte(0); i < 63; i++ { // set to 63 elements to have a node split on the next insert
		parent.set([]byte{myPrefix, i}, []byte{i})
	}
	it, _ := parent.reverseIterator([]byte{myPrefix, 32}, []byte{myPrefix, 50}) // ValidatorsPowerStoreIterator
	if !it.Valid() {
		t.Fatal("expected valid iterator")
	}
	it.Close() <--- adding this does not cause panic to happen
	for i := 0; it.Valid() && i < 2; it.Next() {
		i++
		parent.set([]byte{otherPrefix, byte(i)}, []byte("any value")) // SetLastValidatorPower
	}
}

@fdymylja proposed to copy the tree in the iterator which fixes the test

func newMemIterator(start, end []byte, tree *btree.BTreeG[item], ascending bool) *memIterator {
	iter := tree.Copy().Iter()

From the btree code doc:

// Copy the tree. This is a copy-on-write operation and is very fast because
// it only performs a shadowed copy.

This has the interesting effect that the validator set returned on AppManager.InitGenesis is empty now. When I did some debugging, I saw the ApplyAndReturnValidatorSetUpdates is called twice. On init genesis and on end block. In the second call, the val power has not changed so an empty list is returned.

Cosmos SDK Version

main

How to reproduce?

I can provide a sims2 branch for server v2