Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix the nondeterminism sim to not fail when the validator set is empty. #2124

Open
4 tasks
SpicyLemon opened this issue Aug 14, 2024 · 0 comments
Open
4 tasks
Labels
simulation Issues related to simulation testing
Milestone

Comments

@SpicyLemon
Copy link
Contributor

Summary

Update the nondeterminism sim test (make test-sim-nondeterminism) so that, if it gets panic: validator set is empty after InitGenesis, ..., it can still pass.

Problem Definition

The nondeterminism sim picks a few random seeds each time it runs. That is different from the other sims that use a pre-defined list of seeds to use. But the result is intermittent failures when the randomness aligns a certain way.

One way it shows up is by getting a panic like this:

--- FAIL: TestAppStateDeterminism (105.78s)
panic: validator set is empty after InitGenesis, please ensure at least one validator is initialized with a delegation greater than or equal to the DefaultPowerReduction ({824644944992}) [recovered]
	panic: validator set is empty after InitGenesis, please ensure at least one validator is initialized with a delegation greater than or equal to the DefaultPowerReduction ({824644944992})

goroutine 9 [running]:
testing.tRunner.func1.2({0x3034cc0, 0xc002370fb0})
	/opt/hostedtoolcache/go/1.21.13/x64/src/testing/testing.go:1545 +0x238
testing.tRunner.func1()
	/opt/hostedtoolcache/go/1.21.13/x64/src/testing/testing.go:1548 +0x397
panic({0x3034cc0?, 0xc002370fb0?})
	/opt/hostedtoolcache/go/1.21.13/x64/src/runtime/panic.go:914 +0x21f
github.com/cosmos/cosmos-sdk/x/simulation.initChain(0xc001348f30?, {0x3fd09d6113f22504, _, _, {_, _, _}, {_, _}, {_, ...}}, ...)
	/home/runner/go/pkg/mod/github.com/provenance-io/cosmos-sdk@v0.50.7-pio-1/x/simulation/simulate.go:48 +0x39f
github.com/cosmos/cosmos-sdk/x/simulation.SimulateFromSeedProv({_, _}, {_, _}, _, _, _, {0xc00bda8000, 0x5a, 0x80}, ...)
	/home/runner/go/pkg/mod/github.com/provenance-io/cosmos-sdk@v0.50.7-pio-1/x/simulation/simulate.go:99 +0x4ef
github.com/cosmos/cosmos-sdk/x/simulation.SimulateFromSeed(...)
	/home/runner/go/pkg/mod/github.com/provenance-io/cosmos-sdk@v0.50.7-pio-1/x/simulation/simulate.go:68
github.com/provenance-io/provenance/app.TestAppStateDeterminism(0xc000c7fa00)
	/home/runner/work/provenance/provenance/app/sim_test.go:591 +0xc0d
testing.tRunner(0xc000c7fa00, 0x5130fb8)
	/opt/hostedtoolcache/go/1.21.13/x64/src/testing/testing.go:1595 +0xff
created by testing.(*T).Run in goroutine 1
	/opt/hostedtoolcache/go/1.21.13/x64/src/testing/testing.go:1648 +0x3ad
FAIL	github.com/provenance-io/provenance/app	105.858s
FAIL

That was the result from seed 7069437441164333710.

I thought there were things already in place to handle that specific panic, but either they aren't being applied to the nondeterminism test or else it's not being applied correctly.

Proposal

Update the nondeterminism sim test to handle that specific panic. Hopefully, we can make it just eat the panic and move on to the rest of the test process allowing it to run the needed comparisons and pass/fail based on that. If eating that panic will always cause another failure, though, we're probably fine leaving things as is. But we'll have to dig into it to know for sure.


For Admin Use

  • Not duplicate issue
  • Appropriate labels applied
  • Appropriate contributors tagged
  • Contributor assigned/self-assigned
@SpicyLemon SpicyLemon added the simulation Issues related to simulation testing label Aug 14, 2024
@SpicyLemon SpicyLemon added this to the backlog milestone Aug 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
simulation Issues related to simulation testing
Projects
Development

No branches or pull requests

1 participant