Skip to content

v6.4.0 - Fast storage bug fixes

Compare
Choose a tag to compare
@p0mvn p0mvn released this 21 Feb 02:56
· 8 commits to v6.x since this release
86d1a03

The main focus of this upgrade is stability improvements with the IAVL patch of v6.3.0 The big points are:

  • Fixes the "version X was already saved to a different hash" and "insufficient funds" bugs that nodes were sometimes seeing after crashes.
  • Reduce the RAM overhead during the initial IAVL migration. Node operators should still have 32GB of RAM for all of their mainnet nodes.
  • Improving the log messages during migrations

The bugs people were seeing for nodes being saved to a different hash, etc. happen after restoring the node post-crash during commit (the crash could be caused by something else). The bug on restart stems from the long-standing issue that SDK commits are actually not-atomic. This manifested in a failure to flush the latest height when an error during commit occurs, causing the state invalidities on restart. We mitigated the non-atomicity problem by flushing the metadata about the stores that successfully committed a new height. As a result, on a restart, the SDK now knows which stores/modules are already on the newly committed height.

There is still a known issue related to a misconfiguration between pruning and snapshot settings. During state sync, it is now possible to apply the blocks fast enough to begin pruning a height that is currently being snapshotted.

We are working on a solution but for now, we suggest setting your app.toml to the following:
pruning-keep-recent=10000
pruning-keep-every=0
pruning-interval=<random small prime under 100>
snapshot-interval=1500
snapshot-keep-recent=2

It is also possible to set pruning-keep-every == snapshot-interval. This ensures that a snapshotted height is never pruned. However, there are other problems known to occur when pruning-keep-every is set. Therefore, we recommend increasing pruning-keep-recent to 10000 for now instead.

What's Changed

  • Move app.setupUpgradeHandlers() after module manager and configurator (backport #884) by @mergify in #885
  • Upgrade IAVL and SDK with RAM improvements and bug fixes for v6.4 by @p0mvn in #907

Full Changelog: v6.3.1...v6.4.0