[FRAME] Remove Gov V1 from Polkadot and Kusama & unlock/unreserve funds

[kianenigma]

Originally posted by @kianenigma in paritytech/polkadot#6701 (comment)

Tasks

• Merge migrations to unreserve & unlock funds from Gov V1 pallets
  • elections-phragmen: migration unreserving Members, RunnerUps, Candidates, and Voters deposits and unlocking Voter stakes.
    • migration(elections-phragmen): unreserve deposits and clear locks substrate#14218
  • democracy: migration unreserving all accs in DepositOf and unlock all Voter funds stored under this pallet.
    • migration(democracy): unreserve deposits and clear locks substrate#14226
  • tips: migration unreserving all OpenTips
    • migration(tips): unreserve deposits substrate#14241
  • collective: balances migration not necessary
  • membership: balances migration not necessary
• Create RemovePallet migration utility (RemovePallet migration utility struct substrate#13835)
• Next Kusama runtime upgrade unreserve & unlock funds from all Gov V1 pallets
  • migration: unlock/unreserve Gov v1 balances, remove Gov V1 pallets from polkadot runtime, and remove Gov V1 pallet key/values from storage polkadot#7314
• Verify Kusama unreserve & unlock of all funds was successful
• Once pallets are no longer in the Kusama runtime, remove the storage with the RemoveStorage migration
• Next Polkadot runtime upgrade unreserve & unlock funds from all Gov V1 pallets
• Verify Polkadot unreserve & unlock of all funds was successful
• Once pallets are no longer in the Polkadot runtime, remove the storage with the RemoveStorage migration
• Fully document the transition from Gov v1 -> Gov v2

Unreserving / unlocking details

We need to unreserve / unlock all funds in the pallets before we delete the storage.

more info from @kianenigma :

do a quick analysis of the code of these pallets and find all deposit instances within FRAME, every time we reserve a deposit, we alway store it somewhere in the storage of the pallet as well (namely because we only recently started to use named deposits, so it was hard to keep record). Example: https://github.com/paritytech/substrate/blob/master/frame/elections-phragmen/src/lib.rs#L154

Then you can create some kind fake storage map for these maps (using #[storage_alist] macro), iterate the map, unreserve.

For locks it is similar, but simpler because the locks are all stored in the balances pallet; you just need to recover the PalletId of all of these pallets, then call into Balances::remove_lock.

[Gioyik]

@kianenigma I would like to work on this if that is ok. A few questions:

• I will use runtime/kusama/src/lib.rs as reference for the polkadot section. Does the same needs to be done for rococo and westend libs?
• I can identify clean up in node/service/src/chain_spec.rs and runtime/**/src/xcm_config.rs. Is there another place I might be missing?

[Gioyik]

fyi, I have been working on this, in this branch: https://github.com/Gioyik/polkadot/tree/gio-opengov1-storage-cleanup

[liamaharon]

Hey @Gioyik, the task isn't about removing the pallets from the runtime code (already done in paritytech/polkadot#6701) but rather about cleaning up the DB storage used by those pallets. I've updated the issue description with some more details, still up for grabs if you want it but obviously no obligation :).

[muharem]

Whats the status? Anyone working on this?

[ggwpez]

Whats the status? Anyone working on this?

No. Gov V1 must first be properly disabled & undeployed as XLC commented here paritytech/polkadot#6977 (comment)
I am not sure what the proposed timeline for this is, i imagine there is an overlap period also no Polkadot for Gov V1.

[muharem]

Removal of Kusama Gov V1 already in master.
And I think this migration can be part of the pallet, we will need it for Polkadot as well. Also might be useful for others.

[liamaharon]

Removal of Kusama Gov V1 already in master. And I think this migration can be part of the pallet, we will need it for Polkadot as well. Also might be useful for others.

Given Gov V1 is removed from Kusama (correct is us we're wrong @ggwpez) I'm going to start working on a migration to unlock gov v1 funds and remove the storage in the next couple days.

[ggwpez]

Okay i missed that we already un-deployed it on Kusama in paritytech/polkadot#6701.
Then yes, please go ahead @liamaharon

[xlc]

The upcoming Kusama runtime 9420 have removed gov1 yet does not refund locked tokens. It means people still have token locked in gov1 will not able to access those token.

[liamaharon]

Thanks for bumping this @xlc, prioritising it now

[xlc]

@liamaharon can we do those in a lazy migration way?
i.e. add a permissionless extrinsic to return deposits so users can call those to get their funds back
this makes this migration suitable for parachains without worrying big pov issue.

[liamaharon]

Valid concern @xlc, thanks for raising.

If we find an instance where one of the migrations is too risky to run due to PoV issues I can convert it to be multi-block, finalizing multi-block migrations is high on the priority stack and this seems like a perfect application for it. If for some reason multi-block migrations are delayed and parachains need to migrate, then we should consider adding a lazy option as you suggest.

[ggwpez]

Yea I am working on a small version of #198. The migration gets in an optional Cursor and can advance that to its liking.

[liamaharon]

Migration guide published: https://forum.polkadot.network/t/gov-v1-to-opengov-a-technical-migration-guide/4116

[jam10o-new]

Heya.

I'm sitting on a Kusama account that still has a democracy lock on it. Not completely sure what's going on there, but it looks like many, many other accounts are affected. Somewhere to start looking around could be accounts with votes that were expired (ready to unlock) but not unlocked during the migration (ie, they would not be visible in VotingOf or DepositOf, the storage you checked in your migration, and just needed a simple call to democracy.unlock()), but I also notice some high-conviction votes that wouldn't have been expired at the time of migration with this issue, so it's probably not that simple.

To illustrate the scale of this issue, you can go to any gov1 referendum:

https://kusama.subscan.io/referenda

and look through the voters. Almost all voters who have votes that are not labelled as expired also still have a democracy lock, but this is not a rule.

(for illustration, here's the final referendum: https://kusama.subscan.io/referenda/273, and the first voter you see there: https://kusama.subscan.io/account/D3Fh5td8SjuWn3GR7WEQyZx6TncxwFLaix2YTvygMyxDcU9, has a substantial lock still on their account. This can also be verified in the apps UI in case you don't trust block explorers for whatever reason)

fwiw, I suspect that the reason why folks haven't been noticing this is because it primarily effects older members of the community and active or previously high conviction voters 🙃 - our wallets and gov UI's generally automatically set vote sizes to the amount currently already locked for voting, so unless you stopped voting and let other locks expire like I have, you wouldn't notice that you have a permanent gov1 lock on your account.

I'm frankly not interested in participating in this community anymore tbqh, just want my tokens unlocked, so forgive me for not speaking up about/looking into this earlier, that's it for my free labor, good luck. 🤷

[liamaharon]

Hey @jam10o-new, the runtimes have not been deployed yet, that is responsibility of the fellowship. Deployment can be tracked on their repo here: polkadot-fellows/runtimes#20

[jam10o-new]

apologies.

Verify Kusama unreserve & unlock of all funds was successful

Threw me off, as it seemed to imply these changes had already been deployed - I just realized that Polkadot still has Gov1 pallets but has apparently also been verified, so I guess y'all are talking about tests.