Elastic scaling: runtime v2 descriptor support #5423

sandreim · 2024-08-20T15:52:53Z

Closes #5045 and #5046

~~On top of #5362~~

TODO:

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

paritytech-cicd-pr · 2024-09-05T16:22:11Z

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable 1/3
Logs: https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/7276883

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

polkadot/primitives/src/vstaging/mod.rs

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

polkadot/runtime/parachains/src/inclusion/mod.rs

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

…reim/runtime_v2_descriptor_support

alindima

Looking good! Have you tested the runtime migration with an upgrade in zombienet?
I've always discovered bugs by doing that in the past 😆

polkadot/primitives/src/vstaging/mod.rs

polkadot/runtime/parachains/src/inclusion/mod.rs

polkadot/runtime/parachains/src/paras_inherent/mod.rs

polkadot/runtime/parachains/src/shared.rs

polkadot/runtime/parachains/src/shared/migration.rs

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

sandreim · 2024-09-13T12:53:20Z

Looking good! Have you tested the runtime migration with an upgrade in zombienet? I've always discovered bugs by doing that in the past 😆

Not yet, but I don't really expect any issues, it is a very simple migration.

sandreim · 2024-09-13T20:59:19Z

polkadot/primitives/src/vstaging/mod.rs

+	claim_queue: BTreeMap<CoreIndex, VecDeque<Id>>,
+) -> BTreeMap<ParaId, BTreeMap<u8, BTreeSet<CoreIndex>>> {


Some type alias should help here.

Agree :)
Also, I find it a bit strange to have a BTreeMap<u8, BTreeSet<_>>
instead of a Vec<BTreeSet<CoreIndex>> where the index in the Vec represents the depth. Given that the depth is supposed to be a number 0..x and without gaps IIUC

I was using a VecDeque initially, but switched to BTreeMap. There are 2 reasons to do it:

we don't need to pop like with the claim. queue

future proofing in case we want to support gaps for on-demand, where currently we need to double the assignment in case the core is not used at all, as the para might not have enough time to utilize it if we add it at claim queue offset 0

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

sandreim · 2024-09-17T15:00:18Z

Runtime upgrade successful in Zombienet using a Westend local chain.

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

alindima

Nice work!

ordian

Looking good overall!
Left some questions for my understanding and potential corner-cases

ordian · 2024-09-19T15:16:36Z

polkadot/primitives/src/vstaging/mod.rs

+
+		self.selected_core().and_then(|(core_selector, _cq_offset)| {
+			let core_index =
+				**assigned_cores.get(core_selector.0 as usize % assigned_cores.len())?;


what i didn't get from the RFC is why do we need to do % assigned_cores and not assume that the core_selector < assigned_cores.len()?

Core selector is a sequence number that wraps around. We use the LSB of block numbers to fill it in.

ordian · 2024-09-19T15:19:57Z

polkadot/runtime/parachains/src/inclusion/mod.rs

+		{
+			let (upward_messages, ump_signals) = upward_messages.split_at(separator_index);
+
+			if ump_signals.len() > 2 {


is 0 ump_signals allowed in case there's a separator?
is multiple separators allowed? we only look here for the last one
my thinking it should be no in both cases - might be good to validate those

0 is not allowed and this should be tested by the check_core_index function and selected_core should return None. I will have to double check and add another check here if this is not covered.

If we have multiple separators that is fine, it means that the parachain wants to hurt itself. Those will count as XCM messages and are limited by a param in relay chain configuration.

Currently it seems we are not really strict, we just consider that the candidate does not commit to a core as we are using Option. I will add stricter checks.

polkadot/runtime/parachains/src/shared.rs

ordian · 2024-09-19T15:33:10Z

polkadot/primitives/src/vstaging/mod.rs

+	claim_queue: BTreeMap<CoreIndex, VecDeque<Id>>,
+) -> BTreeMap<ParaId, BTreeMap<u8, BTreeSet<CoreIndex>>> {


Agree :)
Also, I find it a bit strange to have a BTreeMap<u8, BTreeSet<_>>
instead of a Vec<BTreeSet<CoreIndex>> where the index in the Vec represents the depth. Given that the depth is supposed to be a number 0..x and without gaps IIUC

sandreim added 30 commits August 12, 2024 15:07

WIP primitives

793142e

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

WIP

3a29fdf

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

Working version.

4a53577

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

Better version

8285ae7

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

Add missing primitives and fix things

2831c5e

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

Implement v2 receipts in polkadot-runtime-parachains

c767d60

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

add missing stuff

96999e3

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

Switch parachains runtime to use new primitives

c5f2dc3

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

use vstaging primitives

dbb0160

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

update rococo and westend

5efab68

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

client keeps using the old primitives

c2232e4

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

no unsafe pls

87b079f

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

move async backing primtiives to own file

00e8c13

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

fix

cd4d02f

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

fix test build

5509e33

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

fix test-runtime

f8b86d2

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

self review feedback

fe2fbfb

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

review feedback

975e13b

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

feedback

1c7ac55

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

feedback

653873b

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

clippy

dc98149

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

chores

0a6bce3

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

Filter v2 candidate descriptors

5e4dac2

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

fix

f12ca7a

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

fix prospective parachains tests

13734de

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

fix fix

effb1cc

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

fmt

3f75cba

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

fix comment

75a47bb

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

another one

12ed853

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

fix build

f2c0882

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

sandreim added 4 commits September 5, 2024 16:17

feedback

43f6de7

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

sir, make it faster

70e48d2

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

fix

1e26c73

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

one last fix

f4e3fb5

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

sandreim added 2 commits September 5, 2024 23:34

fixes

2e87ad3

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

remove println

54432be

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

sandreim commented Sep 5, 2024

View reviewed changes

polkadot/primitives/src/vstaging/mod.rs Outdated Show resolved Hide resolved

sandreim added 2 commits September 6, 2024 18:09

add prdoc

cfbecb0

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

fix comment

3a518f2

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

sandreim added the T8-polkadot This PR/Issue is related to/affects the Polkadot network. label Sep 6, 2024

refactor map_candidates_to_cores

54106e2

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

sandreim commented Sep 6, 2024

View reviewed changes

polkadot/runtime/parachains/src/inclusion/mod.rs Outdated Show resolved Hide resolved

sandreim added 2 commits September 9, 2024 14:14

doc updates

b44a604

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

Merge branch 'master' of github.com:paritytech/polkadot-sdk into sand…

4c5c707

…reim/runtime_v2_descriptor_support

sandreim mentioned this pull request Sep 11, 2024

Switch node side to v2 candidate receipts #5679

Draft

1 task

alindima reviewed Sep 13, 2024

View reviewed changes

sandreim added 2 commits September 13, 2024 15:42

feedback

caff543

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

refactor

218f530

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

sandreim commented Sep 13, 2024

View reviewed changes

sandreim added 2 commits September 14, 2024 00:03

fix try-runtime

216937a

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

check ump signal count and test

c0aee8c

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

alindima mentioned this pull request Sep 18, 2024

runtime: remove ttl #5461

Draft

7 tasks

sandreim added 3 commits September 18, 2024 22:14

remove unused

1ef7952

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

fix prdoc

5790b8e

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

more tests cases

ba9d3ff

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

alindima approved these changes Sep 19, 2024

View reviewed changes

ordian reviewed Sep 19, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Elastic scaling: runtime v2 descriptor support #5423

Elastic scaling: runtime v2 descriptor support #5423

sandreim commented Aug 20, 2024 •

edited

Loading

paritytech-cicd-pr commented Sep 5, 2024

alindima left a comment

sandreim commented Sep 13, 2024

sandreim Sep 13, 2024

ordian Sep 19, 2024

sandreim Sep 20, 2024

sandreim commented Sep 17, 2024

alindima left a comment

ordian left a comment

ordian Sep 19, 2024

sandreim Sep 20, 2024

ordian Sep 19, 2024

sandreim Sep 20, 2024

sandreim Sep 20, 2024

ordian Sep 19, 2024

		claim_queue: BTreeMap<CoreIndex, VecDeque<Id>>,
		) -> BTreeMap<ParaId, BTreeMap<u8, BTreeSet<CoreIndex>>> {

Elastic scaling: runtime v2 descriptor support #5423

Are you sure you want to change the base?

Elastic scaling: runtime v2 descriptor support #5423

Conversation

sandreim commented Aug 20, 2024 • edited Loading

paritytech-cicd-pr commented Sep 5, 2024

alindima left a comment

Choose a reason for hiding this comment

sandreim commented Sep 13, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sandreim commented Sep 17, 2024

alindima left a comment

Choose a reason for hiding this comment

ordian left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sandreim commented Aug 20, 2024 •

edited

Loading