async_backing: Candidate timeouts on group rotation boundary in versi

[alexggh]

When running tick and glutton parachains on versi with async-backing ever 2-3 minutes candidate timeout on backing and that blocks the core for about 2 min.

Some logs regarding 1 candidate: https://grafana.teleport.parity.io/goto/Nke60BpSR?orgId=1.

Root cause

It seems that at group rotation boundary there is a problem in the way group assignment work and we end up in a situation where the collator and the backing subsystem end up using the group assignment before the rotation, but since the candidate is backed in a block after rotation the availability will use a different group for fetching the chunk which results in the candidate timing out.

The main culprit for this problem seems to be the backed assumption in runtime function validator_groups

polkadot-sdk/polkadot/runtime/parachains/src/runtime_api_impl/v7.rs

Line 46 in 8a8f6f9

pub fn validator_groups<T: initializer::Config>(

that when are backing a candidate is going to be included on chain next block, which is not the case with async_backing.

	let now = <frame_system::Pallet<T>>::block_number() + One::one();

	let groups = <scheduler::Pallet<T>>::validator_groups();
	let rotation_info = <scheduler::Pallet<T>>::group_rotation_info(now);

	(groups, rotation_info)

Hence the usage of validator_groups in backing subsystem and group_responsible

polkadot-sdk/polkadot/runtime/parachains/src/runtime_api_impl/v7.rs

Line 108 in 8a8f6f9

group_responsible: group_responsible_for(

from CoreState in availability-distribution will give us different groups, so the candidate never passes the availability part.

@rphmeier: Thoughts on how to fix this ?

[eskimor]

The backing time should be irrelevant for this. As I described, I think on the parathreads ticket for determining the backing group we have to use the relay parent of the candidate. If this is not the case, we need to fix that. I don't see how anything else can work.

[eskimor]

Consequences in this comment.

[alexggh]

If this is not the case, we need to fix that

I confirmed that is not the case, reproducibility steps:

1. Start any zombienet with a config where async backing is enabled and a group rotation frequency lower than the session lenght which is 10 blocks on zombienet. E.g:

[relaychain.genesis.runtimeGenesis.patch.configuration.config]
  scheduling_lookahead = 2
  group_rotation_frequency = 4

2. Run the test.
3. Open polkadot.js observe candidates time out.
4. Open the logs and you are going to see a lot of for the candidates that timed out:

Validator did not have our chunk

5. The core will be stuck for the entire duration of this group rotation and nothing else get to be executed on it.

[rphmeier]

Nice find, and thanks!

The linked PR fixes the OBO correctly. The backing group is determined based on the relay-parent and not the number the block was backed in.

[Polkadot-Forum]

This issue has been mentioned on Polkadot Forum. There might be relevant details there:

https://forum.polkadot.network/t/async-backing-development-updates/6176/1