Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Track block proposals in the validator monitor #3643

Closed
paulhauner opened this issue Oct 15, 2022 · 3 comments
Closed

Track block proposals in the validator monitor #3643

paulhauner opened this issue Oct 15, 2022 · 3 comments

Comments

@paulhauner
Copy link
Member

Description

Making the validator monitor track block proposals would help us raise alerts if there has been a missed proposal by a local/monitored validator. This has been requested several times and makes a lot of sense, especially now we have builders as an additional failure point.

Implementation

Considering this briefly, I think we could track missed proposals in two ways:

  1. Use ValidatorMonitor::process_valid_state to reading the proposers for the current epoch and then look at latest_block_roots for skip slots.
  2. Add a new ValidatorMonitor::process_head_state method which works similar to the previous method, but only takes states which were elected as the head (rather than all states).

The issue with (1) would be that relatively weight-less, obscure heads may trigger a rather alarming "missed block" alert.

A downside both approaches is that it would only register a missed block once a subsequent block has been imported. I'm not hugely concerned with this since we should already be raising CRIT logs whenever a block production fails. The edge-case here would be a failure in block propagation, which presently raises no logs. All this considered, we'd likely be registering the proposal failure within 12-24s and given the relatively infrequent nature of block proposal I'm comfortable with this slight delay.

@michaelsproul
Copy link
Member

Sounds like a nice addition. @tthebst has done some related work in #3635 which we could build on to track misses as you suggest

@paulhauner
Copy link
Member Author

paulhauner commented Nov 15, 2022

#3727 and #3728 are some small changes to the validator monitor that I want to get included before we start on this one.

@paulhauner
Copy link
Member Author

Closing as a duplicate with #3414

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants