Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add block v3 endpoint. #339

Merged
merged 6 commits into from
Jul 28, 2023
Merged

Add block v3 endpoint. #339

merged 6 commits into from
Jul 28, 2023

Conversation

mcdee
Copy link
Contributor

@mcdee mcdee commented Jul 16, 2023

As per #309 this adds a version 3 block endpoint that consolidates the current blinded and unblinded endpoints. Features of this endpoint:

  • a single endpoint to fetch blocks regardless of type
  • allows the beacon node to decide (with help from the EL) if it should use MEV relays for execution payload
  • provides the returned block's payload to the proposer
  • metadata allows for simpler handling of decoding the returned data

@@ -421,3 +424,13 @@ components:
required: true
schema:
$ref: '#/components/schemas/ConsensusVersion'
Eth-Blinded-Payload:
description: Required in response so client can deserialize returned json or ssz data to the correct object.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
description: Required in response so client can deserialize returned json or ssz data to the correct object.
description: Required in response so client can deserialize returned JSON or SSZ data to the correct object.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The existing Eth-Consensus-Version uses lower-case so I stuck with it for consistency.

Comment on lines 50 to 55
Eth-Consensus-Version:
$ref: '../../beacon-node-oapi.yaml#/components/headers/Eth-Consensus-Version'
Eth-Blinded-Payload:
$ref: '../../beacon-node-oapi.yaml#/components/headers/Eth-Blinded-Payload'
Eth-Payload-Value:
$ref: '../../beacon-node-oapi.yaml#/components/headers/Eth-Payload-Value'
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Headers are required for SSZ returns. Should the added two headers be also present as JSON values to the application/json response type? Otherwise, why include version as property but not the payload value.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, wasn't sure about that. In the past we added the version to the JSON to allow for decoding, but it has pretty much been replaced with the header values now (as you don't need to do any mucking around with parsing streams to obtain the values from the headers). I'm not averse to putting these new values in the JSON metadata, but not sure if they add significant value any more.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we do something controversial and just return the json object not wrapped in a 'data' and have it just the json object or the ssz object determined by header?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we do something controversial and just return the json object not wrapped in a 'data' and have it just the json object or the ssz object determined by header?

It would definitely break the standard from other routes and prevent allowing to add more fields in the future. I would mirror the header values in the JSON type for overall consistency. It's easier to parse a JSON reply than headers for client side tooling that deals with JSON only

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dapplion yeah I'm thinking the same. If the metadata fields are in the JSON then it will allow parsing of the JSON separate from the HTTP response (e.g. if the JSON is saved and parsed later). I'll update accordingly.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated. One question that rises from this is the naming of the fields: do we want to explicitly state which values are related to the execution payload, i.e.

  • blinded-execution-payload rather than blinded-payload
  • execution-payload-value rather than payload-value

(and equivalent change for the header fields). Obviously this makes them a little more verbose, but is safer if we ever end up with other payloads in the beacon block. Thoughts?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if we went that way, i'd probably suggest something like

execution-payload-value
execution-payload-blinded

i don't mind either way on whether we expand it...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have updated to use these names, agree that they are a bit clearer. I think that this looks like the best way forward;
@dapplion @rolfyone agree/disagree?

Copy link
Collaborator

@dapplion dapplion left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Minor nit on mirroring header values

Copy link
Collaborator

@rolfyone rolfyone left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM will just make sure others are happy...

Copy link
Collaborator

@dapplion dapplion left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

Copy link
Contributor

@g11tech g11tech left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. can't find produceBlindedBlockV3 in PR, is this a mistake or intentional
    ahh realized this is replacement for both, so then can be add another query param like blockSource ("builder", "execution" or "any")
  2. can be also add shouldOverrideBuilder flag in response so that validator driven architectures of making the choice (like lodestar validator or may be vouch) can consume this info
  3. can we also add the blockSource ("builder" or "execution") to the response apart from the query to reflect the selection here (especially in case of any), also gives a way to figure out that if the publishBlock call is binded to a particular beacon in case of blinded response

@mcdee
Copy link
Contributor Author

mcdee commented Jul 25, 2023

  1. can't find produceBlindedBlockV3 in PR, is this a mistake or intentional

Not sure to what this refers. There is a produceBlindedBlock operation, but that is specifically for producing blinded blocks alone. The operation ID of this call is produceBlockV3.

  1. can be also add shouldOverrideBuilder flag in response so that validator driven architectures of making the choice (like lodestar validator or may be vouch) can consume this info

Beacon nodes will by default return the highest value block from either the relay(s) or the local payload. However, if the beacon node's circuit breaker is triggered or the execution client specifies that the local payload should be used then the local payload will be returned regardless of value. Does knowing why the full payload was returned (higher value / consensus circuit breaker / execution shouldOverrideBuilder) have any value here?

  1. can we also add the blockSource ("builder" or "execution") to the response

This can be inferred from the returned payload:

  • blinded => comes from the builder
  • unblinded => comes from the execution node

@g11tech
Copy link
Contributor

g11tech commented Jul 25, 2023

  1. lodestar validator and potentially other validator driven approaches use the info to decide what to use in validator and don't decide between execution and builder in beacon node, so would be good to flow data
    i. in previous discussions we have (it has been some time now) we sort of agreed to let these two kind of approaches live side by side and this extra info will help that and is harmless and quite small to pass around
    ii. thats why a similar blinded version is required (although we can transition to this but am inclined towards validator driven approach atleast for implementation diversity)
  2. even execution can return blinded which will be important for the big payloads specially in deneb, so the inference should better be explicit

imo, this only needs to pass certain more flags around which are very small in data size and doesn't require any extra implementation work.

@mcdee
Copy link
Contributor Author

mcdee commented Jul 25, 2023

  1. I think that although the idea of having validator-controlled and beacon node-controlled selection of blocks as separate streams was nice in theory, in reality the beacon node is the entity that decides from where to fetch the execution payload. The validator client ultimately has no choice in the matter; it can only use what the beacon node returns, regardless of what it asks for. As such, I'm not sure what the purpose of the suggested flags will serve (specifically, in which situation would the validator client not ask for "any" as their block, given they want to maximize the value of the execution payload).

  2. as @arnetheduck says in Consolodating blinded/unblinded flow #309 (comment) we want the beacon node to be stateless as much as we can manage, which would require returning full blocks whenever possible

@g11tech
Copy link
Contributor

g11tech commented Jul 25, 2023

  1. I think that although the idea of having validator-controlled and beacon node-controlled selection of blocks as separate streams was nice in theory, in reality the beacon node is the entity that decides from where to fetch the execution payload. The validator client ultimately has no choice in the matter; it can only use what the beacon node returns, regardless of what it asks for. As such, I'm not sure what the purpose of the suggested flags will serve (specifically, in which situation would the validator client not ask for "any" as their block, given they want to maximize the value of the execution payload).

    1. as @arnetheduck says in Consolodating blinded/unblinded flow #309 (comment) we want the beacon node to be stateless as much as we can manage, which would require returning full blocks whenever possible

shouldn't these be left to implementation and support the implementation types. again lodestar can transition to this paradigm but should be nice to have and support implementation diversity.

@mcdee
Copy link
Contributor Author

mcdee commented Jul 25, 2023

Which flags are you after here? The one that was under discussion was to tell the beacon node to not override use of a relay payload with a local payload (when it had been informed to either from its own circuit breaker or from the execution API shouldOverrideBuilder flag), but it seems to me that it makes more sense to make this a configuration option for the beacon node itself rather than a parameter passed each time a block is requested.

@g11tech
Copy link
Contributor

g11tech commented Jul 26, 2023

Which flags are you after here? The one that was under discussion was to tell the beacon node to not override use of a relay payload with a local payload (when it had been informed to either from its own circuit breaker or from the execution API shouldOverrideBuilder flag), but it seems to me that it makes more sense to make this a configuration option for the beacon node itself rather than a parameter passed each time a block is requested.

lodestar current impl needs all the flags i mentioned before, but may be we will change our impl to push the block selection to beacon node.

but then atleast blockSource should be there in the response so that if the beacon node chooses blinded response even for executon block, (configured for e.g. via a startup flag for beacon node) then validator knows that the publish call is bound to the same beacon node. For lodestar this matters as we have sometimes seen latency for block communication between beacon and validator and have even lead to some miss scenaros

@rolfyone rolfyone merged commit 20980bf into ethereum:master Jul 28, 2023
2 checks passed
required: true
schema:
type: boolean
Eth-Execution-Payload-Value:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Without the consensus value, there's no (reasonable) way for a VC to compare blocks for profit - should we add this here?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does Nimbus already compute this value during block proposal? Lighthouse sort of does, but approximates slightly (if there are slashable attestations we don't keep track of the rewards from both).

It would make VC block choosing more fair, as the heuristics Vouch uses currently seem to favour Prysm despite Prysm's blocks being less profitable than Lighthouse and Nimbus on average (according to my blockprint nodes, public dash coming soon).

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could this be a new PR? this one is closed.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does Nimbus already compute this value during block proposal?

We do it for offline analysis for historical blocks (to pick out more detail than is needed for state transitions) but making it "online" would probably not be too hard - indeed, the VC currently has to do a lot of guesswork and it will likely always be wrong / skewed - further exacerbating the problem is that in that guesswork, it's not working with eth-denominated data usually but rather "fresh attestation count" and similar proxies which prevents the value from being compared with the execution profit (ie low cl+high el vs low cl+high el score)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could this be a new PR? this one is closed.

#358

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants