-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use p2p to fetch blocks and headers #1516
Conversation
6f014e8
to
9ee4c87
Compare
Might fix issues related to https://github.com/casey/ord/issues/1377, since those look like the RPC threads are falling over from too many requests. I haven't fully groked the indexing logic for fetching raw transactions, but if we are doing a lot of |
This is awesome! Keep in mind that https://github.com/casey/ord/issues/1364 will wind up making sync faster. Some thoughts!
|
Thanks @casey. This will make syncing faster even when skipping txs, since it fetches headers 2k at a time instead of incrementally.
|
9ee4c87
to
e690f4c
Compare
e690f4c
to
90de650
Compare
.first() | ||
.unwrap() | ||
.to_string(), | ||
}; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure what the best way to parse the url is. By default it is 127.0.0.1:8332/wallet/ord
, which doesn't parse as a url, and in tests it is http://127.0.0.1:...
which won't parse with a split(':')
. Maybe regex?
Maybe pass --rpc-port
as an option instead of --rpc-url
? But that would be a breaking change.
90de650
to
963fc75
Compare
@casey I think this is ready for a closer look. |
19527e3
to
0dff7ed
Compare
@andrewtoth Awesome, thank you! Putting it on the project backlog so it doesn't fall off our radar. |
WOW thank you! I was ticking <1 block per second. Found this and pulled your changes, now at ~50-100 per second. This should definitely be a top priority as most questions on the discord are about this right now (3 days - 2 weeks to index is rough). Quick note: commit seems to happen every 5k blocks & takes about 3 minutes on my machine (I believe that's what was causing the json-rpc errors before this patch). This may confuse users when they see the block count hang for some time - I killed & restarted indexing myself a few times thinking that it had frozen. |
0dff7ed
to
0220355
Compare
0220355
to
9a831a8
Compare
After 16 hours the index is at block I am on |
This has got a lot of successful testing from users in #1648. |
With |
One user had set |
@casey We should definitely consider this, did an initial review and test. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewed conceptually and high-level flow and logic - looks good to me and ready for more detailed review.
I'm sorry for taking so long to collect my thoughts on this! Bitcoin's RPC and P2P interfaces are very different. The P2P interfaces is an untrusted interface used between full nodes on the bitcoin network. The RPC interfaces is an authenticated interface between a client and a trusted node. Using the P2P interface for applications where the node must be trusted is dicey. You can use it to directly connect to a trusted node, but you can also use it to connect to a totally untrusted node. I think that expressing this subtle difference to our users would be hard, and that lots of people, because running a full node is challenging, would try to connect to some random node, and have their security degraded as a result. Also #1364 fixes the same issue as this would, namely, speeding up initial sync, and is a very clean and low-impact chain that doesn't introduce another connection to manage. So, I would like to see that implemented first, before deciding if also using P2P or REST is a good idea. Also, keeping ord simple to configure is important (The degens are dying out there 😆) and every new outgoing connection introduces code, configuration, and maintainence complexity. Sorry for not figuring this out sooner, it's only been in the last few days that I've been able to sit down and review PRs in detail, and previously I didn't give it enough attention. Let's make sure we nail down the following in an issue before anyone writes a big PR:
Sorry to close! We're still figuring out this whole massive open source project thing ourselves! |
@casey disappointing conclusion. I disagree with your reasoning. This requires no other configuration options. It works out of the box. It's been tested by many users already without problems. The only configuration issue was someone who had maxuploadtarget already set, which is obviously an advanced configuration option. |
Does anyone know how much space ord index files take and where they're located? |
Uses the p2p interface to fetch block hashes and headers much more efficiently. Fetches headers 2k at a time, and fetches blocks in chunks of 10.
Syncs the first 200k or so blocks much faster, but haven't benchmarked the whole chain. I don't suspect it will be much more gain for the whole chain since most of the time will be spent indexing to the db, and blocks are already being fetched in the background. However, this is a much more efficient way of fetching this data.
Still have to fix tests because they don't support connecting via p2p yet.
Closes https://github.com/casey/ord/issues/1502.