ord index is very slow

[77656233]

Hey,

my index is syncing for 2 days already and each second only 1 point of the 775k gets added. this would take 190 days to full sync the index. The first 100k starts to sync in 10 minutes and after it gets super slow.

I am on debian.

Any idea how to fix that or to speed that up?

Thanks

[andrewtoth]

This can be fixed by merging #1516 and #1636 and then reverting #1357.

[joel-carter]

@andrewtoth I have merged https://github.com/casey/ord/pull/1516 and https://github.com/casey/ord/pull/1636 and reverted https://github.com/casey/ord/pull/1357. It gets to the value set in the --first-inscription-height parameter very quickly. However, afterwards it is extremely slow taking around 1 minute per block. I have built using the cargo build -r command. Any other ideas?

[andrewtoth]

Did you start bitcoind with -rest or add rest=1 in bitcoin.conf?

[joel-carter]

Yes bitcoind is running with the -rest flag

[andrewtoth]

If you run with RUST_LOG=warn before the ord command does it warn that it can't connect to REST?

[joel-carter]

No there are no warnings output at all. See below

Update: It does seem to be using the rest endpoint as when I stop bitcoind running I get the warning that it can't connect to REST.

[andrewtoth]

Ok, the first blocks are the slowest because every input has to be fetched. As it indexes more blocks, it will already have the previous inputs so less inputs need to be fetched from disk.
It is definitely slower than what I experienced though. 1 minute for a block is too long. What type of hard drive are you using?

[joel-carter]

It is a virtual HDD with google cloud. Seems to max out at around 80 MiB/s (~600mbps if I am correct), but ord and bitcoind are using a fraction of that. (5-15 MiB/s at most). cpu utilisation is about 1-3% for both processes together too. Ram is 16GB and only using around 600MB

[77656233]

got it working with compile 0.4.2 and using a bootrap file from community

[joel-carter]

@ChristianGrieger could you share further details please?

[77656233]

1. git clone https://github.com/casey/ord
2. cd folder
3. git checkout 0.4.2
4. cargo build -r
5. download bootstrap file, rename it to index.redb.gz and place/unzip it in /root/.local/share/ord

wget 'https://s3.amazonaws.com/octal.eth/index.redb.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAVY3HUS3MKQ7N5W5V%2F20230204%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230204T180540Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=db0cf7243e0cced65a5d228714db8fc13f483671bda2860d1264de2ecf84289d'

6. run ord wallet balance
7. done ✅😅

[mmikeww]

This can be fixed by merging #1516 and #1636 and then reverting #1357.

any chance we can get an executable for this? i'm on 2 days of indexing and its a pain in the ass

[jpc4r]

I cloned the repo with git clone main repo link After switched branche to 4.0.2 with git checkout 4.0.2 build it after and added a bootstrap file from community for the db and started it. or what you want to know?

Where did you get the file from?

[so7ow]

This can be fixed by merging #1516 and #1636 and then reverting #1357.

Any help for us git-illiterates?

[andrewtoth]

Any help for us git-illiterates?

Added the speedup improvements in my branch https://github.com/andrewtoth/ord/tree/speedup-improvements if you'd like to help test/benchmark.

git clone https://github.com/andrewtoth/ord
cd ord
git checkout speedup-improvements
cargo build --release

Make sure you set rest=1 in bitcoin.conf or run bitcoind -txindex -rest

[so7ow]

git clone git@github.com:andrewtoth/ord

% git clone git@github.com:andrewtoth/ord
Cloning into 'ord'...
git@github.com: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

[andrewtoth]

Updated the post, use git clone https://github.com/andrewtoth/ord instead.

[so7ow]

Now

Updated the post, use git clone https://github.com/andrewtoth/ord instead.

now fails on cargo line with: zsh: command not found: cargo

[andrewtoth]

Oh I assumed you'd have rust installed. Install rust first, which includes cargo. After the cargo build line you can find ord at ./target/release/ord.

[andrewtoth]

Whoops had a bad merge in there. Fixed. Do a git pull and try again.

[so7ow]

Success!! Will report back.

[so7ow]

I didn't get too far! Running ord:

error: Manual upgrade required. Expected file format version 109, but file is version 107

[andrewtoth]

Ahh this has a newer db version. You'll have to nuke your previous index file. It should be in ~/.local/share/ord if you're on linux. Just rm ~/.local/share/ord/index.redb.

[so7ow]

But... that's 3 day's worth of work! 🥴

Will this new one be fast enough to catch me up?

[andrewtoth]

Ahh you can specify a new index file. Run ord --index ./index.redb index and it will create one in your current directory.

[so7ow]

Well! It just ran through all the indexing in ~10 seconds and returned:

error: Can't assign requested address (os error 49)

[so7ow]

I think I'm wrong, it didn't quite finish all the indexing. It zipped right up to 766244/775968 and then crashed out. If I run it again it goes through a few more blocks and crashes out again.

[andrewtoth]

Yeah it blows through everything until block 767430 in about 10 seconds, then it takes me a little over an hour to sync to tip.
Are you running bitcoind on default port? Do you have rest enabled? Not sure haven't hit that error. Any other configs for your bitcoind?

[so7ow]

Yes, default port. Yes to rest.
txindex=1 in bitcoin.conf
server: true in settings.json

Other than that, vanilla, I think.

[andrewtoth]

Hmm not sure what's happening. What OS are you running? It's connecting properly to the P2P interface to be able to sync all headers. It might be breaking on REST. Try running bitcoind with rest=0 and see if it syncs. It will be much slower but if it doesn't break then there's a bug with my REST patch.

[andrewtoth]

@apemithrandir you need txindex=1 on bitcoind for this branch. Check by running bitcoin-cli getindexinfo. It should show "synced": true.

You can also check by running bitcoin-cli getrawtransaction <txid that fails>.

[apemithrandir]

@apemithrandir you need txindex=1 on bitcoind for this branch. Check by running bitcoin-cli getindexinfo. It should show "synced": true.

You can also check by running bitcoin-cli getrawtransaction <txid that fails>.

Thanks for the reply. I have txindex=1 (I also have a fully indexed fulcrum server running on the same machine). If I run getrawtransaction for the id's that fail i get the expected response.

It started happening around blockheight 770k. The indexer goes for a while then hits a failed to get transaction and exits. Unfort it doesn't save the progress thus far and reverts to the index height it was at beforehand. It keeps happening and for different tx id's each time. I can always get the raw tx id's afterwards.

If I start the index and then ctrl+c escape it after it progresses, I can kind of inch it along but if I leave it to it's own devices it eventually runs into this issue and exits while losing any progress.

[andrewtoth]

@apemithrandir Hmm... I see you have fulcrum running as well, it could be making a lot of RPC calls that will cause ord to fail if using the rest endpoint. Can you try running bitcoind with rpcworkqueue=64?

I will make a fix so that ord will save progress if fetching a tx fails.

[apemithrandir]

@apemithrandir Hmm... I see you have fulcrum running as well, it could be making a lot of RPC calls that will cause ord to fail if using the rest endpoint. Can you try running bitcoind with rpcworkqueue=64?

I will make a fix so that ord will save progress if fetching a tx fails.

Sure. I can try stopping my fulcrum server while it finishes the sync as well.

[apemithrandir]

@apemithrandir Hmm... I see you have fulcrum running as well, it could be making a lot of RPC calls that will cause ord to fail if using the rest endpoint. Can you try running bitcoind with rpcworkqueue=64?
I will make a fix so that ord will save progress if fetching a tx fails.

Sure. I can try stopping my fulcrum server while it finishes the sync as well.

Sync finished but it looks like the memory didn't get automatically cleared afterwards. Also I'm guess since it locks up the ram you shouldn't create a daemon and have --index-stats index running in the background all the time.

[andrewtoth]

@apemithrandir thanks for testing. Doesn't look like the changes on my branch will be merged.

Check out #1759 instead.

[apemithrandir]

@apemithrandir thanks for testing. Doesn't look like the changes on my branch will be merged.

Check out #1759 instead.

I'll test out https://github.com/andrewtoth/ord/tree/batch-tx-requests

[apemithrandir]

The branch compiles fine and I'm running the indexer now. leaving my fulcrum running in the background this time but did add the rpcworkqueue=64 to my bitcoin.conf. will report back later when it finishes.

[andrewtoth]

One of the goals should be to not require configuring anything other than txindex. So if you can unset rpcworkqueue and make sure it works that would be a big help.

[apemithrandir]

One of the goals should be to not require configuring anything other than txindex. So if you can unset rpcworkqueue and make sure it works that would be a big help.

Ok I will stop the index and try that. I also notice on this branch it doesn't go straight to the first inscription block. It is still going through all the blocks.
edit: 22min to do 52k blocks

[Plexzy]

rpcworkqueue=64

Getting the same issue at block 767433. Tried rpcworkqueue=64 but nothing changed. Any fixes?

[apemithrandir]

One of the goals should be to not require configuring anything other than txindex. So if you can unset rpcworkqueue and make sure it works that would be a big help.

Ok I will stop the index and try that. I also notice on this branch it doesn't go straight to the first inscription block. It is still going through all the blocks. edit: 22min to do 52k blocks

Not sure if I did something wrong on my end but this branch:
https://github.com/andrewtoth/ord/tree/batch-tx-requests

Basically stalled at block height 476k. I was surprised when it didn't do like your speed-improvments branch and skip the blocks without inscriptions.

[andrewtoth]

@apemithrandir tell Casey how much faster it is using p2p. https://github.com/casey/ord/pull/1516#issuecomment-1431862594

[apemithrandir]

@apemithrandir tell Casey how much faster it is using p2p. #1516 (comment)

After reading @casey's comment, I would agree with him. I would expect that an indexer should be using the full node on my machine (or a specific node that I tell it to connect to) and not connecting to a random node on the network. I personally don't mind too much if the indexer takes a while, tho if it fails while indexing that is obviously a bigger issue.

I'm not sure if it is possible to skip up the pre-inscriptions blocks while still using RPC, this would likely help the most.

[andrewtoth]

@apemithrandir it's not able to connect to a random node though, that patch uses the same host as the RPC and it connects to RPC first before connecting to P2P. So if you're not running your node over RPC it doesn't start, and it can't connect to a different node because it uses the RPC host.

My patch does skip pre-inscription blocks andrewtoth@3225527. But using RPC to fetch headers and blocks is that much slower than P2P. With P2P you can request 2000 headers at a time in binary. With RPC you have to fetch block height in hex, then block header in hex.

[apemithrandir]

Let me know when you and @casey resolve it. As it stands with my machine I currently cannot successfully run --index-sats index unless I build the speed-improvements branch.

[apemithrandir]

I tried compiling this code (was a detached head but merged a local branch):
andrewtoth@3225527
But it still didn't skip before inscription block. I am back on the speedup-improvements branch now and re-building the index.

[mat888]

Make sure to include the --bitcoin-data-dir /yourdirectory/bitcoin/ otherwise it will be even slower than normal!

[andrewtoth]

@apemithrandir you can't skip blocks if you use --index-sats. That indexes all sat ranges from genesis. So my patch does not help with that, only the normal indexing. If you can test a regular index that would be great.

There are many things that could potentially make index-sats faster, but ultimately it will take longer and longer as sat ranges become more fragmented. So it will always take a long time and there is likely no solution that will make it take only a few hours or so.

[apemithrandir]

Ah ok. I find the index-sats never completes for me. The memory management is not ideal. I would need some way to tell it not to use 100% of the ram since it caused my server to crash/disconnect. I was also getting these errors on the build from the speed-improvements branch:

I will test again with normal index.

[andrewtoth]

Right, there's an issue for memory management https://github.com/casey/ord/issues/1630. I'm working on fixing the json-rpc issues as well.

[apemithrandir]

@apemithrandir you can't skip blocks if you use --index-sats. That indexes all sat ranges from genesis. So my patch does not help with that, only the normal indexing. If you can test a regular index that would be great.

There are many things that could potentially make index-sats faster, but ultimately it will take longer and longer as sat ranges become more fragmented. So it will always take a long time and there is likely no solution that will make it take only a few hours or so.

so if I run ord index on your latest master branch:
https://github.com/andrewtoth/ord
it should run fast or do I still need to checkout speed-improvements? which branch would you like me to test?

[andrewtoth]

https://github.com/andrewtoth/ord/tree/batch-tx-requests

[apemithrandir]

https://github.com/andrewtoth/ord/tree/batch-tx-requests

Buiilt from batch-tx-requests. Ran time index ord
Had some interrupts from HTTP response too short errors (this resulted in reverting to a previous block height), but overall ran quickly. Built index in around 3 hrs.
Edit: to be clear I am not that concerned with an index taking a long time to build. I've built a fulcrum indices and often takes days to build. The main issues are when it exits due to memory mismanagement or JSON errors. ideally you would like to just leave your server running building the index in the background.

[apemithrandir]

Maybe not the place for this but I I have some basic questions about inscriptions/ordinals. In order to "take advantage" of ordinal theory do you need some sort of super-powered coin control? If you found out you had a "rare sat" in one of your Ord Wallet UTXOs is there any commands in ord wallet that would also you to select that sat for inscription? The default ord wallet inscribe behavior is to take the 1st sat from the first input and inscribe to that sat.

If you had a "rare sat" in position K of N in one of your UTXOs you would need to create a sort multi-path transaction that would split your UTXO into a [1,K-1] UTXO and a [K,N-K-1-fee] UTXO, then select the [K,N-K-1-fee] UTXO to perform the ord wallet inscribe transaction. Also if your "rare sat" existed in position [N-fee, N] portion of your UTXO you would need to know to take a second UTXO of size M > fee to combine that first UTXO so that you can split the UTXO into [1, K-1], [K, N+M-fee-1], [N+M-fee, N+M]. You would also need to ensure that N+M-fee-1-K > fee_inscribe + 10000.

[gmart7t2]

each second only 1 point of the 775k gets added. this would take 190 days to full sync the index.

Any idea how to fix that or to speed that up?

Yes:

>>> 775e3 / 60 / 60 / 24
8.969907407407407

Just by doing the calculation correctly I have reduced the sync time from 190 days to 9 days without changing a single line of code!

[DanM3rcurius]

What about start counting from -1206 ? Or count into the negative with the lost ordinals?

[raphjaph]

I'm going to close this since we pulled in most of the improvements. Open a new issue if you have slow index problems :)