Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(core/mempool): improve perf of retrieve transactions #4710

Merged
merged 2 commits into from
Sep 22, 2022

Conversation

sdbondi
Copy link
Member

@sdbondi sdbondi commented Sep 21, 2022

Description

  • improves the performance of transaction retrieval for a block template esp with many inputs
  • improves the performance of published block handling with many inputs
  • adds execution times for each step later debugging in published block handler
  • adds a number of logs with timing

Motivation and Context

Mempool storage has a very long running write lock while retrieving txns (many inputs) for a block and processing a published block. This manifests with the status line freezing and with miners unable to retrieve a block template.

Fairly minor improvements/low hanging fruit, however with many inputs this change greatly reduces the time to complete and I was able to mine these transactions.

Some possible optimisations for future

  • re-look at algo for dependent transactions, the way dependent transactions are currently found requires removal and re-insertion of transactions "after the fact" as transactions are retrieved for a block but perhaps should be detected and cached as transactions are inserted to maximise retrieve's perf. Needs more evaluation.
  • Evaluate caching the block template, cache is invalidated if a new transaction comes in, a new block is received and reorgs
  • if compact inputs are propagated, input.output_hash() will not actually have to generate the hash when doing the O(nmj) comparison. compact input broadcasting from wallets is probably not worth the effort at this stage
  • expand the current mempool benchmark suite.

How Has This Been Tested?

Mempool freeze is reduced and blocks can be mined, however it is still slow for many inputs.

Copy link
Collaborator

@SWvheerden SWvheerden left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@hansieodendaal
Copy link
Contributor

hansieodendaal commented Sep 22, 2022

I have thoroughly tested this code with coin splits, all good there.

ToDo: Test with spending dust in huge transactions

@CjS77 CjS77 merged commit f55762e into tari-project:development Sep 22, 2022
@sdbondi sdbondi deleted the core-mempool-minor-perf branch September 22, 2022 12:02
@hansieodendaal
Copy link
Contributor

I have thoroughly tested this code with spending dust in huge transactions - works perfectly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants