Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add prune old peers functionality for crawler #18310

Merged
merged 4 commits into from
Jul 16, 2024
Merged

Conversation

cmmarslender
Copy link
Contributor

Keeps the database size and write times under control

Purpose:

Currently, the crawler DB will grow infinitely over time. We used to run similar queries when backing up the crawlers, and since they were recreated periodically, our DBs were mostly kept under control. This moves that pruning to the actual crawler's areas of responsibility, and benefits others running the crawler.

@cmmarslender cmmarslender requested a review from a team as a code owner July 15, 2024 22:40
@cmmarslender cmmarslender added Changed Required label for PR that categorizes merge commit message as "Changed" for changelog Crawler labels Jul 15, 2024
Copy link
Member

@hoffmang9 hoffmang9 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hell to the yes.

Copy link
Contributor

File Coverage Missing Lines
chia/seeder/crawl_store.py 94.4% lines 424
chia/seeder/crawler.py 33.3% lines 96-97, 309, 311
Total Missing Coverage
49 lines 5 lines 89%

Copy link

Pull Request Test Coverage Report for Build 9947688177

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

Details

  • 44 of 49 (89.8%) changed or added relevant lines in 4 files are covered.
  • 218 unchanged lines in 7 files lost coverage.
  • Overall coverage decreased (-0.2%) to 90.75%

Changes Missing Coverage Covered Lines Changed/Added Lines %
chia/seeder/crawl_store.py 17 18 94.44%
chia/seeder/crawler.py 2 6 33.33%
Files with Coverage Reduction New Missed Lines %
chia/_tests/core/full_node/test_transactions.py 1 99.11%
chia/server/server.py 2 82.21%
chia/_tests/core/test_farmer_harvester_rpc.py 2 98.02%
chia/full_node/full_node.py 8 85.71%
chia/seeder/peer_record.py 33 45.38%
chia/seeder/crawl_store.py 73 46.89%
chia/seeder/crawler.py 99 33.62%
Totals Coverage Status
Change from base Build 9946298621: -0.2%
Covered Lines: 101290
Relevant Lines: 111538

💛 - Coveralls

@cmmarslender
Copy link
Contributor Author

Coverage drop here is related to specifically NOT running the crawler loop in the background. The loop was causing flaky behavior by doing unexpected crawling tasks in the background and changing the state of the crawl store in the tests in an uncontrolled and unpredictable manner. The "coverage" that was being provided by that running was not true test coverage - nothing is really testing that loop at the moment in a controlled manner.

@Starttoaster Starttoaster merged commit 8b01fdc into main Jul 16, 2024
373 of 374 checks passed
@Starttoaster Starttoaster deleted the crawler-prune branch July 16, 2024 15:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Changed Required label for PR that categorizes merge commit message as "Changed" for changelog Crawler
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants