-
Notifications
You must be signed in to change notification settings - Fork 2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add prune old peers functionality for crawler #18310
Conversation
…ze and write times under control
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hell to the yes.
|
Pull Request Test Coverage Report for Build 9947688177Warning: This coverage report may be inaccurate.This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.
Details
💛 - Coveralls |
Coverage drop here is related to specifically NOT running the crawler loop in the background. The loop was causing flaky behavior by doing unexpected crawling tasks in the background and changing the state of the crawl store in the tests in an uncontrolled and unpredictable manner. The "coverage" that was being provided by that running was not true test coverage - nothing is really testing that loop at the moment in a controlled manner. |
Keeps the database size and write times under control
Purpose:
Currently, the crawler DB will grow infinitely over time. We used to run similar queries when backing up the crawlers, and since they were recreated periodically, our DBs were mostly kept under control. This moves that pruning to the actual crawler's areas of responsibility, and benefits others running the crawler.