Track last seen post timestamp per wiki #100
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rossjrw
force-pushed
the
reopt-post-download
branch
from
95a390f
to
45449ad
Compare
|
Coincidentally looks like this also fixes a bug where the latest timestamp would be none if there are no posts stored for a given wiki, causing an error that prevents any posts from being stored for it at all. I messed up the migration by initting all wikis to 0 when I should have calculated it from the last recorded post for each. Because of that, the new posts getter is temporarily completely deoptimised and the next few runs will overshoot as it tries to download every post it can get its grubby little mitts on. So long as no single wiki takes more than 15 mins to download it'll fix itself over the next few runs, it just means my uptime takes a hit for the next month. |
It can never be reliable to use the notifiable_posts table for any sort of aggregate data, because any and all of it is liable to be deleted. Therefore, using the latest post's timestamp as the 'now' check is actually deleting posts that are a year older than the latest queued notification in my database - which could be anything. Conversely, the max of the post check timestamps across all registered wikis is simply the timestamp of the latest post on Wikidot that the service has ever seen, which is updated every hour and is much more reliable.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
2a85c2c fixed a bug where a post would be skipped being downloaded if a post was stored with a later timestamp than it, which because posts are downloaded ordered by wiki and not by timestamp, was causing almost all posts to be skipped. It fixed it by limiting the 'latest timestamp' search only to posts from the given wiki.
While effective, this is not efficient. Most downloaded posts are deleted during the very same run, meaning on the next run, many of those same posts will be downloaded again, wasting time and networking resources.
This PR fixes this by recording the last time that posts were downloaded from a wiki in the context_wiki table. This recording will be permanent, even if the corresponding post is deleted.