-
Notifications
You must be signed in to change notification settings - Fork 19
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge #530: Improve tracker statistics importation
af7150b feat: [#469] import torrent stats using multiple torrents tracker API endpoint (Jose Celano) 16cbea8 feat: [#469] import torrent statistics in batches (Jose Celano) feffd09 feat: [#469] add update datetime for tracker stasts importation (Jose Celano) Pull request description: Currently, the Index imports statistics for all torrents every hour (1 hour is the default value in the configuration). We need to import stats for all torrents because we allow users to sort torrents by torrent stats (number of seeders and leechers). This PR improves a little bit the process. - [x] Add a new field (`updated_at`) to the table `torrust_torrent_tracker_stats` with the datetime when the stats were imported from the tracker. This is for logging purposes but it also helps to import torrents in batches. Regarding logging, it could help to check that the cronjob is running correctly. - [x] We get all torrents (`get_all_torrents_compact`) from the database. That could be big array of infohashes. We could obtain the 50 records that have not been updated for the longest time and run the importation every 100 milliseconds. We request the tracker API every 100 milliseconds getting 50 torrents. Those values can be adjusted in the future. - [x] A [new filter was added to the tracker API to get statistics for a list of torrents with one request](torrust/torrust-tracker#728). We can use it instead of getting one torrent at a time. **Pros:** - With millions of torrents we don't need to load all of them into memory. - The new field `updated_at` helps to monitor the importation process. - We get torrent stats for 50 torrents in one request instead of one request per torrent. **Cons:** - Every 100 milliseconds we run a query to check which torrent stats are pending to update. ACKs for top commit: josecelano: ACK af7150b Tree-SHA512: af1632282419457e20cc86e447b65d36c8e52dbff47e5c79cc1802fc6f67c759d572568f2846f65d4d5540049240ea82246df21d773ed1e6a285bde681fb423b
- Loading branch information
Showing
10 changed files
with
281 additions
and
22 deletions.
There are no files selected for viewing
4 changes: 4 additions & 0 deletions
4
migrations/mysql/20240312130530_torrust_add_update_data_to_tracker_stats.sql
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
-- New field to track when stats were updated from the tracker | ||
ALTER TABLE torrust_torrent_tracker_stats ADD COLUMN updated_at DATETIME DEFAULT NULL; | ||
UPDATE torrust_torrent_tracker_stats SET updated_at = '1000-01-01 00:00:00'; | ||
ALTER TABLE torrust_torrent_tracker_stats MODIFY COLUMN updated_at DATETIME NOT NULL; |
2 changes: 2 additions & 0 deletions
2
migrations/sqlite3/20240312130530_torrust_add_update_data_to_tracker_stats.sql
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,2 @@ | ||
-- New field to track when stats were updated from the tracker | ||
ALTER TABLE torrust_torrent_tracker_stats ADD COLUMN updated_at TEXT DEFAULT "1000-01-01 00:00:00"; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.