Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

7429/feature/add trending score to solr #9878

Open
wants to merge 6 commits into
base: master
Choose a base branch
from

Conversation

benbdeitch
Copy link
Collaborator

@benbdeitch benbdeitch commented Sep 13, 2024

Closes #7429

This PR adds support for trending scores to Solr, allowing us to better track which works are achieving a statistically notable increase in popularity. It adds several new fields, and comes with two scripts to be run-- one daily, the other hourly, to keep this information constantly up to date.

Currently, it's still in draft mode, as there is currently no code to automatically run the scripts.

Technical

This implementation uses Solr's ability to update documents in place, which requires the new trending fields to not be stored or indexed, and instead treated as a docValue. Essentially, they are left out of Solr's inverted index, and instead treated as a more usual document-to-value mapping.

This is both A) more performant than atomic updates, and B) avoids the issues that atomic updates can have with copyfield values.

  1. Delete your solr container and all related volumes.
  2. Run docker compose up.
  3. Going to your local solr instance, run a search for a work on Solr (e.g. key:"/works/OL54120W"), and check to ensure that the new fields are present.
  4. Save a work to your 'want-to-read' list.
  5. In another instance of the command line, run
docker compose exec web bash
cd scripts
python calculate_trending_scores_hourly.py
  1. After a minute or so, run the search on Solr again, and see if the appropriate trending field has updated.

Screenshot

Stakeholders

@cdrini

@benbdeitch benbdeitch marked this pull request as draft September 13, 2024 22:13
@github-actions github-actions bot added the Priority: 2 Important, as time permits. [managed] label Sep 13, 2024
@benbdeitch benbdeitch force-pushed the 7429/feature/add-trending-score-to-solr branch from 3673c5e to 8fb48c7 Compare September 14, 2024 19:22
@mekarpeles mekarpeles added this to the Sprint 2024-09 milestone Sep 15, 2024
@benbdeitch benbdeitch force-pushed the 7429/feature/add-trending-score-to-solr branch from 74fdc7a to ff3dbf3 Compare October 1, 2024 22:51
@benbdeitch benbdeitch force-pushed the 7429/feature/add-trending-score-to-solr branch from ff3dbf3 to 6d56b8b Compare October 1, 2024 22:51
@benbdeitch benbdeitch marked this pull request as ready for review October 1, 2024 22:55
@benbdeitch benbdeitch force-pushed the 7429/feature/add-trending-score-to-solr branch from ebfa080 to 6aef600 Compare October 1, 2024 22:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Priority: 2 Important, as time permits. [managed]
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add trending score to solr
4 participants