Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/storage/monitoring tool #1248

Merged
merged 13 commits into from
Dec 14, 2023
Merged

Feature/storage/monitoring tool #1248

merged 13 commits into from
Dec 14, 2023

Conversation

Didayolo
Copy link
Collaborator

@Didayolo Didayolo commented Nov 28, 2023

Merge the PR into a branch in our own codalab/codabench repository.

Original PR:


Description

This PR aims to offer a monitoring tool similar to the one available in Codalab.
The following features have been developed:

  • The monitoring tool is only accessible by admins
  • It is accessible under the user menu -> analytics -> storage tab
  • 3 metrics are available
    • Usage history
    • Competitions usage history and details
    • Users usage history and details
  • Metrics can be download using the download buttons
  • The metrics only displays what has been computed by the server beforehand
  • 2 background tasks has been coded:
    • a reset task that get executed every 3 month (based on the value that is set in the settings/base.py). It reset the value of all computed file sizes so they will be re-computed again without any shifting on the next run of the storage analytics task
    • a storage analytic snapshot task that get executed every week (based on the value that is set in the settings/base.py). It measures all files with unset size. Then it measures the storage usage per category (competition, user or admin) and per day. It then check for database <-> storage inconsistency (and created a log file in the logs/ folder). And save everything into the database so it becomes query-able by the front-end.

Issues this PR resolves

#713

Checklist

  • Code review by me
  • Hand tested by me
  • I'm proud of my work
  • Code review by reviewer
  • Hand tested by reviewer
  • CircleCi tests are passing
  • Ready to merge

@Didayolo
Copy link
Collaborator Author

@ihsaan-ullah ihsaan-ullah merged commit 0311135 into develop Dec 14, 2023
@ihsaan-ullah ihsaan-ullah deleted the monitoring-tool branch December 14, 2023 15:11
@ihsaan-ullah ihsaan-ullah mentioned this pull request Jan 25, 2024
41 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants