Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stalled metrics are not reported for backward model #846

Open
Tracked by #164
eu9ene opened this issue Sep 13, 2024 · 0 comments
Open
Tracked by #164

Stalled metrics are not reported for backward model #846

eu9ene opened this issue Sep 13, 2024 · 0 comments
Labels
bug Something is broken or not correct weights and biases Intergration with Weights and Biases

Comments

@eu9ene
Copy link
Collaborator

eu9ene commented Sep 13, 2024

The bleu/chrf/ce-mean-words_stalled metrics are supposed to be on the online training dashboards but they are missing.

https://wandb.ai/moz-translations/zh-en/runs/backwards_E8S5g?nw=nwuserepavlov

I do see the messages in the log https://firefox-ci-tc.services.mozilla.com/tasks/Q5CG-D5kQaqZQGYukCsrcg/runs/1/logs/live/public/logs/live.log

[task 2024-09-13T16:39:03.768Z] [2024-09-13 16:39:03] [valid] Ep. 1 : Up. 360000 : chrf : 31.403 : stalled 2 times (last best: 31.8198)
[task 2024-09-13T16:39:05.636Z] [2024-09-13 16:39:05] [valid] Ep. 1 : Up. 360000 : ce-mean-words : 2.46139 : stalled 2 times (last best: 2.45698)
[task 2024-09-13T16:39:19.532Z] [2024-09-13 16:39:19] [valid] Ep. 1 : Up. 360000 : bleu-detok : 35.3306 : stalled 2 times (last best: 35.4744)
@eu9ene eu9ene added weights and biases Intergration with Weights and Biases bug Something is broken or not correct labels Sep 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something is broken or not correct weights and biases Intergration with Weights and Biases
Projects
None yet
Development

No branches or pull requests

1 participant