Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Student trains too long #864

Open
eu9ene opened this issue Sep 24, 2024 · 0 comments
Open

Student trains too long #864

eu9ene opened this issue Sep 24, 2024 · 0 comments
Labels
cost & perf Speeding up and lowering cost for the pipeline

Comments

@eu9ene
Copy link
Collaborator

eu9ene commented Sep 24, 2024

Training time increased significantly after adding data augmentation. For example https://firefox-ci-tc.services.mozilla.com/tasks/L3zoPepXRpSG9WrEzxwZ8A

It looks like chrF is still improving after 12 days of training https://wandb.ai/moz-translations/en-ru/runs/6tdhv6op?nw=nwuserepavlov

We can probably play with early stopping or learning rate here.

@eu9ene eu9ene added the cost & perf Speeding up and lowering cost for the pipeline label Sep 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cost & perf Speeding up and lowering cost for the pipeline
Projects
None yet
Development

No branches or pull requests

1 participant