Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEATURE]: Can gradient accumulation be used in the pretraining of llama2-70b? #4785

Closed
Pty72 opened this issue Sep 23, 2023 · 1 comment
Closed

[FEATURE]: Can gradient accumulation be used in the pretraining of llama2-70b? #4785

Pty72 opened this issue Sep 23, 2023 · 1 comment
Labels
enhancement New feature or request

Comments

@Pty72
Copy link

Pty72 commented Sep 23, 2023

Describe the feature

Can gradient accumulation be used in the pretraining of llama2-70b?
And if so, how can it be enabled？
@Pty72 Pty72 added the enhancement New feature or request label Sep 23, 2023
@Pty72 Pty72 changed the title [FEATURE]: "Can gradient accumulation be used in the pretraining of llama2-70b?" [FEATURE]: Can gradient accumulation be used in the pretraining of llama2-70b? Sep 23, 2023
@Fridge003 Fridge003 mentioned this issue Sep 27, 2023
Closed
@Fridge003 Fridge003 mentioned this issue Oct 8, 2023
Merged
10 tasks
@Fridge003
Copy link
Contributor

Fridge003 commented Oct 17, 2023 •
edited
Loading

Hello, we have just supported gradient accumulation on Gemini plugin.
Usage can be referred to docs/source/en/features/gradient_accumulation_with_booster.md, our online tutorials will also be updated within a few days.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Fridge003 @Pty72