-
Notifications
You must be signed in to change notification settings - Fork 28
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
LLM Finetune Functionality #277
Conversation
@eldarkurtic As part of your review, please pay special attention to the fsdp handling to ensure it is being applied how you expect it to. If you can pull down and test a model for accuracy/speed, that would also be appreciated. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM overall, just had a few nitpicky comments. I haven't tried running this on machine yet, will give it a go later today
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
looks great overall, see few comments
* add functions to mask weights during finetuneing * update logic for loading weights * update yaml * update mask name * add logic to update batchsize based on gpu count * make sparsify requirements less broad; move sparseml[transformers] to nm deps * remove flash-attn * quality
Summary
finetune
which provides support for all the steps required for llmfoundry finetuning integration--data
cli arg to the sparsify command)finetune.py
includes the steps which can be wrapped inlaunch_ddp
andtrainhook
data
arg containing the yaml file, there are nofinetune
specific train args but this can be added in as greater finetuneing support is flushed outTesting
samples
folderThis produces the following output in the running directory after training is complete.
sparse_transfer_finetune_2023_08_09_01_24_08
All paths from llmfoundry are updated to use the working directory as the root and therefore, the output directory shown above contains the checkpoints and all other llmfoundry-specific outputs.