[plugin]hybrid support zero bubble pipeline #6060

flybird11111 · 2024-09-12T03:59:35Z

📌 Checklist before creating the PR

I have created an issue for this PR for traceability
The title follows the standard format: [doc/gemini/tensor/...]: A concise description
I have added relevant tags if possible for us to better distinguish different PRs
I have installed pre-commit: pip install pre-commit && pre-commit install

🚨 Issue number

Link this PR to your issue with words like fixed to automatically close the linked issue upon merge

e.g. fixed #1234, closed #1234, resolved #1234

📝 What does this PR do?

Summarize your work here.
if you have any plots/diagrams/screenshots/tables, please attach them here.

💥 Checklist before requesting a review

I have linked my PR to an issue (instruction)
My issue clearly describes the problem/feature/proposal, with diagrams/charts/table/code if possible
I have performed a self-review of my code
I have added thorough tests.
I have added docstrings for all the functions/methods I implemented

⭐️ Do you enjoy contributing to Colossal-AI?

🌝 Yes, I do.
🌚 No, I don't.

Tell us more if you don't enjoy contributing to Colossal-AI.

fix

for more information, see https://pre-commit.ci

colossalai/pipeline/schedule/zero_bubble_pp.py

… support-zbv

for more information, see https://pre-commit.ci

… support-zbv

[pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci fix

for more information, see https://pre-commit.ci

… support-zbv

for more information, see https://pre-commit.ci

… support-zbv

for more information, see https://pre-commit.ci

… support-zbv

for more information, see https://pre-commit.ci

[pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci fix

for more information, see https://pre-commit.ci

… support-zbv

* hybrid support zbv * fix fix * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix * Update zero_bubble_pp.py * fix * fix-ci * fix [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci fix * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix * fix * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix * fix * fix * fix * [zerobubble]Support ZeroBubble Pipeline (hpcaitech#6034) * [feat] add zerobubble pp (just a frame now); add POC test for dx_dw; add test for zerobubble; * [feat] add dw test; * [fix] fix weight not close; * [update] update text; * [feat] add test run_fwd_bwd automatic scheduling; * [feat] split communication and calculation; fix pop empty send_bwd_buffer error; * [feat] add test for p & p grad; * [feat] add comments for ZBV func; * [fix] rm useless assign and comments; * [fix] fix ci test; add pytest; * [feat] add run_fwd_bwd_with_microbatch (replace input) & test; add p&p.grad assert close test & all pass; * [feat] add apply v_schedule graph; p & p.grad assert err exist; * [fix] update * [feat] fix ci; add assert; * [feat] fix poc format * [feat] fix func name & ci; add comments; * [fix] fix poc test; add comments in poc; * [feat] add optim backward_b_by_grad * [feat] fix optimizer bwd b & w; support return accum loss & output * [feat] add fwd_bwd_step, run_fwd_only; * [fix] fix optim bwd; add license for v_schedule; remove redundant attributes; fix schedule loop "while"--> "for"; add communication dict; * [fix] fix communication_map; * [feat] update test; rm comments; * [fix] rm zbv in hybridplugin * [fix] fix optim bwd; * [fix] fix optim bwd; * [fix] rm output.data after send fwd; * [fix] fix bwd step if condition; remove useless comments and format info; * [fix] fix detach output & release output; * [fix] rm requir_grad for output; * [fix] fix requir grad position and detach position and input&output local buffer append position; * [feat] add memory assertation; * [fix] fix mem check; * [fix] mem assertation' * [fix] fix mem assertation * [fix] fix mem; use a new model shape; only assert mem less and equal than theo; * [fix] fix model zoo import; * [fix] fix redundant detach & clone; add buffer assertation in the end; * [fix] add output_obj_grad assert None at bwd b step; replace input_obj.require_grad_ with treemap; * [fix] update optim state dict assert (include param group & state); fix mem assert after add optim; * [fix] add testcase with microbatch 4; * hybrid support zbv * fix fix * fix * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update zero_bubble_pp.py * fix * fix-ci * fix [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci fix * fix * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix * fix * fix * fix * fix * fix * fix * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix * fix * fix * fix --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: duanjunwen <935724073@qq.com>

flybird11111 added 2 commits September 12, 2024 03:48

hybrid support zbv

6911938

fix

e6da1aa

fix

flybird11111 requested a review from a team as a code owner September 12, 2024 03:59

[pre-commit.ci] auto fixes from pre-commit.com hooks

6d5b32b

for more information, see https://pre-commit.ci

duanjunwen reviewed Sep 12, 2024

View reviewed changes

colossalai/pipeline/schedule/zero_bubble_pp.py Outdated Show resolved Hide resolved

flybird11111 and others added 8 commits September 12, 2024 10:55

fix

4404775

Merge branch 'support-zbv' of github.com:flybird11111/ColossalAI into…

e993144

… support-zbv

[pre-commit.ci] auto fixes from pre-commit.com hooks

3feda3b

for more information, see https://pre-commit.ci

fix

6d0122d

Merge branch 'support-zbv' of github.com:flybird11111/ColossalAI into…

24727de

… support-zbv

Update zero_bubble_pp.py

9802a7d

fix

9c59e6c

fix-ci

37d9623

flybird11111 force-pushed the support-zbv branch 3 times, most recently from d4287a6 to 37d9623 Compare September 13, 2024 06:34

fix

5965f8b

[pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci fix

flybird11111 force-pushed the support-zbv branch from a67a441 to 5965f8b Compare September 13, 2024 08:25

pre-commit-ci bot and others added 12 commits September 13, 2024 08:26

[pre-commit.ci] auto fixes from pre-commit.com hooks

f99fc6d

for more information, see https://pre-commit.ci

fix

e169f97

Merge branch 'support-zbv' of github.com:flybird11111/ColossalAI into…

1684d6d

… support-zbv

[pre-commit.ci] auto fixes from pre-commit.com hooks

7f78272

for more information, see https://pre-commit.ci

fix

885ace7

Merge branch 'support-zbv' of github.com:flybird11111/ColossalAI into…

95c4b31

… support-zbv

fix

629c76d

[pre-commit.ci] auto fixes from pre-commit.com hooks

1fcc3a6

for more information, see https://pre-commit.ci

fix

88b068f

Merge branch 'support-zbv' of github.com:flybird11111/ColossalAI into…

6550e80

… support-zbv

fix

0509712

fix

28f581f

flybird11111 and others added 25 commits September 25, 2024 02:33

fix

b55279c

[pre-commit.ci] auto fixes from pre-commit.com hooks

3efd8d4

for more information, see https://pre-commit.ci

Update zero_bubble_pp.py

1839030

fix

433c8a9

fix-ci

3fb1e42

fix

cd2e34b

[pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci fix

fix

4e0f212

[pre-commit.ci] auto fixes from pre-commit.com hooks

fa358b2

for more information, see https://pre-commit.ci

fix

0bde0bf

[pre-commit.ci] auto fixes from pre-commit.com hooks

a2f187b

for more information, see https://pre-commit.ci

fix

a9ac438

fix

8c3cdea

fix

f07b305

fix

29ec566

fix

b57e78d

fix

083ea31

fix

83d0766

fix

d3e83c3

Merge branch 'feature/zerobubble' into support-zbv

8ad2b72

[pre-commit.ci] auto fixes from pre-commit.com hooks

9b3c266

for more information, see https://pre-commit.ci

fix

f288930

Merge branch 'support-zbv' of github.com:flybird11111/ColossalAI into…

454e236

… support-zbv

fix

93557f5

fix

90a82e2

fix

7403580

ver217 approved these changes Sep 27, 2024

View reviewed changes

ver217 merged commit af6aa9e into hpcaitech:feature/zerobubble Sep 27, 2024
3 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[plugin]hybrid support zero bubble pipeline #6060

[plugin]hybrid support zero bubble pipeline #6060

flybird11111 commented Sep 12, 2024

[plugin]hybrid support zero bubble pipeline #6060

[plugin]hybrid support zero bubble pipeline #6060

Conversation

flybird11111 commented Sep 12, 2024

📌 Checklist before creating the PR

🚨 Issue number

📝 What does this PR do?

💥 Checklist before requesting a review

⭐️ Do you enjoy contributing to Colossal-AI?