Global_steps taking too much space

[pratikkumar018]

Hi Team,
I am using huggingface Trainer & deepspeed(zero 3 stage).
When my model is training using deepspeed I see there is a checkpoint folder say checkpoint-800 & inside that there are various checkpoints like pytorch_model-00001-of-00003.bin & so on. Apart from that there is a global_step800 folder. My size of checkpoint-800 folder is 170GB out of which global_step800 folder occupies around 146 GB.

I believe my pytorch_model-00001-of-00003.bin,pytorch_model-00002-of-00003.bin & pytorch_model-00003-of-00003.bin are sufficient to use this model for inferencing and global_step folder can be deleted once that training step is completed. Is there any automated way or flag by which global_step800 folder will be automatically deleted by deepspeed.
I have having disk storage issue because of this larger size folder.

Thanks in advance.

[tjruwase]

@pratikkumar018, can you please confirm you are running with zero stage 3 because pytorch_model*.bin is a pytorch checkpoint naming convention. Moreover, zero stage 3 creates multiple checkpoint files, one for each training.

[pratikkumar018]

Yes, this is the config i m using.
{ "bf16": { "enabled": "auto" }, "optimizer": { "type": "AdamW", "params": { "lr": "auto", "betas": "auto", "eps": "auto", "weight_decay": "auto" } }, "scheduler": { "type": "WarmupLR", "params": { "warmup_min_lr": "auto", "warmup_max_lr": "auto", "warmup_num_steps": "auto" } }, "zero_optimization": { "stage": 3, "overlap_comm": true, "contiguous_gradients": true, "sub_group_size": 1e9, "reduce_bucket_size": 1e6, "stage3_prefetch_bucket_size": 0.94e6, "stage3_param_persistence_threshold": 1e4, "stage3_max_live_parameters": 1e9, "stage3_max_reuse_distance": 1e9, "stage3_gather_16bit_weights_on_model_save": true, "offload_optimizer": { "device": "cpu" }, "offload_param": { "device": "cpu" } }, "gradient_accumulation_steps": "auto", "gradient_clipping": "auto", "steps_per_print": 10, "train_batch_size": "auto", "train_micro_batch_size_per_gpu": "auto", "wall_clock_breakdown": false }

[tjruwase]

@pratikkumar018, regarding your main question, no, deepspeed does not delete files.

[pratikkumar018]

Ok. Got it. My issue was these global_step files occupy a lot of space & I only need pytorch*bin for inference. I can manually delete global_step folder. But was looking if there is an automated way/flag.

[YerongLi]

Did you find a way to delete the global_steps with deepspeed?