Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Evolution --resume fix #4802

Merged
merged 1 commit into from
Sep 15, 2021
Merged

Evolution --resume fix #4802

merged 1 commit into from
Sep 15, 2021

Conversation

glenn-jocher
Copy link
Member

@glenn-jocher glenn-jocher commented Sep 15, 2021

Also disable /weights dir creation when evolving as no weights are saved and empty folder causes user expectations of weights.

πŸ› οΈ PR Summary

Made with ❀️ by Ultralytics Actions

🌟 Summary

Improved directory management and options handling during model training and evolution.

πŸ“Š Key Changes

  • Modified weight directory creation to accommodate evolution runs.
  • Tweaked command-line options to better manage the --evolve flag, ensuring separate runs do not overwrite each other.

🎯 Purpose & Impact

  • 🧰 Enhanced Evolve Feature: The change in directory creation ensures that when evolutionary training is underway (--evolve), the script sets up directories correctly without clashes, leading to a more organized training process.
  • πŸ› οΈ Improved Argument Handling: Adjustments to the options fix potential issues where resuming training could conflict with existing saved models, streamlining user experience and script robustness.

This update should not affect normal training runs but will make evolving models more reliable and user-friendly.

Also disable `/weights` dir creation when evolving as no weights are saved and empty folder causes user expectations of weights.
@glenn-jocher glenn-jocher self-assigned this Sep 15, 2021
@glenn-jocher glenn-jocher linked an issue Sep 15, 2021 that may be closed by this pull request
@glenn-jocher glenn-jocher merged commit 621b6d5 into master Sep 15, 2021
@glenn-jocher glenn-jocher deleted the fix/evolve_resume branch September 15, 2021 11:13
@glenn-jocher
Copy link
Member Author

glenn-jocher commented Sep 15, 2021

Resuming YOLOv5 πŸš€ evolution is a bit different than resuming a normal training run with python train.py --resume. If you started an evolution run which was interrupted, or finished normally, and you would like to continue for additional generations where you left off, then you pass --resume and specify the --name of the evolution you want to resume, i.e.:

Start Evolution

Assume you evolve YOLOv5s on COCO128 for 2 epochs for 3 generations:

python train.py --epochs 10 --data coco128.yaml --weights yolov5s.pt --evolve 3

If this is your first evolution a new directory runs/evolve/exp will be created to save your results.

# β”œβ”€β”€ yolov5
#     └── runs
#         └── evolve
#             └── exp  ← evolution saved here

Start a Second Evolution

Now assume you want to start a completely separate evolution: YOLOv5s on VOC for 5 epochs for 3 generations. You simply start evolving, and your new evolution will again be logged to a new directory runs/evolve/exp2:

python train.py --epochs 5 --data VOC.yaml --weights yolov5s.pt --evolve 3

You will now have two evolution runs saved:

# β”œβ”€β”€ yolov5
#     └── runs
#         └── evolve
#             β”œβ”€β”€ exp  ← first evolution (COCO128)
#             └── exp2  ← second evolution (VOC)

Notebook example: Open In Colab Open In Kaggle
Screenshot 2021-09-15 at 12 23 13

Resume an Evolution

If you want to resume the first evolution (COCO128 saved to runs/evolve/exp), then you use the same exact command you started with plus --resume --name exp, passing the additional number of generations you want, i.e. --evolve 30 for 30 more generations:

python train.py --epochs 10 --data coco128.yaml --weights yolov5s.pt --evolve 30 --resume --name exp

Evolution will run for an additional 30 generations and all new results will be added to the existing runs/evolve/exp/evolve.csv.

Good luck and let us know if you have any other questions!

CesarBazanAV pushed a commit to CesarBazanAV/yolov5 that referenced this pull request Sep 29, 2021
Also disable `/weights` dir creation when evolving as no weights are saved and empty folder causes user expectations of weights.
BjarneKuehl pushed a commit to fhkiel-mlaip/yolov5 that referenced this pull request Aug 26, 2022
Also disable `/weights` dir creation when evolving as no weights are saved and empty folder causes user expectations of weights.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

How to use --bucket parameter in colab?
1 participant