-
-
Notifications
You must be signed in to change notification settings - Fork 16.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add generator and worker seed #8602
Add generator and worker seed #8602
Conversation
for more information, see https://pre-commit.ci
@UnglvKitDe I'm seeing identical reproducible results with master and torch>=1.12.0. It doesn't seem we need any more modifications. i.e. in Colab with |
@UnglvKitDe are these DDP-specific requirements for reproducibility? |
@glenn-jocher Mh, so I implemented it in my version because it was recommended and I didn't want to take any risks with edge cases. On the other hand, because I once read about a bug like this. But it seems that it has been fixed in the meantime. I could not recreate the case, but it is also still described in tutorials like here that it can come to this problem especially with many workers. |
@UnglvKitDe I can verify that I am not seeing reproducible results with DDP trainings, we probably do need this PR merged then. I'm worried about the same seed on all workers though, I think this may impact augmentation by repeating the same augmentations. Perhaps we should set worker seeds equal to their RANK? |
@UnglvKitDe PR is merged. Thank you for your contributions to YOLOv5 🚀 and Vision AI ⭐ |
@UnglvKitDe tested DDP training but I do not see reproducible results after this PR unfortunately. It seems something else is missing. |
@glenn-jocher Each worker has a different worker_seed. Here is a small example:
|
@UnglvKitDe oh perfect, got it! I tried to implement an additional PR #8688 but still don't get reproducible DDP results even afterwards. Not sure unfortunately. |
* Add generator and worker seed * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update dataloaders.py * Update dataloaders.py * Update dataloaders.py Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Glenn Jocher <glenn.jocher@ultralytics.com>
Worker seed and generator inserted into the dataloader as described in #8601
🛠️ PR Summary
Made with ❤️ by Ultralytics Actions
🌟 Summary
Enhance randomness control in YOLOv5 dataloaders for more consistent training results. 🔄
📊 Key Changes
seed_worker
function to initialize the random seeds for dataloader workers.🎯 Purpose & Impact