-
Notifications
You must be signed in to change notification settings - Fork 6
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Filter positivity datapoints in hyperopt #2145
base: master
Are you sure you want to change the base?
Conversation
@Cmurilochem, are you actually including penalties in losses? nnpdf/n3fit/runcards/hyperopt_studies/restricted_search_space_renew_hyperopt.yml Line 131 in e37aa4e
At some point in then draft you mentioned that this is not the case. |
Hi @Radonirinaunimi I'm afraid most of our GPU budget is burnt and I'm not sure the folks at SURF are willing to give us more again... About the penalties in the hyperloss: I'm pretty sure they are excluded. Also, I received an invite for the slack channel but it won't allow me in! |
Hi @goord, I hope it was not our runs that burnt the GPUs 😬 For the paper, I think it would be realistic to only perform 250 trials. This is because from the PDF point of view we are doing a proof of concept, and we use restricted parameter space anyway. Do you think this would not even be possible? At some point, @Cmurilochem was planning to run 3 more sets of 250.
Ok, good! As it should be. I will modify the card here.
Hmhm, what is the message that you received? Maybe @juanrojochacon knows how to solve this? |
Well we can at least run a 5-day job, there is enough budget for that. In the meantime we can explore our options for more compute (Leonardo or new pilot project on Snellius?). @Cmurilochem maybe you can find the time to start a job? Regarding the slack: tried again and now it works |
yes @goord @Cmurilochem Slack has been a mess in the last few weeks but it is sorted out now, back to our Pro plan so all communication can proceed via there as usual now. Thanks! |
Hi @Radonirinaunimi and @goord. Yes. I excluded penalties in all runs. So, if this is the problem, nothing to be worried about. Also, @goord is right. We have a limited budget and I suspect that we left a 3-4 days job. Tomorrow I am back home and will submit it again. But we currently have more than 250 trials for sure. |
Perfect, thanks! Please use this branch for the runs.
I fear that we cannot use those, unfortunately. But we should always make backups of them, just in case. |
Hi @Radonirinaunimi and @goord. Just submitted the new hyperopt from this branch; we currently have just ~3.5 days of budget. I am currently on holidays with family, but will find some time to give you feedback on the progress of the calculation. |
n3fit/runcards/hyperopt_studies/restricted_search_space_renew_hyperopt.yml
Outdated
Show resolved
Hide resolved
Unrelated: it looks like the polarized theories have been updated (cc, @giacomomagni, @scarlehoff)? Now the C-factors are no longer present. This is the reason why the tests are failing. |
b303842
to
d2feade
Compare
maybe I've forgot them during my last update... I'll check it EDIT: Something went wrong when removing the |
d25b744
to
6075488
Compare
Update various things for the hyperopt run card.
@Cmurilochem, @goord, I fear that we have to re-run the hyperopt, again. The previous models fail badly the positivity selection, and the following "might" fix that.