Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Intermediate merge of Aron's stuff #1775

Merged
merged 158 commits into from
Jul 17, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
158 commits
Select commit Hold shift + click to select a range
cf8afeb
Add theories 398 397 to theory.db
niclaurenti May 22, 2023
6b40d90
Name input layer xgrids
APJansen May 12, 2023
57f6552
Rename and simplify dense_me -> neural_network
APJansen May 12, 2023
d81814d
Rename i->i_replica and layer_seed->replica_seed
APJansen May 12, 2023
557b14c
Rename preprocessing -> prefactor
APJansen May 12, 2023
e087a38
Create apply_prefactor layer (can't name yet as it is created twice r…
APJansen May 12, 2023
ca187e4
Merge msr.py into layers/msr_normalization.py
APJansen May 12, 2023
97d7409
Turn msr_impose into class method
APJansen May 12, 2023
e0fb2f5
Use MSR_Normalization class attributes
APJansen May 12, 2023
c8e7e63
Create integration grid in MSR_Normalizations init
APJansen May 12, 2023
c5551f1
Renamings
APJansen May 12, 2023
6241614
Prepare call method in MSR_Normalization
APJansen May 12, 2023
4ecb1ba
Share more layers, add names
APJansen May 15, 2023
767977b
Create named layers in msr_normalization
APJansen May 15, 2023
d6ed37c
Add pdf_integrated step
APJansen May 15, 2023
396fc6f
Move computation of xdivided to init, move integration into call
APJansen May 15, 2023
472abed
remove tempcall
APJansen May 15, 2023
08c8876
Fix bug introduced after renaming preprocessing to prefactor
APJansen May 16, 2023
5337d18
Join neural network layers into their own NN_i model
APJansen May 16, 2023
ed85389
Clarify layer names
APJansen May 16, 2023
6bc651f
Revert all changes to MSR
APJansen May 17, 2023
96328a2
Rewrite msr into a model that takes as inputs the pdf, pdf on integra…
APJansen May 17, 2023
82cf95b
Set shape of integration grid to (None, 1) rather than (2000, 1) to d…
APJansen May 17, 2023
7b475ba
Revert renaming of Preprocessing to Prefactor (now use PreprocessingF…
APJansen May 17, 2023
7153e93
Clean up model creation code
APJansen May 17, 2023
dfb4eed
Reorganize input options, keeping effect the same
APJansen May 22, 2023
9dc1384
Make preprocessing layers dependent on which option is used (scaling,…
APJansen May 22, 2023
7c6d8ae
Fix msr with scaling
APJansen May 22, 2023
cbce181
Fix msr with scaling, shape of integration grid
APJansen May 22, 2023
3296aa4
Set shape of x=1 input to None so tensorflow displays proper shapes
APJansen May 22, 2023
853dd2e
Merge from master
APJansen May 23, 2023
fba7827
Add dummy ph_replica argument to MSR_Normalization layer
APJansen May 25, 2023
38e1984
Factor out scaler into own module scaler.py
APJansen May 23, 2023
d0065df
Do scaling of [0,1] to [-1, 1] inside scaler
APJansen May 23, 2023
2d2f8a0
Replace None layers with identity functions
APJansen May 23, 2023
08d26f2
Add option to save model plots (to current directory for now)
APJansen May 23, 2023
577d5d8
Add test for xDivide
APJansen May 25, 2023
f798c22
Update xDivide documentation to include v15 in the default settings, …
APJansen May 25, 2023
00a4974
Simplify xDivide layer
APJansen May 25, 2023
4c94100
Clarify xDivide documentation.
APJansen May 25, 2023
4f3d949
Turn off plotting of model for now
APJansen May 25, 2023
96e71b2
Merge branch 'master' into model_refactor
APJansen May 30, 2023
b279068
Manually change regression weights hdf5 file structure to fit new model
APJansen May 31, 2023
c1bb0ac
Manually change developing_weights.h5 file structure to fit new model
APJansen May 31, 2023
7d80a61
Remove inp option in pdfNN_layer_generator, instead enforcing adding …
APJansen Jun 1, 2023
38a1ccb
Incorporate Juan's comments
APJansen Jun 2, 2023
789a4c3
Incorporate Roy's comments
APJansen Jun 2, 2023
fe84657
Remove plotting of model from model_trainer
APJansen Jun 5, 2023
1f4537e
Remove unnecessary get_original layer from msr if scaler is not used
APJansen Jun 5, 2023
6353db3
Add model plot example script to documentation.
APJansen Jun 5, 2023
559d480
Merge master
APJansen Jun 5, 2023
0168360
Apply black to all changed files.
APJansen Jun 5, 2023
801366a
Remove sentence about plotting full model.
APJansen Jun 7, 2023
78c31b4
add nf=3 option
scarlehoff Jun 9, 2023
fe60a68
load only .exportgrid files corresponding to the fit
scarlehoff Jun 9, 2023
135862e
create a separate Q,nf entry when crossing a threshold
scarlehoff Jun 9, 2023
d63bda0
add a default q0 grid for nf0=3 and associated tests
scarlehoff Jun 9, 2023
70458a3
Merge branch 'master' into QED-tests
niclaurenti Jun 12, 2023
c6914a1
Merge branch 'master' into qed-tests
niclaurenti Jun 13, 2023
fcc3e8e
Add qed test in test_evolven3fit.py
niclaurenti Jun 13, 2023
5e7176e
Add qed test in test_fit
niclaurenti Jun 13, 2023
0af3004
Add qed test in test_fit 2
niclaurenti Jun 13, 2023
2256664
Refector QED tests
niclaurenti Jun 14, 2023
5a319fc
Fix test
niclaurenti Jun 14, 2023
9758018
Add fiatlux precision in the runcard
niclaurenti Jun 14, 2023
0a926fd
Rename abs_err -> eps_base
niclaurenti Jun 14, 2023
f99b77b
Remove now unused scatter_to_zero.
APJansen Jun 14, 2023
32e1a86
Use .json from linux machine
niclaurenti Jun 14, 2023
e2954ae
Fix very small bug in interp1d
niclaurenti Jun 14, 2023
3d1783d
Download theoryIDs
niclaurenti Jun 15, 2023
296e2d4
Add whitelines
niclaurenti Jun 16, 2023
22085f9
Merge branch 'qed-tests' of github.com:NNPDF/nnpdf into qed-tests
niclaurenti Jun 16, 2023
7585eea
Load theoryID from fit
niclaurenti Jun 16, 2023
57a5343
Use test theory in test_compute
niclaurenti Jun 16, 2023
077e2ed
Fix charm mass in test
niclaurenti Jun 16, 2023
388be7b
speed up photon computation in test
niclaurenti Jun 16, 2023
ddc57d3
Fix charm mass in test again
niclaurenti Jun 16, 2023
dba334a
Fix testing of alpha
niclaurenti Jun 16, 2023
3fa81ba
Fix test_compute
niclaurenti Jun 16, 2023
bb3a1fd
Add test on photon computation
niclaurenti Jun 16, 2023
3901ba5
Remove unused dependencies
niclaurenti Jun 16, 2023
f3de6b0
Polish test
niclaurenti Jun 16, 2023
7550e2d
Polish test again
niclaurenti Jun 16, 2023
98d919f
Pass test theory to test_structurefunctions.py
niclaurenti Jun 16, 2023
4b0ed9c
Run isort and black on test_structurefunctions.py
niclaurenti Jun 16, 2023
10dd7e2
modify the call to generate_block to make it compatible with eko 0.13.5
scarlehoff Jun 16, 2023
5ede6f4
Test init parameters of test_structurefunctions.py
niclaurenti Jun 16, 2023
52e9130
Run isort and black on test_structurefunctions.py
niclaurenti Jun 16, 2023
370d0dd
Use API to install pdfset
niclaurenti Jun 16, 2023
b97450d
Refactor test_structurefunctions.py and add docstrings
niclaurenti Jun 16, 2023
3e548a8
Small modification
niclaurenti Jun 16, 2023
b40db35
Use FIATLUX_DEFAULT from photon.compute
niclaurenti Jun 16, 2023
0958b08
Remove unused import
niclaurenti Jun 16, 2023
6a7ef27
Merge branch 'master' into fix_evolven3fit_for_nf=3
scarlehoff Jun 16, 2023
61c5bb9
Remove variable NNPDF40
niclaurenti Jun 19, 2023
53d7c73
Add comment
niclaurenti Jun 19, 2023
95f1e24
Allow categorical variables in smpdf plots
Zaharid Apr 17, 2023
ced3981
Add test and improve variables
Zaharid May 26, 2023
88a00f8
Remove figby in atlas total cross section
Zaharid May 26, 2023
9b861be
Improve comments
Zaharid Jun 19, 2023
e43cac7
Typo
Zaharid Jun 19, 2023
80a99fc
Leftover typo
Zaharid Jun 19, 2023
6089608
Update test image
Zaharid Jun 19, 2023
1e5f78e
Merge pull request #1754 from NNPDF/fix_evolven3fit_for_nf=3
niclaurenti Jun 19, 2023
502997f
Merge branch 'master' into qed-tests
niclaurenti Jun 19, 2023
d5a48f9
Check installation of error pdf in test
niclaurenti Jun 19, 2023
1e237da
Call black and isort
niclaurenti Jun 19, 2023
1e105cd
Fix last commit
niclaurenti Jun 19, 2023
dfc8892
Merge pull request #1715 from NNPDF/smpdfcategorical
Zaharid Jun 19, 2023
7eb0c4e
Merge pull request #1738 from NNPDF/qed-tests
niclaurenti Jun 19, 2023
2b8c806
Merge branch 'master' into model_refactor
APJansen Jun 20, 2023
0bd5582
Pass replica number as additional argument to impose_msr model
APJansen Jun 20, 2023
1071478
First tentative grouping
andreab1997 Jun 21, 2023
d6392ba
Add luminosity channels (#1762)
Zaharid Jun 22, 2023
acd0b06
Use function to generate fiatlux runcard in tests
niclaurenti Jun 22, 2023
40bc7bc
Call api inside tests
niclaurenti Jun 22, 2023
f579752
Merge pull request #1763 from NNPDF/fix-qed-tests
scarlehoff Jun 22, 2023
8dbc98d
Replace indexing with tf.gather since ph_replica is now a tensor
APJansen Jun 23, 2023
a0f6e09
gather -> op_gather
APJansen Jun 23, 2023
20833ca
Revert to commit 2b8c806e
APJansen Jun 24, 2023
6a1ce02
Set photon integral as explicit argument to normalization model
APJansen Jun 24, 2023
a67fdc3
Splitting of DY data sets into CC and NC - all data sets tested and w…
enocera Jun 25, 2023
e1983cb
Remove leftover comments
andreab1997 Jun 26, 2023
b43816a
Remove underscore in group names and add new groups output
andreab1997 Jun 26, 2023
7e17c3d
Merge branch 'master' into add_new_DY_group
andreab1997 Jun 27, 2023
4389a12
Merge pull request #1760 from NNPDF/add_new_DY_group
andreab1997 Jun 29, 2023
768c0c2
Add theory 527 and extend comments in theories 522, 523, 524, 525, 526
niclaurenti Jun 29, 2023
997be4e
Change the order of the process for the output
andreab1997 Jun 30, 2023
0310180
Merge pull request #1766 from NNPDF/add-527
niclaurenti Jul 3, 2023
c8d087b
Change also luxset with vp-nextfitruncard
niclaurenti Jul 3, 2023
2ecddfa
Merge pull request #1767 from NNPDF/qed-nextfitruncard
niclaurenti Jul 3, 2023
6617da2
Remove duplicate imports
APJansen Jul 5, 2023
5aa72fc
Remove duplicate imports
APJansen Jul 5, 2023
a80a77e
Add link to plot_model documentation
APJansen Jul 5, 2023
f82b407
Move photon integrals down to final loop
APJansen Jul 5, 2023
abf98bc
Add theoryids for alpha variation but same thcovmat
andreab1997 Jul 5, 2023
78d83d6
Add theory database with new theories
andreab1997 Jul 5, 2023
4847c56
Merge pull request #1768 from NNPDF/thunc_alphavar
andreab1997 Jul 5, 2023
80ef5d7
Simplify numpy_to_input, removing unused no_reshape and setting gridp…
APJansen Jul 5, 2023
ccfb7ce
Merge branch 'master' into model_refactor
APJansen Jul 6, 2023
0a4d684
Improve numpy_to_input documentation
APJansen Jul 7, 2023
94d6e80
remove mention that vp-comparefit doesn't always have to be the first…
RoyStegeman Jul 10, 2023
883cb19
Implement FkRotation as subclass of Rotation by rewriting using a rot…
APJansen Jul 10, 2023
e7a9140
Change 8 to 9 in docstring
APJansen Jul 10, 2023
0518c7c
Apply black and isort to all affected files
APJansen Jul 10, 2023
3954197
Merge pull request #1771 from NNPDF/update_fitting_docs
RoyStegeman Jul 10, 2023
e92b706
Update n3fit/src/n3fit/backends/keras_backend/operations.py
APJansen Jul 10, 2023
5bcf169
Restore comments in scaler
APJansen Jul 10, 2023
915d2d6
Merge pull request #1772 from NNPDF/rotation_matrix
APJansen Jul 10, 2023
1a9ce30
Add Optional type hint for arguments that can be None
APJansen Jul 12, 2023
9b6a675
Use numpy.typing
APJansen Jul 12, 2023
8566728
Raise error from original error
APJansen Jul 12, 2023
27f9357
Fix error when not imposing sumrule
APJansen Jul 12, 2023
52ddc1d
Fix consistency of lumy plot label
Zaharid Jul 12, 2023
f13a506
Change fiatlux_dis_F{2,L} -> FIATLUX_DIS_F{2,L}
niclaurenti Jul 13, 2023
4395179
Merge pull request #1774 from NNPDF/rename_fiatlux_dis
niclaurenti Jul 13, 2023
694417d
Merge branch 'master' into model_refactor
APJansen Jul 13, 2023
6ddab6e
Merge pull request #1734 from NNPDF/model_refactor
APJansen Jul 14, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
82 changes: 82 additions & 0 deletions buildmaster/filters/ATLASWRAP11.cc
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
/*
This file implements the W production subset of the ATLASWZRAP11CC data set.
This is required to separate CC DY from NC DY.
Implemented by ERN June 2023.
*/

#include "ATLASWRAP11CC.h"

// Central selection
void ATLASWRAP11CCFilter::ReadData()
{

// Opening files
fstream f1;

stringstream datafile("");
datafile << dataPath() << "rawdata/ATLASWZRAP11CC/wzrap11.dat";

f1.open(datafile.str().c_str(), ios::in);

if (f1.fail()) {
cerr << "Error opening data file " << datafile.str() << endl;
exit(-1);
}

// Dummy string
std::string dummy;

// Read the names of systematics
std::string *sysNames = new std::string[fNSys];
sysNames[0] = "UNCORR";
for (int i=0; i<5; i++) f1 >> dummy;
for (int i=0; i<fNSys; i++)
f1 >> sysNames[i];

const int nBins = 2;
const int ndataWZ[nBins] = {11,22}; // Data thresholds for W+ and W-
const double MWZ2[nBins]= {pow(MW,2.0), pow(MW,2.0)}; //Mass squared of W (+ and -)

int low_bin = 0;
for (int b = 0; b < nBins; b++)
{
for (int i = low_bin; i < ndataWZ[b]; i++)
{
double etamin, etamax;

// Kinematics
f1 >> dummy; f1 >> etamin; f1 >> etamax;
fKin1[i] = etamin + (etamax - etamin)/2.0;
fKin2[i] = MWZ2[b];
fKin3[i] = 7000;

// Observable
f1 >> fData[i];
fData[i] *= 1000; // pb -> fb

// Statistical errors - percentage with respect the observable
f1 >> fStat[i];
fStat[i] *= fData[i]*1e-2;

// Correlated systematic errors
for (int l = 0; l < fNSys; l++)
{
f1 >> fSys[i][l].mult;
fSys[i][l].type = MULT;
fSys[i][l].name = sysNames[l];
}

// Additive errors
for (int l = 0; l < fNSys; l++)
fSys[i][l].add = fSys[i][l].mult*fData[i]*1e-2;
}
// Update lowest point in bin
low_bin = ndataWZ[b];
}

delete[] sysNames;

f1.close();
}


130 changes: 130 additions & 0 deletions buildmaster/filters/ATLASWRAP36PB.cc
Original file line number Diff line number Diff line change
@@ -0,0 +1,130 @@
/*
This file implements the W production subset of the ATLASWZRAP36PB data set.
This is required to separate CC DY from NC DY.
Implemented by ERN June 2023.
*/

#include "ATLASWRAP36PB.h"

void ATLASWRAP36PBFilter::ReadData()
{
// Opening files
fstream fWZ[2];

stringstream datafile("");
datafile << dataPath() << "rawdata/ATLASWZRAP36PB/ATLAS-36pb-Wplrap.data";
fWZ[0].open(datafile.str().c_str(), ios::in);

if (fWZ[0].fail()) {
cerr << "Error opening data file " << datafile.str() << endl;
exit(-1);
}

stringstream datafile2("");
datafile2 << dataPath() << "rawdata/ATLASWZRAP36PB/ATLAS-36pb-Wmlrap.data";
fWZ[1].open(datafile2.str().c_str(), ios::in);

if (fWZ[1].fail()) {
cerr << "Error opening data file " << datafile2.str() << endl;
exit(-1);
}

// Starting filter
const double lcorr = 1.0187; // correction factor due to luminosity upgrade
const int ndataWZ[2] = {11,11}; //Number of data for W+, W-
const double convfac = lcorr*1000.; // Must multiply from pb to fb
const double MWZ2[2] = {pow(MW,2.0), pow(MW,2.0)}; //Mass squared of W (+ and -) and Z

string line;
int idat = 0;
double etamin,etamax,tmp;

for (int iWZ = 0; iWZ < 2; iWZ++)
{
// rapidity
getline(fWZ[iWZ],line);
istringstream lstream(line);
for (int i = 0; i < ndataWZ[iWZ]; i++)
{
lstream >> etamin >> etamax;
fKin1[idat+i] = etamin + (etamax-etamin)*0.5;
}

// M_W
for (int i = 0; i < ndataWZ[iWZ]; i++)
fKin2[idat+i] = MWZ2[iWZ];

// sqrt(s)
for (int i = 0; i < ndataWZ[iWZ]; i++)
fKin3[idat+i] = 7000;

// obs
getline(fWZ[iWZ],line);
istringstream lstream2(line);
for (int i = 0; i < ndataWZ[iWZ]; i++)
{
lstream2 >> fData[idat+i];
fData[idat+i] *= convfac;
}

// stat (%, converted later)
getline(fWZ[iWZ],line);
istringstream lstream3(line);
for (int i = 0; i < ndataWZ[iWZ]; i++)
lstream3 >> fStat[idat+i];

// uncorrelated sys
getline(fWZ[iWZ],line);
istringstream lstream4(line);
for (int i = 0; i < ndataWZ[iWZ]; i++)
{
lstream4 >> fSys[idat+i][0].mult;
fSys[idat+i][0].name = "UNCORR";
}

// total correlated sys (unused)
getline(fWZ[iWZ],line);

// total uncertainty (unused)
getline(fWZ[iWZ],line);

// correlated systematics
for (int isys = 2; isys < fNSys; isys++) //2 to skip uncorr and lumi
{
getline(fWZ[iWZ],line);
istringstream lstream(line);
lstream >> tmp;
for (int i = 0; i < ndataWZ[iWZ]; i++)
{
lstream >> fSys[idat+i][isys].mult;
ostringstream sysname;
sysname << "ATLASWZRAP36PB_" << isys-2;
fSys[idat+1][isys].name = sysname.str();
}
}

// luminosity: 3.4%
for (int i = 0; i < ndataWZ[iWZ]; i++)
{
fSys[idat+i][1].mult = 3.5;
fSys[idat+i][1].name = "ATLASLUMI10";
}

idat+=ndataWZ[iWZ];
}

// Convert additive uncertainties to absolute form
for (int i = 0; i < fNData; i++)
{
fStat[i] *= fData[i]*1e-2;
for(int l = 0; l < fNSys; l++)
{
fSys[i][l].type = MULT; // All systematics multiplicative
fSys[i][l].add = fSys[i][l].mult*fData[i]*1e-2;
}
}


fWZ[0].close();
fWZ[1].close();
}
84 changes: 84 additions & 0 deletions buildmaster/filters/ATLASZRAP11.cc
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
/*
This file implements the Z production subset of the ATLASWZRAP11CC data set.
This is required to separate NC DY from CC DY.
Implemented by ERN June 2023.
*/

#include "ATLASZRAP11CC.h"

// Central selection
void ATLASZRAP11CCFilter::ReadData()
{

// Opening files
fstream f1;

stringstream datafile("");
datafile << dataPath() << "rawdata/ATLASWZRAP11CC/wzrap11.dat";

f1.open(datafile.str().c_str(), ios::in);

if (f1.fail()) {
cerr << "Error opening data file " << datafile.str() << endl;
exit(-1);
}

// Dummy string
std::string dummy;

// Read the names of systematics
std::string *sysNames = new std::string[fNSys];
sysNames[0] = "UNCORR";
for (int i=0; i<5; i++) f1 >> dummy;
for (int i=0; i<fNSys; i++)
f1 >> sysNames[i];

const int nBins = 3;
const int ndataWZ[nBins] = {6,18,24}; // Data thresholds for (Z_low, Z_peak, Z_high) respectively
const double MWZ2[nBins]= {pow(56.0,2.0), pow(91.0,2.0), pow(133.0,2.0)}; //Mass squared of (Z_low, Z_peak, Z_high)

string line;
for(int iline=0; iline<23; iline++)
getline(f1,line);

int low_bin = 0;
for (int b = 0; b < nBins; b++)
{
for (int i = low_bin; i < ndataWZ[b]; i++)
{
double etamin, etamax;

// Kinematics
f1 >> dummy; f1 >> etamin; f1 >> etamax;
fKin1[i] = etamin + (etamax - etamin)/2.0;
fKin2[i] = MWZ2[b];
fKin3[i] = 7000;

// Observable
f1 >> fData[i];
fData[i] *= 1000; // pb -> fb

// Statistical errors - percentage with respect the observable
f1 >> fStat[i];
fStat[i] *= fData[i]*1e-2;

// Correlated systematic errors
for (int l = 0; l < fNSys; l++)
{
f1 >> fSys[i][l].mult;
fSys[i][l].type = MULT;
fSys[i][l].name = sysNames[l];
}

// Additive errors
for (int l = 0; l < fNSys; l++)
fSys[i][l].add = fSys[i][l].mult*fData[i]*1e-2;
}
// Update lowest point in bin
low_bin = ndataWZ[b];
}

delete[] sysNames;

f1.close();
}
Loading