Read shapes in from pandas dataframe

[shane-breeze]

I kept converting pandas dataframes into TH1s for my binned shape fits so instead I've included this conversion in combine and others might find it useful (shapes can be saved in human readable csv/json files or even excel spreadsheets e.g.).

Changed ShapeTools.py to interpret files with the extensions [".csv", ".json", ".html", ".pkl", ".xlsx", ".h5", ".parquet"] as a pandas dataframe (see here for IO). Any other extensions are dealt with as before, i.e. as ROOT files. Note that multiindexed dataframes are used and some file extensions need to be converted to multiindex where all but the last 2 columns are used for indexing.

DataFrameWrapper.py adds a class to wrap pandas dataframe so there's a Get method which acts in a similar way to ROOT::TFile::Get in return TH1s.

Example csv file included and the following commands gave the same results (apart from file names/CP time):

combine -M MultiDimFit --algo singles data/tutorials/shapes/simple-shapes-TH1.txt -v 4
combine -M MultiDimFit --algo singles data/tutorials/shapes/simple-shapes-df.txt -v 4

Excel spreadsheets depends on openpyxl and xlrd, parquet depends on pyarrow or fastparquet, and hdf depends on pytables (can be installed through pip).

This can be extended to unbinned fits by having a similar multiindex for channel/process categorisation and a column(s) with the observable(s).

[nucleosynthesis]

Thanks @shane-breeze . Will this also support the use of the autoMCStats directive in the datacards?

[shane-breeze]

Yes. There's a variance column in the dataframe that fills the error of the histograms. Just tested the simple shapes card with autoMCStats and got the same results with TH1s and the dataframe.

[shane-breeze]

Added a cast for the index selection in the datacard to the index dtypes. This is taken as a string inside the datacard but the dataframe can have int, float, ...

[nsmith-]

Just wondering about the label, what work needs to be done on this?

[amarini]

@nsmith- , the action items on this PR were the following in the last discussion:

1. resolve the conflicts
2. add a warning when pandas shapes are loaded that non all the features may be available

[amarini]

@shane-breeze , can you allow editing from maintainers?

[shane-breeze]

@amarini, I have unarchived my fork of this repository. It should now be writable.

[nsmith-]

non all the features may be available

autoMCStats is available (I checked it still works), so I think there is no missing feature. Unbinned data is anyway detected by the output of getShape so it shouldn't cause any datacard-level requirement to fail iiuc.

[hcombbot]

Pull Request Test.
Summary
========
Running options:
* MODE : cmssw
* COMBINE_TAG : 102x
* COMBINE_REPO : cms-analysis
* COMBINE_MERGE : shane-breeze/shapes-df
* GITHUB_PR : 548


Ratio to reference values:
--------
| comb_2019_hbb_boosted_standalone | comb_2019_hgg | comb_2019_hmm | comb_2019_htt | comb_2019_hww | comb_2019_tth_hbb | comb_2019_tth_hgg | comb_2019_tth_multilepton | comb_2019_vh_htt | comb_2019_vhbb | comb_2019_vhbb2017 |
| ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------ |
| 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |

You can find more detail at https://gitlab.cern.ch/cms-hcg/performances/ci/-/pipelines/3810969