[WIP] Branch for closure test chi2 table #109

ecole41 · 2025-12-28T20:28:53Z

This branch edits simunet_analysis.py to include a write_datasets_chi2_csv validphys function which calculates the $\chi^2$ per dataset using contaminated level 0 data and predictions produced with fitted PDF and mean BSM factors.
The load_datasets_contamination function is also edited to be able to handle multiple contamination parameters.

To do:

Use correct covariance matrix for $\chi^2$ calculation

This is added here as the delta_chi2_table function is unable to read pineapple FK-tables as below:

When trying to produce a chi2 table for a closure test fit, I get this error when loading the underlying pdf in produce_fitunderlyinglaw:

[ERROR]: Bad configuration encountered:
Bad input type for parameter 'pdf': Value 'NNPDF40_nnlo_as_01180' is not of type str, but of type 'PDF'.

After editing this function to parse this, I then receive an error when trying to load the new pineapple FK-tables, I have been trying to edit this to work around this bug but haven't yet found a full solution. With this current branch I am getting this error:

  File "/Users/ellacole/codes/simunet/simunet_git/SIMUnet/validphys2/src/validphys/core.py", line 690, in load
    loaded_data = dataset.load()
  File "/Users/ellacole/codes/simunet/simunet_git/SIMUnet/validphys2/src/validphys/core.py", line 531, in load
    fkset = FKSet(FKSet.parseOperator(self.op), fktables)
  File "/Users/ellacole/miniconda3/envs/simunet_mac/lib/python3.9/site-packages/NNPDF/nnpdf.py", line 3516, in __init__
    _nnpdf.FKSet_swiginit(self, _nnpdf.new_FKSet(*args))
TypeError: Wrong number or type of arguments for overloaded function 'new_FKSet'.
  Possible C/C++ prototypes are:
    NNPDF::FKSet::FKSet(NNPDF::SigmaOp,std::vector< NNPDF::FKTable *,std::allocator< NNPDF::FKTable * > > const &)
    NNPDF::FKSet::FKSet(NNPDF::FKSet const &)
    NNPDF::FKSet::FKSet(NNPDF::FKSet const &,std::vector< int,std::allocator< int > > const &)

I was wondering if anyone might know a cleaner way to load the new FK-tables here as it might have been done before.

For reference, this is the runcard I am using:

dataset_inputs:
# ----------------------------------------------------------------------------------------------
# DIS sector
- {dataset: NMCPD_dw_ite, frac: 0.75}
- {dataset: NMC, frac: 0.75}
- {dataset: SLACP_dwsh, frac: 0.75}
- {dataset: SLACD_dw_ite, frac: 0.75}
- {dataset: BCDMSP_dwsh, frac: 0.75}
- {dataset: BCDMSD_dw_ite, frac: 0.75}
- {dataset: CHORUSNUPb_dw_ite, frac: 0.75}
- {dataset: CHORUSNBPb_dw_ite, frac: 0.75}
- {dataset: NTVNUDMNFe_dw_ite, frac: 0.75, cfac: [MAS]}
- {dataset: NTVNBDMNFe_dw_ite, frac: 0.75, cfac: [MAS]}
- {dataset: HERACOMBNCEM, frac: 0.75}
- {dataset: HERACOMBNCEP460, frac: 0.75}
- {dataset: HERACOMBNCEP575, frac: 0.75}
- {dataset: HERACOMBNCEP820, frac: 0.75}
- {dataset: HERACOMBNCEP920, frac: 0.75}
- {dataset: HERACOMBCCEM, frac: 0.75}
- {dataset: HERACOMBCCEP, frac: 0.75}
- {dataset: HERACOMB_SIGMARED_C, frac: 0.75}
- {dataset: HERACOMB_SIGMARED_B, frac: 0.75}
- {dataset: CDFZRAP_NEW, frac: 0.75, cfac: [QCD]}
- {dataset: D0ZRAP_40, frac: 0.75, cfac: [QCD]}
- {dataset: D0WMASY, frac: 0.75, cfac: [QCD]}
# ----------------------------------------------------------------------------------------------
# Others
- {dataset: ATLAS_WZ_TOT_13TEV, frac: 0.75, cfac: [NRM, QCD]}
- {dataset: ATLAS_WP_JET_8TEV_PT, frac: 0.75, cfac: [QCD]}
- {dataset: ATLAS_WM_JET_8TEV_PT, frac: 0.75, cfac: [QCD]}
- {dataset: ATLASPHT15_SF, frac: 0.75, cfac: [QCD, EWK]}
# ----------------------------------------------------------------------------------------------
# DRELL-YAN sector
- {dataset: ATLASWZRAP11CC, frac: 0.75, cfac: [QCD]}
- {dataset: ATLASWZRAP36PB, frac: 0.75, cfac: [QCD]}
- {dataset: ATLASWZRAP11CF, frac: 0.75, cfac: [QCD]}
- {dataset: ATLASZPT8TEVMDIST, frac: 0.75, cfac: [QCD], sys: 10}
- {dataset: ATLASZPT8TEVYDIST, frac: 0.75, cfac: [QCD], sys: 10}
- {dataset: CMSZDIFF12, frac: 0.75, cfac: [QCD, NRM], sys: 10}
- {dataset: CMSWEASY840PB, frac: 0.75, cfac: [QCD]}
- {dataset: CMSWMASY47FB, frac: 0.75, cfac: [QCD]}
- {dataset: CMSWMU8TEV, frac: 0.75, cfac: [QCD]}
- {dataset: LHCBZ940PB, frac: 0.75, cfac: [QCD]}
- {dataset: LHCBZEE2FB_40, frac: 0.75, cfac: [QCD]}
- {dataset: LHCBWZMU7TEV, frac: 0.75, cfac: [NRM, QCD]}
- {dataset: LHCBWZMU8TEV, frac: 0.75, cfac: [NRM, QCD]}
- {dataset: LHCB_Z_13TEV_DIMUON, frac: 0.75, cfac: [QCD]}
- {dataset: LHCB_Z_13TEV_DIELECTRON, frac: 0.75, cfac: [QCD]}
- {dataset: DYE886R_dw_ite, frac: 0.75, cfac: [QCD]}
- {dataset: DYE886P, frac: 0.75, cfac: [QCD]}
- {dataset: DYE605_dw_ite, frac: 0.75, cfac: [QCD]}
- {dataset: DYE906R_dw_ite, frac: 0.75, cfac: [ACC, QCD]}
- {dataset: ATLAS_DY_2D_8TEV_LOWMASS, frac: 0.75, cfac: [QCD]}
- {dataset: HLLHC_FWDY_14TEV_CCm_DIF_RAP, frac: 0.75, cfac: ['QCD'], new_commondata: True}
- {dataset: HLLHC_FWDY_14TEV_CCp_DIF_RAP, frac: 0.75, cfac: ['QCD'], new_commondata: True}
- {dataset: HLLHC_FWDY_14TEV_NC_DIF_RAP, frac: 0.75, cfac: ['QCD'], new_commondata: True}
- {dataset: ATLASDY2D8TEV, frac: 0.75, cfac: ['QCDEWK'], contamination: EFT_LO}
- {dataset: ATLASLOMASSDY11EXT, frac: 0.75, cfac: [QCD]}
- {dataset: CMSDY2D11, frac: 0.75, cfac: ['QCD'], contamination: EFT_LO}
- {dataset: CMSDY1D12, frac: 0.75, cfac: ['QCD', 'EWK'], contamination: EFT_LO}
- {dataset: ATLASZHIGHMASS49FB, frac: 0.75, cfac: ['QCD'], contamination: EFT_LO} # HMDY
- {dataset: CMS_HMDY_13TEV, frac: 0.75, cfac: ['QCD', 'EWK'], contamination: EFT_LO}
- {dataset: HLLHC_HMDY_NC_EL_FINAL, frac: 0.75, cfac: ['QCD', 'EWK'], contamination: EFT_LO} # HLLHC projections
- {dataset: HLLHC_HMDY_NC_MU_FINAL, frac: 0.75, cfac: ['QCD', 'EWK'], contamination: EFT_LO}
- {dataset: HLLHC_HMDY_CC_EL_FINAL, frac: 0.75, cfac: ['QCD', 'EWK'], contamination: EFT_LO}
# ----------------------------------------------------------------------------------------------
##### Top sector #####
# ttbar
- {dataset: ATLASTTBARTOT7TEV, frac: 0.75, cfac: [QCD], contamination: 'EFT_LO'}
- {dataset: ATLASTTBARTOT8TEV, frac: 0.75, cfac: [QCD], contamination: 'EFT_LO'}
- {dataset: ATLAS_TTB_DIFF_8TEV_LJ_TTRAPNORM, frac: 0.75, cfac: [QCD], contamination: 'EFT_LO'}
- {dataset: ATLAS_TOPDIFF_DILEPT_8TEV_TTMNORM, frac: 0.75, cfac: [QCD], contamination: 'EFT_LO'}
- {dataset: ATLAS_TTBAR_8TEV_LJETS_TOTAL, frac: 0.75, cfac: [QCD], contamination: 'EFT_LO'}
- {dataset: ATLAS_TTBAR_13TEV_TTMNORM, frac: 0.75, cfac: [QCD], contamination: 'EFT_LO'}
- {dataset: ATLAS_TTBAR_13TEV_HADRONIC_2D_TTM_ABSYTTNORM, frac: 0.75, cfac: [QCD], contamination: 'EFT_LO'}
- {dataset: ATLAS_TTBAR_13TEV_LJETS_TOTAL, frac: 0.75, cfac: [QCD], contamination: 'EFT_LO'}
- {dataset: ATLAS_TTBAR_13TEV_DILEPTON_TOTAL, frac: 0.75, cfac: [QCD], contamination: 'EFT_LO'}
- {dataset: ATLAS_TTBAR_13TEV_HADRONIC_TOTAL, frac: 0.75, cfac: [QCD], contamination: 'EFT_LO'}
- {dataset: CMSTTBARTOT5TEV, frac: 0.75, cfac: [QCD], contamination: 'EFT_LO'}
- {dataset: CMSTTBARTOT7TEV, frac: 0.75, cfac: [QCD], contamination: 'EFT_LO'}
- {dataset: CMSTTBARTOT8TEV, frac: 0.75, cfac: [QCD], contamination: 'EFT_LO'}
- {dataset: CMSTTBARTOT13TEV, frac: 0.75, cfac: [QCD], contamination: 'EFT_LO'}
- {dataset: CMSTOPDIFF8TEVTTRAPNORM, frac: 0.75, cfac: [QCD], contamination: 'EFT_LO'}
- {dataset: CMS_TTBAR_2D_DIFF_MTT_TTRAP_NORM, frac: 0.75, cfac: [QCD], contamination: 'EFT_LO'}
- {dataset: CMS_TTB_DIFF_13TEV_2016_2L_TTMNORM, frac: 0.75, cfac: [QCD], contamination: 'EFT_LO'}
- {dataset: CMS_TTBAR_13TEV_TTMNORM, frac: 0.75, cfac: [QCD], contamination: 'EFT_LO'}
- {dataset: CMS_TTBAR_13TEV_LJETS_TOTAL, frac: 0.75, cfac: [QCD], contamination: 'EFT_LO'}
- {dataset: HLLHC_TTBAR_14TEV_LJ_DIF_MTTBAR, frac: 0.75, cfac: [QCD], contamination: 'EFT_LO', new_commondata: True}
# ttbar ASY
- {dataset: ATLAS_TTBAR_8TEV_ASY, frac: 0.75, cfac: [QCD], contamination: 'EFT_LO'}
- {dataset: ATLAS_TTBAR_13TEV_ASY_2022, frac: 0.75, cfac: [QCD], contamination: 'EFT_LO'}
- {dataset: CMS_TTBAR_8TEV_ASY, frac: 0.75, cfac: [QCD], contamination: 'EFT_LO'}
- {dataset: CMS_TTBAR_13TEV_ASY, frac: 0.75, cfac: [QCD], contamination: 'EFT_LO'}
- {dataset: ATLAS_CMS_TTBAR_8TEV_ASY, frac: 0.75, cfac: [QCD], contamination: 'EFT_LO'}
# ttZ
- {dataset: ATLAS_TTBARZ_8TEV_TOTAL, frac: 0.75, contamination: 'EFT_LO'}
- {dataset: ATLAS_TTBARZ_13TEV_TOTAL, frac: 0.75, cfac: [QCD], contamination: 'EFT_LO'}
- {dataset: ATLAS_TTBARZ_13TEV_PTZNORM, frac: 0.75, contamination: 'EFT_LO'}
- {dataset: CMS_TTBARZ_8TEV_TOTAL, frac: 0.75, contamination: 'EFT_LO'}
- {dataset: CMS_TTBARZ_13TEV_TOTAL, frac: 0.75, cfac: [QCD], contamination: 'EFT_LO'}
- {dataset: CMS_TTBARZ_13TEV_PTZNORM, frac: 0.75, contamination: 'EFT_LO'}
# ttW
- {dataset: ATLAS_TTBARW_8TEV_TOTAL, frac: 0.75, contamination: 'EFT_LO'}
- {dataset: ATLAS_TTBARW_13TEV_TOTAL, frac: 0.75, cfac: [QCD], contamination: 'EFT_LO'}
- {dataset: CMS_TTBARW_8TEV_TOTAL, frac: 0.75, contamination: 'EFT_LO'}
- {dataset: CMS_TTBARW_13TEV_TOTAL, frac: 0.75, cfac: [QCD], contamination: 'EFT_LO'}
# Single top # NOT IMPACTED BY SMEFT
- {dataset: ATLAS_SINGLETOP_TCH_7TEV_T, frac: 0.75, cfac: [QCD]}
- {dataset: ATLAS_SINGLETOP_TCH_7TEV_TB, frac: 0.75, cfac: [QCD]}
- {dataset: ATLAS_SINGLETOP_TCH_DIFF_7TEV_T_RAP_NORM, frac: 0.75, cfac: [QCD]}
- {dataset: ATLAS_SINGLETOP_TCH_DIFF_7TEV_TBAR_RAP_NORM, frac: 0.75, cfac: [QCD]}
- {dataset: ATLAS_SINGLETOP_TCH_8TEV_T, frac: 0.75, cfac: [QCD]}
- {dataset: ATLAS_SINGLETOP_TCH_8TEV_TB, frac: 0.75, cfac: [QCD]}  
- {dataset: ATLAS_SINGLETOP_TCH_DIFF_8TEV_T_RAP_NORM, frac: 0.75, cfac: [QCD]}
- {dataset: ATLAS_SINGLETOP_TCH_DIFF_8TEV_TBAR_RAP_NORM, frac: 0.75, cfac: [QCD]}
- {dataset: ATLAS_SINGLETOP_SCH_8TEV_TOTAL, frac: 0.75, cfac: [QCD]}
- {dataset: ATLAS_SINGLETOP_TCH_13TEV_T, frac: 0.75, cfac: [QCD]}
- {dataset: ATLAS_SINGLETOP_TCH_13TEV_TB, frac: 0.75, cfac: [QCD]}
- {dataset: ATLAS_SINGLETOP_SCH_13TEV_TOTAL, frac: 0.75, cfac: [QCD]}
- {dataset: CMS_SINGLETOP_TCH_TOT_7TEV, frac: 0.75, cfac: [QCD]}
- {dataset: CMS_SINGLETOP_TCH_8TEV_T, frac: 0.75, cfac: [QCD]}
- {dataset: CMS_SINGLETOP_TCH_8TEV_TB, frac: 0.75, cfac: [QCD]}
- {dataset: CMS_SINGLETOP_SCH_8TEV_TOTAL, frac: 0.75, cfac: [QCD]}
- {dataset: CMS_SINGLETOP_TCH_13TEV_T, frac: 0.75, cfac: [QCD]}
- {dataset: CMS_SINGLETOP_TCH_13TEV_TB, frac: 0.75, cfac: [QCD]}
- {dataset: CMS_SINGLETOP_TCH_13TEV_YTNORM, frac: 0.75, cfac: [QCD]}
# tW # NOT IMPACTED BY SMEFT
- {dataset: ATLAS_SINGLETOPW_8TEV_TOTAL, frac: 0.75}
- {dataset: ATLAS_SINGLETOPW_8TEV_SLEP_TOTAL, frac: 0.75}
- {dataset: ATLAS_SINGLETOPW_13TEV_TOTAL, frac: 0.75}
- {dataset: CMS_SINGLETOPW_8TEV_TOTAL, frac: 0.75}
- {dataset: CMS_SINGLETOPW_13TEV_TOTAL, frac: 0.75}
- {dataset: CMS_SINGLETOPW_13TEV_SLEP_TOTAL, frac: 0.75}
# ----------------------------------------------------------------------------------------------
# Jet sector
# - {dataset: ATLAS_1JET_8TEV_R06_DEC, frac: 0.75, cfac: [QCD]} # Removed to test contamination
# - {dataset: ATLAS_2JET_7TEV_R06, frac: 0.75, cfac: [QCD]}
# - {dataset: CMS_2JET_7TEV, frac: 0.75, cfac: [QCD]}
# - {dataset: CMS_1JET_8TEV, frac: 0.75, cfac: [QCD]}
# ----------------------------------------------------------------------------------------------

closuretest:
  filterseed: 0                               # Random seed to be used in filtering data partitions
  fakedata: true                              # true = to use FAKEPDF to generate pseudo-data
  fakepdf: "NNPDF40_nnlo_as_01180_1000"             # Theory input for pseudo-data
  errorsize: 1.0                              # uncertainties rescaling
  fakenoise: false                            # true = to add random fluctuations to pseudo-data
  rancutprob: 1.0                             # Fraction of data to be included in the fit
  rancutmethod: 0                             # Method to select rancutprob data fraction
  rancuttrnval: false                         # 0(1) to output training(valiation) chi2 in report
  printpdf4gen: false                         # To print info on PDFs during minimization
  contamination_parameters:
  - name: 'W'
    # value: 0.
    value: 0.00008
    linear_combination:
      Olq3: -15.94
  - name: 'Y'
    value: 0.
    # value: 0.00015
    linear_combination: 
      Olq1: 1.51606
      Oed: -6.0606
      Oeu: 12.1394
      Olu: 6.0606
      Old: -3.0394
      Oqe: 3.0394
  - name: 'Z'
    value: 0.
    # value: 0.0004
    linear_combination:
      O81qq: -164.705
      O8ut:  -164.705
      O8dt:  -164.705
      O8qt:  -164.705
      O8qu:  -164.705
      O8qd:  -164.705

seed: 0
rngalgo: 0

use_cuts: internal
use_fitcommondata: True

fits: 
  - {id: PDFcont_W8e-5_1000GeVcut, label: "PDFcont_W8e-5_1000GeVcut"}

pdfs: 
  - {id: PDFcont_W8e-5_1000GeVcut, label: "PDFcont_W8e-5_1000GeVcut"}


template_text: |
  {@delta_chi2_table@}

actions_:
  - report(main=True)

ecole41 · 2025-12-28T21:13:12Z

validphys2/src/validphys/core.py

+                fktable = p.load()
+            except Exception as e:
+                from validphys.pineparser import pineappl_reader
+                fktable = pineappl_reader(p)


I think that a fix would be here, I am not sure how to load the FK-table when it is a new pineappl format as p.load() currently gives an error. I believe this is because p.load() gives an object of type class 'NNPDF.nnpdf.FKTable' for appl tables but pineappl_reader(p) gives a the FKdata as a type 'validphys.coredata.FKTableData', this then causes an error when creating the FK set

^ This is for the delta_chi2_table function but it unnecessary for the new write_datasets_chi2_csv function

Edited loading fk tables

4b58c06

ecole41 added the bug Something isn't working label Dec 28, 2025

ecole41 commented Dec 28, 2025

View reviewed changes

ecole41 added 2 commits December 31, 2025 16:10

Added level0 bsm chi2 function

0af4ae5

Edited contamination function to include multiple parameters

1c366be

ecole41 added enhancement New feature or request and removed bug Something isn't working labels Dec 31, 2025

ecole41 requested a review from ElieHammou December 31, 2025 16:42

ecole41 and others added 10 commits December 31, 2025 16:50

revert files

626c7f3

Added PDF fit covmat

03912be

Full covmat

6280484

T0 and exp comparison

f67c9c7

Removed scaling

4497918

Added global chi2

c463cdc

bias=True in covmat

7d58b83

fixed bug

08dc2d1

Fixed load contamination bug

c2ddd58

Removed unused variables

dd75d95

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Branch for closure test chi2 table #109

[WIP] Branch for closure test chi2 table #109

Uh oh!

ecole41 commented Dec 28, 2025 •

edited

Loading

Uh oh!

ecole41 Dec 28, 2025 •

edited

Loading

Uh oh!

ecole41 Dec 31, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

[WIP] Branch for closure test chi2 table #109

Are you sure you want to change the base?

[WIP] Branch for closure test chi2 table #109

Uh oh!

Conversation

ecole41 commented Dec 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ecole41 Dec 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ecole41 Dec 31, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ecole41 commented Dec 28, 2025 •

edited

Loading

ecole41 Dec 28, 2025 •

edited

Loading