Skip to content

Conversation

@MarcoColonna
Copy link
Contributor

@MarcoColonna MarcoColonna commented Jun 6, 2025

If you are creating this PR in order to submit a draft of your paper, please name your PR with Paper: <title>. An editor will then add a draft label; this will trigger GitHub Actions to run automated checks on your paper and build a preview. You may then work to resolve failed checks and ensure the preview build looks correct. If you have any questions, please tag the proceedings team in a comment on your PR with @scipy-conference/2025-proceedings.

See the project readme for more information.

Hi, I am Sanhita Joshi, @sanhitamj . I will serve as the editor for this submission. Reach out to me if any assistance needed.

@github-actions
Copy link

github-actions bot commented Jun 6, 2025

Curvenote Preview

Directory Preview Checks Updated (UTC)
papers/marco_colonna 🔍 Inspect 50 checks passed (6 optional) Oct 14, 2025, 6:41 PM

@rowanc1 rowanc1 added paper This indicates that the PR in question is a paper draft This triggers Curvenote Preview actions labels Jun 6, 2025
@MarcoColonna MarcoColonna changed the title Paper: ReDist, a python tool for model-agnostics binned-likelihood fits in High Energy Physics Paper: Redist, a python tool for model-agnostics binned-likelihood fits in High Energy Physics Jun 13, 2025
@ameyxd
Copy link
Contributor

ameyxd commented Jun 23, 2025

Inviting reviewers: @pranoy-ray and @yash91sharma

@ameyxd
Copy link
Contributor

ameyxd commented Jul 2, 2025

@sanhitamj will serve as editor for this paper.

A Negative-Logarithmic-Likelihood (NLL) fit is performed to find the optimal point, then further fits are applied over a grid of points in the phase space of the real and imaginary parts of the NP Wilson Coefficients.
For each point, the NP Wilson Coefficients are fixed, while the rest of the parameters, for instance the SM contributions, are free to float to maximize the NLL.
If the grid of phase space is defined with sufficiently fine granularity, this method allows inferring 2-Dimensional (2D) Confidence Intervals (CI) correctly, taking into account correlations among the parameters as shown in [Figure %s](#fig:scan_SLL).
:::{figure} ./figure1.png

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello, the figure1.png is not rendering along with the rest of the content.

I am using "myst start" to render the paper.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see the images appearing correctly. I am not a super expert of myst, do you have hints of where the issue might be?

where the hi_temp and lo_temp are obtained from the shapes of the contributions with different FFs or WCs injected.
An example of this usage, where the two alternative shapes hi_temp and lo_temp have been defined with respect to different injections of a FF parameter ($\Delta\rho^2$), is shown in [Figure %s](#fig:histsys_with_Hammer).

:::{figure} ./figure2.png

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same problem as the figure1.png above

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see the images appearing correctly. I am not a super expert of myst, do you have hints of where the issue might be?

What has been shown in this document represents one of the many applications, in HEP and beyond, of the Redist weighting method for model building.
Future efforts will integrate other popular theoretical backends in flavour physics like Flavio [@straub2018flaviopythonpackageflavour], finally allowing data analysts effortless use of, and possibly toggling between, all the different packages.

Fields outside HEP can benefit from the Redist method for any problem requiring the HistFactory model building and the flexibility given by the custom modifier.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1-2 concrete examples of scientific domains using this beyond HEP would be very helpful in making this more relevant to readers from other disciplines.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The method is, as pyhf, used mainly in HEP.
As Redist will grow in popularity we hope it will become used also outside of the field: possible usages are data-driven interpretation of anomalies in complex systems (minimizing dependencies on the model).

I add a phrase about this.

Further applications of the Redist package in different fields, along the lines of the Redist-HAMMER interface in High Energy Physics, to extend *pyhf* models are possible and straightforward to implement.
---

## Introduction

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The readability of this paper could be greatly increased by adding a system diagram explaining where Redist fits with other existing tools/libraries/concepts (HAMMER, pyhf, etc).

That would also highlight the gap in the system, which Redist is filling in. Also clarify what already exists and what is being created as a part of this paper.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I add a diagram as you suggested.
I include in the diagram showing where Redist is placed in the framework between HAMMER and pyhf. On the note of another comment I include also a small scheme of the nested structure of the package.

```python
import *pyhf*
import json
from redist import modifier

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add information about how can someone use/install redist?

Or link to any github repo or pypi librariy? If not then consider creating and adding one.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps adding a "Software Availability" section somewhere might be better.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I originally included a reference to the git repo but it got me error due to the missing doi.
I can maybe include the explicit link but I would like to make it a reference (seems like myst requires all the references to have a doi...)

Can we include the git repo as a reference without including a doi?

I include as a link in the next push




## Outlook

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are the limitations of Redist?

  • Cases where this tool might fail or perform poorly?
  • computational, pythonic, dependency limitations?
  • Types of physics analyses where the approach wouldn't be suitable?

You have mentioned some future improvements/integrations like Flavio, but clearly highlighting and listing limitations would be better.


Python is becoming increasingly popular in High Energy Physics (HEP) analysis due to its flexibility, ease of use and the growing number of scientific tool implementet on it.
At the same time, the HAMMER package is gaining traction in the community as a powerful framework to reinterpret previously analyzed datasets and explore BSM scenarios through FFs reweighting.
The Redist-HAMMER is the first full Pythonic interface between the HAMMER package and a fitting environment, by integrating it with the pyhf framework [@pyhf] [@pyhf_joss].

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are there any gaps between the HAMMER package functionalities and this pythonic interface?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hammer offers a very wide set of methods to address a wide set of studies one would like to do in Semileptonic Analysis.
One of the most important is the possibility of creating Histograms weighting the events to different Form Factor parametrizations, or NP injections.

Redist-HAMMER allows to use the Histograms in the pyhf fitting environment in a coherent and efficient way.
Hammer offers also other functionalities (for example the computation of the decay rate, that is used also in the Histogram creation) that can be used in an analysis using directly the Hammer-python-interface.

These are anyway characteristics of the Hammer-python-interface that we partially take advantage to in the Redist-HAMMER interface that are discussed in the HAMMER documentation.

At the same time, the HAMMER package is gaining traction in the community as a powerful framework to reinterpret previously analyzed datasets and explore BSM scenarios through FFs reweighting.
The Redist-HAMMER is the first full Pythonic interface between the HAMMER package and a fitting environment, by integrating it with the pyhf framework [@pyhf] [@pyhf_joss].
This allows HAMMER-processed samples to be used directly within pyhf’s binned-likelihood models [@Cranmer:2012sba], via the Redist modifiers of the likelihood, encoding the BSM and FF dependence of the shapes as defined by the HAMMER theoretical backend.
Other efforts, such as the RooHammerModel [@Garc_a_Pardi_as_2022], have been spent to address similar functionalities in the C++-based RooFit-HistFactory framework [@Verkerke:2003ir].

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does this pythonic implementation compare to C++ implementation in terms of performance and flexibility?

Python should be easy to use and adapt, but any drawbacks?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I add a phrase on the advantages of Python.

The two interfaces (ours and RooHammerModel) serves similar tasks but with few differences:

  • obviously the fitting interfaces are different (RooFit vs pyhf)
  • some design decisions are different, in redist-HAMMER we are also taking care of the other histograms (non weighted) that composes the template-model while the RooHammerModel is designed to take care only of the Hammer-processed histograms (the rest of the modeling is left to the reader)

The drawbacks might be related to the C++ vs python comparisons. Possibly we can discuss further on these topics.

More generically, following the same prescription, the pyhf model is able to handle arbitrarily complex degrees of freedom where the effect of changing the templates is completely determined by the initial and alternative hypothesis we define as simple python functions.

This method has direct application in HEP analysis, in particular for the reinterpretation of semileptonic analyses.
Taking, for example, the semileptonic modes of the beauty mesons $\overline{B^0} \to D^{*+}\mu^{-}\overline{\nu_{\mu}}$ and $\overline{B^0} \to D^{*+}\tau^{-}\overline{\nu_{\tau}}$, notably, tesion exist between the expected ratio of the abundance of the two decay modes and the measured ratio of the respective Branching Fractions (BF):

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo in "tesion" ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed in the next push

The HAMMER-weighted sample can be accessed, and a template fit can be applied to study the sensitivity of a given Wilson Coefficient given a certain statistical power, which is the number of events of the template dataset used in the fit.
The phase space of the real and imaginary parts of the Wilson Coefficient associated with a scalar-like NP contribution, coupling particles and antiparticles in the same chiral state, has been inspected using a template generated with the RapidSim tool [@Cowan_2017] and containing $B^0 \to D^{*}\tau\nu_{\tau}$ events.
The fit has been done using a pseudo-dataset identical to the template itself, with different injections of NP, to demonstrate the application of the Redist-HAMMER interface in a small and simple sensitivity study context.
This represents a simple and practical example of a use case in flavour physics data analysis, demostrating the functionalities of maximum-likelihood-fits, generation of pseudodatasets and phase space scanning for confidence level determination.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo in "demostrating"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed in the next push


## Outlook
We present the Redist-HAMMER interface, an extension of the already existing Redist module to interface HAMMER with the fitting environment of pyhf.
The package has been developed to address the challenge of direct NP measurements through the reinterpretation of the LHCb and Belle datasets, which will be a crucial step for getting a further and deeper understanding of tensions of HEP mesurements with SM predictions.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo in "mesurements".

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed in the next push

"formfactors": {
"delta_RhoSq": 1.5,
"delta_cSt": 2.0,
...

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this meant to specify other optional fields which are not shown?

It would be great to add minimal, complete file somewhere in the paper, or a link to one.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The "..." are representing the other (possible) Form Factors to be used in the Histogram weighting.
The names of the form factors (delta_RhoSq,...) are dependent on the parametrization you are using as their meaning.

The number of Form Factors (and later Wilson Coefficients) can be very large, one can also have just a subset in the definition of the json file (you just do not allow yourself to use the others that you are not specifying in the fit later).

I would personally avoid to add 10 rows for 10 Form Factors, and 10 rows for 10 Wilson Coefficients in the json, I would suggest to either:

  • leave things like they are, maybe specifying in the text that the Form Factors are meant to be defined there.
  • remove the "..." and assume that one (in the example) wants to use only the RhoSq and cSt form factors parameters

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the explanation, that makes sense. I would suggest that we remove the "...". it can add further confusion for the reader. And add a text stating that other Form Factors can be used as well.

@pranoy-ray
Copy link
Collaborator

pranoy-ray commented Jul 9, 2025

@MarcoColonna here is my review:

  1. Improve Clarity and Structure by visualizing the architecture: Add a diagram to illustrate the nested class structure of Redist-HAMMER. This will help clarify the "seemingly convoluted" design for any general reader. Next, connect theory to practice: Explicitly link the mathematical likelihood formulas in Section 2 to the practical Python code and JSON configurations shown later. Next, please unify terminology: Use a single, clear term like "parameters of interest" to consistently refer to Wilson Coefficients, Form Factors, and other New Physics degrees of freedom for better reader comprehension.

  2. Need to improve Technical Details and Figures: Please clarify code snippets: Briefly explain the origin of placeholder variables (like Nnorm in the new_params definition) in the code examples to provide better context. Next, analyze Figure 1 results: State in the body text that the fit in Figure 1 correctly identifies the injected New Physics values, which demonstrates the tool's effectiveness. Finally, explain Figure 2's Power: Emphasize in the text that Figure 2 showcases the seamless integration of a complex theoretical uncertainty (a Form Factor parameter) as a standard histosys nuisance parameter within pyhf.

  3. Overall Content and Impact needs some updation: Please work on strengthening the overall outlook: Make the outlook more concrete by speculating on specific benefits of combining datasets, such as simultaneously constraining shared Form Factor parameters while measuring the Wilson Coefficients relevant to the R(D*) anomaly. Next, provide broader examples: To substantiate the claim of broader impact, briefly mention a hypothetical, non-HEP application, such as in astrophysics or systems biology.

The Redist-HAMMER ([github repository](https://github.com/lorenzennio/redist)) is the first full Pythonic interface between the HAMMER package and a fitting environment, by integrating it with the *pyhf* framework [@pyhf] [@pyhf_joss].
This allows HAMMER-processed samples to be used directly within pyhf’s binned-likelihood models [@Cranmer:2012sba], via the Redist modifiers of the likelihood, encoding the BSM and FF dependence of the shapes as defined by the HAMMER theoretical backend.
Other efforts, such as the RooHammerModel [@Garc_a_Pardi_as_2022], have been spent to address similar functionalities in the C++-based RooFit-HistFactory framework [@Verkerke:2003ir].
Redist-HAMMER builds on this idea by bringing it into the Python ecosystem, providing a flexible reinterpretation tool that supports combination and fitting workflows entirely in Python allowing for a easy to start and felxible interface to apply reinterpretation of HEP datasets.
Copy link

@sanhitamj sanhitamj Jul 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo: ... an easy to start and flexible interface...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed in the next push

System diagram of the Redist package allowing HAMMER processed samples to be used in *pyhf* inference. The nested structure of the Redist-HAMMER interface is summarized.
:::

The most inner object in the nested structure is the HammerCacher, a class that uses the Python HAMMER interface functions to read from a file a HAMMER-processed histograms, which are created applying HAMMER to the simulations and contain the HAMMER weighting parameters (generally ~10-20 new degrees of freedom for describing the FF parametrization and BSM injection) and the mathematical tools to extrapulate the variations of the shape as function of them.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo: ... tools to extrapolate the variations ...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed in the next push

The HammerCacher stores the bin content of the HAMMER-processed histogram and updates it on the basis of any set of NP and FF parameters that were defined at HAMMER processing time.
To avoid useless iterations with the HAMMER weighting, a caching procedure for the values of the parameters has been used: the values of the NP and FF parameters are stored, and the histogram bin content is updated only when the parameters are effectively varied.
Multiple HammerCacher instances can be embedded in the same object and their parameters varied at the same time using the MultiHammerCacher.
In addition the HammerNuisWrapper is defined, it which contains a HammerCacher and attaches a set of multiplicative parameters to vary the bin content of the HAMMER histogram.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Slightly awkward sentence construction. How about the following:

In addition the HammerNuisWrapper is defined as the one that contains a HammerCatcher and attaches a set of multiplicative parameters to vary the bin content of the HAMMER histogram.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed in the next push

A similar structure has been built to include non-HAMMER-weighted contributions in the model, meaning regular histograms produced from HEP simulations; the shapes of these contributions are defined by reading a file, and from the set of multiplicative parameters—representing, for example, the yields.
Each NuisWrapper is further contained in a Template class, which defines an object allowing access to the bin content of the contributions and enabling changes to the parameters.
The Template class also contains options to generate toy samples or to return the whole template histogram for a specific injection of parameters.
Multiple Templates are stored in a single Fitter object, which is used to represent what in pyhf would be defined as a Channel, and it is used to interface the preceding nested structure of classes to Redist.
Copy link

@sanhitamj sanhitamj Jul 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: Put the word pyhf in the same special format as before.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed in the next push


Given the JSON file, a Fitter can be easily produced simply by using the Reader object, letting the machinery compose the nested structure on top of which the Fitter sits:
```python
import *pyhf*

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does pyhf here need asterisk around, in the real code? If not, the asterisks around can go, for the code snippet.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed in the next push

A scalefactor is attached to the contributions to apply a fixed pre-scaling to the yield of the contributions.
Finally, information useful at plotting time, such as axis titles and binnings, is included.

Given the JSON file, a Fitter can be easily produced simply by using the Reader object, letting the machinery compose the nested structure on top of which the Fitter sits:

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question: Do you mean give a JSON file with the format from above, or the JSON file has to be exactly like what's given above?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am more clear on the format to be the same, the options depend on the samples you are including

```
where $M(\vec{p})$ is the shape of the template as a function of the parameters $\vec{p}$, $f_0$ is the reference hypothesis distribution, and $f$ is the alternative distribution.
A set of parameters is then defined and the custom modifier created.
The custom modifier is assigned a name to distinguish it from other types of standard modifiers and is added later to an already existing, but not custom, pyhf model through the expand_pdf attribute:

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pyhf with asterisks

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also expand_pdf in asterisk

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed in the next push

A set of parameters is then defined and the custom modifier created.
The custom modifier is assigned a name to distinguish it from other types of standard modifiers and is added later to an already existing, but not custom, pyhf model through the expand_pdf attribute:
```python
new_params = {"Re_S_qLlL": {"inits": (0.0,),"bounds": ((-3.0, 3.0),),"paramset_type": "unconstrained"},}
Copy link

@sanhitamj sanhitamj Jul 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: Take a look at the preview here

The first 2 lines of this code block and the last line need slight horizontal scrolling. It would be easier to read if the code is split in multiple lines, following pep-8 rules.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i fix this in the next push

More generically, following the same prescription, the pyhf model is able to handle arbitrarily complex degrees of freedom where the effect of changing the templates is completely determined by the initial and alternative hypothesis we define as simple python functions.

This method has direct application in HEP analysis, in particular for the reinterpretation of semileptonic analyses.
Taking, for example, the semileptonic modes of the beauty mesons $\overline{B^0} \to D^{*+}\mu^{-}\overline{\nu_{\mu}}$ and $\overline{B^0} \to D^{*+}\tau^{-}\overline{\nu_{\tau}}$, notably, tension exist between the expected ratio of the abundance of the two decay modes and the measured ratio of the respective Branching Fractions (BF):

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Grammar: ... notably, tension exists between the expected ratio ...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed in the next push

Negative-Log-Likelihood phase-space scan of the scalar Wilson Coefficient on a template-dataset with no NP injection (left) and on a template dataset with the NP injection of $Re(S_{qLlL})=0.2$ and $Im(S_{qLlL})=0.8$ (right). The confidence levels correspontent to 1, 3 and 5 standard deviations are overlaid to both the scans. Both the fits correctly identify the injected New Physics values showing the effectiveness of the method.
:::

In fact, the custom modifiers applying the weighting can be mixed with the already existing shape modifiers in pyhf.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Decorator for pyhf

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed in the next push



## Outlook
We present the Redist-HAMMER interface, an extension of the already existing Redist module to interface HAMMER with the fitting environment of pyhf.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Decorator for pyhf

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed in the next push




Python is becoming increasingly popular in High Energy Physics (HEP) analysis due to its flexibility, ease of use and the growing number of scientific tool implementet on it.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

type: should be implemented

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed in the next push


An additional class named Reader is taking care of model construction using the configurations included in a single JSON file:

```{code-block} c

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a suggestion: {code-block} could be changes to json for better rendering and syntax highlight.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I change this. Thanks!

```
then the Fitter object allows easy handling of the HAMMER-processed contributions and their degrees of freedom—for example, generating pseudo-data where, for a given injection of NP and FF parameters, a binned distribution is produced according to a Poissonian distribution for each bin:
```python
params_0 = {"SM" : 1.,"Re_S_qLlL" : 0.,"Im_S_qLlL" : 0., ... ,

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider adding comments or placeholder definitions for these variables.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the next push I will change also the parameter definition in the json to remove the "..." as you suggested above.
As well as before here to add all the possible variables would be long and probably not very useful.

Redist has been designed to handle all the Form Factors and Wilson Coefficients at the same time but it works also with subsets of them.

What has been shown in this document represents one of the many applications, in HEP and beyond, of the Redist weighting method for model building.
Future efforts will integrate other popular theoretical backends in flavour physics like Flavio [@straub2018flaviopythonpackageflavour], finally allowing data analysts effortless use of, and possibly toggling between, all the different packages.

Fields outside HEP can benefit from the Redist method for building model-independent fits, enabling interpretation of observed patterns in complex systems without relying on predefined models. No newline at end of file

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add 1-2 examples of those fields outside HEP?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pyhf has not yet been used widely in fields outside of HEP (or at least I do not know).

On one side the HistFactory and Redist methods are general methods that can be implemented in multiple fields (Astriphysics, Medical Physics, Complex Systems...), on the other pyhf is currently mainly used in HEP and also the main efforts to implement new techniques, as the Redist-HAMMER (as far as I know) are made by the HEP community.

I would state, as I did that complex system studies can benefit from our statistical methods which is a very general statement as "complex systems" covers many applications.

I would personally avoid to claim the usage of Redist to extremely specific problems (unpacking a bit the complex systems) as it would not be true, as now the theoretical packages we interface (and also the ones we plan to implement) are for HEP.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it. So I think, it might be useful to specifically state that in future we could explore other similar areas/fields/usecases which could reuse this solution.


Allowing for a correct reinterpretation of data also opens the possibility of combining different datasets in a single coherent fitting environment.
The Redist-HAMMER interface has been validated, and studies on the benefits of a coherent combination in a simultaneous fit of different datasets, representing different decay modes, are currently ongoing.
By combining different datasets in a simultaneous fit, the degrees of freedom that are shared lead to an enhancement of the sensitivity and possibly to the cancellation of biases that might arise in a post-fit average of different results.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any limitations when it comes to dataset size? I believe HEP datasets can be massive. Is the size limited to what can fit in the memory or can Redist support some kind of scaling beyond just one machine?

If this is a limitation for now, then consider adding it too.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As you correctly say HEP datasets are massive.
On the other hand when we do analysis we heavily preprocess data (with selections and other treatments as the HAMMER weigthing).

When it comes to the fit itself we are very far from memory usage problems.

Copy link

@yash91sharma yash91sharma left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the changes. I have one additional suggestion: I strongly suggest to please add a "limitations" section, which highlights use cases, situations where Redist is not suitable for, or where it might not work properly. More details in this comment: #1083 (comment)

Other than above, this paper looks good. I am approving this.
Great work!

@sanhitamj
Copy link

@yash91sharma and @pranoy-ray thank you both so much for help with this submission and maintain the quality for the Proceedings.
@MarcoColonna thank you so much for the submission and diligently working with the reviewers.

@fwkoch fwkoch added approved This triggers Curvenote Submission action and removed draft This triggers Curvenote Preview actions labels Oct 14, 2025
@fwkoch fwkoch merged commit 5681ac3 into scipy-conference:2025 Oct 14, 2025
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved This triggers Curvenote Submission action assigned-editor paper This indicates that the PR in question is a paper

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants