Searching for new phenomena with profile likelihood ratio tests

Abstract

Likelihood ratio tests are standard statistical tools used in particle physics to perform tests of hypotheses. The null distribution of the likelihood ratio test statistic is often assumed to be χ2, following Wilks’ theorem. However, in many circumstances relevant to modern experiments this theorem is not applicable. In this Expert Recommendation, we overview practical ways to identify these situations and provide guidelines on how to construct valid inference. We use examples from particle physics, but the statistical constructs discussed here can be used in any scientific discipline that relies on data analysis.

Access options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Fig. 1: Illustration of the example model used throughout this paper, and studies of its log-likelihood test statistic distribution.
Fig. 2: Illustration of the upcrossings of a process and a comparison of the local and global P values in the example discussed in the section on non-identifiability and look-elsewhere effects.
Fig. 3: Illustration of P-value approximations.

References

  1. 1.

    Neyman, J. & Pearson, E. S. On the problem of the most efficient tests of statistical hypotheses. Phil. Trans. R. Soc. Lond. A 231, 289–337 (1933).

    ADS  Article  Google Scholar 

  2. 2.

    Karlin, S. & Rubin, H. The theory of decision procedures for distributions with monotone likelihood ratio. Ann. Math. Stat. 27, 272–299 (1956).

    MathSciNet  Article  Google Scholar 

  3. 3.

    Aad, G. et al. Observation of a new particle in the search for the standard model Higgs boson with the ATLAS detector at the LHC. Phys. Lett. B716, 1–29 (2012).

    ADS  Article  Google Scholar 

  4. 4.

    Chatrchyan, S. et al. Observation of a new boson at a mass of 125 GeV with the CMS experiment at the LHC. Phys. Lett. B716, 30–61 (2012).

    ADS  Article  Google Scholar 

  5. 5.

    An, F. P. et al. Observation of electron-antineutrino disappearance at Daya Bay. Phys. Rev. Lett. 108, 171803 (2012).

    ADS  Article  Google Scholar 

  6. 6.

    Aprile, E. et al. Dark matter search results from a one ton-year exposure of XENON1T. Phys. Rev. Lett. 121, 111302 (2018).

    ADS  Article  Google Scholar 

  7. 7.

    PandaX-II Collaboration et al. Dark matter results from 54-ton-day exposure of PandaX-II experiment. Phys. Rev. Lett. 119, 181302 (2017).

    ADS  Article  Google Scholar 

  8. 8.

    Akerib, D. S. et al. Results from a search for dark matter in the complete LUX exposure. Phys. Rev. Lett. 118, 021303 (2017).

    ADS  Article  Google Scholar 

  9. 9.

    Abdallah, H. et al. Search for γ-ray line signals from dark matter annihilations in the inner galactic halo from 10 years of observations with H.E.S.S. Phys. Rev. Lett. 120, 201101 (2018).

    ADS  Article  Google Scholar 

  10. 10.

    Ackermann, M. et al. Updated search for spectral lines from galactic dark matter interactions with pass 8 data from the Fermi Large Area Telescope. Phys. Rev. D 91, 122002 (2015).

    ADS  Article  Google Scholar 

  11. 11.

    Wilks, S. The large-sample distribution of the likelihood ratio for testing composite hypotheses. Ann. Math. Stat. 9, 60–62 (1938).

    Article  Google Scholar 

  12. 12.

    Cox, D. R. & Hinkley, D. V. Theoretical Statistics (Chapman and Hall/CRC, 1979).

  13. 13.

    Protassov, R., Van Dyk, D. A., Connors, A., Kashyap, V. L. & Siemiginowska, A. Statistics, handle with care: detecting multiple model components with the likelihood ratio test. Astrophys. J. 571, 545 (2002).

    ADS  Article  Google Scholar 

  14. 14.

    Reid, N. & Fraser, D. in Statistics for the 21st Century: Methodologies for Applications of the Future (eds Rao, C. & Székely, G.) 351–366 (Marcel Dekker AG, 2000).

  15. 15.

    Brazzale, A. R. & Valentina, M. Likelihood Asymptotics in Nonregular Settings. A Review with Emphasis on the Likelihood Ratio. Working Paper Series 4 (Department of Statistical Sciences, Univ. Padova, 2018).

  16. 16.

    Severini, T. A. An empirical adjustment to the likelihood ratio statistic. Biometrika 86, 235–247 (1999).

    MathSciNet  Article  Google Scholar 

  17. 17.

    He, H. & Severini, T. A. et al. Higher-order asymptotic normality of approximations to the modified signed likelihood ratio statistic for regular models. Ann. Stat. 35, 2054–2074 (2007).

    MathSciNet  Article  Google Scholar 

  18. 18.

    Cowan, G., Cranmer, K., Gross, E. & Vitells, O. Asymptotic formulae for likelihood-based tests of new physics. Eur. Phys. J. C 71, 1554 (2011); erratum 73, 2501 (2013).

  19. 19.

    Cowan, G., Cranmer, K., Gross, E. & Vitells, O. Asymptotic distribution for two-sided tests with lower and upper boundaries on the parameter of interest. Preprint at http://arxiv.org/abs/arXiv:1210.6948 (2012).

  20. 20.

    Chernoff, H. On the distribution of the likelihood ratio. Ann. Math. Stat. 25, 573–578 (1954).

    MathSciNet  Article  Google Scholar 

  21. 21.

    Self, S. G. & Liang, K.-Y. Asymptotic properties of maximum likelihood estimators and likelihood ratio tests under nonstandard conditions. J. Am. Stat. Assoc. 82, 605–610 (1987).

    MathSciNet  Article  Google Scholar 

  22. 22.

    Cavaliere, G., Nielsen, H. B., Pedersen, R. S. & Rahbek, A. Bootstrap inference on the boundary of the parameter space with application to conditional volatility models. Discussion Papers 18-10, University of Copenhagen. Department of Economics.

  23. 23.

    Andrews, D. W. Inconsistency of the bootstrap when a parameter is on the boundary of the parameter space. Econometrica 68, 399–405 (2000).

    MathSciNet  Article  Google Scholar 

  24. 24.

    Geyer, C. J. Likelihood Ratio Tests and Inequality Contraints Technical Report 610 http://hdl.handle.net/11299/205465 (Univ. Minnesota, 1995).

  25. 25.

    Efron, B. Large-scale Inference: Empirical Bayes Methods for Estimation, Testing, and Prediction (Cambridge Univ. Press, 2012).

  26. 26.

    Gross, E. & Vitells, O. Trial factors for the look elsewhere effect in high energy physics. Eur. Phys. J. C 70, 525–530 (2010).

    ADS  Article  Google Scholar 

  27. 27.

    Davies, R. B. Hypothesis testing when a nuisance parameter is present only under the alternative. Biometrika 74, 33–43 (1987).

    MathSciNet  MATH  Google Scholar 

  28. 28.

    Ghosh, J. K. & Sen, P. K. On the Asymptotic Performance of the Log Likelihood Ratio Statistic for the Mixture Model and Related Results Institute of Statistics mimeo series 1467 https://repository.lib.ncsu.edu/handle/1840.4/3493 (North Carolina State Univ., Department of Statistics, 1984).

  29. 29.

    Algeri, S., Conrad, J. & van Dyk, D. A. A method for comparing non-nested models with application to astrophysical searches for new physics. Mon. Not. R. Astron. Soc. 458, L84–L88 (2016).

    ADS  Article  Google Scholar 

  30. 30.

    Vitells, O. & Gross, E. Estimating the significance of a signal in a multi-dimensional search. Astropart. Phys. 35, 230–234 (2011).

    ADS  Article  Google Scholar 

  31. 31.

    Algeri, S. & van Dyk, D. A. Testing one hypothesis multiple times: the multidimensional case. J. Comput. Graph. Stat. https://doi.org/10.1080/10618600.2019.1677474 (2019).

  32. 32.

    Aharonian, F. et al. Spectrum and variability of the Galactic Center VHE γ-ray source HESS J1745−290. Astron. Astrophys. 503, 817–825 (2009).

    ADS  Article  Google Scholar 

  33. 33.

    Cox, D. R. Tests of separate families of hypotheses. In Proc. Fourth Berkeley Symposium on Mathematical Statistics and Probability Vol. 1, 105–123 (1961).

  34. 34.

    Algeri, S. Detecting new signals under background mismodeling. Phys. Rev. D 101, 015003 (2020).

  35. 35.

    Dauncey, P., Kenzie, M., Wardle, N. & Davies, G. Handling uncertainties in background shapes: the discrete profiling method. J. Instrum. 10, P04015–P04015 (2015).

    Article  Google Scholar 

  36. 36.

    Aad, G. et al. Measurement of Higgs boson production in the diphoton decay channel in pp collisions at center-of-mass energies of 7 and 8 TeV with the ATLAS detector. Phys. Rev. D 90, 112015 (2014).

    ADS  Article  Google Scholar 

  37. 37.

    Priel, N., Rauch, L., Landsman, H., Manfredini, A. & Budnik, R. A model independent safeguard for unbinned likelihood. J. Cosmol. Astropart. Phys. 2017, 013–013 (2017).

    Article  Google Scholar 

  38. 38.

    Yellin, S. Finding an upper limit in the presence of unknown background. Phys. Rev. D 66, 032005 (2002).

    ADS  Article  Google Scholar 

Download references

Acknowledgements

J.C., J.A. and K.D.M. acknowledge support from the Knut and Alice Wallenberg Foundation, and the Swedish Research Council.

Author information

Affiliations

Authors

Contributions

S.A. mainly contributed to the sections: ‘Wilks’ theorem and its conditions’, ‘Insufficient data’, ‘Parameters with bounds’, ‘Non-identifiability and look-elsewhere effects’, ‘Non-nestedness’, ‘Uncertain models and nuisance parameters’, ‘Recommendations’, Figs 2 and 3a, and Table 1. J.A. mainly contributed to the introduction and the sections ‘Insufficient data’ and ‘Parameters with bounds’, Fig. 1 and Table 1. K.D.M. mainly contributed to the sections ‘Paramaters with bounds’, ‘Non-identifiability and look-elsewhere effects’, ‘Uncertain models and nuisance parameters’ and Fig. 3b. J.C. mainly contributed to the section ‘Recommendations’, had the idea of writing the Expert Recommendation and coordinated its overall development. All the authors have read, discussed and extensively revised subsequent drafts of the manuscript in all its components.

Corresponding author

Correspondence to Jan Conrad.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Peer review information

Nature Reviews Physics thanks Nicholas Wardle, Michael Schmelling and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Algeri, S., Aalbers, J., Morå, K.D. et al. Searching for new phenomena with profile likelihood ratio tests. Nat Rev Phys 2, 245–252 (2020). https://doi.org/10.1038/s42254-020-0169-5

Download citation