Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Expert Recommendation
  • Published:

Searching for new phenomena with profile likelihood ratio tests

Nature Reviews Physics volume 2, pages 245–252 (2020)Cite this article

Subjects

Abstract

Likelihood ratio tests are standard statistical tools used in particle physics to perform tests of hypotheses. The null distribution of the likelihood ratio test statistic is often assumed to be χ2, following Wilks’ theorem. However, in many circumstances relevant to modern experiments this theorem is not applicable. In this Expert Recommendation, we overview practical ways to identify these situations and provide guidelines on how to construct valid inference. We use examples from particle physics, but the statistical constructs discussed here can be used in any scientific discipline that relies on data analysis.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Illustration of the example model used throughout this paper, and studies of its log-likelihood test statistic distribution.
Fig. 2: Illustration of the upcrossings of a process and a comparison of the local and global P values in the example discussed in the section on non-identifiability and look-elsewhere effects.
Fig. 3: Illustration of P-value approximations.

Similar content being viewed by others

References

  1. Neyman, J. & Pearson, E. S. On the problem of the most efficient tests of statistical hypotheses. Phil. Trans. R. Soc. Lond. A 231, 289–337 (1933).

    Article  ADS  Google Scholar 

  2. Karlin, S. & Rubin, H. The theory of decision procedures for distributions with monotone likelihood ratio. Ann. Math. Stat. 27, 272–299 (1956).

    Article  MathSciNet  Google Scholar 

  3. Aad, G. et al. Observation of a new particle in the search for the standard model Higgs boson with the ATLAS detector at the LHC. Phys. Lett. B716, 1–29 (2012).

    Article  ADS  Google Scholar 

  4. Chatrchyan, S. et al. Observation of a new boson at a mass of 125 GeV with the CMS experiment at the LHC. Phys. Lett. B716, 30–61 (2012).

    Article  ADS  Google Scholar 

  5. An, F. P. et al. Observation of electron-antineutrino disappearance at Daya Bay. Phys. Rev. Lett. 108, 171803 (2012).

    Article  ADS  Google Scholar 

  6. Aprile, E. et al. Dark matter search results from a one ton-year exposure of XENON1T. Phys. Rev. Lett. 121, 111302 (2018).

    Article  ADS  Google Scholar 

  7. PandaX-II Collaboration et al. Dark matter results from 54-ton-day exposure of PandaX-II experiment. Phys. Rev. Lett. 119, 181302 (2017).

    Article  ADS  Google Scholar 

  8. Akerib, D. S. et al. Results from a search for dark matter in the complete LUX exposure. Phys. Rev. Lett. 118, 021303 (2017).

    Article  ADS  Google Scholar 

  9. Abdallah, H. et al. Search for γ-ray line signals from dark matter annihilations in the inner galactic halo from 10 years of observations with H.E.S.S. Phys. Rev. Lett. 120, 201101 (2018).

    Article  ADS  Google Scholar 

  10. Ackermann, M. et al. Updated search for spectral lines from galactic dark matter interactions with pass 8 data from the Fermi Large Area Telescope. Phys. Rev. D 91, 122002 (2015).

    Article  ADS  Google Scholar 

  11. Wilks, S. The large-sample distribution of the likelihood ratio for testing composite hypotheses. Ann. Math. Stat. 9, 60–62 (1938).

    Article  Google Scholar 

  12. Cox, D. R. & Hinkley, D. V. Theoretical Statistics (Chapman and Hall/CRC, 1979).

  13. Protassov, R., Van Dyk, D. A., Connors, A., Kashyap, V. L. & Siemiginowska, A. Statistics, handle with care: detecting multiple model components with the likelihood ratio test. Astrophys. J. 571, 545 (2002).

    Article  ADS  Google Scholar 

  14. Reid, N. & Fraser, D. in Statistics for the 21st Century: Methodologies for Applications of the Future (eds Rao, C. & Székely, G.) 351–366 (Marcel Dekker AG, 2000).

  15. Brazzale, A. R. & Valentina, M. Likelihood Asymptotics in Nonregular Settings. A Review with Emphasis on the Likelihood Ratio. Working Paper Series 4 (Department of Statistical Sciences, Univ. Padova, 2018).

  16. Severini, T. A. An empirical adjustment to the likelihood ratio statistic. Biometrika 86, 235–247 (1999).

    Article  MathSciNet  Google Scholar 

  17. He, H. & Severini, T. A. et al. Higher-order asymptotic normality of approximations to the modified signed likelihood ratio statistic for regular models. Ann. Stat. 35, 2054–2074 (2007).

    Article  MathSciNet  Google Scholar 

  18. Cowan, G., Cranmer, K., Gross, E. & Vitells, O. Asymptotic formulae for likelihood-based tests of new physics. Eur. Phys. J. C 71, 1554 (2011); erratum 73, 2501 (2013).

  19. Cowan, G., Cranmer, K., Gross, E. & Vitells, O. Asymptotic distribution for two-sided tests with lower and upper boundaries on the parameter of interest. Preprint at http://arxiv.org/abs/arXiv:1210.6948 (2012).

  20. Chernoff, H. On the distribution of the likelihood ratio. Ann. Math. Stat. 25, 573–578 (1954).

    Article  MathSciNet  Google Scholar 

  21. Self, S. G. & Liang, K.-Y. Asymptotic properties of maximum likelihood estimators and likelihood ratio tests under nonstandard conditions. J. Am. Stat. Assoc. 82, 605–610 (1987).

    Article  MathSciNet  Google Scholar 

  22. Cavaliere, G., Nielsen, H. B., Pedersen, R. S. & Rahbek, A. Bootstrap inference on the boundary of the parameter space with application to conditional volatility models. Discussion Papers 18-10, University of Copenhagen. Department of Economics.

  23. Andrews, D. W. Inconsistency of the bootstrap when a parameter is on the boundary of the parameter space. Econometrica 68, 399–405 (2000).

    Article  MathSciNet  Google Scholar 

  24. Geyer, C. J. Likelihood Ratio Tests and Inequality Contraints Technical Report 610 http://hdl.handle.net/11299/205465 (Univ. Minnesota, 1995).

  25. Efron, B. Large-scale Inference: Empirical Bayes Methods for Estimation, Testing, and Prediction (Cambridge Univ. Press, 2012).

  26. Gross, E. & Vitells, O. Trial factors for the look elsewhere effect in high energy physics. Eur. Phys. J. C 70, 525–530 (2010).

    Article  ADS  Google Scholar 

  27. Davies, R. B. Hypothesis testing when a nuisance parameter is present only under the alternative. Biometrika 74, 33–43 (1987).

    MathSciNet  MATH  Google Scholar 

  28. Ghosh, J. K. & Sen, P. K. On the Asymptotic Performance of the Log Likelihood Ratio Statistic for the Mixture Model and Related Results Institute of Statistics mimeo series 1467 https://repository.lib.ncsu.edu/handle/1840.4/3493 (North Carolina State Univ., Department of Statistics, 1984).

  29. Algeri, S., Conrad, J. & van Dyk, D. A. A method for comparing non-nested models with application to astrophysical searches for new physics. Mon. Not. R. Astron. Soc. 458, L84–L88 (2016).

    Article  ADS  Google Scholar 

  30. Vitells, O. & Gross, E. Estimating the significance of a signal in a multi-dimensional search. Astropart. Phys. 35, 230–234 (2011).

    Article  ADS  Google Scholar 

  31. Algeri, S. & van Dyk, D. A. Testing one hypothesis multiple times: the multidimensional case. J. Comput. Graph. Stat. https://doi.org/10.1080/10618600.2019.1677474 (2019).

  32. Aharonian, F. et al. Spectrum and variability of the Galactic Center VHE γ-ray source HESS J1745−290. Astron. Astrophys. 503, 817–825 (2009).

    Article  ADS  Google Scholar 

  33. Cox, D. R. Tests of separate families of hypotheses. In Proc. Fourth Berkeley Symposium on Mathematical Statistics and Probability Vol. 1, 105–123 (1961).

  34. Algeri, S. Detecting new signals under background mismodeling. Phys. Rev. D 101, 015003 (2020).

  35. Dauncey, P., Kenzie, M., Wardle, N. & Davies, G. Handling uncertainties in background shapes: the discrete profiling method. J. Instrum. 10, P04015–P04015 (2015).

    Article  Google Scholar 

  36. Aad, G. et al. Measurement of Higgs boson production in the diphoton decay channel in pp collisions at center-of-mass energies of 7 and 8 TeV with the ATLAS detector. Phys. Rev. D 90, 112015 (2014).

    Article  ADS  Google Scholar 

  37. Priel, N., Rauch, L., Landsman, H., Manfredini, A. & Budnik, R. A model independent safeguard for unbinned likelihood. J. Cosmol. Astropart. Phys. 2017, 013–013 (2017).

    Article  Google Scholar 

  38. Yellin, S. Finding an upper limit in the presence of unknown background. Phys. Rev. D 66, 032005 (2002).

    Article  ADS  Google Scholar 

Download references

Acknowledgements

J.C., J.A. and K.D.M. acknowledge support from the Knut and Alice Wallenberg Foundation, and the Swedish Research Council.

Author information

Authors and Affiliations

  1. School of Statistics, University of Minnesota, Minneapolis, MN, USA

    Sara Algeri

  2. Physics Department, Stockholm University, Stockholm, Sweden

    Jelle Aalbers & Knut Dundas Morå

  3. Oskar Klein Centre, Stockholm University, Stockholm, Sweden

    Jelle Aalbers & Knut Dundas Morå

  4. Physics Department, Columbia University, New York, NY, USA

    Knut Dundas Morå & Jan Conrad

Authors
  1. Sara Algeri
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jelle Aalbers
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Knut Dundas Morå
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jan Conrad
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

S.A. mainly contributed to the sections: ‘Wilks’ theorem and its conditions’, ‘Insufficient data’, ‘Parameters with bounds’, ‘Non-identifiability and look-elsewhere effects’, ‘Non-nestedness’, ‘Uncertain models and nuisance parameters’, ‘Recommendations’, Figs 2 and 3a, and Table 1. J.A. mainly contributed to the introduction and the sections ‘Insufficient data’ and ‘Parameters with bounds’, Fig. 1 and Table 1. K.D.M. mainly contributed to the sections ‘Paramaters with bounds’, ‘Non-identifiability and look-elsewhere effects’, ‘Uncertain models and nuisance parameters’ and Fig. 3b. J.C. mainly contributed to the section ‘Recommendations’, had the idea of writing the Expert Recommendation and coordinated its overall development. All the authors have read, discussed and extensively revised subsequent drafts of the manuscript in all its components.

Corresponding author

Correspondence to Jan Conrad.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Peer review information

Nature Reviews Physics thanks Nicholas Wardle, Michael Schmelling and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Algeri, S., Aalbers, J., Morå, K.D. et al. Searching for new phenomena with profile likelihood ratio tests. Nat Rev Phys 2, 245–252 (2020). https://doi.org/10.1038/s42254-020-0169-5

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s42254-020-0169-5

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing