# Searching for new phenomena with profile likelihood ratio tests

## Abstract

Likelihood ratio tests are standard statistical tools used in particle physics to perform tests of hypotheses. The null distribution of the likelihood ratio test statistic is often assumed to be χ2, following Wilks’ theorem. However, in many circumstances relevant to modern experiments this theorem is not applicable. In this Expert Recommendation, we overview practical ways to identify these situations and provide guidelines on how to construct valid inference. We use examples from particle physics, but the statistical constructs discussed here can be used in any scientific discipline that relies on data analysis.

## Access options

from\$8.99

All prices are NET prices.

## References

1. 1.

Neyman, J. & Pearson, E. S. On the problem of the most efficient tests of statistical hypotheses. Phil. Trans. R. Soc. Lond. A 231, 289–337 (1933).

2. 2.

Karlin, S. & Rubin, H. The theory of decision procedures for distributions with monotone likelihood ratio. Ann. Math. Stat. 27, 272–299 (1956).

3. 3.

Aad, G. et al. Observation of a new particle in the search for the standard model Higgs boson with the ATLAS detector at the LHC. Phys. Lett. B716, 1–29 (2012).

4. 4.

Chatrchyan, S. et al. Observation of a new boson at a mass of 125 GeV with the CMS experiment at the LHC. Phys. Lett. B716, 30–61 (2012).

5. 5.

An, F. P. et al. Observation of electron-antineutrino disappearance at Daya Bay. Phys. Rev. Lett. 108, 171803 (2012).

6. 6.

Aprile, E. et al. Dark matter search results from a one ton-year exposure of XENON1T. Phys. Rev. Lett. 121, 111302 (2018).

7. 7.

PandaX-II Collaboration et al. Dark matter results from 54-ton-day exposure of PandaX-II experiment. Phys. Rev. Lett. 119, 181302 (2017).

8. 8.

Akerib, D. S. et al. Results from a search for dark matter in the complete LUX exposure. Phys. Rev. Lett. 118, 021303 (2017).

9. 9.

Abdallah, H. et al. Search for γ-ray line signals from dark matter annihilations in the inner galactic halo from 10 years of observations with H.E.S.S. Phys. Rev. Lett. 120, 201101 (2018).

10. 10.

Ackermann, M. et al. Updated search for spectral lines from galactic dark matter interactions with pass 8 data from the Fermi Large Area Telescope. Phys. Rev. D 91, 122002 (2015).

11. 11.

Wilks, S. The large-sample distribution of the likelihood ratio for testing composite hypotheses. Ann. Math. Stat. 9, 60–62 (1938).

12. 12.

Cox, D. R. & Hinkley, D. V. Theoretical Statistics (Chapman and Hall/CRC, 1979).

13. 13.

Protassov, R., Van Dyk, D. A., Connors, A., Kashyap, V. L. & Siemiginowska, A. Statistics, handle with care: detecting multiple model components with the likelihood ratio test. Astrophys. J. 571, 545 (2002).

14. 14.

Reid, N. & Fraser, D. in Statistics for the 21st Century: Methodologies for Applications of the Future (eds Rao, C. & Székely, G.) 351–366 (Marcel Dekker AG, 2000).

15. 15.

Brazzale, A. R. & Valentina, M. Likelihood Asymptotics in Nonregular Settings. A Review with Emphasis on the Likelihood Ratio. Working Paper Series 4 (Department of Statistical Sciences, Univ. Padova, 2018).

16. 16.

Severini, T. A. An empirical adjustment to the likelihood ratio statistic. Biometrika 86, 235–247 (1999).

17. 17.

He, H. & Severini, T. A. et al. Higher-order asymptotic normality of approximations to the modified signed likelihood ratio statistic for regular models. Ann. Stat. 35, 2054–2074 (2007).

18. 18.

Cowan, G., Cranmer, K., Gross, E. & Vitells, O. Asymptotic formulae for likelihood-based tests of new physics. Eur. Phys. J. C 71, 1554 (2011); erratum 73, 2501 (2013).

19. 19.

Cowan, G., Cranmer, K., Gross, E. & Vitells, O. Asymptotic distribution for two-sided tests with lower and upper boundaries on the parameter of interest. Preprint at http://arxiv.org/abs/arXiv:1210.6948 (2012).

20. 20.

Chernoff, H. On the distribution of the likelihood ratio. Ann. Math. Stat. 25, 573–578 (1954).

21. 21.

Self, S. G. & Liang, K.-Y. Asymptotic properties of maximum likelihood estimators and likelihood ratio tests under nonstandard conditions. J. Am. Stat. Assoc. 82, 605–610 (1987).

22. 22.

Cavaliere, G., Nielsen, H. B., Pedersen, R. S. & Rahbek, A. Bootstrap inference on the boundary of the parameter space with application to conditional volatility models. Discussion Papers 18-10, University of Copenhagen. Department of Economics.

23. 23.

Andrews, D. W. Inconsistency of the bootstrap when a parameter is on the boundary of the parameter space. Econometrica 68, 399–405 (2000).

24. 24.

Geyer, C. J. Likelihood Ratio Tests and Inequality Contraints Technical Report 610 http://hdl.handle.net/11299/205465 (Univ. Minnesota, 1995).

25. 25.

Efron, B. Large-scale Inference: Empirical Bayes Methods for Estimation, Testing, and Prediction (Cambridge Univ. Press, 2012).

26. 26.

Gross, E. & Vitells, O. Trial factors for the look elsewhere effect in high energy physics. Eur. Phys. J. C 70, 525–530 (2010).

27. 27.

Davies, R. B. Hypothesis testing when a nuisance parameter is present only under the alternative. Biometrika 74, 33–43 (1987).

28. 28.

Ghosh, J. K. & Sen, P. K. On the Asymptotic Performance of the Log Likelihood Ratio Statistic for the Mixture Model and Related Results Institute of Statistics mimeo series 1467 https://repository.lib.ncsu.edu/handle/1840.4/3493 (North Carolina State Univ., Department of Statistics, 1984).

29. 29.

Algeri, S., Conrad, J. & van Dyk, D. A. A method for comparing non-nested models with application to astrophysical searches for new physics. Mon. Not. R. Astron. Soc. 458, L84–L88 (2016).

30. 30.

Vitells, O. & Gross, E. Estimating the significance of a signal in a multi-dimensional search. Astropart. Phys. 35, 230–234 (2011).

31. 31.

Algeri, S. & van Dyk, D. A. Testing one hypothesis multiple times: the multidimensional case. J. Comput. Graph. Stat. https://doi.org/10.1080/10618600.2019.1677474 (2019).

32. 32.

Aharonian, F. et al. Spectrum and variability of the Galactic Center VHE γ-ray source HESS J1745−290. Astron. Astrophys. 503, 817–825 (2009).

33. 33.

Cox, D. R. Tests of separate families of hypotheses. In Proc. Fourth Berkeley Symposium on Mathematical Statistics and Probability Vol. 1, 105–123 (1961).

34. 34.

Algeri, S. Detecting new signals under background mismodeling. Phys. Rev. D 101, 015003 (2020).

35. 35.

Dauncey, P., Kenzie, M., Wardle, N. & Davies, G. Handling uncertainties in background shapes: the discrete profiling method. J. Instrum. 10, P04015–P04015 (2015).

36. 36.

Aad, G. et al. Measurement of Higgs boson production in the diphoton decay channel in pp collisions at center-of-mass energies of 7 and 8 TeV with the ATLAS detector. Phys. Rev. D 90, 112015 (2014).

37. 37.

Priel, N., Rauch, L., Landsman, H., Manfredini, A. & Budnik, R. A model independent safeguard for unbinned likelihood. J. Cosmol. Astropart. Phys. 2017, 013–013 (2017).

38. 38.

Yellin, S. Finding an upper limit in the presence of unknown background. Phys. Rev. D 66, 032005 (2002).

## Acknowledgements

J.C., J.A. and K.D.M. acknowledge support from the Knut and Alice Wallenberg Foundation, and the Swedish Research Council.

## Author information

Authors

### Contributions

S.A. mainly contributed to the sections: ‘Wilks’ theorem and its conditions’, ‘Insufficient data’, ‘Parameters with bounds’, ‘Non-identifiability and look-elsewhere effects’, ‘Non-nestedness’, ‘Uncertain models and nuisance parameters’, ‘Recommendations’, Figs 2 and 3a, and Table 1. J.A. mainly contributed to the introduction and the sections ‘Insufficient data’ and ‘Parameters with bounds’, Fig. 1 and Table 1. K.D.M. mainly contributed to the sections ‘Paramaters with bounds’, ‘Non-identifiability and look-elsewhere effects’, ‘Uncertain models and nuisance parameters’ and Fig. 3b. J.C. mainly contributed to the section ‘Recommendations’, had the idea of writing the Expert Recommendation and coordinated its overall development. All the authors have read, discussed and extensively revised subsequent drafts of the manuscript in all its components.

## Ethics declarations

### Competing interests

The authors declare no competing interests.

### Peer review information

Nature Reviews Physics thanks Nicholas Wardle, Michael Schmelling and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

### Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

## Rights and permissions

Reprints and Permissions