Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Measuring the capabilities of quantum computers

Nature Physics volume 18, pages 75–79 (2022)Cite this article

Subjects

Abstract

Quantum computers can now run interesting programs, but each processor’s capability—the set of programs that it can run successfully—is limited by hardware errors. These errors can be complicated, making it difficult to accurately predict a processor’s capability. Benchmarks can be used to measure capability directly, but current benchmarks have limited flexibility and scale poorly to many-qubit processors. We show how to construct scalable, efficiently verifiable benchmarks based on any program by using a technique that we call circuit mirroring. With it, we construct two flexible, scalable volumetric benchmarks based on randomized and periodically ordered programs. We use these benchmarks to map out the capabilities of twelve publicly available processors, and to measure the impact of program structure on each one. We find that standard error metrics are poor predictors of whether a program will run successfully on today’s hardware, and that current processors vary widely in their sensitivity to program structure.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: A scalable method for benchmarking a quantum computer’s capability.
Fig. 2: Randomized benchmarks do not predict structured circuit performance.
Fig. 3: Empirical capability regions.

Similar content being viewed by others

Data availability

All data used in this work are available at https://doi.org/10.5281/zenodo.5197499.

Code availability

The data analysis code used to produce all the results presented in this work is available at https://doi.org/10.5281/zenodo.5197499. The circuit sampling code is available in pyGSTi39,40,41.

References

  1. Arute, F. et al. Quantum supremacy using a programmable superconducting processor. Nature 574, 505–510 (2019).

    Article  ADS  Google Scholar 

  2. Preskill, J. Quantum computing in the NISQ era and beyond. Quantum 2, 79 (2018).

    Article  Google Scholar 

  3. Sarovar, M. et al. Detecting crosstalk errors in quantum information processors. Quantum 4, 321 (2020).

    Article  Google Scholar 

  4. Huang, E., Doherty, A. C. & Flammia, S. Performance of quantum error correction with coherent errors. Phys. Rev. A 99, 022313 (2019).

    Article  ADS  Google Scholar 

  5. Kueng, R., Long, D. M., Doherty, A. C. & Flammia, S. T. Comparing experiments to the fault-tolerance threshold. Phys. Rev. Lett. 117, 170502 (2016).

    Article  ADS  Google Scholar 

  6. Murphy, D. C. & Brown, K. R. Controlling error orientation to improve quantum algorithm success rates. Phys. Rev. A 99, 032318 (2019).

    Article  ADS  Google Scholar 

  7. Mavadia, S. et al. Experimental quantum verification in the presence of temporally correlated noise. NPJ Quantum Inf. 4, 7 (2018).

    Article  ADS  Google Scholar 

  8. Proctor, T. et al. Detecting and tracking drift in quantum information processors. Nat. Commun. 11, 5396 (2020).

    Article  ADS  Google Scholar 

  9. Erhard, A. et al. Characterizing large-scale quantum computers via cycle benchmarking. Nat. Commun. 10, 5347 (2019).

    Article  ADS  Google Scholar 

  10. Flammia, S. T. & Wallman, J. J. Efficient estimation of Pauli channels. ACM Trans. Quant. Comp. 1, 3 (2020).

    Google Scholar 

  11. Harper, R., Flammia, S. T. & Wallman, J. J. Efficient learning of quantum noise. Nat. Phys. 16, 1184–1188 (2020).

    Article  Google Scholar 

  12. Blume-Kohout, R. et al. Demonstration of qubit operations below a rigorous fault tolerance threshold with gate set tomography. Nat. Commun. 8, 14485 (2017).

    Article  Google Scholar 

  13. Gambetta, J. M. et al. Characterization of addressability by simultaneous randomized benchmarking. Phys. Rev. Lett. 109, 240504 (2012).

    Article  ADS  Google Scholar 

  14. Michielsen, K. et al. Benchmarking gate-based quantum computers. Comput. Phys. Commun. 220, 44–55 (2017).

    Article  ADS  MathSciNet  Google Scholar 

  15. Proctor, T. J. et al. Direct randomized benchmarking for multiqubit devices. Phys. Rev. Lett. 123, 030503 (2019).

    Article  ADS  Google Scholar 

  16. McKay, D. C., Sheldon, S., Smolin, J. A., Chow, J. M. & Gambetta, J. M. Three-qubit randomized benchmarking. Phys. Rev. Lett. 122, 200502 (2019).

    Article  ADS  Google Scholar 

  17. Boixo, S. et al. Characterizing quantum supremacy in near-term devices. Nat. Phys. 14, 595–600 (2018).

    Article  Google Scholar 

  18. Cross, A. W., Bishop, L. S., Sheldon, S., Nation, P. D. & Gambetta, J. M. Validating quantum computers using randomized model circuits. Phys. Rev. A 100, 032328 (2019).

    Article  ADS  Google Scholar 

  19. Magesan, E., Gambetta, J. M. & Emerson, J. Scalable and robust randomized benchmarking of quantum processes. Phys. Rev. Lett. 106, 180504 (2011).

    Article  ADS  Google Scholar 

  20. Linke, N. M. et al. Experimental comparison of two quantum computing architectures. Proc. Natl Acad. Sci. USA 114, 3305–3310 (2017).

    Article  Google Scholar 

  21. Harrow, A. W. & Montanaro, A. Quantum computational supremacy. Nature 549, 203–209 (2017).

    Article  ADS  Google Scholar 

  22. Wright, K. et al. Benchmarking an 11-qubit quantum computer. Nat. Commun. 10, 5464 (2019).

    Article  ADS  Google Scholar 

  23. Murali, P. et al. Full-stack, real-system quantum computer studies: architectural comparisons and design insights, in Proc. 46th Annual International Symposium on Computer Architecture (ISCA) 527–540 (Assoc. for Computing Machinery, 2019).

  24. McCaskey, A. J. et al. Quantum chemistry as a benchmark for near-term quantum computers. NPJ Quantum Inf. 5, 99 (2019).

    Article  ADS  Google Scholar 

  25. Ferracin, S., Kapourniotis, T. & Datta, A. Accrediting outputs of noisy intermediate-scale quantum computing devices. New J. Phys. 21, 113038 (2019).

    Article  ADS  MathSciNet  Google Scholar 

  26. Tuckett, D. K., Bartlett, S. D. & Flammia, S. T. Ultrahigh error threshold for surface codes with biased noise. Phys. Rev. Lett. 120, 050505 (2018).

    Article  ADS  Google Scholar 

  27. Tuckett, D. K., Bartlett, S. D., Flammia, S. T. & Brown, B. J. Fault-tolerant thresholds for the surface code in excess of 5% under biased noise. Phys. Rev. Lett. 124, 130501 (2020).

    Article  ADS  Google Scholar 

  28. Loschmidt, J. Über den Zustand des Wärmegleichgewichts eines Systems von Körpern mit Rücksicht auf die Schwerkraft. Sitzungsber. Akad. Wiss. 2, 128–142 (1876).

    Google Scholar 

  29. Emerson, J., Alicki, R. & Życzkowski, K. Scalable noise estimation with random unitary operators. J. Opt. B 7, S347 (2005).

    Article  ADS  MathSciNet  Google Scholar 

  30. McKay, D. C., Wood, C. J., Sheldon, S., Chow, J. M. & Gambetta, J. M. Efficient Z gates for quantum computing. Phys. Rev. A 96, 022330 (2017).

    Article  ADS  Google Scholar 

  31. IBM Q Experience (IBM Q team, accessed 10 November 2019); https://quantum-computing.ibm.com

  32. Quantum Cloud Services (Rigetti Computing, accessed 1 November 2019); https://docs.rigetti.com/qcs/?cacheBust=1634728803487

  33. Blume-Kohout, R. & Young, K. C. A volumetric framework for quantum computer benchmarks. Quantum 4, 362 (2020).

    Article  Google Scholar 

  34. Kohn, W. & Luttinger, J. M. Quantum theory of electrical transport phenomena. Phys. Rev. 108, 590 (1957).

    Article  ADS  MathSciNet  Google Scholar 

  35. Wallman, J. J. & Emerson, J. Noise tailoring for scalable quantum computation via randomized compiling. Phys. Rev. A 94, 052325 (2016).

    Article  ADS  Google Scholar 

  36. Campbell, E. Shorter gate sequences for quantum computing by mixing unitaries. Phys. Rev. A 95, 042306 (2017).

    Article  ADS  MathSciNet  Google Scholar 

  37. Knill, E. Quantum computing with realistically noisy devices. Nature 434, 39–44 (2005).

    Article  ADS  Google Scholar 

  38. Carignan-Dugas, A., Wallman, J. J. & Emerson, J. Bounding the average gate fidelity of composite channels using the unitarity. New J. Phys. 21, 053016 (2019).

    Article  ADS  MathSciNet  Google Scholar 

  39. Nielsen, E. et al. Probing quantum processor performance with pyGSTi. Quantum Sci. Technol. 5, 044002 (2020).

    Article  ADS  Google Scholar 

  40. Nielsen, E. et al. PyGSTi Version 0.9.10 (Zenodo, 2021); https://doi.org/10.5281/zenodo.5546759

  41. Proctor, T., Rudinger, K., Nielsen, E., Young, K. & Blume-Kohout, R. Measuring the capabilities of quantum computers. Zenodo https://doi.org/10.5281/zenodo.5197499 (2021).

Download references

Acknowledgements

This work was supported by the United States Department of Energy, Office of Science, Office of Advanced Scientific Computing Research through the Quantum Testbed programme and the Accelerated Research in Quantum Computing (ARQC) programme, and the Laboratory-Directed Research and Development programme at Sandia National Laboratories. Sandia National Laboratories is a multi-programme laboratory managed and operated by National Technology and Engineering Solutions of Sandia, LLC., a wholly owned subsidiary of Honeywell International, Inc., for the United States Department of Energy’s National Nuclear Security Administration under contract DE-NA-0003525. All statements of fact, opinion or conclusions contained herein are those of the authors and should not be construed as representing the official views or policies of the United States Department of Energy or the United States Government, or the views of IBM or Rigetti Computing. We thank both the IBM Q and Rigetti Computing teams for extensive technical support, in particular A. Brown, J. Chow, J. Gambetta, S. Hassinger, A. Javadi, F. J. Martin Fernandez, P. Karalekas, R. Karle, D. McClure, D. McKay, P. Nation, N. Ochem, C. Osborn, E. Peterson, D. Moreda Rodriguez, M. Skilbeck, M. Tod and C. Wood.

Author information

Authors and Affiliations

  1. Quantum Performance Laboratory, Sandia National Laboratories, Albuquerque, NM, 87185, USA

    Timothy Proctor, Kenneth Rudinger, Kevin Young, Erik Nielsen & Robin Blume-Kohout

  2. Quantum Performance Laboratory, Sandia National Laboratories, Livermore, CA, 94550, USA

    Timothy Proctor, Kenneth Rudinger, Kevin Young, Erik Nielsen & Robin Blume-Kohout

Authors
  1. Timothy Proctor
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kenneth Rudinger
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kevin Young
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Erik Nielsen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Robin Blume-Kohout
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

T.P., K.Y. and R.B.-K. developed the methods, designed the experiments, analysed the data and wrote the manuscript. T.P., K.R., K.Y. and E.N. wrote the circuit sampling, data collection and data analysis code. K.R. ran the experiments.

Corresponding author

Correspondence to Timothy Proctor.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Peer review information Nature Physics thanks the anonymous reviewers for their contribution to the peer review of this work

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Supplementary Notes 1–9 and Figs. 1–5.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Proctor, T., Rudinger, K., Young, K. et al. Measuring the capabilities of quantum computers. Nat. Phys. 18, 75–79 (2022). https://doi.org/10.1038/s41567-021-01409-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41567-021-01409-7

This article is cited by

Search

Advanced search

Quick links

Nature Briefing AI and Robotics

Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing: AI and Robotics