Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Optimizing quantum annealing schedules with Monte Carlo tree search enhanced with neural networks

Nature Machine Intelligence volume 4, pages 269–278 (2022)Cite this article

Subjects

Matters Arising to this article was published on 06 October 2022

A preprint version of the article is available at arXiv.

Abstract

Quantum annealing is a practical approach to approximately implement the adiabatic quantum computational model in a real-world setting. The goal of an adiabatic algorithm is to prepare the ground state of a problem-encoded Hamiltonian at the end of an annealing path. This is typically achieved by driving the dynamical evolution of a quantum system slowly to enforce adiabaticity. Properly optimized annealing schedules often considerably accelerate the computational process. Inspired by the recent success of deep reinforcement learning such as DeepMind’s AlphaZero, we propose a Monte Carlo tree search (MCTS) algorithm and its enhanced version boosted by neural networks—which we name QuantumZero (QZero)—to automate the design of annealing schedules in a hybrid quantum–classical framework. Both the MCTS and QZero algorithms perform remarkably well in discovering effective annealing schedules even when the annealing time is short for the 3-SAT examples considered in this study. Furthermore, the flexibility of neural networks allows us to apply transfer-learning techniques to boost QZero’s performance. We demonstrate in benchmark studies that MCTS and QZero perform more efficiently than other reinforcement learning algorithms in designing annealing schedules.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Hybrid quantum–classical framework for designing annealing schedules.
Fig. 2: Set-up of MCTS.
Fig. 3: Success probability of solving several 3-SAT instances with different structures.
Fig. 4: Illustration of transferring annealing schedules.
Fig. 5
Fig. 6: Comparing the learning efficiency among RL algorithms.

Similar content being viewed by others

Data availability

The data that support the results in this paper is available at https://github.com/yutuer21/quantumzero.

Code availability

The code that support the results in this paper is available at https://github.com/yutuer21/quantumzero(ref. 71).

References

  1. Jiang, S., Britt, K. A., McCaskey, A. J., Humble, T. S. & Kais, S. Quantum annealing for prime factorization. Sci. Rep. 8, 17667 (2018).

    Article  Google Scholar 

  2. King, A. D. et al. Observation of topological phenomena in a programmable lattice of 1,800 qubits. Nature 560, 456–460 (2018).

    Article  Google Scholar 

  3. Harris, R. et al. Phase transitions in a programmable quantum spin glass simulator. Science 361, 162–165 (2018).

    Article  MathSciNet  Google Scholar 

  4. Willsch, D., Willsch, M., De Raedt, H. & Michielsen, K. Support vector machines on the D-wave quantum annealer. Comput. Phys. Commun. 248, 107006 (2020).

    Article  MathSciNet  Google Scholar 

  5. Mott, A., Job, J., Vlimant, Jean-Roch, Lidar, D. & Spiropulu, M. Solving a Higgs optimization problem with quantum annealing for machine learning. Nature 550, 375–379 (2017).

    Article  Google Scholar 

  6. Li, R. Y., Di Felice, R., Rohs, R. & Lidar, D. A. Quantum annealing versus classical machine learning applied to a simplified computational biology problem. npj Quant. Inf. 4, 14 (2018).

    Article  Google Scholar 

  7. Hormozi, L., Brown, E. W., Carleo, G. & Troyer, M. Nonstoquastic Hamiltonians and quantum annealing of an Ising spin glass. Phys. Rev. B 95, 184416 (2017).

    Article  Google Scholar 

  8. Herr, D. et al. Optimizing schedules for quantum annealing. Preprint at https://arxiv.org/abs/1705.00420 (2017).

  9. Zeng, L., Zhang, J. & Sarovar, M. Schedule path optimization for adiabatic quantum computing and optimization. J Phys. A 49, 165305 (2016).

    Article  MathSciNet  MATH  Google Scholar 

  10. Susa, Y., Yamashiro, Y., Yamamoto, M. & Nishimori, H. Exponential speedup of quantum annealing by inhomogeneous driving of the transverse field. J. Phys. Soc. Jpn 87, 023002 (2018).

    Article  Google Scholar 

  11. Albash, T. & Lidar, D. A. Adiabatic quantum computation. Rev. Mod. Phys. 90, 015002 (2018).

    Article  MathSciNet  Google Scholar 

  12. Hauke, P. et al. Perspectives of quantum annealing: Methods and implementations. Rep. Progr. Phys. 83, 054401 (2020).

    Article  Google Scholar 

  13. Farhi, E., Goldstone, J., Gutmann, S. & Sipser, M. Quantum computation by adiabatic evolution. Preprint at https://arxiv.org/abs/quant-ph/0001106 (2000).

  14. Farhi, E. et al. A quantum adiabatic evolution algorithm applied to random instances of an NP-complete problem. Science 292, 472–475 (2001).

    Article  MathSciNet  MATH  Google Scholar 

  15. Das, A. & Chakrabarti, B. K. Quantum Annealing and Related Optimization Methods (Springer, 2005).

  16. Childs, A. M., Farhi, E. & Preskill, J. Robustness of adiabatic quantum computation. Phys. Rev. A 65, 012322 (2001).

    Article  Google Scholar 

  17. Aharonov, D. et al. Adiabatic quantum computation is equivalent to standard quantum computation. SIAM Rev. 50, 755–787 (2008).

    Article  MathSciNet  MATH  Google Scholar 

  18. Susa, Y. & Nishimori, H. Variational optimization of the quantum annealing schedule for the Lechner–Hauke–Zoller scheme. Phys. Rev. A 103, 022619 (2021).

    Article  Google Scholar 

  19. Herr, D. et al. Optimizing schedules for quantum annealing. Preprint at https://arxiv.org/abs/1705.00420 (2017).

  20. Schiffer, B. F., Tura, J. & Cirac, J. I. Adiabatic spectroscopy and a variational quantum adiabatic algorithm. Preprint at https://arxiv.org/abs/2103.01226 (2021).

  21. Boixo, S. et al. Eigenpath traversal by phase randomization. Quant. Inf. Comput. 9, 833–855 (2009).

    MathSciNet  MATH  Google Scholar 

  22. Coulom, R. in Computers and Games 72–83 (Springer, 2007); https://doi.org/10.1007/978-3-540-75538-8_7

  23. Kocsis, L. & Szepesvári, C. Bandit based Monte-Carlo planning. In European Conference on Machine Learning 282–293 (Springer, 2006).

  24. Lee, Chang-Shing et al. The computational intelligence of Mogo revealed in Taiwan’s computer Go tournaments. IEEE Trans. Comput. Intell. AI Games 1, 73–89 (2009).

    Article  Google Scholar 

  25. Silver, D. et al. Mastering the game of go without human knowledge. Nature 550, 354–359 (2017).

    Article  Google Scholar 

  26. Silver, D. et al. A general reinforcement learning algorithm that masters Chess, Shogi, and Go through self-play. Science 362, 1140–1144 (2018).

    Article  MathSciNet  MATH  Google Scholar 

  27. Peruzzo, A. et al. A variational eigenvalue solver on a photonic quantum processor. Nat. Commun. 5, 4213 (2014).

    Article  Google Scholar 

  28. Kandala, A. et al. Hardware-efficient variational quantum eigensolver for small molecules and quantum magnets. Nature 549, 242–246 (2017).

    Article  Google Scholar 

  29. Farhi, E., Goldstone, J. & Gutmann, S. A quantum approximate optimization algorithm. Preprint at https://arxiv.org/abs/1411.4028 (2014).

  30. McClean, J. R., Romero, J., Babbush, R. & Aspuru-Guzik, A. The theory of variational hybrid quantum–classical algorithms. New. J. Phys. 18, 023023 (2016).

    Article  MATH  Google Scholar 

  31. Preskill, J. Quantum computing in the NISQ era and beyond. Quantum 2, 79 (2018).

    Article  Google Scholar 

  32. Cao, Y. et al. Quantum chemistry in the age of quantum computing. Chem. Rev. 119, 10856–10915 (2019).

    Article  Google Scholar 

  33. Chen, M.-C. et al. Demonstration of adiabatic variational quantum computing with a superconducting quantum coprocessor. Phys. Rev. Lett. 125, 180501 (2020).

    Article  Google Scholar 

  34. Dalgaard, M., Motzoi, F., Sørensen, J. J. & Sherson, J. Global optimization of quantum dynamics with AlphaZero deep exploration. npj Quant. Inf. 6, 6 (2020).

    Article  Google Scholar 

  35. Zhang, X.-M., Wei, Z., Asad, R., Yang, X.-C. & Wang, X. When reinforcement learning stands out in quantum control? A comparative study on state preparation. npj Quant. Inf. 5, 85 (2019).

    Article  Google Scholar 

  36. Chen, C., Dong, D., Li, H.-X., Chu, J. & Tarn, T.-J. Fidelity-based probabilistic Q-learning for control of quantum systems. IEEE Trans. Neural Netw. Learn. Syst. 25, 920–933 (2014).

    Article  Google Scholar 

  37. Bukov, M. et al. Reinforcement learning in different phases of quantum control. Phys. Rev. X 8, 031086 (2018).

    Google Scholar 

  38. Niu, M. Y., Boixo, S., Smelyanskiy, V. N. & Neven, H. Universal quantum control through deep reinforcement learning. npj Quant. Inf. 5, 33 (2019).

    Article  Google Scholar 

  39. McKiernan, K. A., Davis, E., Alam, M. S. & Rigetti, C. Automated quantum programming via reinforcement learning for combinatorial optimization. Preprint at https://arxiv.org/abs/1908.08054 (2019).

  40. Khairy, S., Shaydulin, R., Cincio, L., Alexeev, Y. & Balaprakash, P. Reinforcement-learning-based variational quantum circuits optimization for combinatorial problems. Preprint at https://arxiv.org/abs/1911.04574 (2019).

  41. Lin, J., Lai, Z. Y. & Li, X. Quantum adiabatic algorithm design using reinforcement learning. Phys. Rev. A 101, 052327 (2020).

    Article  Google Scholar 

  42. Beloborodov, D. et al. Reinforcement learning enhanced quantum-inspired algorithm for combinatorial optimization. Mach. Learn. Sci. Technol. 2, 025009 (2020).

    Article  Google Scholar 

  43. Ayanzadeh, R., Halem, M. & Finin, T. Reinforcement quantum annealing: a quantum-assisted learning automata approach. Preprint at https://arxiv.org/abs/2001.00234 (2020).

  44. Sutton, R. S. and Barto, A. G. Reinforcement Learning: An Introduction (Bradford, 2018).

  45. Kaelbling, L. P., Littman, M. L. & Moore, A. W. Reinforcement learning: a survey. J. Artif. Intell. Res. 4, 237–285 (1996).

    Article  Google Scholar 

  46. van Otterlo, M. & Wiering, M. in Adaptation, Learning, and Optimization 3–42 (Springer, 2012); https://doi.org/10.1007/978-3-642-27645-3_1

  47. Mnih, V. et al. Playing Atari with deep reinforcement learning. Preprint at https://arxiv.org/abs/1312.5602 (2013).

  48. Mnih, V. et al. Human-level control through deep reinforcement learning. Nature 518, 529–533 (2015).

    Article  Google Scholar 

  49. Schulman, J., Wolski, F., Dhariwal, P., Radford, A. & Klimov, O. Proximal policy optimization algorithms. Preprint at https://arxiv.org/abs/1707.06347 (2017)

  50. Vodopivec, T., Samothrakis, S. & Ster, B. On Monte Carlo tree search and reinforcement learning. J. Artif. Intell. Res. 60, 881–936 (2017).

    Article  MathSciNet  MATH  Google Scholar 

  51. Nautrup, HendrikPoulsen, Delfosse, N., Dunjko, V., Briegel, H. J. & Friis, N. Optimizing quantum error correction codes with reinforcement learning. Quantum 3, 215 (2019).

    Article  Google Scholar 

  52. Kanno, S. & Tada, T. Many-body calculations for periodic materials via restricted Boltzmann machine-based VQE. Quant. Sci. Technol. 6, 025015 (2021).

    Article  Google Scholar 

  53. Morales, M. E. S., Biamonte, J. & Zimborás, Z. On the universality of the quantum approximate optimization algorithm. Quant. Inf. Process. 19, 291 (2020).

    Article  MathSciNet  Google Scholar 

  54. Caneva, T., Calarco, T. & Montangero, S. Chopped random-basis quantum optimization. Phys. Rev. A 84, 022326 (2011).

    Article  Google Scholar 

  55. Fösel, T., Tighineanu, P., Weiss, T. & Marquardt, F. Reinforcement learning with neural networks for quantum feedback. Phys. Rev. X 8, 031084 (2018).

    Google Scholar 

  56. Xu, H. et al. Generalizable control for quantum parameter estimation through reinforcement learning. npj Quant. Inf. 5, 82 (2019).

    Article  Google Scholar 

  57. Wallnöfer, J. et al. Machine learning for long-distance quantum communication. PRX Quant. 1, 010301 (2020).

    Article  Google Scholar 

  58. Karanikolas, V. & Kawabata, S. Improved performance of quantum annealing by a diabatic pulse application. Preprint at https://arxiv.org/abs/1806.08517 (2018).

  59. King, J. et al. Quantum annealing amid local ruggedness and global frustration. J. Phys. Soc. Jpn 88, 061007 (2019).

    Article  Google Scholar 

  60. Hogg, T. Adiabatic quantum computing for random satisfiability problems. Phys. Rev. A 67, 022314 (2003).

    Article  Google Scholar 

  61. Žnidarič, M. Scaling of the running time of the quantum adiabatic algorithm for propositional satisfiability. Phys. Rev. A 71, 062305 (2005).

    Article  MathSciNet  Google Scholar 

  62. Kirkpatrick, S. & Selman, B. Critical behavior in the satisfiability of random boolean expressions. Science 264, 1297–1301 (1994).

    Article  MathSciNet  MATH  Google Scholar 

  63. Monasson, R., Zecchina, R., Kirkpatrick, S., Selman, B. & Troyansky, L. Determining computational complexity from characteristic ‘phase transitions’. Nature 400, 133–137 (1999).

    Article  MathSciNet  MATH  Google Scholar 

  64. Brockman, G. et al. OpenAI gym. Preprint at https://arxiv.org/abs/1606.01540 (2016).

  65. Dhariwal, P. et al. OpenAI Baselines (GitHub, 2017); https://github.com/openai/baselines

  66. Lloyd, S. Quantum approximate optimization is computationally universal. Preprint at https://arxiv.org/abs/1812.11075 (2018).

  67. Farhi, E., Goldstone, J. & Gutmann, S. Quantum adiabatic evolution algorithms versus simulated annealing. Preprint at https://arxiv.org/abs/quant-ph/0201031 (2002).

  68. Kong, L. & Crosson, E. The performance of the quantum adiabatic algorithm on spike Hamiltonians. Int. J. Quant. Inf. 15, 1750011 (2017).

    Article  MathSciNet  MATH  Google Scholar 

  69. Roland, J. & Cerf, N. J. Quantum search by local adiabatic evolution. Phys. Rev. A 65, 042308 (2002).

    Article  Google Scholar 

  70. Brady, L. T., Baldwin, C. L., Bapat, A., Kharkov, Y. & Gorshkov, A. V. Optimal protocols in quantum annealing and quantum approximate optimization algorithm problems. Phys. Rev. Lett. 126, 070505 (2021).

    Article  MathSciNet  Google Scholar 

  71. Chen, Y. et al. yutuer21/quantumzero: Quantumzero (Zenodo, 2021); https://doi.org/10.5281/zenodo.5749588

Download references

Author information

Authors and Affiliations

  1. Tencent Quantum Laboratory, Tencent, Shenzhen, Guangdong, China

    Yu-Qin Chen, Chee-Kong Lee, Shengyu Zhang & Chang-Yu Hsieh

  2. The Chinese University of Hong Kong, Hong Kong, Hong Kong

    Yu Chen

Authors
  1. Yu-Qin Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yu Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chee-Kong Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shengyu Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Chang-Yu Hsieh
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Y.Q.C. performed the simulations and analysed the data. Y.Q.C. and Y.C. wrote the code. All authors contributed to interpreting data and engaged in useful scientific discussions. C.Y.H. conceived and supervised this project. All authors contributed to the writing.

Corresponding author

Correspondence to Chang-Yu Hsieh.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review

Peer review information

Nature Machine Intelligence thanks Evert van Nieuwenburg and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Supplementary Figs. 1–7.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, YQ., Chen, Y., Lee, CK. et al. Optimizing quantum annealing schedules with Monte Carlo tree search enhanced with neural networks. Nat Mach Intell 4, 269–278 (2022). https://doi.org/10.1038/s42256-022-00446-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s42256-022-00446-y

This article is cited by

Search

Advanced search

Quick links

Nature Briefing AI and Robotics

Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing: AI and Robotics