Abstract
Recently, inspired by quantum annealing, many solvers specialized for unconstrained binary quadratic programming problems have been developed. For further improvement and application of these solvers, it is important to clarify the differences in their performance for various types of problems. In this study, the performance of four quadratic unconstrained binary optimization problem solvers, namely DWave Hybrid Solver Service (HSS), Toshiba Simulated Bifurcation Machine (SBM), Fujitsu Digital Annealer (DA), and simulated annealing on a personal computer, was benchmarked. The problems used for benchmarking were instances of real problems in MQLib, instances of the SATUNSAT phase transition point of random notallequal 3SAT (NAE 3SAT), and the Ising spin glass SherringtonKirkpatrick (SK) model. Concerning MQLib instances, the HSS performance ranked first; for NAE 3SAT, DA performance ranked first; and regarding the SK model, SBM performance ranked first. These results may help understand the strengths and weaknesses of these solvers.
Similar content being viewed by others
Introduction
Quantum annealing (QA)^{1,2}, which is a quantum heuristic algorithm for solving combinatorial optimization problems, has attracted a great deal of attention because it is implemented using real quantum systems by DWave Systems Inc.^{3,4}, aiming at becoming more powerful than classical algorithms such as simulated annealing (SA)^{5,6}. To use the current DWave’s QA device, a combinatorial optimization problem must be mapped to a quadratic unconstrained binary optimization (QUBO) problem. QUBO is an optimization problem of binary variables \({\mathrm{x}}_{{\mathrm{i}}} \in \left\{ {0, \, 1} \right\}\), where \({\mathrm{i}} \in \left\{ {{1},{ 2}, \ldots ,{\mathrm{ N}}} \right\}\), and its cost function to be minimized is defined as
where Q_{i, j} is a real number called QUBO matrix element. In general, QUBO is NPhard^{7}, and many NPcomplete problems and combinatorial optimization problems are mapped to QUBO^{8}.
Although current QA devices have limited capability owing to hardware implementation limitations, in anticipation of future developments of QA devices, methods using QUBO models for solving realworld problems in a variety of fields have been actively studied^{9,10,11,12,13,14,15}. Inspired by this trend, several sophisticated heuristic QUBO solvers have been developed and commercialized^{16,17,18,19}. It is highly nontrivial to determine whether a particular algorithm is more powerful than another because the performance of heuristic algorithms varies depending on the target problem. For successful application to realworld problems and further development of these QUBO solvers, it is necessary to clarify the strengths and weaknesses of each solver for various types of QUBO problems. In this study, we benchmarked the performance of three commercialized QUBO solvers including one using a real QA device: DWave Hybrid Solver Service (HSS), Toshiba Simulated Bifurcation Machine (SBM), and Fujitsu Digital Annealer (DA). In order to understand the characteristics of the solvers, we benchmark various types of problems, including Ising spin glass problems and realworld problems. This is in contrast to a similar benchmark study reported recently^{20}, which used only a single kind of constraint satisfaction problem (specifically, 3regular 3XORSAT). While in Ref.^{20}, the size dependence of the time to obtain an optimal solution with a certain probability is analyzed in detail, in this study, the performance of the solvers is evaluated by comparing the value of the cost function obtained for a given execution time. Such a performance evaluation will be helpful in application cases where approximate solutions are acceptable.
The remainder of this paper is organized as follows. In “QUBO solvers” section, we briefly explain the solvers benchmarked. In “Problem instances for benchmarking” section, the definition of the problem instances used for benchmarking are provided. In “Results” section, we present the results of the benchmarking experiment. Concluding remarks are given in “Discussion and conclusion” section.
QUBO solvers
In this section, we briefly explain the four solvers used in this study. Three commercial solvers were benchmarked. For comparison, we also experimented with SA on a personal computer.
The first solver is HSS, commercialized by DWave Systems Inc.^{16}. This solver is a socalled quantum–classical hybrid algorithm that employs QA as an accelerator. Note that the actual implementation of the algorithm is not open to the public. Thus, it is unclear how QA is used internally. We used HSS hybrid BQM solver, version 2.0, which can manage up to 10^{6} variables and 2 × 10^{8} couplings^{21}. We accessed HSS via Leap cloud.
The second solver is SBM, commercialized by Toshiba^{18}. The QA inspired algorithm of SBM, socalled simulated bifurcation (SB) algorithm, uses the adiabatic time evolution of Kerrnonlinear parametric oscillators (KPOs)^{22}. The dynamics in the classical limit of KPOs can be quickly computed in classical computers by solving the independent equations of motion in parallel^{18}. To overcome accuracy degradation caused by analog errors due to the use of the dynamics of continuous variables, a variant of the SB algorithm called ballistic SB (bSB) algorithm was developed, which mitigate the analog error by modifying the potential term of the equation of motion. As a further improvement of the bSB algorithm, the discrete SB (dSB) algorithm was also developed, which reduces the analog error by discretizing the potential term of the bSB algorithm^{23}. We use SBM evaluation version 1.5.1 (which is not publicly available), that uses dSB algorithm and can manage alltoall coupling of up to 10^{6} variables and 10^{8} nonzero couplings. Parallelization is 80 or 160 per GPU. In this study, we used the autoising solver; hyperparameters are automatically searched by the solver. We accessed SBM via evaluation version directly from Toshiba.
The third solver is DA, commercialized by Fujitsu^{19}. DA uses an SAspecific hardware architecture to accelerate the parallel tempering Markov chain Monte Carlo (MCMC) calculation^{24,25}. Although DA does not use quantum algorithms, it is inspired by DWave devices in the sense that the hardware is specialized for QUBO solving. We used fujitsuDA2PT solver, which can manage alltoall coupling of up to 8192 variables. We accessed DA via DA Center Japan.
For comparison with these commercial solvers, we ran SA using the opensource software DWave neal, version 0.5.7^{26}, on a personal computer with Ubuntu 20.04.3 LTS and Python 3.8.2. DWave neal implements SA with MCMC without parallel tempering method. The CPU used in the experiment was Intel Core i99900K, and singlethreaded runs were performed.
Problem instances for benchmarking
In this section, we explain the three problem sets used in the conducted benchmarking.
MQLib repository instances
We used the same set of 45 problems used in the benchmarks presented in HHS’s white paper^{16,27}. This problem set is extracted from the MQLib repository, and some of the problems have their origin in realworld problems, such as image segmentation^{28}. This problem set was reported to be timeconsuming to solve because of all the heuristics contained in the MQLib library. Concerning benchmarking, a 20minute run is recommended for each problem^{16}. The 45 problems are uniformly classified into nine classes: three classes according to size (small: 1000 ≤ N ≤ 2500, medium: 2500 < N ≤ 5000, and large: 5000 < N ≤ 10000) and three classes according to edge density (sparse: d ≤ 0.1, medium: 0.1 < d ≤ 0.5, and dense: 0.5 < d), where d is the number of edges divided by the number of edges in a complete graph of the same size^{16}.
NotAllEqual 3SAT
Satisfiability problem (SAT) is one of the most fundamental NPhard problems and therefore it is good benchmark problem for heuristic solvers. Notallequal 3SAT (NAE 3SAT) is a variant of the Boolean SAT problem and is an NPcomplete problem^{29}. NAE 3SAT requires at least one literal to be true and at least one literal to be false in each clause with three literals. The cost function of a random NAE 3SAT with N variables and M clauses is expressed in a straightforward manner in the Ising model with σ_{i} ∈ {− 1, 1}, where 1 ≤ i ≤ N:
where i_{m,l} ∈ {1, 2, … ,N} and ζ_{m,l} ∈ {− 1, 1} for 1 ≤ m ≤ M and 1 ≤ l ≤ 3 are random variables that follow a discrete uniform distribution; ζ_{m,l} = − 1 corresponds to the negation of the lth Boolean variable in clause m. Each clause has three different variables, i.e., i_{m,l} ̸ = i_{m,l}′ if l ̸ = l′. If the minimum of E(σ) in Eq. (2) is 0 for a given formula, it is satisfiable (SAT); otherwise, it is unsatisfiable (UNSAT). The QUBO formulation as in Eq. (1) can be easily obtained from this Ising formulation by the variable transformation x_{i} = (σ_{i} + 1)/2. Because NAE 3SAT has such a natural QUBO representation, it is a suitable benchmark problem for QUBO solvers amongst SAT variants. When the clausetovariable ratio is M/N = 2.11, the SATUNSAT phase transition occurs, and problem instances are most difficult to solve^{30,31,32}. In this study, we used randomly generated instances with this critical clausetovariable ratio for benchmarking.
Sherrington–Kirkpatrick model
The Sherrington–Kirkpatrick (SK) model is an Ising spin glass model with infinite spatial dimensions^{33,34}. The cost function of N variables with no external field is expressed as
where J_{i, j} is a random Gaussian variable. As previously explained, the QUBO formulation can be easily obtained. The mean field analysis shows that the energy landscape of the SK model has a manyvalley structure separated by asymptotically infinitely large energy barriers, which implies that it is extremely difficult to find the exact solution^{35}. In this study, we used randomly generated instances with J_{i, j} presenting zero mean and unity standard deviation for benchmarking.
Results
In this section, we present benchmarking results for each of the three problem sets introduced in the previous section. In the results shown below, the network time required to send the instance and receive the result was ignored in the measurement of execution time. Regarding HSS, the number of seconds specified in time_limit was used as the execution time. For SBM, the time specified in timeout was used as the execution time. Concerning DA, there was no parameter to specify the execution time directly, so total_elapsed_time recorded in the response file was used as the execution time. Finally, for SA with DWave neal, we measured the time taken for the sample function to finish.
MQLib instances
First, we present the results for a 5min experiment of the instances from the MQLib repository. For HSS and SBM, the execution time was set to 5 min. For DA, number_replicas was set to 128 and number_iterations was adjusted for each instance so that the deviation of execution time in 5 min was within 20 s. Concerning SA, num_sweeps was adjusted for each instance such that the execution time was 5 min.
Figure 1 shows the number of wins for each solver; this number was counted when the solver obtained the best solution. If there was more than one solver with the best solution, the number of wins was counted for all of them. The total result for all classes was that HHS won most of the problems (22), followed by DA (20), SBM (16), and SA (7). The results for each class classified by size show that HSS won the most for the small class, while DA won the most for the medium and large classes. The results for each class classified by edge density show that, for Sparse class, HSS won the most, for Medium class, DA won the most, and HSS and DA won the most. The number of wins of SA was only 2 at most, and most of the time, it was 0 or 1 for each of the nine classes.
Furthermore, we evaluate the quality of the obtained solution using a score defined as the ratio of the value of cost function
Tables 1, 2, 3, 4, 5, 6, 7, 8 and 9 show the score for each instance, and Fig. 1 shows the average of the scores for Small, Medium, and Large classes, and for all instances. The original lowest values of the cost function found in this benchmarking are listed in Table 10. The average scores of HSS and SBM are almost identical and higher than other solvers. This implies that HSS and SBM have stable performance on a wide range of problems. On the other hand, DA has an exceptionally bad solution for the instance g001345, which is why the average score drops significantly in the Large class. In addition, in the Small and Medium classes, the average score of DA is about 0.01 lower than the other solvers. This implies that DA is slightly less stable, because even for SA, which has the fewest wins, the difference in average score from HSS is within 0.001.
NAE 3SAT instances
Next, we present the results for the random NAE 3SAT instances with a number of variables N = 8192 and a number of clauses M = 17285, i.e., instances with a clausetovariable ratio N/M ≈ 2.11. Figure 2a shows the average of ten randomly generated instances of the cost function as a function of the execution time. As a reference, Fig. 2b shows the results for ten different instances. Given that each data point was obtained from an independent run, a longer run may lead to a worse solution than a shorter run. In the range 100–600 s, DA presented the lowest value of the cost function, closely followed by SBM and SA; HSS presented the highest value. In the region below 100 s, SBM and SA showed lower energy than DA. After a longtime calculation of about 1000 s, HSS finally reaches the same performance as SBM and SA, but still not as good as the result of 100 s run of DA. Interestingly, the performance of SBM and SA is almost identical for a wide range of execution time.
SK model
Finally, we present the results for the SK model with 8192 variables. As with the NAE 3SAT instances, the experiments were performed by varying the execution time. Figure 3a shows the average of six randomly generated instances of the cost function as a function of the execution time. As a reference, Fig. 3b shows the results for six different instances. SBM clearly outperformed the other solvers, achieving the best solutions at the 100s mark, with little energy change for longer runs. HSS and DA showed almost the same time dependence, although HSS provided a slightly better solution. It is interesting that this pair is different from the pair, SBM and SA, that exhibits similar performance in NAE 3SAT instances. In runs longer than 600 s, SA obtained as good solutions as HSS and DA, but due to the alltoall coupling, its preprocessing calculation was expensive, requiring at least approximately 500 s for the total calculation time.
Discussion and conclusion
We benchmarked the heuristic QUBO solvers, HSS, SBM, DA, and SA, using the instances from the MQLib repository, random NAE 3SAT, and the SK model. Benchmarking with problems of various origins revealed some of the characteristics of the strengths and weaknesses of each solver. For MQLib instances, which are a set of various problem instances including realworld problems, HSS showed the best performance on average, and SBM also showed stable performance that was not so different from HSS. DA outperformed other solvers on large instances, but it gave slightly poor solutions to some instances. It is rather natural result that the performance of DA varied depending on the instances because the performance of heuristic algorithms strongly depends on the problem instances in general, and it is somewhat surprising that HSS and SBM showed stable performance. In this experiment, with a run time of 5 min, we find that the difference in the value of cost function of the obtained solutions is often less than 0.01%, which is probably negligible in some application cases. Therefore, a possible direction for further study is to investigate how the results change in experiments with shorter run times. For random NAE 3SAT instances at the SATUNSAT transition point, which is a typical hard optimization problem, DA performed best for most of the execution times. The performance of SBM and SA was almost the same, and HSS was the worst. It is believed that local search methods such as the parallel tempering method used in DA do not work well for SAT instances at the SATUNSAT transition point that have very few solutions^{31}, and there is probably no efficient algorithm. Therefore, the result that DA still performed best implies that other solvers are also not particularly effective, which is as expected. For SK model, which is a typical hard problem originated from the spin glass, SBM exhibited a clear advantage over other solvers, while HSS and DA showed similar performance. Since the parallel tempering method is considered to work relatively well for the SK model, it is a bit surprising that SBM, rather than DA, showed outstanding performance as opposed to the case of NAE 3SAT. It is an important challenge to understand the characteristics of each solver found in this study from the viewpoint of their algorithm and hardware architecture.
Data availability
All other data used in this study are available from the corresponding authors upon reasonable request. The problem instaces of MQlib is available from the MQLib repository^{36}. The NAE 3SAT and SK model instance was generated reproduced by the python program shown in Listings 1 and 2 with Python 3.8.2 on Ubuntu20.04.3 LTS.
References
Farhi, E. et al. A quantum adiabatic evolution algorithm applied to random instances of an npcomplete problem. Science 292, 472–475. https://doi.org/10.1126/science.1057726 (2001).
Das, A. & Chakrabarti, B. K. Colloquium: Quantum annealing and analog quantum computation. Rev. Mod. Phys. 80, 1061–1081. https://doi.org/10.1103/RevModPhys.80.1061 (2008).
Johnson, M. et al. Quantum annealing with manufactured spins. Nature 473, 194–198. https://doi.org/10.1038/nature10012 (2011).
Harris, R. et al. Experimental investigation of an eightqubit unit cell in a superconducting optimization processor. Phys. Rev. B 82, 024511. https://doi.org/10.1103/PhysRevB.82.024511 (2010).
Kirkpatrick, S., Gelatt, C. D. & Vecchi, M. P. Optimization by simulated annealing. Science 220, 671–680. https://doi.org/10.1126/science.220.4598.671 (1983).
Fu, Y. & Anderson, P. W. Application of statistical mechanics to NPcomplete problems in combinatorial optimisation. J. Phys. A: Math. Gen. 19, 1605–1620. https://doi.org/10.1088/03054470/19/9/033 (1986).
Barahona, F. On the computational complexity of ising spin glass models. J. Phys. A: Math. Gen. 15, 3241–3253. https://doi.org/10.1088/03054470/15/10/028 (1982).
Lucas, A. Ising formulations of many np problems. Front. Phys. 2, 5. https://doi.org/10.3389/fphy.2014.00005 (2014).
PerdomoOrtiz, A., Dickson, N., DrewBrook, M., Rose, G. & AspuruGuzik, A. Finding lowenergy conformations of lattice protein models by quantum annealing. Sci. Rep. 2, 571. https://doi.org/10.1038/srep00571 (2012).
Garnerone, S., Zanardi, P. & Lidar, D. A. Adiabatic quantum algorithm for search engine ranking. Phys. Rev. Lett. 108, 230506. https://doi.org/10.1103/PhysRevLett.108.230506 (2012).
Babbush, R., Love, P. J. & AspuruGuzik, A. Adiabatic quantum simulation of quantum chemistry. Sci. Reports 4, DOI: https://doi.org/10.1038/srep06603 (2014).
Venturelli, D., Marchand, D. J. J. & Rojo, G. Quantum annealing implementation of jobshop scheduling (2016). 1506.08479.
Mott, A., Job, J., Vlimant, J. R., Lidar, D. & Spiropulu, M. Solving a Higgs optimization problem with quantum annealing for machine learning. Nature 550, 375–379. https://doi.org/10.1038/nature24047 (2017).
Benedetti, M., RealpeGómez, J., Biswas, R. & PerdomoOrtiz, A. Quantumassisted learning of hardwareembedded probabilistic graphical models. Phys. Rev. X 7, 041052. https://doi.org/10.1103/PhysRevX.7.041052 (2017).
Li, R. Y., Di Felice, R., Rohs, R. & Lidar, D. A. Quantum annealing versus classical machine learning applied to a simplified computational biology problem. NPJ Quantum Inf. 4, 14. https://doi.org/10.1038/s4153401800608 (2018).
Dwave hybrid solver service: An overview. https://www.dwavesys.com/sites/default/files/141039AA_DWave_Hybrid_Solver_Service_An_Overview.pdf.
Inagaki, T. et al. A coherent ising machine for 2000node optimization problems. Science 354, 603–606. https://doi.org/10.1126/science.aah4243 (2016).
Goto, H., Tatsumura, K. & Dixon, A. R. Combinatorial optimization by simulating adiabatic bifurcations in nonlinear Hamiltonian systems. Sci. Adv. https://doi.org/10.1126/sciadv.aav2372 (2019).
Aramon, M. et al. Physicsinspired optimization for quadratic unconstrained problems using a digital annealer. Front. Phys. 7, 48. https://doi.org/10.3389/fphy.2019.00048 (2019).
Kowalsky, M., Albash, T., Hen, I. & Lidar, D. A. 3regular 3xorsat planted solutions benchmark of classical and quantum heuristic optimizers (2021). 2103.08464.
Dwave hybrid solver service + advantage: Technology update. https://www.dwavesys.com/sites/default/files/141048AA_DWave_Hybrid_Solver_Service_plus_Advantage_Technology_Update.pdf.
Goto, H. Bifurcationbased adiabatic quantum computation with a nonlinear oscillator network: Toward quantum soft computing. Sci. Rep. https://doi.org/10.1038/srep21686 (2016).
Goto, H. et al. Highperformance combinatorial optimization based on classical mechanics. Sci. Adv. 7, eabe7953. https://doi.org/10.1126/sciadv.abe7953 (2021).
Matsubara, S. et al. Isingmodel optimizer with paralleltrial bitsieve engine. In Complex, Intelligent, and Software Intensive Systems (eds Barolli, L. & Terzo, O.) 432–438 (Springer, 2018).
Tsukamoto, S., Takatsu, M., Matsubara, S. & Tamura, H. An accelerator architecture for combinatorial optimization problems. Fujitsu Sci. Tech. J. 53, 8–13 (2017).
dwaveneal. https://github.com/dwavesystems/dwaveneal.
Hss overview whitepaper benchmarks. https://github.com/dwavesystems/hssoverviewbenchmarks.
Dunning, I., Gupta, S. & Silberholz, J. What works best when? A systematic evaluation of heuristics for maxcut and QUBO. INFORMS J. Comput. 30, 608–624 (2018).
Darmann, A. & Döcker, J. On simplified npcomplete variants of notallequal 3SAT and 3SAT (2019). 1908.04198.
Achlioptas, D., Chtcherba, A., Istrate, G. & Moore, C. The phase transition in 1ink SAT and NAE 3SAT. In Proceedings of the Annual ACMSIAM Symposium on Discrete Algorithms. https://doi.org/10.1145/365411.365760 (2001).
Clark, D. A. et al. Local search and the number of solutions. In Principles and Practice of Constraint Programming—CP96 (ed. Freuder, E. C.) 119–133 (Springer, 1996).
Gent, I. P. & Walsh, T. The sat phase transition. In ECAI (1994).
Sherrington, D. & Kirkpatrick, S. Solvable model of a spinglass. Phys. Rev. Lett. 35, 1792–1796. https://doi.org/10.1103/PhysRevLett.35.1792 (1975).
Mezard, M., Parisi, G. & Virasoro, M. Spin Glass Theory and Beyond: An Introduction to the Replica Method and its Applications. World Scientific Lecture Notes in Physics (World Scientific Publishing Company, 1987).
Thouless, D. J., Anderson, P. W. & Palmer, R. G. Solution of ‘solvable model of a spin glass’. Philos. Mag. A J. Theor. Exp. Appl. Phys. 35, 593–601. https://doi.org/10.1080/14786437708235992 (1977).
Mqlib. https://github.com/MQLib/MQLib.
Acknowledgements
We thank Murray Thom, Catherine McGeoch, Hayato Goto, and Yoshihiko Nishikawa for fruitful discussion on our benchmark tests. In addition, we acknowledge research supports on various aspects from DWave Systems Inc. and TOSHIBA CORPORATION. M. O. thanks financial support from JSPS KAKENHI Grant Number 20H02168, the Next Generation HighPerformance Computing Infrastructures and Applications R & D Program by MEXT, and MEXTQuantum Leap Flagship Program Grant Number JPMXS0120352009.
Author information
Authors and Affiliations
Contributions
H.O. and M.O. wrote the main manuscript. All authors reviewed the manuscript.
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing interests.
Additional information
Publisher's note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Oshiyama, H., Ohzeki, M. Benchmark of quantuminspired heuristic solvers for quadratic unconstrained binary optimization. Sci Rep 12, 2146 (2022). https://doi.org/10.1038/s41598022060705
Received:
Accepted:
Published:
DOI: https://doi.org/10.1038/s41598022060705
This article is cited by

Performance of quantum annealing inspired algorithms for combinatorial optimization problems
Communications Physics (2024)

Message passing variational autoregressive network for solving intractable Ising models
Communications Physics (2024)

On good encodings for quantum annealer and digital optimization solvers
Scientific Reports (2023)

An optimization case study for solving a transport robot scheduling problem on quantumhybrid and quantuminspired hardware
Scientific Reports (2023)

Travel time optimization on multiAGV routing by reverse annealing
Scientific Reports (2022)
Comments
By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.