Ensuring that autonomous vehicles do not cause accidents remains a challenge. We present a formal verification technique for guaranteeing legal safety in arbitrary urban traffic situations. Legal safety means that autonomous vehicles never cause accidents although other traffic participants are allowed to perform any behaviour in accordance with traffic rules. Our technique serves as a safety layer for existing motion planning frameworks that provide intended trajectories for autonomous vehicles. We verify whether intended trajectories comply with legal safety and provide fallback solutions in safety-critical situations. The benefits of our verification technique are demonstrated in critical urban scenarios, which have been recorded in real traffic. The autonomous vehicle executed only safe trajectories, even when using an intended trajectory planner that was not aware of other traffic participants. Our results indicate that our online verification technique can drastically reduce the number of traffic accidents.
Subscribe to Journal
Get full journal access for 1 year
only $8.25 per issue
All prices are NET prices.
VAT will be added later in the checkout.
Tax calculation will be finalised during checkout.
Rent or Buy article
Get time limited or full article access on ReadCube.
All prices are NET prices.
All data gathered and reported in this study are available in the Supplementary data file. This includes the environment model, the intended trajectory and the verification result of each verification cycle for all scenarios.
The code to visualize and analyse the gathered data and obtained results of this study are included in the Supplementary data file.
Favarò, F., Eurich, S. & Nader, N. Autonomous vehicles’ disengagements: trends, triggers and regulatory limitations. Accid. Anal. Prev. 110, 136–148 (2018).
Anderson, J. M. et al. Autonomous Vehicle Technology: A Guide for Policymakers (Rand Corporation, 2016).
Koopman, P. & Wagner, M. Autonomous vehicle safety: an interdisciplinary challenge. IEEE Intell. Transportation Syst. Mag. 9, 90–96 (2017).
Kalra, N. & Paddock, S. M. Driving to safety: how many miles of driving would it take to demonstrate autonomous vehicle reliability? Transportation Res. A Policy Practice 94, 182–193 (2016).
Seshia, S. A., Sadigh, D. & Sastry, S. S. Towards verified artificial intelligence. Preprint at https://arxiv.org/abs/1606.08514 (2017).
Schwarting, W., Alonso-Mora, J. & Rus, D. Planning and decision-making for autonomous vehicles. Annu. Rev. Control Robot. Autonomous Syst. 1, 187–210 (2018).
United Nations Economic Commission for Europe. Convention on Road Traffic. United Nations Conference on Road Traffic (United Nations, 1968); consolidated version of 2006.
Vanholme, B., Gruyer, D., Lusetti, B., Glaser, S. & Mammar, S. Highly automated driving on highways based on legal safety. IEEE Trans. Intell. Transportation Syst. 14, 333–347 (2013).
Althoff, M. & Dolan, J. M. Online verification of automated road vehicles using reachability analysis. IEEE Trans. Robotics 30, 903–918 (2014).
Koopman, P. & Wagner, M. Challenges in autonomous vehicle testing and validation. SAE Int. J. Transportation Safety 4, 15–24 (2016).
Dahl, J., de Campos, G. R., Olsson, C. & Fredriksson, J. Collision avoidance: a literature review on threat-assessment techniques. IEEE Trans. Intell. Vehicles 4, 101–113 (2019).
Tumova, J., Hall, G. C., Karaman, S., Frazzoli, E. & Rus, D. Least-violating control strategy synthesis with safety rules. In Proceedings of the 16th International Conference on Hybrid Systems: Computation and Control 1–10 (HSCC, 2013).
Kress-Gazit, H., Fainekos, G. E. & Pappas, G. J. Temporal-logic-based reactive mission and motion planning. IEEE Trans. Robotics 25, 1370–1381 (2009).
Fraichard, T. & Asama, H. Inevitable collision states—a step towards safer robots? In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems 388–393 (IEEE, 2003).
Chan, N., Kuffner, J. & Zucker, M. Improved motion planning speed and safety using regions of inevitable collision. In 17th CISM-IFToMM Symposium on Robot Design, Dynamics and Control 103–114 (Springer, 2008).
Koller, T., Berkenkamp, F., Turchetta, M. & Krause, A. Learning-based model predictive control for safe exploration. In Proceedings of the 2018 IEEE International Conference on Decision and Control 6059–6066 (IEEE, 2018).
Wabersich, K. P. & Zeilinger, M. N. Linear model predictive safety certification for learning-based control. In Proceedings of the IEEE International Conference on Decision and Control 7130–7135 (IEEE, 2018).
Sadraddini, S. & Belta, C. A provably correct MPC approach to safety control of urban traffic networks. In Proceedings of the American Control Conference 1679–1684 (2016).
Ames, A. D. et al. Control barrier functions: theory and applications. In Proceedings of the 18th European Control Conference 3420–3431 (IEEE, 2019).
Tedrake, R., Manchester, I. R., Tobenkin, M. & Roberts, J. W. LQR-trees: feedback motion planning via sums-of-squares verification. Int. J. Robotics Res. 29, 1038–1052 (2010).
Li, W., Sadigh, D., Sastry, S. S. & Seshia, S. A. Synthesis for human-in-the-loop control systems. In Proceedings of the International Conference on Tools and Algorithms for the Construction and Analysis of Systems 470–484 (Springer, 2014).
Jalalmaab, M., Fidan, B., Jeon, S. & Falcone, P. Guaranteeing persistent feasibility of model predictive motion planning for autonomous vehicles. In Proceedings of the 2017 IEEE Intelligent Vehicles Symposium 843–848 (IEEE, 2017).
Danielson, C., Weiss, A., Berntorp, K. & Di Cairano, S. Path planning using positive invariant sets. In Proceedings of the 55th International Conference on Decision and Control 5986–5991 (IEEE, 2016).
Herbert, S. L. et al. FaSTrack: a modular framework for fast and guaranteed safe motion planning. In Proceedings of the 56th International Conference on Decision and Control 1517–1522 (IEEE, 2017).
Falcone, P., Ali, M. & Sjöberg, J. Predictive threat assessment via reachability analysis and set invariance theory. IEEE Trans. Intell. Transportation Syst. 12, 1352–1361 (2011).
Vaskov, S. et al. Towards provably not-at-fault control of autonomous robots in arbitrary dynamic environments. In Proc. Robotics: Science and Systems 1–9 (2019).
Lefèvre, S., Vasquez, D. & Laugier, C. A survey on motion prediction and risk assessment for intelligent vehicles. ROBOMECH J. 1, 1–14 (2014).
Gindele, T., Brechtel, S. & Dillmann, R. Learning driver behavior models from traffic observations for decision making and planning. IEEE Intell. Transportation Syst. Mag. 7, 69–79 (2015).
Bahram, M., Hubmann, C., Lawitzky, A., Aeberhard, M. & Wollherr, D. A combined model- and learning-based framework for interaction-aware maneuver prediction. IEEE Trans. Intell. Transportation Syst. 17, 1538–1550 (2016).
Deo, N., Rangesh, A. & Trivedi, M. M. How would surround vehicles move? A unified framework for maneuver classification and motion prediction. IEEE Trans. Intell. Vehicles 3, 129–140 (2018).
Ghahramani, Z. Probabilistic machine learning and artificial intelligence. Nature 521, 452–459 (2015).
Tang, C., Chen, J. & Tomizuka, M. Adaptive probabilistic vehicle trajectory prediction through physically feasible Bayesian recurrent neural network. In Proceedings of the 2019 IEEE International Conference on Robotics and Automation 3846–3852 (IEEE, 2019).
Pool, E. A. I., Kooij, J. F. P. & Gavrila, D. M. Context-based cyclist path prediction using recurrent neural networks. In Proceedings of the 2019 IEEE Intelligent Vehicles Symposium 824–830 (IEEE, 2019).
Wu, A. & How, J. Guaranteed infinite horizon avoidance of unpredictable, dynamically constrained obstacles. Autonomous Robots 32, 227–242 (2012).
Bouraine, S., Fraichard, T. & Salhi, H. Provably safe navigation for mobile robots with limited field-of-views in dynamic environments. Autonomous Robots 32, 267–283 (2012).
Yang, Y., Zhang, J., Cai, K. & Prandini, M. Multi-aircraft conflict detection and resolution based on probabilistic reach sets. IEEE Trans. Control Syst. Technol. 25, 309–316 (2017).
Nager, Y., Censi, A. & Frazzoli, E. What lies in the shadows? Safe and computation-aware motion planning for autonomous vehicles using intent-aware dynamic shadow regions. In Proceedings of the 2019 IEEE International Conference on Robotics and Automation 5800–5806 (IEEE, 2019).
McNaughton, M., Urmson, C., Dolan, J. M. & Lee, J.-W. Motion planning for autonomous driving with a conformal spatiotemporal lattice. In Proceedings of the 2011 IEEE International Conference on Robotics and Automation 4889–4895 (IEEE, 2011).
Werling, M., Kammel, S., Ziegler, J. & Gröll, L. Optimal trajectories for time-critical street scenarios using discretized terminal manifolds. Int. J. Robotics Res. 31, 346–359 (2012).
Zucker, M. et al. CHOMP: covariant Hamiltonian optimization for motion planning. Int. J. Robotics Res. 32, 1164–1193 (2013).
Ziegler, J., Bender, P., Dang, T. & Stiller, C. Trajectory planning for Bertha—a local, continuous method. In Proceedings of the 2014 IEEE Intelligent Vehicles Symposium 450–457 (IEEE, 2014).
Hult, R., Zanon, M., Gros, S. & Falcone, P. An MIQP-based heuristic for optimal coordination of vehicles at intersections. In Proceedings of the 2018 IEEE International Conference on Decision and Control 2783–2790 (IEEE, 2018).
Sun, Z., Hsu, D., Jiang, T., Kurniawati, H. & Reif, J. H. Narrow passage sampling for probabilistic roadmap planning. IEEE Trans. Robotics 21, 1105–1115 (2005).
LaValle, S. M. in Planning Algorithms 79–80 (Cambridge Univ. Press, 2006).
Schouwenaars, T., De Moor, B., Feron, E. & How, J. Mixed integer programming for multi-vehicle path planning. In Proceedings of the 2001 European Control Conference 2603–2608 (IEEE, 2001).
Qian, X., Altché, F., Bender, P., Stiller, C. & de La Fortelle, A. Optimal trajectory planning for autonomous driving integrating logical constraints: an MIQP perspective. In Proceedings of the IEEE 19th International Conference on Intelligent Transportation Systems 205–210 (IEEE, 2016).
Park, J., Karumanchi, S. & Iagnemma, K. Homotopy-based divide-and-conquer strategy for optimal trajectory planning via mixed-integer programming. IEEE Trans. Robotics 31, 1101–1115 (2015).
Gutjahr, B., Gröll, L. & Werling, M. Lateral vehicle trajectory optimization using constrained linear time-varying MPC. IEEE Trans. Intell. Transportation Syst. 18, 1586–1595 (2016).
Zhan, W., Chen, J., Chan, C.-Y., Liu, C. & Tomizuka, M. Spatially-partitioned environmental representation and planning architecture for on-road autonomous driving. In Proceedings of the 2017 IEEE Intelligent Vehicles Symposium 632–639 (IEEE, 2017).
Mohy-ud-Din, H. & Muhammad, A. Detecting narrow passages in configuration spaces via spectra of probabilistic roadmaps. In Proceedings of the 2010 ACM Symposium on Applied Computing 1294–1298 (ACM, 2010).
Do, Q. H., Mita, S. & Yoneda, K. Narrow passage path planning using fast marching method and support vector machine. In Proceedings of the 2014 IEEE Intelligent Vehicles Symposium 630–635 (IEEE, 2014).
Bender, P., Taş, Ö. S., Ziegler, J. & Stiller, C. The combinatorial aspect of motion planning: maneuver variants in structured environments. In Proceedings of the 2015 IEEE Intelligent Vehicles Symposium 1386–1392 (IEEE, 2015).
Archer, J. & Vogel, K. The Traffic Safety Problems in Urban Areas. Technical Report (KTH Stockholm, 2000).
Shalev-Shwartz, S., Shammah, S. & Shashua, A. On a formal model of safe and scalable self-driving cars. Preprint at https://arxiv.org/pdf/1708.06374.pdf (2018).
Liebenwein, L. et al. Compositional and contract-based verification for autonomous driving on road networks. In Robotics Research, Springer Proceedings in Advanced Robotics Vol. 10, 163–181 (Springer, 2020).
Trautman, P. & Krause, A. Unfreezing the robot: navigation in dense, interacting crowds. In Proceedings of the 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems 797–803 (IEEE, 2010).
Menéndez-Romero, C., Winkler, F., Dornhege, C. & Burgard, W. Maneuver planning for highly automated vehicles. In Proceedings of the 2017 IEEE Intelligent Vehicles Symposium 1458–1464 (IEEE, 2017).
Althoff, M. & Magdici, S. Set-based prediction of traffic participants on arbitrary road networks. IEEE Trans. Intell. Vehicles 1, 187–202 (2016).
Koschi, M. & Althoff, M. SPOT: a tool for set-based prediction of traffic participants. In Proceedings of the 2017 IEEE Intelligent Vehicles Symposium 1686–1693 (IEEE, 2017).
Koschi, M., Pek, C., Beikirch, M. & Althoff, M. Set-based prediction of pedestrians in urban environments considering formalized traffic rules. In Proceedings of the 21st International Conference on Intelligent Transportation Systems 2704–2711 (IEEE, 2018).
Pek, C. & Althoff, M. Computationally efficient fail-safe trajectory planning for self-driving vehicles using convex optimization. In Proceedings of the 2018 IEEE International Conference on Intelligent Transportation Systems 1447–1454 (IEEE, 2018).
Manzinger, S., Pek, C. & Althoff, M. Using reachable sets for trajectory planning of automated vehicles. IEEE Trans. Intell. Vehicles https://doi.org/10.1109/TIV.2020.3017342 (2020).
Paden, B., Čáp, M., Yong, S. Z., Yershov, D. & Frazzoli, E. A survey of motion planning and control techniques for self-driving urban vehicles. IEEE Trans. Intell. Vehicles 1, 33–55 (2016).
González, D., Pérez, J., Milanés, V. & Nashashibi, F. A review of motion planning techniques for automated vehicles. IEEE Trans. Intell. Transportation Syst. 17, 1135–1145 (2016).
Magdici, S., Ye, Z. & Althoff, M. Determining the maximum time horizon for vehicles to safely follow a trajectory. In Proceedings of the 20th International Conference on Intelligent Transportation Systems 1893–1899 (IEEE, 2017).
Héry, E., Masi, S., Xu, P. & Bonnifait, P. Map-based curvilinear coordinates for autonomous vehicles. In Proceedings of the 20th International Conference on Intelligent Transportation Systems 1–7 (IEEE, 2017).
Schürmann, B. et al. Ensuring drivability of planned motions using formal methods. In Proceedings of the 20th International Conference on Intelligent Transportation Systems 1661–1668 (IEEE, 2017).
We thank S. Kaster for his support in implementing the prediction and S. Steyer for providing the object detection and tracking algorithms. We also thank C. Schürmann for the voice-overs in the Supplementary Videos. This work was partially supported by the BMW Group within the CAR@TUM project, the German Federal Ministry of Economics and Technology through the research initiative Ko-HAF, and the German Research Foundation (DFG) under grants AL 1185/4-2 and AL 1185/3-2.
The authors declare no competing interests.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary information with Figs. 1–6, results with Figs. 7–10, methods with Fig. 11 and Tables 1–6, description of the data file, description of Videos 1–3.
Verification results of presented scenarios.
Illustration of computation steps during a single verification cycle.
Comparing the results of different intended planners.
Recorded scenarios, obtained solutions and visualization software.
About this article
Cite this article
Pek, C., Manzinger, S., Koschi, M. et al. Using online verification to prevent autonomous vehicles from causing accidents. Nat Mach Intell 2, 518–528 (2020). https://doi.org/10.1038/s42256-020-0225-y