Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Using online verification to prevent autonomous vehicles from causing accidents

Abstract

Ensuring that autonomous vehicles do not cause accidents remains a challenge. We present a formal verification technique for guaranteeing legal safety in arbitrary urban traffic situations. Legal safety means that autonomous vehicles never cause accidents although other traffic participants are allowed to perform any behaviour in accordance with traffic rules. Our technique serves as a safety layer for existing motion planning frameworks that provide intended trajectories for autonomous vehicles. We verify whether intended trajectories comply with legal safety and provide fallback solutions in safety-critical situations. The benefits of our verification technique are demonstrated in critical urban scenarios, which have been recorded in real traffic. The autonomous vehicle executed only safe trajectories, even when using an intended trajectory planner that was not aware of other traffic participants. Our results indicate that our online verification technique can drastically reduce the number of traffic accidents.

This is a preview of subscription content, access via your institution

Access options

Rent or buy this article

Prices vary by article type

from$1.95

to$39.95

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Verification of legal safety.
Fig. 2: Verification during replanning.
Fig. 3: Results of Scenario I (urban intersection).
Fig. 4: Results of Scenario II (jaywalking pedestrian).
Fig. 5: Results of the verification technique with different intended planners.
Fig. 6: Computation steps of the verification technique.

Similar content being viewed by others

Data availability

All data gathered and reported in this study are available in the Supplementary data file. This includes the environment model, the intended trajectory and the verification result of each verification cycle for all scenarios.

Code availability

The code to visualize and analyse the gathered data and obtained results of this study are included in the Supplementary data file.

References

  1. Favarò, F., Eurich, S. & Nader, N. Autonomous vehicles’ disengagements: trends, triggers and regulatory limitations. Accid. Anal. Prev. 110, 136–148 (2018).

    Article  Google Scholar 

  2. Anderson, J. M. et al. Autonomous Vehicle Technology: A Guide for Policymakers (Rand Corporation, 2016).

  3. Koopman, P. & Wagner, M. Autonomous vehicle safety: an interdisciplinary challenge. IEEE Intell. Transportation Syst. Mag. 9, 90–96 (2017).

    Article  Google Scholar 

  4. Kalra, N. & Paddock, S. M. Driving to safety: how many miles of driving would it take to demonstrate autonomous vehicle reliability? Transportation Res. A Policy Practice 94, 182–193 (2016).

    Article  Google Scholar 

  5. Seshia, S. A., Sadigh, D. & Sastry, S. S. Towards verified artificial intelligence. Preprint at https://arxiv.org/abs/1606.08514 (2017).

  6. Schwarting, W., Alonso-Mora, J. & Rus, D. Planning and decision-making for autonomous vehicles. Annu. Rev. Control Robot. Autonomous Syst. 1, 187–210 (2018).

    Article  Google Scholar 

  7. United Nations Economic Commission for Europe. Convention on Road Traffic. United Nations Conference on Road Traffic (United Nations, 1968); consolidated version of 2006.

  8. Vanholme, B., Gruyer, D., Lusetti, B., Glaser, S. & Mammar, S. Highly automated driving on highways based on legal safety. IEEE Trans. Intell. Transportation Syst. 14, 333–347 (2013).

    Article  Google Scholar 

  9. Althoff, M. & Dolan, J. M. Online verification of automated road vehicles using reachability analysis. IEEE Trans. Robotics 30, 903–918 (2014).

    Article  Google Scholar 

  10. Koopman, P. & Wagner, M. Challenges in autonomous vehicle testing and validation. SAE Int. J. Transportation Safety 4, 15–24 (2016).

    Article  Google Scholar 

  11. Dahl, J., de Campos, G. R., Olsson, C. & Fredriksson, J. Collision avoidance: a literature review on threat-assessment techniques. IEEE Trans. Intell. Vehicles 4, 101–113 (2019).

    Article  Google Scholar 

  12. Tumova, J., Hall, G. C., Karaman, S., Frazzoli, E. & Rus, D. Least-violating control strategy synthesis with safety rules. In Proceedings of the 16th International Conference on Hybrid Systems: Computation and Control 1–10 (HSCC, 2013).

  13. Kress-Gazit, H., Fainekos, G. E. & Pappas, G. J. Temporal-logic-based reactive mission and motion planning. IEEE Trans. Robotics 25, 1370–1381 (2009).

    Article  Google Scholar 

  14. Fraichard, T. & Asama, H. Inevitable collision states—a step towards safer robots? In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems 388–393 (IEEE, 2003).

  15. Chan, N., Kuffner, J. & Zucker, M. Improved motion planning speed and safety using regions of inevitable collision. In 17th CISM-IFToMM Symposium on Robot Design, Dynamics and Control 103–114 (Springer, 2008).

  16. Koller, T., Berkenkamp, F., Turchetta, M. & Krause, A. Learning-based model predictive control for safe exploration. In Proceedings of the 2018 IEEE International Conference on Decision and Control 6059–6066 (IEEE, 2018).

  17. Wabersich, K. P. & Zeilinger, M. N. Linear model predictive safety certification for learning-based control. In Proceedings of the IEEE International Conference on Decision and Control 7130–7135 (IEEE, 2018).

  18. Sadraddini, S. & Belta, C. A provably correct MPC approach to safety control of urban traffic networks. In Proceedings of the American Control Conference 1679–1684 (2016).

  19. Ames, A. D. et al. Control barrier functions: theory and applications. In Proceedings of the 18th European Control Conference 3420–3431 (IEEE, 2019).

  20. Tedrake, R., Manchester, I. R., Tobenkin, M. & Roberts, J. W. LQR-trees: feedback motion planning via sums-of-squares verification. Int. J. Robotics Res. 29, 1038–1052 (2010).

    Article  Google Scholar 

  21. Li, W., Sadigh, D., Sastry, S. S. & Seshia, S. A. Synthesis for human-in-the-loop control systems. In Proceedings of the International Conference on Tools and Algorithms for the Construction and Analysis of Systems 470–484 (Springer, 2014).

  22. Jalalmaab, M., Fidan, B., Jeon, S. & Falcone, P. Guaranteeing persistent feasibility of model predictive motion planning for autonomous vehicles. In Proceedings of the 2017 IEEE Intelligent Vehicles Symposium 843–848 (IEEE, 2017).

  23. Danielson, C., Weiss, A., Berntorp, K. & Di Cairano, S. Path planning using positive invariant sets. In Proceedings of the 55th International Conference on Decision and Control 5986–5991 (IEEE, 2016).

  24. Herbert, S. L. et al. FaSTrack: a modular framework for fast and guaranteed safe motion planning. In Proceedings of the 56th International Conference on Decision and Control 1517–1522 (IEEE, 2017).

  25. Falcone, P., Ali, M. & Sjöberg, J. Predictive threat assessment via reachability analysis and set invariance theory. IEEE Trans. Intell. Transportation Syst. 12, 1352–1361 (2011).

    Article  Google Scholar 

  26. Vaskov, S. et al. Towards provably not-at-fault control of autonomous robots in arbitrary dynamic environments. In Proc. Robotics: Science and Systems 1–9 (2019).

  27. Lefèvre, S., Vasquez, D. & Laugier, C. A survey on motion prediction and risk assessment for intelligent vehicles. ROBOMECH J. 1, 1–14 (2014).

    Article  Google Scholar 

  28. Gindele, T., Brechtel, S. & Dillmann, R. Learning driver behavior models from traffic observations for decision making and planning. IEEE Intell. Transportation Syst. Mag. 7, 69–79 (2015).

    Article  Google Scholar 

  29. Bahram, M., Hubmann, C., Lawitzky, A., Aeberhard, M. & Wollherr, D. A combined model- and learning-based framework for interaction-aware maneuver prediction. IEEE Trans. Intell. Transportation Syst. 17, 1538–1550 (2016).

    Article  Google Scholar 

  30. Deo, N., Rangesh, A. & Trivedi, M. M. How would surround vehicles move? A unified framework for maneuver classification and motion prediction. IEEE Trans. Intell. Vehicles 3, 129–140 (2018).

    Article  Google Scholar 

  31. Ghahramani, Z. Probabilistic machine learning and artificial intelligence. Nature 521, 452–459 (2015).

    Article  Google Scholar 

  32. Tang, C., Chen, J. & Tomizuka, M. Adaptive probabilistic vehicle trajectory prediction through physically feasible Bayesian recurrent neural network. In Proceedings of the 2019 IEEE International Conference on Robotics and Automation 3846–3852 (IEEE, 2019).

  33. Pool, E. A. I., Kooij, J. F. P. & Gavrila, D. M. Context-based cyclist path prediction using recurrent neural networks. In Proceedings of the 2019 IEEE Intelligent Vehicles Symposium 824–830 (IEEE, 2019).

  34. Wu, A. & How, J. Guaranteed infinite horizon avoidance of unpredictable, dynamically constrained obstacles. Autonomous Robots 32, 227–242 (2012).

    Article  Google Scholar 

  35. Bouraine, S., Fraichard, T. & Salhi, H. Provably safe navigation for mobile robots with limited field-of-views in dynamic environments. Autonomous Robots 32, 267–283 (2012).

    Article  Google Scholar 

  36. Yang, Y., Zhang, J., Cai, K. & Prandini, M. Multi-aircraft conflict detection and resolution based on probabilistic reach sets. IEEE Trans. Control Syst. Technol. 25, 309–316 (2017).

    Article  Google Scholar 

  37. Nager, Y., Censi, A. & Frazzoli, E. What lies in the shadows? Safe and computation-aware motion planning for autonomous vehicles using intent-aware dynamic shadow regions. In Proceedings of the 2019 IEEE International Conference on Robotics and Automation 5800–5806 (IEEE, 2019).

  38. McNaughton, M., Urmson, C., Dolan, J. M. & Lee, J.-W. Motion planning for autonomous driving with a conformal spatiotemporal lattice. In Proceedings of the 2011 IEEE International Conference on Robotics and Automation 4889–4895 (IEEE, 2011).

  39. Werling, M., Kammel, S., Ziegler, J. & Gröll, L. Optimal trajectories for time-critical street scenarios using discretized terminal manifolds. Int. J. Robotics Res. 31, 346–359 (2012).

    Article  Google Scholar 

  40. Zucker, M. et al. CHOMP: covariant Hamiltonian optimization for motion planning. Int. J. Robotics Res. 32, 1164–1193 (2013).

    Article  Google Scholar 

  41. Ziegler, J., Bender, P., Dang, T. & Stiller, C. Trajectory planning for Bertha—a local, continuous method. In Proceedings of the 2014 IEEE Intelligent Vehicles Symposium 450–457 (IEEE, 2014).

  42. Hult, R., Zanon, M., Gros, S. & Falcone, P. An MIQP-based heuristic for optimal coordination of vehicles at intersections. In Proceedings of the 2018 IEEE International Conference on Decision and Control 2783–2790 (IEEE, 2018).

  43. Sun, Z., Hsu, D., Jiang, T., Kurniawati, H. & Reif, J. H. Narrow passage sampling for probabilistic roadmap planning. IEEE Trans. Robotics 21, 1105–1115 (2005).

    Article  Google Scholar 

  44. LaValle, S. M. in Planning Algorithms 79–80 (Cambridge Univ. Press, 2006).

  45. Schouwenaars, T., De Moor, B., Feron, E. & How, J. Mixed integer programming for multi-vehicle path planning. In Proceedings of the 2001 European Control Conference 2603–2608 (IEEE, 2001).

  46. Qian, X., Altché, F., Bender, P., Stiller, C. & de La Fortelle, A. Optimal trajectory planning for autonomous driving integrating logical constraints: an MIQP perspective. In Proceedings of the IEEE 19th International Conference on Intelligent Transportation Systems 205–210 (IEEE, 2016).

  47. Park, J., Karumanchi, S. & Iagnemma, K. Homotopy-based divide-and-conquer strategy for optimal trajectory planning via mixed-integer programming. IEEE Trans. Robotics 31, 1101–1115 (2015).

    Article  Google Scholar 

  48. Gutjahr, B., Gröll, L. & Werling, M. Lateral vehicle trajectory optimization using constrained linear time-varying MPC. IEEE Trans. Intell. Transportation Syst. 18, 1586–1595 (2016).

    Google Scholar 

  49. Zhan, W., Chen, J., Chan, C.-Y., Liu, C. & Tomizuka, M. Spatially-partitioned environmental representation and planning architecture for on-road autonomous driving. In Proceedings of the 2017 IEEE Intelligent Vehicles Symposium 632–639 (IEEE, 2017).

  50. Mohy-ud-Din, H. & Muhammad, A. Detecting narrow passages in configuration spaces via spectra of probabilistic roadmaps. In Proceedings of the 2010 ACM Symposium on Applied Computing 1294–1298 (ACM, 2010).

  51. Do, Q. H., Mita, S. & Yoneda, K. Narrow passage path planning using fast marching method and support vector machine. In Proceedings of the 2014 IEEE Intelligent Vehicles Symposium 630–635 (IEEE, 2014).

  52. Bender, P., Taş, Ö. S., Ziegler, J. & Stiller, C. The combinatorial aspect of motion planning: maneuver variants in structured environments. In Proceedings of the 2015 IEEE Intelligent Vehicles Symposium 1386–1392 (IEEE, 2015).

  53. Archer, J. & Vogel, K. The Traffic Safety Problems in Urban Areas. Technical Report (KTH Stockholm, 2000).

  54. Shalev-Shwartz, S., Shammah, S. & Shashua, A. On a formal model of safe and scalable self-driving cars. Preprint at https://arxiv.org/pdf/1708.06374.pdf (2018).

  55. Liebenwein, L. et al. Compositional and contract-based verification for autonomous driving on road networks. In Robotics Research, Springer Proceedings in Advanced Robotics Vol. 10, 163–181 (Springer, 2020).

  56. Trautman, P. & Krause, A. Unfreezing the robot: navigation in dense, interacting crowds. In Proceedings of the 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems 797–803 (IEEE, 2010).

  57. Menéndez-Romero, C., Winkler, F., Dornhege, C. & Burgard, W. Maneuver planning for highly automated vehicles. In Proceedings of the 2017 IEEE Intelligent Vehicles Symposium 1458–1464 (IEEE, 2017).

  58. Althoff, M. & Magdici, S. Set-based prediction of traffic participants on arbitrary road networks. IEEE Trans. Intell. Vehicles 1, 187–202 (2016).

    Article  Google Scholar 

  59. Koschi, M. & Althoff, M. SPOT: a tool for set-based prediction of traffic participants. In Proceedings of the 2017 IEEE Intelligent Vehicles Symposium 1686–1693 (IEEE, 2017).

  60. Koschi, M., Pek, C., Beikirch, M. & Althoff, M. Set-based prediction of pedestrians in urban environments considering formalized traffic rules. In Proceedings of the 21st International Conference on Intelligent Transportation Systems 2704–2711 (IEEE, 2018).

  61. Pek, C. & Althoff, M. Computationally efficient fail-safe trajectory planning for self-driving vehicles using convex optimization. In Proceedings of the 2018 IEEE International Conference on Intelligent Transportation Systems 1447–1454 (IEEE, 2018).

  62. Manzinger, S., Pek, C. & Althoff, M. Using reachable sets for trajectory planning of automated vehicles. IEEE Trans. Intell. Vehicles https://doi.org/10.1109/TIV.2020.3017342 (2020).

  63. Paden, B., Čáp, M., Yong, S. Z., Yershov, D. & Frazzoli, E. A survey of motion planning and control techniques for self-driving urban vehicles. IEEE Trans. Intell. Vehicles 1, 33–55 (2016).

    Article  Google Scholar 

  64. González, D., Pérez, J., Milanés, V. & Nashashibi, F. A review of motion planning techniques for automated vehicles. IEEE Trans. Intell. Transportation Syst. 17, 1135–1145 (2016).

    Article  Google Scholar 

  65. Magdici, S., Ye, Z. & Althoff, M. Determining the maximum time horizon for vehicles to safely follow a trajectory. In Proceedings of the 20th International Conference on Intelligent Transportation Systems 1893–1899 (IEEE, 2017).

  66. Héry, E., Masi, S., Xu, P. & Bonnifait, P. Map-based curvilinear coordinates for autonomous vehicles. In Proceedings of the 20th International Conference on Intelligent Transportation Systems 1–7 (IEEE, 2017).

  67. Schürmann, B. et al. Ensuring drivability of planned motions using formal methods. In Proceedings of the 20th International Conference on Intelligent Transportation Systems 1661–1668 (IEEE, 2017).

Download references

Acknowledgements

We thank S. Kaster for his support in implementing the prediction and S. Steyer for providing the object detection and tracking algorithms. We also thank C. Schürmann for the voice-overs in the Supplementary Videos. This work was partially supported by the BMW Group within the CAR@TUM project, the German Federal Ministry of Economics and Technology through the research initiative Ko-HAF, and the German Research Foundation (DFG) under grants AL 1185/4-2 and AL 1185/3-2.

Author information

Authors and Affiliations

Authors

Contributions

C.P., S.M. and M.K. developed the verification technique during replanning. M.K. developed the concept and algorithms for the set-based prediction. C.P. and S.M. developed the concept and algorithms for the drivable area computation, driving corridor identification and fail-safe trajectory planning. M.A. developed the main concept of online verification by integrating set-based prediction and fail-safe trajectory generation. He also developed the underlying algorithms for reachability analysis and led the research project. C.P., S.M. and M.K. designed and conducted the experiments and collected the data. The Article and the Supplementary Information were written by C.P., S.M. and M.K.

Corresponding authors

Correspondence to Christian Pek, Stefanie Manzinger or Markus Koschi.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Supplementary information with Figs. 1–6, results with Figs. 7–10, methods with Fig. 11 and Tables 1–6, description of the data file, description of Videos 1–3.

Supplementary Video 1

Verification results of presented scenarios.

Supplementary Video 2

Illustration of computation steps during a single verification cycle.

Supplementary Video 3

Comparing the results of different intended planners.

Supplementary Data File

Recorded scenarios, obtained solutions and visualization software.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Pek, C., Manzinger, S., Koschi, M. et al. Using online verification to prevent autonomous vehicles from causing accidents. Nat Mach Intell 2, 518–528 (2020). https://doi.org/10.1038/s42256-020-0225-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s42256-020-0225-y

This article is cited by

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing