How to Certify Machine Learning Based Safety-critical Systems? A Systematic Literature Review

Daniels ZA, Metaxas D (2018) Scenarionet: An interpretable data-driven model for scene understanding. In: IJCAI Workshop on Explainable Artificial Intelligence (XAI) 2018

2018

Cited alongside, same era.

Fisac JF, Akametalu AK, Zeilinger MN, Kaynama S, Gillula J, Tomlin CJ (2019) A general safety framework for learning-based control in uncertain robotic systems. IEEE Transactions on Automatic Control 64(7):2737–2752, DOI 10.1109/TAC.2018.2876389

2018

Cited alongside, same era.

François-Lavet V, Henderson P, Islam R, Bellemare MG, Pineau J (2018) An Introduction to Deep Reinforcement Learning. Foundations and Trends® in Machine Learning 11(3-4):219–354

2018

Cited alongside, same era.

Gauerhof L, Munk P, Burton S (2018) Structuring validation targets of a machine learning function applied to automated driving. In: Gallina B, Skavhaug A, Bitsch F (eds) Computer Safety, Reliability, and Security, Springer International Publishing, pp 45–58

2018

Cited alongside, same era.

Gehr T, Mirman M, Drachsler-Cohen D, Tsankov P, Chaudhuri S, Vechev M (2018) Ai2: Safety and robustness certification of neural networks with abstract interpretation. In: 2018 IEEE Symposium on Security and Privacy (SP), IEEE, pp 3–18

2018

Cited alongside, same era.

Ghosh S, Berkenkamp F, Ranade G, Qadeer S, Kapoor A (2018a) Verifying controllers against adversarial examples with bayesian optimization. In: 2018 IEEE International Conference on Robotics and Automation (ICRA), IEEE, pp 7306–7313

2018

Cited alongside, same era.

Ghosh S, Jha S, Tiwari A, Lincoln P, Zhu X (2018b) Model, data and reward repair: Trusted machine learning for markov decision processes. In: 2018 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks Workshops (DSN-W), pp 194–199

2018

Cited alongside, same era.

Göpfert JP, Hammer B, Wersing H (2018) Mitigating concept drift via rejection. In: International Conference on Artificial Neural Networks, Springer, pp 456–467

2018

Cited alongside, same era.

Pedroza G, Adedjouma M (2019) Safe-by-Design Development Method for Artificial Intelligent Based Systems. In: SEKE 2019 : The 31st International Conference on Software Engineering and Knowledge Engineering, Lisbon, Portugal, pp 391–397

2019

Later among the works it cites.

Peng W, Ye ZS, Chen N (2019) Bayesian deep-learning-based health prognostics toward prognostics uncertainty. IEEE Transactions on Industrial Electronics 67(3):2283–2293

2019

Later among the works it cites.

Postels J, Ferroni F, Coskun H, Navab N, Tombari F (2019) Sampling-free epistemic uncertainty estimation using approximated variance propagation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 2931–2940

2019

Later among the works it cites.

Rahimi M, Guo JL, Kokaly S, Chechik M (2019) Toward requirements specification for machine-learned components. In: 2019 IEEE 27th International Requirements Engineering Conference Workshops (REW), pp 241–244

2019

Later among the works it cites.

Remeli V, Morapitiye S, Rövid A, Szalay Z (2019) Towards verifiable specifications for neural networks in autonomous driving. In: 2019 IEEE 19th International Symposium on Computational Intelligence and Informatics and 7th IEEE International Conference on Recent Achievements in Mechatronics, Automation, Computer Sciences and Robotics (CINTI-MACRo), IEEE, pp 000175–000180

2019

Later among the works it cites.

Ren H, Chandrasekar SK, Murugesan A (2019a) Using quantifier elimination to enhance the safety assurance of deep neural networks. In: 2019 IEEE/AIAA 38th Digital Avionics Systems Conference (DASC), IEEE, pp 1–8

2019

Later among the works it cites.

Ruan W, Wu M, Sun Y, Huang X, Kroening D, Kwiatkowska M (2019) Global robustness evaluation of deep neural networks with provable guarantees for the hamming distance. In: IJCAI2019

2019

Later among the works it cites.

Salay R, Angus M, Czarnecki K (2019) A safety analysis method for perceptual components in automated driving. In: 2019 IEEE 30th International Symposium on Software Reliability Engineering (ISSRE), IEEE, pp 24–34

2019

Later among the works it cites.

Sehwag V, Bhagoji AN, Song L, Sitawarin C, Cullina D, Chiang M, Mittal P (2019) Analyzing the robustness of open-world machine learning. In: Proceedings of the 12th ACM Workshop on Artificial Intelligence and Security, ACM, New York, NY, USA, AISec’19, p 105–116

2019

Later among the works it cites.

Sekhon J, Fleming C (2019) Towards improved testing for deep learning. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER), pp 85–88

2019

Later among the works it cites.

Sena LH, Bessa IV, Gadelha MR, Cordeiro LC, Mota E (2019) Incremental bounded model checking of artificial neural networks in cuda. In: 2019 IX Brazilian Symposium on Computing Systems Engineering (SBESC), IEEE, pp 1–8

2019

Later among the works it cites.

Sun Y, Huang X, Kroening D, Sharp J, Hill M, Ashmore R (2019) DeepConcolic: Testing and debugging deep neural networks. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Companion Proceedings (ICSE-Companion), pp 111–114

2019

Later among the works it cites.

Sun Y, Huang X, Kroening D, Sharp J, Hill M, Ashmore R (2019) Structural test coverage criteria for deep neural networks. ACM Trans Embed Comput Syst 18(5s)

2019

Later among the works it cites.

Tran HD, Musau P, Lopez DM, Yang X, Nguyen LV, Xiang W, Johnson TT (2019) Parallelizable reachability analysis algorithms for feed-forward neural networks. In: 2019 IEEE/ACM 7th International Conference on Formal Methods in Software Engineering (FormaliSE), IEEE, pp 51–60

2019

Later among the works it cites.

Wagner J, Kohler JM, Gindele T, Hetzel L, Wiedemer JT, Behnke S (2019) Interpretable and fine-grained visual explanations for convolutional neural networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 9097–9107

2019

Later among the works it cites.

Xiang W, Lopez DM, Musau P, Johnson TT (2019) Reachable set estimation and verification for neural network models of nonlinear dynamic systems. In: Safe, autonomous and intelligent vehicles, Springer, pp 123–144

2019

Later among the works it cites.

Xie X, Ma L, Juefei-Xu F, Xue M, Chen H, Liu Y, Zhao J, Li B, Yin J, See S (2019) Deephunter: A coverage-guided fuzz testing framework for deep neural networks. In: Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis, ACM, New York, NY, USA, ISSTA 2019, p 146–157

2019

Later among the works it cites.

Xu H, Chen Z, Wu W, Jin Z, Kuo S, Lyu M (2019) Nv-dnn: Towards fault-tolerant dnn systems with n-version programming. In: 2019 49th Annual IEEE/IFIP International Conference on Dependable Systems and Networks Workshops (DSN-W), pp 44–47

2019

Later among the works it cites.

Yaghoubi S, Fainekos G (2019) Gray-box adversarial testing for control systems with machine learning components. In: Proceedings of the 22nd ACM International Conference on Hybrid Systems: Computation and Control, ACM, New York, NY, USA, HSCC ’19, pp 179––184

2019

Later among the works it cites.

Yan Y, Pei Q (2019) A robust deep-neural-network-based compressed model for mobile device assisted by edge server. IEEE Access 7:179104–179117

2019

Later among the works it cites.

Zhang JM, Harman M, Ma L, Liu Y (2020b) Machine learning testing: Survey, landscapes and horizons. IEEE Transactions on Software Engineering pp 1–1, DOI 10.1109/TSE.2019.2962027

2019

Later among the works it cites.

Zhang P, Dai Q, Ji S (2019c) Condition-guided adversarial generative testing for deep learning systems. In: 2019 IEEE International Conference On Artificial Intelligence Testing (AITest), pp 71–72

2019

Later among the works it cites.

Anderson BG, Ma Z, Li J, Sojoudi S (2020) Tightened convex relaxations for neural network robustness certification. In: 2020 59th IEEE Conference on Decision and Control (CDC), IEEE, pp 2190–2197

2020

Later among the works it cites.

Arcaini P, Bombarda A, Bonfanti S, Gargantini A (2020) Dealing with robustness of convolutional neural networks for image classification. In: 2020 IEEE International Conference On Artificial Intelligence Testing (AITest), pp 7–14, DOI 10.1109/AITEST49225.2020.00009

2020

Later among the works it cites.

Arrieta AB, Díaz-Rodríguez N, Del Ser J, Bennetot A, Tabik S, Barbado A, García S, Gil-López S, Molina D, Benjamins R, et al. (2020) Explainable artificial intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Information Fusion 58:82–115

2020

Later among the works it cites.

Ayers EW, Eiras F, Hawasly M, Whiteside I (2020) Parot: a practical framework for robust deep neural network training. In: NASA Formal Methods Symposium, Springer, pp 63–84

2020

Later among the works it cites.

Bacci E, Parker D (2020) Probabilistic guarantees for safe deep reinforcement learning. In: International Conference on Formal Modeling and Analysis of Timed Systems, Springer, pp 231–248

2020

Later among the works it cites.

Baheri A, Nageshrao S, Tseng HE, Kolmanovsky I, Girard A, Filev D (2019) Deep reinforcement learning with enhanced safety for autonomous highway driving. In: 2020 IEEE Intelligent Vehicles Symposium (IV), IEEE, pp 1550–1555

2020

Later among the works it cites.

Bar A, Klingner M, Varghese S, Huger F, Schlicht P, Fingscheidt T (2020) Robust semantic segmentation by redundant networks with a layer-specific loss contribution and majority vote. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp 332–333

2020

Later among the works it cites.

Bunel R, Lu J, Turkaslan I, Torr PH, Kohli P, Kumar MP (2020) Branch and bound for piecewise linear neural network verification. Journal of Machine Learning Research 21(42):1–39

2020

Later among the works it cites.

Chen Z, Narayanan N, Fang B, Li G, Pattabiraman K, DeBardeleben N (2020b) Tensorfi: A flexible fault injection framework for tensorflow applications. In: 2020 IEEE 31st International Symposium on Software Reliability Engineering (ISSRE), pp 426–435, DOI 10.1109/ISSRE5003.2020.00047

2020

Later among the works it cites.

Cofer D, Amundson I, Sattigeri R, Passi A, Boggs C, Smith E, Gilham L, Byun T, Rayadurgam S (2020) Run-time assurance for learning-based aircraft taxiing. In: 2020 AIAA/IEEE 39th Digital Avionics Systems Conference (DASC), pp 1–9, DOI 10.1109/DASC50938.2020.9256581

2020

Later among the works it cites.

Dapello J, Marques T, Schrimpf M, Geiger F, Cox DD, DiCarlo JJ (2020) Simulating a primary visual cortex at the front of CNNs improves robustness to image perturbations. bioRxiv DOI 10.1101/2020.06.16.154542

2020

Later among the works it cites.

Dean S, Matni N, Recht B, Ye V (2020) Robust guarantees for perception-based control. In: Proceedings of the 2nd Conference on Learning for Dynamics and Control, PMLR, vol 120, pp 350–360

2020

Later among the works it cites.

Dey S, Dasgupta P, Gangopadhyay B (2020) Safety augmentation in decision trees. In: AISafety@ IJCAI

2020

Later among the works it cites.

Fan DD, Nguyen J, Thakker R, Alatur N, Agha-mohammadi Aa, Theodorou EA (2020) Bayesian learning-based adaptive control for safety critical systems. In: 2020 IEEE International Conference on Robotics and Automation (ICRA), pp 4093–4099, DOI 10.1109/ICRA40945.2020.9196709

2020

Later among the works it cites.

Feng Y, Shi Q, Gao X, Wan J, Fang C, Chen Z (2020) Deepgini: Prioritizing massive tests to enhance the robustness of deep neural networks. In: Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis, Association for Computing Machinery, New York, NY, USA, ISSTA 2020, p 177–188, DOI 10.1145/3395363.3397357

2020

Later among the works it cites.

Fremont DJ, Chiu J, Margineantu DD, Osipychev D, Seshia SA (2020) Formal analysis and redesign of a neural network-based aircraft taxiing system with verifai. In: International Conference on Computer Aided Verification, Springer, pp 122–134

2020

Later among the works it cites.

Gauerhof L, Hawkins R, Picardi C, Paterson C, Hagiwara Y, Habli I (2020) Assuring the safety of machine learning for pedestrian detection at crossings. In: Casimiro A, Ortmeier F, Bitsch F, Ferreira P (eds) Computer Safety, Reliability, and Security, Springer International Publishing, Cham, pp 197–212

2020

Later among the works it cites.

Gladisch C, Heinzemann C, Herrmann M, Woehrle M (2020) Leveraging combinatorial testing for safety-critical computer vision datasets. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp 1314–1321

2020

Later among the works it cites.

Hasanbeig M, Kroening D, Abate A (2020) Towards verifiable and safe model-free reinforcement learning. In: CEUR Workshop Proceedings, CEUR Workshop Proceedings

2020

Later among the works it cites.

Henne M, Schwaiger A, Roscher K, Weiss G (2020) Benchmarking uncertainty estimation methods for deep learning with safety-related metrics. In: SafeAI@ AAAI, pp 83–90

2020

Later among the works it cites.

Jain D, Anumasa S, Srijith P (2020) Decision making under uncertainty with convolutional deep gaussian processes. In: Proceedings of the 7th ACM IKDD CoDS and 25th COMAD, pp 143–151

2020

Later among the works it cites.

Jeddi A, Shafiee MJ, Karg M, Scharfenberger C, Wong A (2020) Learn2perturb: An end-to-end feature perturbation learning to improve adversarial robustness. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp 1238–1247, DOI 10.1109/CVPR42600.2020.00132

2020

Later among the works it cites.

Jin W, Ma Y, Liu X, Tang X, Wang S, Tang J (2020) Graph structure learning for robust graph neural networks. In: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ACM, New York, NY, USA, KDD ’20, p 66–74

2020

Later among the works it cites.

Julian KD, Lee R, Kochenderfer MJ (2020) Validation of image-based neural network controllers through adaptive stress testing. In: 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), pp 1–7, DOI 10.1109/ITSC45102.2020.9294549

2020

Later among the works it cites.

Kuppers F, Kronenberger J, Shantia A, Haselhoff A (2020) Multivariate confidence calibration for object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp 326–327

2020

Later among the works it cites.

Liang S, Li Y, Srikant R (2020) Enhancing the reliability of out-of-distribution image detection in neural networks. ArXiv preprint arXiv:1706.02690

Original

2020

Later among the works it cites.

Loquercio A, Segu M, Scaramuzza D (2020) A general framework for uncertainty estimation in deep learning. IEEE Robotics and Automation Letters 5(2):3153–3160, DOI 10.1109/LRA.2020.2974682

2020

Later among the works it cites.

Lyu Z, Ko CY, Kong Z, Wong N, Lin D, Daniel L (2020) Fastened crown: Tightened neural network robustness certificates. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol 34, pp 5037–5044

2020

Later among the works it cites.

Marvi Z, Kiumarsi B (2020) Safe off-policy reinforcement learning using barrier functions. In: 2020 American Control Conference (ACC), IEEE, pp 2176–2181

2020

Later among the works it cites.

Naseer M, Minhas MF, Khalid F, Hanif MA, Hasan O, Shafique M (2020) Fannet: formal analysis of noise tolerance, training bias and input sensitivity in neural networks. In: 2020 Design, Automation & Test in Europe Conference & Exhibition (DATE), IEEE, pp 666–669

2020

Later among the works it cites.

Nguyen HH, Matschek J, Zieger T, Savchenko A, Noroozi N, Findeisen R (2020) Towards nominal stability certification of deep learning-based controllers. In: 2020 American Control Conference (ACC), IEEE, pp 3886–3891

2020

Later among the works it cites.

O’Brien M, Goble W, Hager G, Bukowski J (2020) Dependable neural networks for safety critical tasks. In: International Workshop on Engineering Dependable and Secure Machine Learning Systems, Springer, pp 126–140

2020

Later among the works it cites.

Rajabli N, Flammini F, Nardone R, Vittorini V (2021) Software verification and validation of safe autonomous cars: A systematic literature review. IEEE Access 9:4797–4819, DOI 10.1109/ACCESS.2020.3048047

2020

Later among the works it cites.

Ren K, Zheng T, Qin Z, Liu X (2020) Adversarial attacks and defenses in deep learning. Engineering 6(3):346–360

2020

Later among the works it cites.

Revay M, Wang R, Manchester IR (2020) A convex parameterization of robust recurrent neural networks. IEEE Control Systems Letters 5(4):1363–1368

2020

Later among the works it cites.

Sheikholeslami F, Jain S, Giannakis GB (2020) Minimum uncertainty based detection of adversaries in deep neural networks. In: 2020 Information Theory and Applications Workshop (ITA), IEEE, pp 1–16

2020

Later among the works it cites.

Tian Y, Zhong Z, Ordonez V, Kaiser G, Ray B (2020) Testing dnn image classifiers for confusion & bias errors. In: Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, Association for Computing Machinery, New York, NY, USA, ICSE ’20, p 1122–1134, DOI 10.1145/3377811.3380400

2020

Later among the works it cites.

Törnblom J, Nadjm-Tehrani S (2020) Formal verification of input-output mappings of tree ensembles. Science of Computer Programming 194:102450

2020

Later among the works it cites.

Tran HD, Yang X, Lopez DM, Musau P, Nguyen LV, Xiang W, Bak S, Johnson TT (2020) NNV: The neural network verification tool for deep neural networks and learning-enabled cyber-physical systems. In: International Conference on Computer Aided Verification, Springer, pp 3–17

2020

Later among the works it cites.

Udeshi S, Jiang X, Chattopadhyay S (2020) Callisto: Entropy-based test generation and data quality assessment for machine learning systems. In: 2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST), pp 448–453

2020

Later among the works it cites.

Varghese S, Bayzidi Y, Bar A, Kapoor N, Lahiri S, Schneider JD, Schmidt NM, Schlicht P, Huger F, Fingscheidt T (2020) Unsupervised temporal consistency metric for video segmentation in highly-automated driving. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp 336–337

2020

Later among the works it cites.

Wen M, Topcu U (2020) Constrained cross-entropy method for safe reinforcement learning. IEEE Transactions on Automatic Control

2020

Later among the works it cites.

Wu M, Wicker M, Ruan W, Huang X, Kwiatkowska M (2020) A game-based approximate verification of deep neural networks with provable guarantees. Theoretical Computer Science 807:298–329

2020

Later among the works it cites.

Yan M, Wang L, Fei A (2020) ARTDL: Adaptive random testing for deep learning systems. IEEE Access 8:3055–3064

2020

Later among the works it cites.

Yang Y, Vamvoudakis KG, Modares H (2020) Safe reinforcement learning for dynamical games. International Journal of Robust and Nonlinear Control 30(9):3706–3726

2020

Later among the works it cites.

Zhang J, Li J (2020) Testing and verification of neural-network-based safety-critical control software: A systematic literature review. Information and Software Technology 123:106296, DOI https://doi.org/10.1016/j.infsof.2020.106296

2020

Later among the works it cites.

Delseny H, Gabreau C, Gauffriau A, Beaudouin B, Ponsolle L, Alecu L, Bonnin H, Beltran B, Duchel D, Ginestet JB, Hervieu A, Martinez G, Pasquet S, Delmas K, Pagetti C, Gabriel JM, Chapdelaine C, Picard S, Damour M, Cappi C, Gardès L, Grancey FD, Jenn E, Lefevre B, Flandin G, Gerchinovitz S, Mamalet F, Albore A (2021) White paper machine learning in certified systems. ArXiv preprint arXiv:2103.10529

Original

2021

Closest in time.

Everett M, Lütjens B, How JP (2021) Certifiable robustness to adversarial state uncertainty in deep reinforcement learning. IEEE Transactions on Neural Networks and Learning Systems pp 1–15, DOI 10.1109/TNNLS.2021.3056046

2021

Closest in time.

Gualo F, Rodriguez M, Verdugo J, Caballero I, Piattini M (2021) Data quality certification using ISO/IEC 25012: Industrial experiences. Journal of Systems and Software 176:110938

2021

Closest in time.

Hendrycks D, Carlini N, Schulman J, Steinhardt J (2021) Unsolved problems in ml safety. 2109.13916

2021

Closest in time.

Pauli P, Koch A, Berberich J, Kohler P, Allgöwer F (2022) Training robust neural networks using lipschitz bounds. IEEE Control Systems Letters 6:121–126, DOI 10.1109/LCSYS.2021.3050444

2021

Closest in time.

Roh Y, Heo G, Whang SE (2021) A survey on data collection for machine learning: A big data - ai integration perspective. IEEE Transactions on Knowledge and Data Engineering 33(4):1328–1347

2021

Closest in time.

Vidot G, Gabreau C, Ober I, Ober I (2021) Certification of embedded systems based on machine learning: A survey. 2106.07221

2021

Closest in time.

Wabersich KP, Hewing L, Carron A, Zeilinger MN (2021) Probabilistic model predictive safety certification for learning-based control. IEEE Transactions on Automatic Control

2021

Closest in time.

Croce F, Andriushchenko M, Hein M (2019) Provable robustness of relu networks via maximization of linear regions. In: the 22nd International Conference on Artificial Intelligence and Statistics, PMLR, pp 2057–2066

2066

Closest in time.