2020

CADET: Debugging and Fixing Misconfigurations using Counterfactual Reasoning

Krishna, Rahul, Iqbal, Md Shahriar, Javidian, Mohammad Ali et al.

Understand

Modern computing platforms are highly-configurable with thousands of interacting configurations.

However, configuring these systems is challenging.
Erroneous configurations can cause unexpected non-functional faults.
This paper proposes CADET (short for Causal Debugging Toolkit) that enables users to identify, explain, and fix the root cause of non-functional faults early and in a principled fashion.

Built on

Causation, prediction, and search
Peter Spirtes, Clark N Glymour, Richard Scheines, and David Heckerman. 2000 · 2000
Earlier work this paper cites.
Ancestral graph Markov models
Thomas Richardson, Peter Spirtes, et al · 2002
Earlier work this paper cites.
Computer systems: a programmer’s perspective . Vol. 2
Randal E Bryant, O’Hallaron David Richard, and O’Hallaron David Richard. 2003 · 2003
Earlier work this paper cites.
A smart hill-climbing algorithm for application server configuration. In 13th International Conference on World Wide Web (WWW) . ACM, 287–296
Bowei Xi, Zhen Liu, Mukund Raghavachari, Cathy H Xia, and Li Zhang. 2004 · 2004
Earlier work this paper cites.
Optimizing system configurations quickly by guessing at the performance. In Int’l Conference on Measurement and Modeling of Computer Systems (SIGMETRICS)
Takayuki Osogami and Sei Kato. 2007 · 2007
Earlier work this paper cites.
Active learning of causal networks with intervention experiments and optimal designs
Yang-Bo He and Zhi Geng. 2008 · 2008
Earlier work this paper cites.
Configuration-aware regression testing: an empirical study of sampling and prioritization. In Proceedings of the 2008 international symposium on Software testing and analysis
Xiao Qu, Myra B Cohen, and Gregg Rothermel. 2008 · 2008
Earlier work this paper cites.
HOLMES: Effective statistical debugging via efficient path profiling. In 2009 IEEE 31st International Conference on Software Engineering . IEEE, 34–44
Trishul M Chilimbi, Ben Liblit, Krishna Mehra, Aditya V Nori, and Kapil Vaswani. 2009 · 2009
Earlier work this paper cites.
The Art of Application Performance Testing: Help for Programmers and Quality Assurance.[Sl]:" O’Reilly Media
I Molyneaux. 2009 · 2009
Earlier work this paper cites.
Causality
Judea Pearl. 2009 · 2009
Earlier work this paper cites.
Automating Configuration Troubleshooting with Dynamic Information Flow Analysis.. In OSDI , Vol. 10. 1–14
Mona Attariyan and Jason Flinn. 2010 · 2010
Earlier work this paper cites.
Causal inference for statistical fault localization. In Proceedings of the 19th international symposium on Software testing and analysis . 73–84
George K Baah, Andy Podgurski, and Mary Jean Harrold. 2010 · 2010
Earlier work this paper cites.
Iterative delta debugging
Cyrille Artho. 2011 · 2011
Earlier work this paper cites.
Automated algorithm configuration and parameter tuning
Holger H Hoos. 2011 · 2011
Earlier work this paper cites.
Sequential model-based optimization for general algorithm configuration. In International conference on learning and intelligent optimization . Springer, 507–523
Frank Hutter, Holger H Hoos, and Kevin Leyton-Brown. 2011 · 2011
Earlier work this paper cites.
X-ray: Automating root-cause diagnosis of performance anomalies in production software. In Presented as part of the 10th { \{ USENIX } \} Symposium on Operating Systems Design and Implementation ( { \{ OSDI } \} 12) . 307–320
Mona Attariyan, Michael Chow, and Jason Flinn. 2012 · 2012
Earlier work this paper cites.
Method and computer system for thermal throttling protection
Chia-Chang Chiu and Yi-Sheng Chueh. 2012 · 2012
Earlier work this paper cites.
Learning high-dimensional directed acyclic graphs with latent and selection variables
Diego Colombo, Marloes H Maathuis, Markus Kalisch, and Thomas S Richardson. 2012 · 2012
Earlier work this paper cites.
Predicting performance via automated feature-interaction detection. In 2012 34th International Conference on Software Engineering (ICSE) . IEEE, 167–177
Norbert Siegmund, Sergiy S Kolesnikov, Christian Kästner, Sven Apel, Don Batory, Marko Rosenmüller, and Gunter Saake. 2012 · 2012
Earlier work this paper cites.
Practical bayesian optimization of machine learning algorithms. In Advances in neural information processing systems . 2951–2959
Jasper Snoek, Hugo Larochelle, and Ryan P Adams. 2012 · 2012
Earlier work this paper cites.
An empirical study of bugs in machine learning systems. In 2012 IEEE 23rd International Symposium on Software Reliability Engineering . IEEE, 271–280
Ferdian Thung, Shaowei Wang, David Lo, and Lingxiao Jiang. 2012 · 2012
Earlier work this paper cites.
Counterfactual reasoning and learning systems: The example of computational advertising
Léon Bottou, Jonas Peters, Joaquin Quiñonero-Candela, Denis X Charles, D Max Chickering, Elon Portugaly, Dipankar Ray, Patrice Simard, and Ed Snelson. 2013 · 2013
Earlier work this paper cites.
Variability-aware performance prediction: A statistical learning approach. In 2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE
Jianmei Guo, Krzysztof Czarnecki, Sven Apel, Norbert Siegmund, and Andrzej Wąsowski. 2013 · 2013
Earlier work this paper cites.
Discovering, reporting, and fixing performance bugs. In 10th working conference on mining software repositories
Adrian Nistor, Tian Jiang, and Lin Tan. 2013 · 2013
Earlier work this paper cites.
Towards machine learning-based auto-tuning of mapreduce. In 2013 IEEE 21st International Symposium on Modelling, Analysis and Simulation of Computer and Telecommunication Systems . IEEE, 11–20
Nezih Yigitbasi, Theodore L Willke, Guangdeng Liao, and Dick Epema. 2013 · 2013
Earlier work this paper cites.
Automated diagnosis of software configuration errors. In 2013 35th International Conference on Software Engineering
Sai Zhang and Michael D Ernst. 2013 · 2013
Earlier work this paper cites.
Order-independent constraint-based causal structure learning
Diego Colombo and Marloes H Maathuis. 2014 · 2014
Earlier work this paper cites.
Markovian acyclic directed mixed graphs for discrete data
Robin J Evans and Thomas S Richardson. 2014 · 2014
Earlier work this paper cites.
Deep speech: Scaling up end-to-end speech recognition
Original
Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, et al · 2014
Earlier work this paper cites.
Statistical debugging for real-world performance problems
Linhai Song and Shan Lu. 2014 · 2014
Earlier work this paper cites.
Encore: Exploiting system environment and correlation information for misconfiguration detection. In Proceedings of the 19th international conference on Architectural support for programming languages and operating systems . 687–700
Jiaqi Zhang, Lakshminarayanan Renganarayana, Xiaolan Zhang, Niyu Ge, Vasanth Bala, Tianyin Xu, and Yuanyuan Zhou. 2014 · 2014
Earlier work this paper cites.
Coz: Finding code that counts with causal profiling. In Proceedings of the 25th Symposium on Operating Systems Principles . 184–197
Charlie Curtsinger and Emery D Berger. 2015 · 2015
Earlier work this paper cites.
Automated multi-objective control for self-adaptive software design. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering . 13–24
Antonio Filieri, Henry Hoffmann, and Martina Maggio. 2015 · 2015
Earlier work this paper cites.
Principles and practice of structural equation modeling
Rex B Kline. 2015 · 2015
Earlier work this paper cites.
Fail at scale: Reliability in the face of rapid change
Ben Maurer. 2015 · 2015
Earlier work this paper cites.
Caramel: Detecting and fixing performance problems that have non-intrusive fixes. In 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering
Adrian Nistor, Po-Chun Chang, Cosmin Radoi, and Shan Lu. 2015 · 2015
Earlier work this paper cites.

Similar

Taking the human out of the loop: A review of Bayesian optimization
Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P Adams, and Nando De Freitas. 2015 · 2015
Cited alongside, same era.
Performance-influence models for highly configurable systems. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering . 284–294
Norbert Siegmund, Alexander Grebhahn, Sven Apel, and Christian Kästner. 2015 · 2015
Cited alongside, same era.
Holistic configuration management at facebook. In Proceedings of the 25th Symposium on Operating Systems Principles . 328–343
Chunqiang Tang, Thawan Kooburat, Pradeep Venkatachalam, Akshay Chander, Zhe Wen, Aravind Narayanan, Patrick Dowell, and Robert Karl. 2015 · 2015
Cited alongside, same era.
Deep parameter optimisation. In Proceedings of the 2015 Annual Conference on Genetic and Evolutionary Computation . 1375–1382
Fan Wu, Westley Weimer, Mark Harman, Yue Jia, and Jens Krinke. 2015 · 2015
Cited alongside, same era.
Hey, you have given me too many knobs!: understanding and dealing with over-designed configuration in system software. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering
Tianyin Xu, Long Jin, Xuepeng Fan, Yuanyuan Zhou, Shankar Pasupathy, and Rukma Talwadker. 2015 · 2015
Cited alongside, same era.
Fisher’s exact test
Lynne M Connelly. 2016 · 2016
Cited alongside, same era.
Causal models for debugging and control in cloud computing
Philipp Geiger, Lucian Carata, and Bernhard Schölkopf. 2016 · 2016
Cited alongside, same era.
An empirical study on performance bugs for highly configurable software systems. In Proceedings of the 10th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement
Xue Han and Tingting Yu. 2016 · 2016
Cited alongside, same era.
Predictive entropy search for multi-objective bayesian optimization. In International Conference on Machine Learning . 1492–1501
Daniel Hernández-Lobato, Jose Hernandez-Lobato, Amar Shah, and Ryan Adams. 2016 · 2016
Cited alongside, same era.
An Uncertainty-Aware Approach to Optimal Configuration of Stream Processing Systems. In Proc. Int’l Symp. on Modeling, Analysis and Simulation of Computer and Telecommunication Systems (MASCOTS) (London, UK). IEEE, 10
Pooyan Jamshidi and Giuliano Casale. 2016a · 2016
Cited alongside, same era.
An uncertainty-aware approach to optimal configuration of stream processing systems. In 2016 IEEE 24th International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems (MASCOTS) . IEEE, 39–48
Pooyan Jamshidi and Giuliano Casale. 2016b · 2016
Cited alongside, same era.
Ldx: Causality inference by lightweight dual execution. In Proceedings of the Twenty-First International Conference on Architectural Support for Programming Languages and Operating Systems . 503–515
Yonghwi Kwon, Dohyeong Kim, William Nick Sumner, Kyungtae Kim, Brendan Saltaformaggio, Xiangyu Zhang, and Dongyan Xu. 2016 · 2016
Cited alongside, same era.

Then

The book of why: the new science of cause and effect
Judea Pearl and Dana Mackenzie. 2018 · 2018
Later among the works it cites.
Understanding and auto-adjusting performance-sensitive configurations
Shu Wang, Chi Li, Henry Hoffmann, Shan Lu, William Sentosa, and Achmad Imam Kistijantoro. 2018 · 2018
Later among the works it cites.
99% of misconfiguration incidents in the cloud go unnoticed
2019 · 2019
Later among the works it cites.
Jetson Nano not responding or responding very slow over SSH
2019 · 2019
Later among the works it cites.
Inforence: effective fault localization based on information-theoretic analysis and statistical causal inference
Farid Feyzi and Saeed Parsa. 2019 · 2019
Later among the works it cites.
Review of causal discovery methods based on graphical models
Clark Glymour, Kun Zhang, and Peter Spirtes. 2019 · 2019
Later among the works it cites.
Predicting Performance of Software Configurations: There is no Silver Bullet
Original
Alexander Grebhahn, Norbert Siegmund, and Sven Apel. 2019 · 2019
Later among the works it cites.
Test them all, is it worth it? Assessing configuration sampling on the JHipster Web development stack
Axel Halin, Alexandre Nuttinck, Mathieu Acher, Xavier Devroey, Gilles Perrouin, and Benoit Baudry. 2019 · 2019
Later among the works it cites.
Transfer Learning for Performance Modeling of Deep Neural Network Systems. In 2019 { \{ USENIX } \} Conference on Operational Machine Learning (OpML 19) . 43–46
Md Shahriar Iqbal, Lars Kotthoff, and Pooyan Jamshidi. 2019 · 2019
Later among the works it cites.
Transfer learning for performance modeling of configurable systems: A causal analysis
Original
Mohammad Ali Javidian, Pooyan Jamshidi, and Marco Valtorta. 2019 · 2019
Later among the works it cites.
Tradeoffs in modeling performance of highly configurable software systems
Sergiy Kolesnikov, Norbert Siegmund, Christian Kästner, Alexander Grebhahn, and Sven Apel. 2019 · 2019
Later among the works it cites.
HyperMapper: a Practical Design Space Exploration Framework. In 2019 IEEE 27th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS) . IEEE, 425–426
Luigi Nardi, Artur Souza, David Koeplinger, and Kunle Olukotun. 2019 · 2019
Later among the works it cites.
Learning Software Configuration Spaces: A Systematic Literature Review
Original
Juliana Alves Pereira, Hugo Martin, Mathieu Acher, Jean-Marc Jézéquel, Goetz Botterweck, and Anthony Ventresque. 2019 · 2019
Later among the works it cites.
Configcrusher: White-box performance analysis for configurable systems
Original
Miguel Velez, Pooyan Jamshidi, Florian Sattler, Norbert Siegmund, Sven Apel, and Christian Kastner. 2019 · 2019
Later among the works it cites.
CUDA performance issue on TX2
2020 · 2020
Closest in time.
General Performance Problems
2020 · 2020
Closest in time.
High CPU usage on Jetson TX2 with GigE fully loaded
2020 · 2020
Closest in time.
Keeping the gate locked on your IoT devices: Vulnerabilities found on Amazon’s Alexa
Dikla Barda, Roman Zaikin, and Yaara Shriki. 2020 · 2020
Closest in time.
Alexa Hack Jeopardized Echo Users Network
Andrie Ene. 2020 · 2020
Closest in time.
Causality-Guided Adaptive Interventional Debugging. In Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data . 431–446
Anna Fariha, Suman Nath, and Alexandra Meliou. 2020 · 2020
Closest in time.
Cloud misconfigurations cost companies nearly $5 trillion
Jonathan Greig. 2020 · 2020
Closest in time.
FlexiBO: Cost-Aware Multi-Objective Optimization of Deep Neural Networks
Original
Md Shahriar Iqbal, Jianhai Su, Lars Kotthoff, and Pooyan Jamshidi. 2020 · 2020
Closest in time.
Causal Testing: Understanding Defects’ Root Causes. In Proceedings of the 2020 International Conference on Software Engineering
Brittany Johnson, Yuriy Brun, and Alexandra Meliou. 2020 · 2020
Closest in time.
Too Many Knobs to Tune? Towards Faster Database Tuning by Pre-selecting Important Knobs. In 12th { \{ USENIX } \} Workshop on Hot Topics in Storage and File Systems (HotStorage 20)
Konstantinos Kanellis, Ramnatthan Alagappan, and Shivaram Venkataraman. 2020 · 2020
Closest in time.
Applications of Common Entropy for Causal Inference
Murat Kocaoglu, Sanjay Shakkottai, Alexandros Dimakis, Constantine Caramanis, and Sriram Vishwanath. 2020 · 2020
Closest in time.
Whence to Learn? Transferring Knowledge in Configurable Systems using BEETLE
Rahul Krishna, Vivek Nair, Pooyan Jamshidi, and Tim Menzies. 2020 · 2020
Closest in time.
Statically inferring performance properties of software configurations. In Proceedings of the Fifteenth European Conference on Computer Systems . 1–16
Chi Li, Shu Wang, Henry Hoffmann, and Shan Lu. 2020 · 2020
Closest in time.
BugDoc: A System for Debugging Computational Pipelines. In Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data . 2733–2736
Raoni Lourenço, Juliana Freire, and Dennis Shasha. 2020 · 2020
Closest in time.
Rex: Preventing bugs and misconfiguration in large services using correlated change analysis. In 17th { \{ USENIX } \} Symposium on Networked Systems Design and Implementation
Sonu Mehta, Ranjita Bhagwan, Rahul Kumar, Chetan Bansal, Chandra Maddila, B Ashok, Sumit Asthana, Christian Bird, and Aditya Kumar. 2020 · 2020
Closest in time.
TANDEM: A Taxonomy and a Dataset of Real-World Performance Bugs
Ana B Sánchez, Pedro Delgado-Pérez, Inmaculada Medina-Bulo, and Sergio Segura. 2020 · 2020
Closest in time.
Dimensions of software configuration: on the configuration context in modern software development. In Proceedings of the 28th ESEC/FSE . 338–349
Norbert Siegmund, Nicolai Ruckel, and Janet Siegmund. 2020 · 2020
Closest in time.
Transferring pareto frontiers across heterogeneous hardware environments. In Proceedings of the ACM/SPEC International Conference on Performance Engineering . 12–23
Pavel Valov, Jianmei Guo, and Krzysztof Czarnecki. 2020 · 2020
Closest in time.
Challenges and Opportunities with Causal Discovery Algorithms: Application to Alzheimer’s Pathophysiology
Shen Xinpeng, Ma Sisi, Vemuri Prashanthi, Gyorgy Simon, Michael W Weiner, Paul Aisen, Ronald Petersen, Clifford R Jack, Andrew J Saykin, William Jagust, et al · 2020
Closest in time.
White-Box Analysis over Machine Learning: Modeling Performance of Configurable Systems
Original
Miguel Velez, Pooyan Jamshidi, Norbert Siegmund, Sven Apel, and Christian Kästner. 2021 · 2021
Closest in time.
An Evolutionary Study of Configuration Design and Implementation in Cloud Systems. In Proceedings of International Conference on Software Engineering (ICSE’21)
Yuanliang Zhang, Haochen He, Owolabi Legunsen, Shanshan Li, Wei Dong, and Tianyin Xu. 2021 · 2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…