2020

Loghub: A Large Collection of System Log Datasets for AI-driven Log Analytics

Zhu, Jieming, He, Shilin, He, Pinjia et al.

Understand

Logs have been widely adopted in software system development and maintenance because of the rich runtime information they record.

In recent years, the increase of software size and complexity leads to the rapid growth of the volume of logs.
To handle these large volumes of logs efficiently and effectively, a line of research focuses on developing intelligent and automated log analysis techniques.
However, only a few of these techniques have reached successful deployments in industry due to the lack of public log datasets and open benchmarking upon them.

Built on

Y. Liu, X. Zhang, S. He, H. Zhang, L. Li, Y. Kang, Y. Xu, M. Ma, Q. Lin, Y. Dang, S. Rajmohan, and D. Zhang, “Uniparser: A unified log parser for heterogeneous log data,” in Proceedings of the ACM Web Conference 2022 , ser. WWW ’22. Association for Computing Machinery, 2022, p. 1893–1901
1901
Earlier work this paper cites.
M. M. Breunig, H.-P. Kriegel, R. T. Ng, and J. Sander, “Lof: identifying density-based local outliers,” in ACM sigmod record , vol. 29, no. 2. ACM, 2000, pp. 93–104
2000
Earlier work this paper cites.
B. Schölkopf, J. C. Platt, J. Shawe-Taylor, A. J. Smola, and R. C. Williamson, “Estimating the support of a high-dimensional distribution,” Neural Computation , vol. 13, no. 7, pp. 1443–1471, 2001
2001
Earlier work this paper cites.
K. Hätönen, J. Boulicaut, M. Klemettinen, M. Miettinen, and C. Masson, “Comprehensive log compression with frequent patterns,” in Data Warehousing and Knowledge Discovery, 5th International Conference, DaWaK 2003, Prague, Czech Republic, September 3-5,2003, Proceedings , 2003, pp. 360–370
2003
Earlier work this paper cites.
R. Vaarandi, “A data clustering algorithm for mining patterns from event logs,” in IP Operations & Management, 2003.(IPOM 2003). 3rd IEEE Workshop on . IEEE, 2003, pp. 119–126
2003
Earlier work this paper cites.
M. Chen, A. X. Zheng, J. Lloyd, M. I. Jordan, and E. Brewer, “Failure diagnosis using decision trees,” in International Conference on Autonomic Computing (ICAC) . IEEE, 2004, pp. 36–43
2004
Earlier work this paper cites.
B. Rácz and A. Lukács, “High density compression of log files,” in 2004 Data Compression Conference (DCC 2004), 23-25 March 2004, Snowbird, UT, USA , 2004, p. 557
2004
Earlier work this paper cites.
Y. Liang, Y. Zhang, A. Sivasubramaniam, R. K. Sahoo, J. Moreira, and M. Gupta, “Filtering failure logs for a bluegene/l prototype,” in 2005 International Conference on Dependable Systems and Networks (DSN’05) . IEEE, 2005, pp. 476–485
2005
Earlier work this paper cites.
Y. Liang, Y. Zhang, H. Xiong, and R. Sahoo, “Failure prediction in ibm bluegene/l event logs,” in the 7th IEEE International Conference on Data Mining (ICDM) . IEEE, 2007, pp. 583–588
2007
Earlier work this paper cites.
A. Oliner and J. Stearley, “What supercomputers say: A study of five system logs,” in DSN , 2007
2007
Earlier work this paper cites.
S. Deorowicz and S. Grabowski, “Sub-atomic field processing for improved web log compression,” in Modern Problems of Radio Engineering, Telecommunications and Computer Science, 2008 Proceedings of International Conference on . IEEE, 2008, pp. 551–556
2008
Earlier work this paper cites.
Z. M. Jiang, A. E. Hassan, P. Flora, and G. Hamann, “Abstracting execution logs to execution events for enterprise applications (short paper),” in 2008 The Eighth International Conference on Quality Software . IEEE, 2008, pp. 181–186
2008
Earlier work this paper cites.
F. T. Liu, K. M. Ting, and Z.-H. Zhou, “Isolation forest,” in the 8th IEEE International Conference on Data Mining (ICDM) . IEEE, 2008, pp. 413–422
2008
Earlier work this paper cites.
C. Manning, P. Raghavan, and H. Schutze, Introduction to Information Retrieval . Cambridge University Press, 2008
2008
Earlier work this paper cites.
Q. Fu, J. Lou, Y. Wang, and J. Li, “Execution anomaly detection in distributed systems through unstructured log analysis,” in ICDM 2009, The Ninth IEEE International Conference on Data Mining, Miami, Florida, USA, 6-9 December 2009 , 2009, pp. 149–158
2009
Earlier work this paper cites.
——, “Execution anomaly detection in distributed systems through unstructured log analysis,” in ICDM 2009, The Ninth IEEE International Conference on Data Mining, Miami, Florida, USA, 6-9 December 2009 , 2009, pp. 149–158
2009
Earlier work this paper cites.
A. Makanju, A. N. Zincir-Heywood, and E. E. Milios, “Clustering event logs using iterative partitioning,” in Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Paris, France, June 28 - July 1, 2009 , 2009, pp. 1255–1264
2009
Earlier work this paper cites.
W. Xu, L. Huang, A. Fox, D. A. Patterson, and M. I. Jordan, “Detecting large-scale system problems by mining console logs,” in SOSP , 2009, pp. 117–132
2009
Earlier work this paper cites.
P. Bodik, M. Goldszmidt, A. Fox, D. B. Woodard, and H. Andersen, “Fingerprinting the datacenter: automated classification of performance crises,” in Proceedings of the 5th European Conference on Computer Systems (EuroSys) . ACM, 2010, pp. 111–124
2010
Earlier work this paper cites.
J.-G. Lou, Q. Fu, S. Yang, Y. Xu, and J. Li, “Mining invariants from console logs for system problem detection.” in USENIX Annual Technical Conference , 2010, pp. 1–14
2010
Earlier work this paper cites.
M. Nagappan and M. A. Vouk, “Abstracting log lines to log event types for mining software system logs,” in 2010 7th IEEE Working Conference on Mining Software Repositories (MSR) , 2010, pp. 114–117
2010
Earlier work this paper cites.
——, “Detecting large-scale system problems by mining console logs,” in Proceedings of the 27th International Conference on Machine Learning (ICML-10), June 21-24, 2010, Haifa, Israel , 2010, pp. 37–46
2010
Earlier work this paper cites.
I. Beschastnikh, Y. Brun, S. Schneider, M. Sloan, and M. D. Ernst, “Leveraging existing instrumentation to automatically infer invariant-constrained models,” in Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering (FSE) . ACM, 2011, pp. 267–277
2011
Earlier work this paper cites.
L. Tang, T. Li, and C. Perng, “Logsig: generating system events from raw textual logs,” in Proceedings of the 20th ACM Conference on Information and Knowledge Management, CIKM 2011, Glasgow, United Kingdom, October 24-28, 2011 , 2011, pp. 785–794
2011
Earlier work this paper cites.
G. Lee, J. J. Lin, C. Liu, A. Lorek, and D. V. Ryaboy, “The unified logging infrastructure for data analytics at twitter,” PVLDB , vol. 5, no. 12, pp. 1771–1780, 2012. [Online]. Available: http://vldb.org/pvldb/vol5/p1771_georgelee_vldb2012.pdf
2012
Earlier work this paper cites.
D. Yuan, S. Park, and Y. Zhou, “Characterizing logging practices in open-source software,” in Proceedings of the 34th International Conference on Software Engineering . IEEE Press, 2012, pp. 102–112
2012
Earlier work this paper cites.
R. Christensen and F. Li, “Adaptive log compression for massive log data,” in Proceedings of the ACM SIGMOD International Conference on Management of Data (SIGMOD) , 2013, pp. 1283–1284
2013
Earlier work this paper cites.

Similar

H. Mi, H. Wang, Y. Zhou, M. R.-T. Lyu, and H. Cai, “Toward fine-grained, unsupervised, scalable performance diagnosis for production cloud computing systems,” IEEE Transactions on Parallel and Distributed Systems , vol. 24, no. 6, pp. 1245–1255, 2013
2013
Cited alongside, same era.
M. Mizutani, “Incremental mining of system log format,” in 2013 IEEE International Conference on Services Computing, Santa Clara, CA, USA, June 28 - July 3, 2013 , 2013, pp. 595–602
2013
Cited alongside, same era.
W. Shang, Z. M. Jiang, H. Hemmati, B. Adams, A. E. Hassan, and P. Martin, “Assisting developers of big data analytics applications when deploying on hadoop clouds,” in 35th International Conference on Software Engineering, ICSE ’13, San Francisco, CA, USA, May 18-26, 2013 , 2013, pp. 402–411
2013
Cited alongside, same era.
R. Ding, Q. Fu, J. G. Lou, Q. Lin, D. Zhang, and T. Xie, “Mining historical issue repositories to heal large-scale online service systems,” in 44th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN) . IEEE, 2014, pp. 311–322
2014
Cited alongside, same era.
Q. Fu, J. Zhu, W. Hu, J. Lou, R. Ding, Q. Lin, D. Zhang, and T. Xie, “Where do developers log? an empirical study on logging practices in industry,” in ICSE’14: Companion Proc. of the 36th International Conference on Software Engineering , 2014, pp. 24–33
2014
Cited alongside, same era.
Q. Fu, J. Zhu, W. Hu, J.-G. Lou, R. Ding, Q. Lin, D. Zhang, and T. Xie, “Where do developers log? an empirical study on logging practices in industry,” in Companion Proceedings of the 36th International Conference on Software Engineering . ACM, 2014, pp. 24–33
2014
Cited alongside, same era.
M.-H. Lim, J.-G. Lou, H. Zhang, Q. Fu, A. B. J. Teoh, Q. Lin, R. Ding, and D. Zhang, “Identifying recurrent and unknown performance issues,” in 2014 IEEE International Conference on Data Mining (ICDM) . IEEE, 2014, pp. 320–329
2014
Cited alongside, same era.
P. Mosendz. (2014) When it goes down, facebook loses $24,420 per minute. [Online]. Available: https://www.theatlantic.com/technology/archive/2014/10/facebook-is-losing-24420-per-minute/382054/
2014
Cited alongside, same era.
J. Zhou, Z. Chen, H. Mi, and J. Wang, “Mtracer: A trace-oriented monitoring framework for medium-scale distributed systems,” in 8th IEEE International Symposium on Service Oriented System Engineering (SOSE) , 2014, pp. 266–271
2014
Cited alongside, same era.
J. Zhou, Z. Chen, J. Wang, Z. Zheng, and M. R. Lyu, “Trace bench: An open data set for trace-oriented monitoring,” in IEEE 6th International Conference on Cloud Computing Technology and Science (CloudCom) , 2014, pp. 519–526
2014
Cited alongside, same era.
H. Lin, J. Zhou, B. Yao, M. Guo, and J. Li, “Cowic: A column-wise independent compression for log stream analysis,” in 15th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid) , 2015, pp. 21–30
2015
Cited alongside, same era.
A. Pecchia, M. Cinque, G. Carrozza, and D. Cotroneo, “Industry practices and event logging: Assessment of a critical software development process,” in Proceedings of the 37th International Conference on Software Engineering-Volume 2 . IEEE Press, 2015, pp. 169–178
2015
Cited alongside, same era.

Then

P. He, J. Zhu, Z. Zheng, and M. R. Lyu, “Drain: An online log parsing approach with fixed depth tree,” in 2017 IEEE International Conference on Web Services, ICWS 2017, Honolulu, HI, USA, June 25-30, 2017 , 2017, pp. 33–40
2017
Later among the works it cites.
H. Li, T. Chen, W. Shang, and A. E. Hassan, “Studying software logging using topic models,” Empirical Software Engineering , 2017
2017
Later among the works it cites.
H. Li, W. Shang, and A. E. Hassan, “Which log level should developers choose for a new logging statement?” Empirical Software Engineering , vol. 22, pp. 1684–1716, 2017
2017
Later among the works it cites.
P. He, Z. Chen, S. He, and M. R. Lyu, “Characterizing the natural language descriptions in software logging statements,” in Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering . ACM, 2018, pp. 178–189
2018
Later among the works it cites.
S. He, Q. Lin, J.-G. Lou, H. Zhang, M. R. Lyu, and D. Zhang, “Identifying impactful service system problems via log analysis,” in Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering , 2018, pp. 60–70
2018
Later among the works it cites.
S. Messaoudi, A. Panichella, D. Bianculli, L. Briand, and R. Sasnauskas, “A search-based approach for accurate identification of log message formats,” in Proceedings of the 26th Conference on Program Comprehension . ACM, 2018, pp. 167–177
2018
Later among the works it cites.
M. S. Rakha, C.-P. Bezemer, and A. E. Hassan, “Revisiting the performance evaluation of automated approaches for the retrieval of duplicate issue reports,” IEEE Transactions on Software Engineering , vol. 44, no. 12, pp. 1245–1268, 2018
2018
Later among the works it cites.
A. Chuvakin. (2019) Public security log sharing site project. [Online]. Available: http://log-sharing.dreamhosters.com/
2019
Later among the works it cites.
CloudLab. (2019) Cloudlab. [Online]. Available: https://cloudlab.us/
2019
Later among the works it cites.
J. Liu, J. Zhu, S. He, P. He, Z. Zheng, and M. R. Lyu, “Logzip: Extracting hidden structures via iterative clustering for log compression,” in 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE, 2019, pp. 863–873
2019
Later among the works it cites.
J. Zhu, S. He, J. Liu, P. He, Q. Xie, Z. Zheng, and M. R. Lyu, “Tools and benchmarks for automated log parsing,” in Proceedings of the 41st International Conference on Software Engineering (ICSE-SEIP) , 2019, pp. 121–130
2019
Later among the works it cites.
Z. Chen, J. Liu, W. Gu, Y. Su, and M. R. Lyu, “Experience report: Deep learning-based system log analysis for anomaly detection,” arXiv preprint arXiv:2107.05908 , 2021
Original
2021
Closest in time.
S. He, P. He, Z. Chen, T. Yang, Y. Su, and M. R. Lyu, “A survey on automated log analysis for reliability engineering,” vol. 54, no. 6. New York, NY, USA: Association for Computing Machinery, jul 2021
2021
Closest in time.
J. Wei, G. Zhang, Y. Wang, Z. Liu, Z. Zhu, J. Chen, T. Sun, and Q. Zhou, “On the feasibility of parser-based log compression in { \{ Large-Scale } \} cloud systems,” in 19th USENIX Conference on File and Storage Technologies (FAST 21) , 2021, pp. 249–262
2021
Closest in time.
X. Zhang, Y. Xu, S. Qin, S. He, B. Qiao, Z. Li, H. Zhang, X. Li, Y. Dang, Q. Lin, M. Chintalapati, S. Rajmohan, and D. Zhang, “Onion: Identifying incident-indicating logs for cloud systems,” in Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering , ser. ESEC/FSE 2021. Association for Computing Machinery, 2021, p. 1253–1263
2021
Closest in time.
S. He, X. Zhang, P. He, Y. Xu, L. Li, Y. Kang, M. Ma, Y. Wei, Y. Dang, S. Rajmohan, and Q. Lin, “An empirical study of log analysis at microsoft,” in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering , ser. ESEC/FSE 2022, 2022, p. 1465–1476
2022
Closest in time.
X. Wang, X. Zhang, L. Li, S. He, H. Zhang, Y. Liu, L. Zheng, Y. Kang, Q. Lin, Y. Dang et al. , “Spine: a scalable log parser with feedback guidance,” in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering , 2022, pp. 1198–1208
2022
Closest in time.
X. Wang, X. Zhang, L. Li, S. He, H. Zhang, Y. Liu, L. Zheng, Y. Kang, Q. Lin, Y. Dang, S. Rajmohan, and D. Zhang, “Spine: A scalable log parser with feedback guidance,” in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering , ser. ESEC/FSE 2022. Association for Computing Machinery, 2022, p. 1198–1208
2022
Closest in time.
Q. Cheng, A. Saha, W. Yang, C. Liu, D. Sahoo, and S. Hoi, “Logai: A library for log analytics and intelligence,” arXiv preprint arXiv:2301.13415 , 2023
Original
2023
Closest in time.
Y. Huo, C. Lee, Y. Su, S. Shan, J. Liu, and M. Lyu, “Evlog: Evolving log analyzer for anomalous logs identification,” arXiv preprint arXiv:2306.01509 , 2023
Original
2023
Closest in time.
Y. Huo, Y. Su, C. Lee, and M. R. Lyu, “Semparser: A semantic parser for log analytics,” in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, 2023, pp. 881–893
2023
Closest in time.
J. Liu, S. He, Z. Chen, L. Li, Y. Kang, X. Zhang, P. He, H. Zhang, Q. Lin, Z. Xu et al. , “Incident-aware duplicate ticket aggregation for cloud systems,” arXiv preprint arXiv:2302.09520 , 2023
Original
2023
Closest in time.
J. Liu, J. Huang, Y. Huo, Z. Jiang, J. Gu, Z. Chen, C. Feng, M. Yan, and M. R. Lyu, “Scalable and adaptive log-based anomaly detection with expert in the loop,” arXiv preprint arXiv:2306.05032 , 2023
Original
2023
Closest in time.
J. Wei, G. Zhang, J. Chen, Y. Wang, W. Zheng, T. Sun, J. Wu, and J. Jiang, “Loggrep: Fast and cheap cloud log storage by exploiting both static and runtime patterns,” in Proceedings of the Eighteenth European Conference on Computer Systems , 2023, pp. 452–468
2023
Closest in time.
J. Xu, Q. Fu, Z. Zhu, Y. Cheng, Z. Li, Y. Ma, and P. He, “Hue: A user-adaptive parser for hybrid logs,” arXiv preprint arXiv:2308.07085 , 2023
Original
2023
Closest in time.
G. Yu, P. Chen, P. Li, T. Weng, H. Zheng, Y. Deng, and Z. Zheng, “Logreducer: Identify and reduce log hotspots in kernel on the fly,” in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, 2023, pp. 1763–1775
2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…