Built on

E. A. Feigenbaum et al. , “The art of artificial intelligence: Themes and case studies of knowledge engineering,” 1977
1977
Earlier work this paper cites.
C.-Y. Lin, “ROUGE: A package for automatic evaluation of summaries,” in Text Summarization Branches Out . Barcelona, Spain: Association for Computational Linguistics, Jul. 2004, pp. 74–81. [Online]. Available: https://aclanthology.org/W04-1013
2004
Earlier work this paper cites.
J. A. Collins and I. R. Olson, “Knowledge is power: How conceptual knowledge transforms visual cognition,” Psychonomic bulletin & review , vol. 21, pp. 843–860, 2014
2014
Earlier work this paper cites.
(2024) Faiss: A library for efficient similarity search by meta. https://engineering.fb.com/2017/03/29/data-infrastructure/faiss-a-library-for-efficient-similarity-search/
2017
Earlier work this paper cites.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems , vol. 30, 2017
2017
Earlier work this paper cites.
A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman, “Glue: A multi-task benchmark and analysis platform for natural language understanding,” arXiv preprint arXiv:1804.07461 , 2018
Original
2018
Earlier work this paper cites.
F. Petroni, T. Rocktäschel, P. Lewis, A. Bakhtin, Y. Wu, A. H. Miller, and S. Riedel, “Language models as knowledge bases?” arXiv preprint arXiv:1909.01066 , 2019
Original
2019
Earlier work this paper cites.
D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, “Measuring massive multitask language understanding,” arXiv preprint arXiv:2009.03300 , 2020
Original
2020
Earlier work this paper cites.
T. B. Brown, “Language models are few-shot learners,” arXiv preprint arXiv:2005.14165 , 2020
Original
2020
Earlier work this paper cites.
P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel et al. , “Retrieval-augmented generation for knowledge-intensive nlp tasks,” Advances in Neural Information Processing Systems , vol. 33, pp. 9459–9474, 2020
2020
Earlier work this paper cites.
L. Weidinger, J. Mellor, M. Rauh, C. Griffin, J. Uesato, P.-S. Huang, M. Cheng, M. Glaese, B. Balle, A. Kasirzadeh et al. , “Ethical and social risks of harm from language models,” arXiv preprint arXiv:2112.04359 , 2021
Original
2021
Earlier work this paper cites.
A. Barbaresi, “Trafilatura: A Web Scraping Library and Command-Line Tool for Text Discovery and Extraction,” in Proceedings of the Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations . Association for Computational Linguistics, 2021, pp. 122–131. [Online]. Available: https://aclanthology.org/2021.acl-demo.15
2021
Earlier work this paper cites.
A. Madaan and A. Yazdanbakhsh, “Text and patterns: For effective chain of thought, it takes two to tango,” arXiv preprint arXiv:2209.07686 , 2022
Original
2022
Earlier work this paper cites.
P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y. Zhang, D. Narayanan, Y. Wu, A. Kumar et al. , “Holistic evaluation of language models,” arXiv preprint arXiv:2211.09110 , 2022
Original
2022
Earlier work this paper cites.
E. Aghaei, X. Niu, W. Shadid, and E. Al-Shaer, “Securebert: A domain-specific language model for cybersecurity,” in International Conference on Security and Privacy in Communication Systems . Springer, 2022, pp. 39–56
2022
Earlier work this paper cites.
S. Cao, J. Shi, L. Pan, L. Nie, Y. Xiang, L. Hou, J. Li, B. He, and H. Zhang, “Kqa pro: A dataset with explicit compositional programs for complex question answering over knowledge base,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , 2022, pp. 6101–6119
2022
Earlier work this paper cites.
T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa, “Large language models are zero-shot reasoners,” Advances in neural information processing systems , vol. 35, pp. 22 199–22 213, 2022
2022
Earlier work this paper cites.
Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi, “Self-instruct: Aligning language models with self-generated instructions,” arXiv preprint arXiv:2212.10560 , 2022
Original
2022
Earlier work this paper cites.
A. Zaboli, S. L. Choi, T.-J. Song, and J. Hong, “Chatgpt and other large language models for cybersecurity of smart grid applications,” arXiv preprint arXiv:2311.05462 , 2023
Original
2023
Earlier work this paper cites.
S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg et al. , “Sparks of artificial general intelligence: Early experiments with gpt-4,” arXiv preprint arXiv:2303.12712 , 2023
Original
2023
Earlier work this paper cites.
J. Yu, X. Wang, S. Tu, S. Cao, D. Zhang-Li, X. Lv, H. Peng, Z. Yao, X. Zhang, H. Li et al. , “Kola: Carefully benchmarking world knowledge of large language models,” in The Twelfth International Conference on Learning Representations , 2023
2023
Earlier work this paper cites.
X. Qi, Y. Zeng, T. Xie, P.-Y. Chen, R. Jia, P. Mittal, and P. Henderson, “Fine-tuning aligned language models compromises safety, even when users do not intend to!” arXiv preprint arXiv:2310.03693 , 2023
Original
2023
Earlier work this paper cites.
A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier et al. , “Mistral 7b,” arXiv preprint arXiv:2310.06825 , 2023
Original
2023
Earlier work this paper cites.

Similar

Then

(2024) Meta llama3-70b. Available at https://huggingface.co/meta-llama/Meta-Llama-3-70B
2024
Closest in time.
(2024) Meta llama3-8b. Available at https://huggingface.co/meta-llama/Meta-Llama-3-8B
2024
Closest in time.
(2024) Gemini models. Available at https://deepmind.google/technologies/gemini/pro/
2024
Closest in time.
(2024) mixtral-8x7b-instruct-v0.1. Available at https://replicate.com/mistralai/mixtral-8x7b-instruct-v0.1
2024
Closest in time.
M. Bayer, P. Kuehn, R. Shanehsaz, and C. Reuter, “Cysecbert: A domain-adapted language model for the cybersecurity domain,” ACM Transactions on Privacy and Security , vol. 27, no. 2, pp. 1–20, 2024
2024
Closest in time.
G. Team, T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre, M. Rivière, M. S. Kale, J. Love et al. , “Gemma: Open models based on gemini research and technology,” arXiv preprint arXiv:2403.08295 , 2024

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2024

SECURE: Benchmarking Large Language Models for Cybersecurity

Bhusal, Dipkamal, Alam, Md Tanvirul, Nguyen, Le et al.

Understand

Large Language Models (LLMs) have demonstrated potential in cybersecurity applications but have also caused lower confidence due to problems like hallucinations and a lack of truthfulness.

Existing benchmarks provide general evaluations but do not sufficiently address the practical and applied aspects of LLM performance in cybersecurity-specific tasks.
To address this gap, we introduce the SECURE (Security Extraction, Understanding \& Reasoning Evaluation), a benchmark designed to assess LLMs performance in realistic cybersecurity scenarios.
SECURE includes six datasets focussed on the Industrial Control System sector to evaluate knowledge extraction, understanding, and reasoning based on industry-standard sources.

Built on

E. A. Feigenbaum et al. , “The art of artificial intelligence: Themes and case studies of knowledge engineering,” 1977
1977
Earlier work this paper cites.
C.-Y. Lin, “ROUGE: A package for automatic evaluation of summaries,” in Text Summarization Branches Out . Barcelona, Spain: Association for Computational Linguistics, Jul. 2004, pp. 74–81. [Online]. Available: https://aclanthology.org/W04-1013
2004
Earlier work this paper cites.
J. A. Collins and I. R. Olson, “Knowledge is power: How conceptual knowledge transforms visual cognition,” Psychonomic bulletin & review , vol. 21, pp. 843–860, 2014
2014
Earlier work this paper cites.
(2024) Faiss: A library for efficient similarity search by meta. https://engineering.fb.com/2017/03/29/data-infrastructure/faiss-a-library-for-efficient-similarity-search/
2017
Earlier work this paper cites.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems , vol. 30, 2017
2017
Earlier work this paper cites.
A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman, “Glue: A multi-task benchmark and analysis platform for natural language understanding,” arXiv preprint arXiv:1804.07461 , 2018
Original
2018
Earlier work this paper cites.
F. Petroni, T. Rocktäschel, P. Lewis, A. Bakhtin, Y. Wu, A. H. Miller, and S. Riedel, “Language models as knowledge bases?” arXiv preprint arXiv:1909.01066 , 2019
Original
2019
Earlier work this paper cites.
D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, “Measuring massive multitask language understanding,” arXiv preprint arXiv:2009.03300 , 2020
Original
2020
Earlier work this paper cites.
T. B. Brown, “Language models are few-shot learners,” arXiv preprint arXiv:2005.14165 , 2020
Original
2020
Earlier work this paper cites.
P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel et al. , “Retrieval-augmented generation for knowledge-intensive nlp tasks,” Advances in Neural Information Processing Systems , vol. 33, pp. 9459–9474, 2020
2020
Earlier work this paper cites.
L. Weidinger, J. Mellor, M. Rauh, C. Griffin, J. Uesato, P.-S. Huang, M. Cheng, M. Glaese, B. Balle, A. Kasirzadeh et al. , “Ethical and social risks of harm from language models,” arXiv preprint arXiv:2112.04359 , 2021
Original
2021
Earlier work this paper cites.
A. Barbaresi, “Trafilatura: A Web Scraping Library and Command-Line Tool for Text Discovery and Extraction,” in Proceedings of the Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations . Association for Computational Linguistics, 2021, pp. 122–131. [Online]. Available: https://aclanthology.org/2021.acl-demo.15
2021
Earlier work this paper cites.
A. Madaan and A. Yazdanbakhsh, “Text and patterns: For effective chain of thought, it takes two to tango,” arXiv preprint arXiv:2209.07686 , 2022
Original
2022
Earlier work this paper cites.
P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y. Zhang, D. Narayanan, Y. Wu, A. Kumar et al. , “Holistic evaluation of language models,” arXiv preprint arXiv:2211.09110 , 2022
Original
2022
Earlier work this paper cites.
E. Aghaei, X. Niu, W. Shadid, and E. Al-Shaer, “Securebert: A domain-specific language model for cybersecurity,” in International Conference on Security and Privacy in Communication Systems . Springer, 2022, pp. 39–56
2022
Earlier work this paper cites.
S. Cao, J. Shi, L. Pan, L. Nie, Y. Xiang, L. Hou, J. Li, B. He, and H. Zhang, “Kqa pro: A dataset with explicit compositional programs for complex question answering over knowledge base,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , 2022, pp. 6101–6119
2022
Earlier work this paper cites.
T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa, “Large language models are zero-shot reasoners,” Advances in neural information processing systems , vol. 35, pp. 22 199–22 213, 2022
2022
Earlier work this paper cites.
Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi, “Self-instruct: Aligning language models with self-generated instructions,” arXiv preprint arXiv:2212.10560 , 2022
Original
2022
Earlier work this paper cites.
A. Zaboli, S. L. Choi, T.-J. Song, and J. Hong, “Chatgpt and other large language models for cybersecurity of smart grid applications,” arXiv preprint arXiv:2311.05462 , 2023
Original
2023
Earlier work this paper cites.
S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg et al. , “Sparks of artificial general intelligence: Early experiments with gpt-4,” arXiv preprint arXiv:2303.12712 , 2023
Original
2023
Earlier work this paper cites.
J. Yu, X. Wang, S. Tu, S. Cao, D. Zhang-Li, X. Lv, H. Peng, Z. Yao, X. Zhang, H. Li et al. , “Kola: Carefully benchmarking world knowledge of large language models,” in The Twelfth International Conference on Learning Representations , 2023
2023
Earlier work this paper cites.
X. Qi, Y. Zeng, T. Xie, P.-Y. Chen, R. Jia, P. Mittal, and P. Henderson, “Fine-tuning aligned language models compromises safety, even when users do not intend to!” arXiv preprint arXiv:2310.03693 , 2023
Original
2023
Earlier work this paper cites.
A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier et al. , “Mistral 7b,” arXiv preprint arXiv:2310.06825 , 2023
Original
2023
Earlier work this paper cites.

Similar

Then

(2024) Meta llama3-70b. Available at https://huggingface.co/meta-llama/Meta-Llama-3-70B
2024
Closest in time.
(2024) Meta llama3-8b. Available at https://huggingface.co/meta-llama/Meta-Llama-3-8B
2024
Closest in time.
(2024) Gemini models. Available at https://deepmind.google/technologies/gemini/pro/
2024
Closest in time.
(2024) mixtral-8x7b-instruct-v0.1. Available at https://replicate.com/mistralai/mixtral-8x7b-instruct-v0.1
2024
Closest in time.
M. Bayer, P. Kuehn, R. Shanehsaz, and C. Reuter, “Cysecbert: A domain-adapted language model for the cybersecurity domain,” ACM Transactions on Privacy and Security , vol. 27, no. 2, pp. 1–20, 2024
2024
Closest in time.
G. Team, T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre, M. Rivière, M. S. Kale, J. Love et al. , “Gemma: Open models based on gemini research and technology,” arXiv preprint arXiv:2403.08295 , 2024

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

M. Sultana, A. Taylor, L. Li, and S. Majumdar, “Towards evaluation and understanding of large language models for cyber operation automation,” in 2023 IEEE Conference on Communications and Network Security (CNS) . IEEE, 2023, pp. 1–6

2023

Cited alongside, same era.

G. Li, Y. Li, W. Guannan, H. Yang, and Y. Yu, “Seceval: A comprehensive benchmark for evaluating cybersecurity knowledge of foundation models,” https://github.com/XuanwuAI/SecEval, 2023

2023

Cited alongside, same era.

Z. Liu, “Secqa: A concise question-answering dataset for evaluating large language models in computer security,” arXiv preprint arXiv:2312.15838 , 2023

Original

2023

Cited alongside, same era.

Y. Miao, Y. Bai, L. Chen, D. Li, H. Sun, X. Wang, Z. Luo, D. Sun, and X. Xu, “An empirical study of netops capability of pre-trained large language models,” arXiv preprint arXiv:2309.05557 , 2023

Original

2023

Cited alongside, same era.

Y. Liu, C. Pei, L. Xu, B. Chen, M. Sun, Z. Zhang, Y. Sun, S. Zhang, K. Wang, H. Zhang et al. , “Opseval: A comprehensive task-oriented aiops benchmark for large language models,” arXiv preprint arXiv:2310.07637 , 2023

Original

2023

Cited alongside, same era.

S. Kim, J. Shin, Y. Cho, J. Jang, S. Longpre, H. Lee, S. Yun, S. Shin, S. Kim, J. Thorne et al. , “Prometheus: Inducing evaluation capability in language models,” in NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following , 2023

2023

Cited alongside, same era.

“Chatarena: Multi-agent language game environments for large language models,” https://github.com/chatarena/chatarena , 2023

2023

Cited alongside, same era.

J. Chen and J. Mueller, “Quantifying uncertainty in answers from any language model and enhancing their trustworthiness,” 2023

2023

Cited alongside, same era.

Original

2024

Closest in time.

A. Shestov, A. Cheshkov, R. Levichev, R. Mussabayev, P. Zadorozhny, E. Maslov, C. Vadim, and E. Bulychev, “Finetuning large language models for vulnerability detection,” arXiv preprint arXiv:2401.17010 , 2024

Original

2024

Closest in time.

R. Fang, R. Bindu, A. Gupta, and D. Kang, “Llm agents can autonomously exploit one-day vulnerabilities,” arXiv preprint arXiv:2404.08144 , 2024

Original

2024

Closest in time.

Z. Li, S. Dutta, and M. Naik, “Llm-assisted static analysis for detecting security vulnerabilities,” arXiv preprint arXiv:2405.17238 , 2024

Original

2024

Closest in time.

N. Tihanyi, M. A. Ferrag, R. Jain, and M. Debbah, “Cybermetric: A benchmark dataset for evaluating large language models knowledge in cybersecurity,” arXiv preprint arXiv:2402.07688 , 2024

Original

2024

Closest in time.

S. Ullah, M. Han, S. Pujar, H. Pearce, A. Coskun, and G. Stringhini, “Llms cannot reliably identify and reason about security vulnerabilities (yet?): A comprehensive evaluation, framework, and benchmarks,” in IEEE Symposium on Security and Privacy , 2024

2024

Closest in time.

M. T. Alam, D. Bhushl, L. Nguyen, and N. Rastogi, “Ctibench: A benchmark for evaluating llms in cyber threat intelligence,” arXiv preprint arXiv:2406.07599 , 2024

Original

2024

Closest in time.

L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing et al. , “Judging llm-as-a-judge with mt-bench and chatbot arena,” Advances in Neural Information Processing Systems , vol. 36, 2024

2024

Closest in time.

(2024) Gpt-4o model openai. https://platform.openai.com/docs/models/gpt-4o . Accessed: 2024-05-24

2024

Closest in time.

(2024) Mitre att&ck. Available at https://attack.mitre.org/

2024

Closest in time.

(2024) Cvss 3.1 calculator. Available at https://nvd.nist.gov/vuln-metrics/cvss/v3-calculator

2024

Closest in time.

V. Adlakha, P. BehnamGhader, X. H. Lu, N. Meade, and S. Reddy, “Evaluating correctness and faithfulness of instruction-following models for question answering,” Transactions of the Association for Computational Linguistics , vol. 12, pp. 775–793, 2024

2024

Closest in time.

(2024) Langchaink. Available at https://python.langchain.com/v0.2/docs/introduction/

2024

Closest in time.

S. Lee, A. Shakir, D. Koenig, and J. Lipp. (2024) Open source strikes bread - new fluffy embeddings model. [Online]. Available: https://www.mixedbread.ai/blog/mxbai-embed-large-v1

2024

Closest in time.

B. Zhang, Z. Liu, C. Cherry, and O. Firat, “When scaling meets llm finetuning: The effect of data, model and finetuning method,” arXiv preprint arXiv:2402.17193 , 2024

Original

2024

Closest in time.

(2024) Common crawl. Available at https://commoncrawl.org/

2024

Closest in time.

(2024) Openai api. Available at https://platform.openai.com/

2024

Closest in time.