Fetching the paper…

Reading the bibliography…

2024

Rethinking CyberSecEval: An LLM-Aided Approach to Evaluation Critique

Hariharan, Suhas, Majid, Zainab Ali, Veuthey, Jaime Raldua et al.

Original
PDF
Post on X

Understand

A key development in the cybersecurity evaluations space is the work carried out by Meta, through their CyberSecEval approach.

While this work is undoubtedly a useful contribution to a nascent field, there are notable features that limit its utility.
Key drawbacks focus on the insecure code detection part of Meta's methodology.
We explore these limitations, and use our exploration as a test case for LLM-assisted benchmark analysis.

Rethinking CyberSecEval: An LLM-Aided Approach to Evaluation Critique · Around

Built on

Purple llama cyberseceval: A secure coding benchmark for language models, 2023
Original
Manish Bhatt, Sahana Chennabasappa, Cyrus Nikolaidis, Shengye Wan, Ivan Evtimov, Dominik Gabi, Daniel Song, Faizan Ahmad, Cornelius Aschermann, Lorenzo Fontana, Sasha Frolov, Ravi Prakash Giri, Dhaval Kapil, Yiannis Kozyrakis, David LeBlanc, James Milazzo, Aleksandar Straumann, Gabriel Synnaeve, Varun Vontimitta, Spencer Whitman, and Joshua Saxe · 2023
Earlier work this paper cites.
Common Weakness Enumeration
MITRE · 2023
Earlier work this paper cites.

Similar

Common Weakness Enumeration
MITRE · 2023
Cited alongside, same era.
Cyberseceval 2: A wide-ranging cybersecurity evaluation suite for large language models, 2024
Original
Manish Bhatt, Sahana Chennabasappa, Yue Li, Cyrus Nikolaidis, Daniel Song, Shengye Wan, Faizan Ahmad, Cornelius Aschermann, Yaohui Chen, Dhaval Kapil, David Molnar, Spencer Whitman, and Joshua Saxe · 2024
Cited alongside, same era.

Then

Semgrep: Lightweight static analysis for many languages
Semgrep Team · 2024
Closest in time.
Cyberseceval 3: Advancing the evaluation of cybersecurity risks and capabilities in large language models, 2024
Original
Shengye Wan, Cyrus Nikolaidis, Daniel Song, David Molnar, James Crnkovich, Jayson Grace, Manish Bhatt, Sahana Chennabasappa, Spencer Whitman, Stephanie Ding, Vlad Ionescu, Yue Li, and Joshua Saxe · 2024
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…