Top Score on the Wrong Exam: On Benchmarking in Machine Learning for Vulnerability Detection

Understand

According to our survey of machine learning for vulnerability detection (ML4VD), 9 in every 10 papers published in the past five years define ML4VD as a function-level binary classification problem: Given a function, does it contain a security flaw? From our experience as security researchers, faced with deciding whether a given function makes the program vulnerable to attacks, we would often first want to understand the context in which this function is called.

• In this paper, we study how often this decision can really be made without further context and study both vulnerable and non-vulnerable functions in the most popular ML4VD datasets.
• We call a function "vulnerable" if it was involved in a patch of an actual security flaw and confirmed to cause the program's vulnerability.
• It is "non-vulnerable" otherwise.