ORAN-Bench-13K: An Open Source Benchmark for Assessing LLMs in Open Radio Access Networks

Understand

Large Language Models (LLMs) can revolutionize how we deploy and operate Open Radio Access Networks (O-RAN) by enhancing network analytics, anomaly detection, and code generation and significantly increasing the efficiency and reliability of a plethora of O-RAN tasks.

• In this paper, we present ORAN-Bench-13K, the first comprehensive benchmark designed to evaluate the performance of Large Language Models (LLMs) within the context of O-RAN.
• Our benchmark consists of 13,952 meticulously curated multiple-choice questions generated from 116 O-RAN specification documents.
• We leverage a novel three-stage LLM framework, and the questions are categorized into three distinct difficulties to cover a wide spectrum of ORAN-related knowledge.