AIBench Scenario: Scenario-distilling AI Benchmarking

Understand

Modern real-world application scenarios like Internet services consist of a diversity of AI and non-AI modules with huge code sizes and long and complicated execution paths, which raises serious benchmarking or evaluating challenges.

• Using AI components or micro benchmarks alone can lead to error-prone conclusions.
• This paper presents a methodology to attack the above challenge.
• We formalize a real-world application scenario as a Directed Acyclic Graph-based model and propose the rules to distill it into a permutation of essential AI and non-AI tasks, which we call a scenario benchmark.