Dyna-bAbI: unlocking bAbI's potential with dynamic synthetic benchmarking

Understand

While neural language models often perform surprisingly well on natural language understanding (NLU) tasks, their strengths and limitations remain poorly understood.

• Controlled synthetic tasks are thus an increasingly important resource for diagnosing model behavior.
• In this work we focus on story understanding, a core competency for NLU systems.
• However, the main synthetic resource for story understanding, the bAbI benchmark, lacks such a systematic mechanism for controllable task generation.