Interactive Task and Concept Learning from Natural Language Instructions and GUI Demonstrations

Understand

Natural language programming is a promising approach to enable end users to instruct new tasks for intelligent agents.

• However, our formative study found that end users would often use unclear, ambiguous or vague concepts when naturally instructing tasks in natural language, especially when specifying conditionals.
• Existing systems have limited support for letting the user teach agents new concepts or explaining unclear concepts.
• In this paper, we describe a new multi-modal domain-independent approach that combines natural language programming and programming-by-demonstration to allow users to first naturally describe tasks and associated conditions at a high level, and then collaborate with the agent to recursively resolve any ambiguities or vagueness through conversations and demonstrations.