Improving Joint Training of Inference Networks and Structured Prediction Energy Networks

Understand

Deep energy-based models are powerful, but pose challenges for learning and inference (Belanger and McCallum, 2016).

• Tu and Gimpel (2018) developed an efficient framework for energy-based models by training "inference networks" to approximate structured inference instead of using gradient descent.
• However, their alternating optimization approach suffers from instabilities during training, requiring additional loss terms and careful hyperparameter tuning.
• In this paper, we contribute several strategies to stabilize and improve this joint training of energy functions and inference networks for structured prediction.