A Comparative Study on Regularization Strategies for Embedding-based Neural Networks

Understand

This paper aims to compare different regularization strategies to address a common phenomenon, severe overfitting, in embedding-based neural networks for NLP.

• We chose two widely studied neural models and tasks as our testbed.
• We tried several frequently applied or newly proposed regularization strategies, including penalizing weights (embeddings excluded), penalizing embeddings, re-embedding words, and dropout.
• We also emphasized on incremental hyperparameter tuning, and combining different regularizations.