Evaluation of Neural Architectures Trained with Square Loss vs Cross-Entropy in Classification Tasks

Understand

Modern neural architectures for classification tasks are trained using the cross-entropy loss, which is widely believed to be empirically superior to the square loss.

• In this work we provide evidence indicating that this belief may not be well-founded.
• We explore several major neural architectures and a range of standard benchmark datasets for NLP, automatic speech recognition (ASR) and computer vision tasks to show that these architectures, with the same hyper-parameter settings as reported in the literature, perform comparably or better when trained with the square loss, even after equalizing computational resources.
• Indeed, we observe that the square loss produces better results in the dominant majority of NLP and ASR experiments.