Improving Prediction Backward-Compatiblility in NLP Model Upgrade with Gated Fusion

Understand

When upgrading neural models to a newer version, new errors that were not encountered in the legacy version can be introduced, known as regression errors.

• This inconsistent behavior during model upgrade often outweighs the benefits of accuracy gain and hinders the adoption of new models.
• To mitigate regression errors from model upgrade, distillation and ensemble have proven to be viable solutions without significant compromise in performance.
• Despite the progress, these approaches attained an incremental reduction in regression which is still far from achieving backward-compatible model upgrade.