Improve Long-term Memory Learning Through Rescaling the Error Temporally

Understand

This paper studies the error metric selection for long-term memory learning in sequence modelling.

• We examine the bias towards short-term memory in commonly used errors, including mean absolute/squared error.
• Our findings show that all temporally positive-weighted errors are biased towards short-term memory in learning linear functionals.
• To reduce this bias and improve long-term memory learning, we propose the use of a temporally rescaled error.