Improving Attention Mechanism with Query-Value Interaction

Understand

Attention mechanism has played critical roles in various state-of-the-art NLP models such as Transformer and BERT.

• It can be formulated as a ternary function that maps the input queries, keys and values into an output by using a summation of values weighted by the attention weights derived from the interactions between queries and keys.
• Similar with query-key interactions, there is also inherent relatedness between queries and values, and incorporating query-value interactions has the potential to enhance the output by learning customized values according to the characteristics of queries.
• However, the query-value interactions are ignored by existing attention methods, which may be not optimal.