A Bayesian Framework for Information-Theoretic Probing

Understand

Pimentel et al.

• (2020) recently analysed probing from an information-theoretic perspective.
• They argue that probing should be seen as approximating a mutual information.
• This led to the rather unintuitive conclusion that representations encode exactly the same information about a target task as the original sentences.