On Linear Identifiability of Learned Representations

Understand

Identifiability is a desirable property of a statistical model: it implies that the true model parameters may be estimated to any desired precision, given sufficient computational resources and data.

• We study identifiability in the context of representation learning: discovering nonlinear data representations that are optimal with respect to some downstream task.
• When parameterized as deep neural networks, such representation functions typically lack identifiability in parameter space, because they are overparameterized by design.
• In this paper, building on recent advances in nonlinear ICA, we aim to rehabilitate identifiability by showing that a large family of discriminative models are in fact identifiable in function space, up to a linear indeterminacy.