Efficient Training of Visual Transformers with Small Datasets

Understand

Visual Transformers (VTs) are emerging as an architectural paradigm alternative to Convolutional networks (CNNs).

• Differently from CNNs, VTs can capture global relations between image elements and they potentially have a larger representation capacity.
• However, the lack of the typical convolutional inductive bias makes these models more data-hungry than common CNNs.
• In fact, some local properties of the visual domain which are embedded in the CNN architectural design, in VTs should be learned from samples.