Sparse DETR: Efficient End-to-End Object Detection with Learnable Sparsity

Understand

DETR is the first end-to-end object detector using a transformer encoder-decoder architecture and demonstrates competitive performance but low computational efficiency on high resolution feature maps.

• The subsequent work, Deformable DETR, enhances the efficiency of DETR by replacing dense attention with deformable attention, which achieves 10x faster convergence and improved performance.
• Deformable DETR uses the multiscale feature to ameliorate performance, however, the number of encoder tokens increases by 20x compared to DETR, and the computation cost of the encoder attention remains a bottleneck.
• In our preliminary experiment, we observe that the detection performance hardly deteriorates even if only a part of the encoder token is updated.