The Better You Learn, The Smarter You Prune: Towards Efficient Vision-language-action Models via Differentiable Token Pruning

B. Zhang and R. Sennrich, “Root mean square layer normalization,” Advances in neural information processing systems , vol. 32, 2019

2019

Earlier work this paper cites.

A. Potapczynski, G. Loaiza-Ganem, and J. P. Cunningham, “Invertible gaussian reparameterization: Revisiting the gumbel-softmax,” Advances in Neural Information Processing Systems , vol. 33, pp. 12 311–12 321, 2020

2020

Earlier work this paper cites.

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. , “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929 , 2020

Original

2020

Earlier work this paper cites.

D. Bolya, C.-Y. Fu, X. Dai, P. Zhang, C. Feichtenhofer, and J. Hoffman, “Token merging: Your vit but faster,” arXiv preprint arXiv:2210.09461 , 2022

Original

2022

Earlier work this paper cites.

S. Kim, S. Shen, D. Thorsley, A. Gholami, W. Kwon, J. Hassoun, and K. Keutzer, “Learned token pruning for transformers,” in Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining , 2022, pp. 784–794

2022

Earlier work this paper cites.

E. J. Hu, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, et al. , “Lora: Low-rank adaptation of large language models,” in International Conference on Learning Representations , 2022

2022

Earlier work this paper cites.

J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,” in International conference on machine learning . PMLR, 2023, pp. 19 730–19 742

2023

Earlier work this paper cites.

W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention,” in Proceedings of the 29th symposium on operating systems principles , 2023, pp. 611–626

2023

Earlier work this paper cites.

B. Liu, Y. Zhu, C. Gao, Y. Feng, Q. Liu, Y. Zhu, and P. Stone, “Libero: Benchmarking knowledge transfer for lifelong robot learning,” Advances in Neural Information Processing Systems , vol. 36, pp. 44 776–44 791, 2023

2023

Earlier work this paper cites.

M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, et al. , “Dinov2: Learning robust visual features without supervision,” arXiv preprint arXiv:2304.07193 , 2023

Original

2023

Earlier work this paper cites.

X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer, “Sigmoid loss for language image pre-training,” in Proceedings of the IEEE/CVF international conference on computer vision , 2023, pp. 11 975–11 986

2023

Earlier work this paper cites.

H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. , “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971 , 2023

Original

2023

Earlier work this paper cites.

M. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, Q. Vuong, T. Kollar, D. Sadigh, S. Levine, and C. Finn, “Openvla: An open-source vision-language-action model,” arXiv preprint arXiv:2406.09246 , 2024

Original

2024

Earlier work this paper cites.

K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, et al. , “pi0: A vision-language-action flow model for general robot control,” arXiv preprint arXiv:2410.24164 , 2024

Original

2024

Earlier work this paper cites.

Q. Li, Y. Liang, Z. Wang, L. Luo, X. Chen, Z. Liao, X. Zhang, Y. Wang, G. Wu, L. Wang, et al. , “Cogact: A foundational vision-language-action model for synergizing cognition and action in robotic manipulation,” arXiv preprint arXiv:2411.19650 , 2024

Original

2024

Earlier work this paper cites.

C. Liang, S. Zhang, L. Zhang, Z. Wang, H. Wang, C.-L. Zhang, K.-W. Chang, T. Wang, and Y. You, “An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models,” in Proceedings of the European Conference on Computer Vision (ECCV) , 2024

2024

Earlier work this paper cites.

Y. Zhang, C.-K. Fan, J. Ma, W. Zheng, T. Huang, K. Cheng, D. Gudovskiy, T. Okuno, Y. Nakata, K. Keutzer, et al. , “Sparsevlm: Visual token sparsification for efficient vision-language model inference,” arXiv preprint arXiv:2410.04417 , 2024

Original

2024

Cited alongside, same era.

Y. Shang, M. Cai, B. Xu, Y. J. Lee, and Y. Yan, “Llava-prumerge: Adaptive token reduction for efficient large multimodal models,” arXiv preprint arXiv:2403.15388 , 2024

Original

2024

Cited alongside, same era.

L. Xing, Q. Huang, X. Dong, J. Lu, P. Zhang, Y. Zang, Y. Cao, C. He, J. Wang, F. Wu, et al. , “Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction,” arXiv preprint arXiv:2410.17247 , 2024

Original

2024

Cited alongside, same era.

Q. Zhang, A. Cheng, M. Lu, R. Zhang, Z. Zhuo, J. Cao, S. Guo, Q. She, and S. Zhang, “Beyond text-visual attention: Exploiting visual cues for effective token pruning in vlms,” arXiv preprint arXiv:2412.01818 , 2024

Original

2024

Cited alongside, same era.

T. Jiang, Q. Dong, Y. Ma, X. Ji, and Y. Liu, “Customizable multimodal trajectory prediction via nodes of interest selection for autonomous vehicles,” Expert Systems with Applications , vol. 288, p. 128222, 2025

2025

Closest in time.

X. Jiang, Y. Ma, P. Li, L. Xu, X. Wen, K. Zhan, Z. Xia, P. Jia, X. Lang, and S. Sun, “Transdiffuser: End-to-end trajectory generation with decorrelated multi-modal representation for autonomous driving,” arXiv preprint arXiv:2505.09315 , 2025

Original

2025

Closest in time.

Y. Yang, L. Zhang, Z. Chen, H. Wang, Y. Gao, Z. Wang, C.-L. Zhang, and K.-W. Chang, “Efficientvla: Training-free acceleration and compression for vision-language-action models,” 2025

2025

Closest in time.

S. Zhang, C.-L. Zhang, Z. Wang, and K.-W. Chang, “Llava-mini: Efficient image and video large multimodal models with one vision token,” 2025

2025

Closest in time.