Fetching the paper…

Reading the bibliography…

2025

TLA: Tactile-Language-Action Model for Contact-Rich Manipulation

Hao, Peng, Zhang, Chaofan, Li, Dingzhe et al.

Original
PDF
Post on X

Understand

Significant progress has been made in vision-language models.

However, language-conditioned robotic manipulation for contact-rich tasks remains underexplored, particularly in terms of tactile sensing.
To address this gap, we introduce the Tactile-Language-Action (TLA) model, which effectively processes sequential tactile feedback via cross-modal language grounding to enable robust policy generation in contact-intensive scenarios.
In addition, we construct a comprehensive dataset that contains 24k pairs of tactile action instruction data, customized for fingertip peg-in-hole assembly, providing essential resources for TLA training and evaluation.

TLA: Tactile-Language-Action Model for Contact-Rich Manipulation · Around

Built on

K. He, X. Zhang, S. Ren et al. , “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 770–778
2016
Earlier work this paper cites.
R. Calandra, A. Owens, D. Jayaraman et al. , “More than a feeling: Learning to grasp and regrasp using vision and touch,” IEEE Robotics and Automation Letters , vol. 3, no. 4, pp. 3300–3307, 2018
2018
Earlier work this paper cites.
F. Torabi, G. Warnell, and P. Stone, “Behavioral cloning from observation,” arXiv preprint arXiv:1805.01954 , 2018
Original
2018
Earlier work this paper cites.
A. Billard and D. Kragic, “Trends and challenges in robot manipulation,” Science , vol. 364, no. 6446, p. eaat8414, 2019
2019
Earlier work this paper cites.
Q. Li, O. Kroemer, Z. Su et al. , “A review of tactile information: Perception and action through touch,” IEEE Transactions on Robotics , vol. 36, no. 6, pp. 1619–1634, 2020
2020
Earlier work this paper cites.
C. Wang, S. Wang, B. Romero et al. , “Swingbot: Learning physical features from in-hand tactile exploration for dynamic swing-up manipulation,” in International Conference on Intelligent Robots and Systems (IROS) , 2020, pp. 5633–5640
2020
Earlier work this paper cites.
M. A. Lee, Y. Zhu, P. Zachares et al. , “Making sense of vision and touch: Learning multimodal representations for contact-rich tasks,” IEEE Transactions on Robotics , vol. 36, no. 3, pp. 582–596, 2020
2020
Earlier work this paper cites.
A. Dosovitskiy, L. Beyer, A. Kolesnikov et al. , “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929 , 2020
Original
2020
Earlier work this paper cites.
S. Cui, R. Wang, J. Hu et al. , “In-hand object localization using a novel high-resolution visuotactile sensor,” IEEE Transactions on Industrial Electronics , vol. 69, no. 6, pp. 6015–6025, 2021
2021
Earlier work this paper cites.
J. Cui and J. Trinkle, “Toward next-generation learned robot manipulation,” Science robotics , vol. 6, no. 54, p. eabd9461, 2021
2021
Earlier work this paper cites.
S. Dong, D. K. Jha, D. Romeres et al. , “Tactile-rl for insertion: Generalization to objects of unknown geometry,” in IEEE International Conference on Robotics and Automation (ICRA) , 2021, pp. 6437–6443
2021
Earlier work this paper cites.
E. J. Hu, Y. Shen, P. Wallis et al. , “Lora: Low-rank adaptation of large language models.” ICLR , p. 3, 2022
2022
Earlier work this paper cites.
H. Qi, B. Yi, S. Suresh et al. , “General in-hand object rotation with vision and touch,” in Conference on Robot Learning , 2023, pp. 2549–2564
2023
Earlier work this paper cites.

Similar

A. Brohan, N. Brown, J. Carbajal et al. , “Rt-2: Vision-language-action models transfer web knowledge to robotic control,” arXiv preprint arXiv:2307.15818 , 2023
Original
2023
Cited alongside, same era.
H. Naveed, A. U. Khan, S. Qiu et al. , “A comprehensive overview of large language models,” arXiv preprint arXiv:2307.06435 , 2023
Original
2023
Cited alongside, same era.
H. Zhou, X. Yao, Y. Meng et al. , “Language-conditioned learning for robotic manipulation: A survey,” arXiv preprint arXiv:2312.10807 , 2023
Original
2023
Cited alongside, same era.
X. Li, M. Liu, H. Zhang et al. , “Vision-language foundation models as effective robot imitators,” arXiv preprint arXiv:2311.01378 , 2023
Original
2023
Cited alongside, same era.
H. Wu, Y. Jing, C. Cheang et al. , “Unleashing large-scale video generative pre-training for visual robot manipulation,” arXiv preprint arXiv:2312.13139 , 2023

Then

F. Yang, C. Feng, Z. Chen et al. , “Binding touch to everything: Learning unified multimodal tactile representations,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 26 340–26 353
2024
Later among the works it cites.
N. Cheng, C. Guan, J. Gao et al. , “Touch100k: A large-scale touch-language-vision dataset for touch-centric multimodal representation,” arXiv preprint arXiv:2406.03813 , 2024
Original
2024
Later among the works it cites.
L. Fu, G. Datta, H. Huang et al. , “A touch, vision, and language dataset for multimodal alignment,” arXiv preprint arXiv:2402.13232 , 2024
Original
2024
Later among the works it cites.
W. Chen, J. Xu, F. Xiang et al. , “General-purpose sim2real protocol for learning contact-rich manipulation with marker-based visuotactile sensors,” IEEE Transactions on Robotics , pp. 1509–1526, 2024
2024
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

2023

Cited alongside, same era.

C. Zhang, S. Cui, S. Wang et al. , “Gelstereo 2.0: An improved gelstereo sensor with multimedium refractive stereo calibration,” IEEE Transactions on Industrial Electronics , vol. 71, no. 7, pp. 7452–7462, 2023

2023

Cited alongside, same era.

S. Cui, Y. Wang, S. Wang et al. , “Tactile imprint simulation of gelstereo visuotactile sensors,” in IEEE International Conference on Mechatronics and Automation (ICMA) , 2023, pp. 650–656

2023

Cited alongside, same era.

C. Chi, Z. Xu, S. Feng et al. , “Diffusion policy: Visuomotor policy learning via action diffusion,” The International Journal of Robotics Research , p. 02783649241273668, 2023

2023

Cited alongside, same era.

O. M. Team, D. Ghosh, H. Walke et al. , “Octo: An open-source generalist robot policy,” arXiv preprint arXiv:2405.12213 , 2024

2024

Cited alongside, same era.

M. J. Kim, K. Pertsch, S. Karamcheti et al. , “Openvla: An open-source vision-language-action model,” arXiv preprint arXiv:2406.09246 , 2024

2024

Cited alongside, same era.

J. Wen, Y. Zhu, J. Li et al. , “Tinyvla: Towards fast, data-efficient vision-language-action models for robotic manipulation,” arXiv preprint arXiv:2409.12514 , 2024

2024

Cited alongside, same era.

A. O’Neill, A. Rehman, A. Maddukuri et al. , “Open x-embodiment: Robotic learning datasets and rt-x models: Open x-embodiment collaboration 0,” in IEEE International Conference on Robotics and Automation (ICRA) , 2024, pp. 6892–6903

2024

Cited alongside, same era.

C.-L. Cheang, G. Chen, Y. Jing et al. , “Gr-2: A generative video-language-action model with web-scale knowledge for robot manipulation,” arXiv preprint arXiv:2410.06158 , 2024

2024

Later among the works it cites.

S. Liu, L. Wu, B. Li et al. , “Rdt-1b: a diffusion foundation model for bimanual manipulation,” arXiv preprint arXiv:2410.07864 , 2024

2024

Later among the works it cites.

K. Black, N. Brown, D. Driess et al. , “ \ p i _ 0 \backslash pi\_0 : A vision-language-action flow model for general robot control,” arXiv preprint arXiv:2410.24164 , 2024

2024

Later among the works it cites.

P. Wang, S. Bai, S. Tan et al. , “Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution,” arXiv preprint arXiv:2409.12191 , 2024

2024

Later among the works it cites.

A. Yang, B. Yang, B. Hui et al. , “Qwen2 technical report,” arXiv preprint arXiv:2407.10671 , 2024

2024

Later among the works it cites.

L. Qian, J. Li, Y. Wu et al. , “Momentor: Advancing video large language model with fine-grained temporal reasoning,” arXiv preprint arXiv:2402.11435 , 2024

2024

Later among the works it cites.

S. Karamcheti, S. Nair, A. Balakrishna et al. , “Prismatic vlms: Investigating the design space of visually-conditioned language models,” in Forty-first International Conference on Machine Learning , 2024

2024

Later among the works it cites.

J. Jones, O. Mees, C. Sferrazza et al. , “Beyond sight: Finetuning generalist robot policies with heterogeneous sensors via language grounding,” arXiv preprint arXiv:2501.04693 , 2025

2025

Closest in time.