Fetching the paper…

Reading the bibliography…

2024

Improving Multi-lingual Alignment Through Soft Contrastive Learning

Park, Minsu, Choi, Seyeon, Choi, Chanyeol et al.

Original
PDF
Post on X

Understand

Making decent multi-lingual sentence representations is critical to achieve high performances in cross-lingual downstream tasks.

In this work, we propose a novel method to align multi-lingual embeddings based on the similarity of sentences measured by a pre-trained mono-lingual embedding model.
Given translation sentence pairs, we train a multi-lingual model in a way that the similarity between cross-lingual embeddings follows the similarity of sentences measured at the mono-lingual teacher model.
Our method can be considered as contrastive learning with soft labels defined as the similarity between sentences.

Improving Multi-lingual Alignment Through Soft Contrastive Learning · Around

Built on

Multilingual universal sentence encoder for semantic retrieval
Original
Yinfei Yang, Daniel Cer, Amin Ahmad, Mandy Guo, Jax Law, Noah Constant, Gustavo Hernandez Abrego, Steve Yuan, Chris Tar, Yun-Hsuan Sung, et al. 2019 · 1907
Earlier work this paper cites.
Sentence-bert: Sentence embeddings using siamese bert-networks
Original
Nils Reimers and Iryna Gurevych. 2019 · 1908
Earlier work this paper cites.
Unsupervised cross-lingual representation learning at scale
Original
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019 · 1911
Earlier work this paper cites.
Making monolingual sentence embeddings multilingual using knowledge distillation
Original
Nils Reimers and Iryna Gurevych. 2020 · 2004
Earlier work this paper cites.
Language-agnostic bert sentence embedding
Original
Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2020 · 2007
Earlier work this paper cites.
Rocketqa: An optimized training approach to dense passage retrieval for open-domain question answering
Original
Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. 2020 · 2010
Earlier work this paper cites.
Parallel data, tools and interfaces in opus
Jörg Tiedemann. 2012 · 2012
Earlier work this paper cites.
Overview of the second bucc shared task: Spotting parallel sentences in comparable corpora
Pierre Zweigenbaum, Serge Sharoff, and Reinhard Rapp. 2017 · 2017
Earlier work this paper cites.

Similar

Learning cross-lingual sentence representations via a multi-task dual-encoder model
Original
Muthuraman Chidambaram, Yinfei Yang, Daniel Cer, Steve Yuan, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2018 · 2018
Cited alongside, same era.
Bert: Pre-training of deep bidirectional transformers for language understanding
Original
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018 · 2018
Cited alongside, same era.
Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond
Mikel Artetxe and Holger Schwenk. 2019 · 2019
Cited alongside, same era.
Cross-lingual language model pretraining
Alexis Conneau and Guillaume Lample. 2019 · 2019
Cited alongside, same era.
Mpnet: Masked and permuted pre-training for language understanding
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2020 · 2020
Cited alongside, same era.

Then

No language left behind: Scaling human-centered machine translation
Original
Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022 · 2022
Later among the works it cites.
Bitext mining using distilled sentence representations for low-resource languages
Original
Kevin Heffernan, Onur Çelebi, and Holger Schwenk. 2022 · 2022
Later among the works it cites.
Mteb: Massive text embedding benchmark
Original
Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers. 2022 · 2022
Later among the works it cites.
Text embeddings by weakly-supervised contrastive pre-training
Original
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2022 · 2022
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Simcse: Simple contrastive learning of sentence embeddings

Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021 · 2021

Cited alongside, same era.

Semantic alignment with calibrated similarity for multilingual sentence embedding

Jiyeon Ham and Eun-Sol Kim. 2021 · 2021

Cited alongside, same era.

Learning transferable visual models from natural language supervision

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021 · 2021

Cited alongside, same era.

Contrastive data and learning for natural language processing

Rui Zhang, Yangfeng Ji, Yue Zhang, and Rebecca J Passonneau. 2022 · 2022

Later among the works it cites.

Sonar: sentence-level multimodal and language-agnostic representations

Paul-Ambroise Duquenne, Holger Schwenk, and Benoît Sagot. 2023 · 2023

Later among the works it cites.

Improving text embeddings with large language models

Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2023 · 2023

Later among the works it cites.

Multilingual e5 text embeddings: A technical report

Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024 · 2024

Closest in time.