2019

CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairs

El-Kishky, Ahmed, Chaudhary, Vishrav, Guzman, Francisco et al.

Understand

Cross-lingual document alignment aims to identify pairs of documents in two distinct languages that are of comparable content or translations of each other.

In this paper, we exploit the signals embedded in URLs to label web documents at scale with an average precision of 94.5% across different language pairs.
We mine sixty-eight snapshots of the Common Crawl corpus and identify web document pairs that are translations of each other.
We release a new web dataset consisting of over 392 million URL pairs from Common Crawl covering documents in 8144 language pairs of which 137 pairs include English.

Built on

Cross-lingual language model pretraining
Original
Guillaume Lample and Alexis Conneau. 2019 · 1901
Earlier work this paper cites.
Wikimatrix: Mining 135m parallel sentences in 1620 language pairs from wikipedia
Original
Holger Schwenk, Vishrav Chaudhary, Shuo Sun, Hongyu Gong, and Francisco Guzmán. 2019 · 1907
Earlier work this paper cites.
Algorithms for the assignment and transportation problems
James Munkres. 1957 · 1957
Earlier work this paper cites.
Cross-language information retrieval based on parallel texts and automatic mining of parallel texts from the web
Jian-Yun Nie, Michel Simard, Pierre Isabelle, and Richard Durand. 1999 · 1999
Earlier work this paper cites.
Mining the web for bilingual text
Philip Resnik. 1999 · 1999
Earlier work this paper cites.
Parallel web text mining for cross-language ir
Jiang Chen and Jian-Yun Nie. 2000 · 2000
Earlier work this paper cites.
Using tf-idf to determine word relevance in document queries
Juan Ramos et al. 2003 · 2003
Earlier work this paper cites.
The web as a parallel corpus
Philip Resnik and Noah A Smith. 2003 · 2003
Earlier work this paper cites.
Europarl: A parallel corpus for statistical machine translation
Philipp Koehn. 2005 · 2005
Earlier work this paper cites.
Improving machine translation performance by exploiting non-parallel corpora
Dragos Stefan Munteanu and Daniel Marcu. 2005 · 2005
Earlier work this paper cites.
Extracting parallel sub-sentential fragments from non-parallel corpora
Dragos Stefan Munteanu and Daniel Marcu. 2006 · 2006
Earlier work this paper cites.
On the use of comparable corpora to improve smt performance
Sadaf Abdul-Rauf and Holger Schwenk. 2009 · 2009
Earlier work this paper cites.

Similar

Mining a comparable text corpus for a vietnamese-french statistical machine translation system
Thi-Ngoc-Diep Do, Viet-Bac Le, Brigitte Bigi, Laurent Besacier, and Eric Castelli. 2009 · 2009
Cited alongside, same era.
United nations general assembly resolutions: A six-language parallel corpus
Alexandre Rafalovitch, Robert Dale, et al. 2009 · 2009
Cited alongside, same era.
Mint: A method for effective and scalable mining of named entity transliterations from large comparable corpora
Raghavendra Udupa, K Saravanan, A Kumaran, and Jagadeesh Jagarlamudi. 2009 · 2009
Cited alongside, same era.
Combining content-based and url-based heuristics to harvest aligned bitexts from multilingual sites with bitextor
Miquel Espla-Gomis and Mikel Forcada. 2010 · 2010
Cited alongside, same era.
Computing Krippendorff’s alpha-reliability
Klaus Krippendorff. 2011 · 2011
Cited alongside, same era.
Dirt cheap web-scale parallel text from the common crawl
Jason R Smith, Herve Saint-Amand, Magdalena Plamada, Philipp Koehn, Chris Callison-Burch, and Adam Lopez. 2013 · 2013
Cited alongside, same era.
N-gram counts and language models from the common crawl
Christian Buck, Kenneth Heafield, and Bas van Ooyen. 2014 · 2014
Cited alongside, same era.
Findings of the wmt 2016 bilingual document alignment shared task
Christian Buck and Philipp Koehn. 2016a · 2016
Cited alongside, same era.
Yoda system for wmt16 shared task: Bilingual document alignment
Aswarth Abhilash Dara and Yiu-Chang Lin. 2016 · 2016
Cited alongside, same era.
First steps towards coverage-based document alignment
Luís Gomes and Gabriel Pereira Lopes. 2016 · 2016
Cited alongside, same era.
The United Nations parallel corpus v1. 0
Michał Ziemski, Marcin Junczys-Dowmunt, and Bruno Pouliquen. 2016 · 2016
Cited alongside, same era.
Bag of tricks for efficient text classification
Armand Joulin, Edouard Grave, and Piotr Bojanowski Tomas Mikolov. 2017 · 2017
Cited alongside, same era.

Then

Findings of the WMT 2018 shared task on parallel corpus filtering
Philipp Koehn, Huda Khayrallah, Kenneth Heafield, and Mikel L. Forcada. 2018 · 2018
Later among the works it cites.
Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing
Taku Kudo and John Richardson. 2018 · 2018
Later among the works it cites.
When and why are pre-trained word embeddings useful for neural machine translation?
Ye Qi, Devendra Sachan, Matthieu Felix, Sarguna Padmanabhan, and Graham Neubig. 2018 · 2018
Later among the works it cites.
Filtering and mining parallel data in a joint multilingual space
Original
Holger Schwenk. 2018 · 2018
Later among the works it cites.
Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond
Mikel Artetxe and Holger Schwenk. 2019 · 2019
Closest in time.
Low-resource corpus filtering using multilingual sentence embeddings
Vishrav Chaudhary, Yuqing Tang, Francisco Guzmán, Holger Schwenk, and Philipp Koehn. 2019 · 2019
Closest in time.
Bert: Pre-training of deep bidirectional transformers for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 · 2019
Closest in time.
Paracrawl: Web-scale parallel corpora for the languages of the eu
Miquel Esplà-Gomis, Mikel L Forcada, Gema Ramírez-Sánchez, and Hieu Hoang. 2019 · 2019
Closest in time.
Hierarchical document encoder for parallel corpus mining
Mandy Guo, Yinfei Yang, Keith Stevens, Daniel Cer, Heming Ge, Yun-hsuan Sung, Brian Strope, and Ray Kurzweil. 2019 · 2019
Closest in time.
The FLORES evaluation datasets for low-resource machine translation: Nepali–English and Sinhala–English
Francisco Guzmán, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, and Marc’Aurelio Ranzato. 2019 · 2019
Closest in time.
Findings of the WMT 2019 shared task on parallel corpus filtering for low-resource conditions
Philipp Koehn, Francisco Guzmán, Vishrav Chaudhary, and Juan Pino. 2019 · 2019
Closest in time.
fairseq: A fast, extensible toolkit for sequence modeling
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019 · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…