2017

SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation

Cer, Daniel, Diab, Mona, Agirre, Eneko et al.

Understand

Semantic Textual Similarity (STS) measures the meaning similarity of sentences.

Applications include machine translation (MT), summarization, generation, question answering (QA), short answer grading, semantic search, dialog and conversational systems.
The STS shared task is a venue for assessing the current state-of-the-art.
The 2017 task focuses on multilingual and cross-lingual pairs with one sub-track exploring MT quality estimation (MTQE) data.

Built on

WordNet: A lexical database for english
George A. Miller. 1995 · 1995
Earlier work this paper cites.
Long short-term memory
Sepp Hochreiter and Jürgen Schmidhuber. 1997 · 1997
Earlier work this paper cites.
The Berkeley FrameNet Project
Collin F. Baker, Charles J. Fillmore, and John B. Lowe. 1998 · 1998
Earlier work this paper cites.
WordNet: An Electronic Lexical Database
Christiane Fellbaum. 1998 · 1998
Earlier work this paper cites.
SemEval-2015 Task 1: Paraphrase and semantic similarity in Twitter (PIT)
Wei Xu, Chris Callison-Burch, and William B. Dolan. 2015 · 2001
Earlier work this paper cites.
Unsupervised construction of large paraphrase corpora: Exploiting massively parallel news sources
Bill Dolan, Chris Quirk, and Chris Brockett. 2004 · 2004
Earlier work this paper cites.
Efficient convolution kernels for dependency and constituent syntactic trees
Alessandro Moschitti. 2006 · 2006
Earlier work this paper cites.
A study of translation edit rate with targeted human annotation
Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006 · 2006
Earlier work this paper cites.
BIT at SemEval-2017 Task 1: Using semantic information space to evaluate semantic textual similarity
Hao Wu, Heyan Huang, Ping Jian, Yuhang Guo, and Chao Su. 2017 · 2007
Earlier work this paper cites.
SemEval-2014 Task 10: Multilingual semantic textual similarity
Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2014 · 2010
Earlier work this paper cites.
Recognizing textual entailment: Rational, evaluation and approaches
Ido Dagan, Bill Dolan, Bernardo Magnini, and Dan Roth. 2010 · 2010
Earlier work this paper cites.
BabelNet: Building a very large multilingual semantic network
Roberto Navigli and Simone Paolo Ponzetto. 2010 · 2010
Earlier work this paper cites.
Gigaword Fifth Edition LDC2011T07
Robert Parker, David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. 2011 · 2011
Earlier work this paper cites.
SemEval-2012 Task 6: A pilot on semantic textual similarity
Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre. 2012 · 2012
Earlier work this paper cites.
Ukp: Computing semantic textual similarity by combining multiple content similarity measures
Daniel Bär, Chris Biemann, Iryna Gurevych, and Torsten Zesch. 2012 · 2012
Earlier work this paper cites.
CompiLIG at SemEval-2017 Task 1: Cross-language plagiarism detection methods for semantic textual similarity
Jérémy Ferrero, Laurent Besacier, Didier Schwab, and Frédéric Agnès. 2017 · 2012
Earlier work this paper cites.
Soft cardinality: A parameterized similarity function for text comparison
Sergio Jimenez, Claudia Becerra, and Alexander Gelbukh. 2012a · 2012
Earlier work this paper cites.
Soft Cardinality: A parameterized similarity function for text comparison
Sergio Jimenez, Claudia Becerra, and Alexander Gelbukh. 2012b · 2012
Earlier work this paper cites.
Takelab: Systems for measuring semantic text similarity
Frane Šarić, Goran Glavaš, Mladen Karan, Jan Šnajder, and Bojana Dalbelo Bašić. 2012a · 2012
Earlier work this paper cites.
TakeLab: Systems for measuring semantic text similarity
Frane Šarić, Goran Glavaš, Mladen Karan, Jan Šnajder, and Bojana Dalbelo Bašić. 2012b · 2012
Earlier work this paper cites.
*SEM 2013 shared task: Semantic Textual Similarity
Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, and Weiwei Guo. 2013 · 2013
Earlier work this paper cites.
UdL at SemEval-2017 Task 1: Semantic textual similarity estimation of english sentence pairs using regression model over pairwise features
Hussein T. Al-Natsheh, Lucie Martinet, Fabrice Muhlenbach, and Djamel Abdelkader ZIGHED. 2017 · 2013
Earlier work this paper cites.
Findings of the 2013 Workshop on Statistical Machine Translation
Ondřej Bojar, Christian Buck, Chris Callison-Burch, Christian Federmann, Barry Haddow, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut, and Lucia Specia. 2013 · 2013
Earlier work this paper cites.
PPDB: The paraphrase database
Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2013 · 2013
Earlier work this paper cites.
Learning deep structured semantic models for web search using clickthrough data
Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013 · 2013
Earlier work this paper cites.
Distributed representations of words and phrases and their compositionality
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b · 2013
Earlier work this paper cites.
Findings of the 2014 workshop on statistical machine translation
Ondrej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Aleš Tamchyna. 2014 · 2014
Earlier work this paper cites.
Distributed representations of sentences and documents
Original
Quoc V. Le and Tomas Mikolov. 2014 · 2014
Earlier work this paper cites.
Dt_team at semeval-2017 task 1: Semantic similarity using alignments, sentence-level embeddings and gaussian mixture model output
Nabin Maharjan, Rajendra Banjade, Dipesh Gautam, Lasang J. Tamang, and Vasile Rus. 2017 · 2014
Earlier work this paper cites.

Similar

The Stanford CoreNLP natural language processing toolkit
Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014 · 2014
Cited alongside, same era.
A SICK cure for the evaluation of compositional distributional semantic models
Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014 · 2014
Cited alongside, same era.
GloVe: Global Vectors for Word Representation
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014 · 2014
Cited alongside, same era.
A latent semantic model with convolutional-pooling structure for information retrieval
Yelong Shen, Xiaodong He, Jianfeng Gao, Li Deng, and Gregoire Mesnil. 2014 · 2014
Cited alongside, same era.
From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014 · 2014
Cited alongside, same era.
A large annotated corpus for learning natural language inference
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015 · 2015
Cited alongside, same era.
Learning bidirectional intent embeddings by convolutional deep structured semantic models for spoken language understanding
Yun-Nung Chen, Dilek Hakkani-Tür, and Xiaodong He. 2015 · 2015
Cited alongside, same era.
cocor: A comprehensive solution for the statistical comparison of correlations
Birk Diedenhofen and Jochen Musch. 2015 · 2015
Cited alongside, same era.
FCICU at SemEval-2017 Task 1: Sense-based language independent semantic textual similarity approach
Basma Hassan, Samir AbdelRahman, Reem Bahgat, and Ibrahim Farag. 2017 · 2015
Cited alongside, same era.
Multi-perspective sentence similarity modeling with convolutional neural networks
Hua He, Kevin Gimpel, and Jimmy Lin. 2015 · 2015
Cited alongside, same era.
OntoNotes: The 90% solution
Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006 · 2015
Cited alongside, same era.
Deep unordered composition rivals syntactic methods for text classification
Mohit Iyyer, Varun Manjunatha, Jordan Boyd-Graber, and Hal Daumé III. 2015 · 2015
Cited alongside, same era.

Then

Task-oriented intrinsic evaluation of semantic textual similarity
Nils Reimers, Philip Beyer, and Iryna Gurevych. 2016 · 2016
Later among the works it cites.
Samsung Poland NLP Team at SemEval-2016 Task 1: Necessity for diversity; combining recursive autoencoders, wordnet and ensemble methods to measure semantic similarity
Barbara Rychalska, Katarzyna Pakulska, Krystyna Chodorowska, Wojciech Walczak, and Piotr Andruszkiewicz. 2016 · 2016
Later among the works it cites.
HCTI at SemEval-2017 Task 1: Use convolutional neural network to evaluate semantic textual similarity
Yang Shao. 2017 · 2016
Later among the works it cites.
Towards universal paraphrastic sentence embeddings
Original
John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2016b · 2016
Later among the works it cites.
A simple but tough-to-beat baseline for sentence embeddings
Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017 · 2017
Closest in time.
Supervised learning of universal sentence representations from natural language inference data
Original
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loïc Barrault, and Antoine Bordes. 2017 · 2017
Closest in time.
Representing sentences as low-rank subspaces
Original
Jiaqi Mu, Suma Bhat, and Pramod Viswanath. 2017 · 2017
Closest in time.
LIM-LIG at SemEval-2017 Task1: Enhancing the semantic similarity for arabic sentences with vectors weighting
El Moatez Billah Nagoudi, Jérémy Ferrero, and Didier Schwab. 2017 · 2017
Closest in time.
Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features
Original
Matteo Pagliardini, Prakhar Gupta, and Martin Jaggi. 2017 · 2017
Closest in time.
Revisiting recurrent networks for paraphrastic sentence embeddings
Original
John Wieting and Kevin Gimpel. 2017 · 2017
Closest in time.
OPI-JSA at SemEval-2017 Task 1: Application of ensemble learning for computing semantic textual similarity
Martyna Śpiewak, Piotr Sobecki, and Daniel Karaś. 2017 · 2018
Closest in time.
Lump at SemEval-2017 Task 1: Towards an interlingua semantic similarity
Cristina España Bonet and Alberto Barrón-Cedeño. 2017 · 2019
Closest in time.
QLUT at SemEval-2017 Task 1: Semantic textual similarity based on word embeddings
Fanqing Meng, Wenpeng Lu, Yuteng Zhang, Jinyong Cheng, Yuehan Du, and Shuwang Han. 2017 · 2020
Closest in time.
ResSim at SemEval-2017 Task 1: Multilingual word representations for semantic textual similarity
Johannes Bjerva and Robert Östling. 2017 · 2021
Closest in time.
ITNLP-AiKF at SemEval-2017 Task 1: Rich features based svr for semantic textual similarity computing
Wenjie Liu, Chengjie Sun, Lei Lin, and Bingquan Liu. 2017 · 2022
Closest in time.
Neobility at SemEval-2017 Task 1: An attention-based sentence similarity model
WenLi Zhuang and Ernie Chang. 2017 · 2023
Closest in time.
SEF @ @ UHH at SemEval-2017 Task 1: Unsupervised knowledge-free semantic textual similarity via paragraph vector
Mirela-Stefania Duma and Wolfgang Menzel. 2017 · 2024
Closest in time.
STS-UHH at SemEval-2017 Task 1: Scoring semantic textual similarity using supervised and unsupervised ensemble
Sarah Kohail, Amr Rekaby Salama, and Chris Biemann. 2017 · 2025
Closest in time.
UMDeep at SemEval-2017 Task 1: End-to-end shared weight LSTM model for semantic textual similarity
Joe Barrow and Denis Peskov. 2017 · 2026
Closest in time.
MITRE at SemEval-2017 Task 1: Simple semantic similarity
John Henderson, Elizabeth Merkhofer, Laura Strickhart, and Guido Zarrella. 2017 · 2027
Closest in time.
DLS@CU: Sentence similarity from word alignment and semantic vector composition
Md Arafat Sultan, Steven Bethard, and Tamara Sumner. 2015 · 2027
Closest in time.
ECNU at SemEval-2017 Task 1: Leverage kernel-based traditional nlp features and neural networks to build a universal model for multilingual and cross-lingual semantic textual similarity
Junfeng Tian, Zhiheng Zhou, Man Lan, and Yuanbin Wu. 2017 · 2028
Closest in time.
PurdueNLP at SemEval-2017 Task 1: Predicting semantic textual similarity with paraphrase and event embeddings
I-Ta Lee, Mahak Goindani, Chang Li, Di Jin, Kristen Marie Johnson, Xiao Zhang, Maria Leonor Pacheco, and Dan Goldwasser. 2017 · 2029
Closest in time.
RTM at SemEval-2017 Task 1: Referential translation machines for predicting semantic similarity
Ergun Biçici. 2017b · 2030
Closest in time.
LIPN-IIMAS at SemEval-2017 Task 1: Subword embeddings, attention recurrent neural networks and cross word alignment for semantic textual similarity
Ignacio Arroyo-Fernández and Ivan Vladimir Meza Ruiz. 2017 · 2031
Closest in time.
L2f/inesc-id at semeval-2017 tasks 1 and 2: Lexical and semantic features in word and textual similarity
Pedro Fialho, Hugo Patinho Rodrigues, Luísa Coheur, and Paulo Quaresma. 2017 · 2032
Closest in time.
SemEval-2015 Task 2: Semantic Textual Similarity, English, Spanish and Pilot on Interpretability
Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Iñigo Lopez-Gazpio, Montse Maritxalar, Rada Mihalcea, German Rigau, Larraitz Uria, and Janyce Wiebe. 2015 · 2045
Closest in time.
Matrix factorization using window sampling and negative sampling for improved word representations
Alexandre Salle, Marco Idiart, and Aline Villavicencio. 2016b · 2068
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…