Fetching the paper…
Reading the bibliography…
Cognitive computing systems require human labeled data for evaluation, and often for training.
The meaning of meaning
Charles Kay Ogden and I.A. Richards. 1923 · 1923
Earlier work this paper cites.
Note on the sampling error of the difference between correlated proportions or percentages
Quinn McNemar. 1947 · 1947
Earlier work this paper cites.
Effective mapping of biomedical text to the UMLS Metathesaurus: the MetaMap program.. In Proceedings of the AMIA Symposium
Alan R Aronson. 2001 · 2001
Earlier work this paper cites.
The unified medical language system (UMLS): integrating biomedical terminology
Olivier Bodenreider. 2004 · 2004
Earlier work this paper cites.
Cascade: crowdsourcing taxonomy creation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Lydia B. Chilton, Greg Little, Darren Edge, Daniel S. Weld, and James A. Landay. 2013 · 2008
Earlier work this paper cites.
Cheap and Fast—but is It Good?: Evaluating Non-expert Annotations for Natural Language Tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing
Rion Snow, Brendan O’Connor, Daniel Jurafsky, and Andrew Y. Ng. 2008 · 2008
Earlier work this paper cites.
Distant supervision for relation extraction without labeled data. In Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2
Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. 2009 · 2009
Earlier work this paper cites.
Annotating named entities in Twitter data with crowdsourcing. In In Proc. NAACL HLT
Tim Finin, Will Murnane, Anand Karandikar, Nicholas Keller, Justin Martineau, and Mark Dredze. 2010 · 2010
Earlier work this paper cites.
The multidimensional wisdom of crowds. In Advances in neural information processing systems
Peter Welinder, Steve Branson, Pietro Perona, and Serge J Belongie. 2010 · 2010
Earlier work this paper cites.
Large Scale Relation Detection. In Proceedings of the NAACL HLT 2010 First International Workshop on Formalisms and Methodology for Learning by Reading
Chris Welty, James Fan, David Gondek, and Andrew Schlaikjer. 2010 · 2010
Earlier work this paper cites.
What Determines Inter-coder Agreement in Manual Annotations? A Meta-analytic Investigation
Petra Saskia Bayerl and Karsten Ingmar Paul. 2011 · 2011
Earlier work this paper cites.
Overcoming barriers to NLP for clinical text: the role of shared tasks and the need for additional creative solutions
Wendy W Chapman, Prakash M Nadkarni, Lynette Hirschman, Leonard W D’Avolio, Guergana K Savova, and Ozlem Uzuner. 2011 · 2011
Cited alongside, same era.
Building a persistent workforce on mechanical turk for multilingual data collection. In Proceedings of The 3rd Human Computation Workshop (HCOMP 2011)
David L Chen and William B Dolan. 2011 · 2011
Cited alongside, same era.
Validating candidate gene-mutation relations in MEDLINE abstracts via crowdsourcing. In Data Integration in the Life Sciences
John D Burger, Emily Doughty, Sam Bayer, David Tresner-Kirsch, Ben Wellner, John Aberdeen, Kyungjoon Lee, Maricel G Kann, and Lynette Hirschman. 2012 · 2012
Cited alongside, same era.
Crowd Truth: Harnessing disagreement in crowdsourcing a relation extraction gold standard
Lora Aroyo and Chris Welty. 2013a · 2013
Cited alongside, same era.
Measuring crowd truth for medical relation extraction. In AAAI 2013 Fall Symposium on Semantics for Big Data
Experiments with crowdsourced re-annotation of a POS tagging data set. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
Dirk Hovy, Barbara Plank, and Anders Søgaard. 2014 · 2014
Later among the works it cites.
CrowdTruth: Machine-Human Computation Framework for Harnessing Disagreement in Gathering Annotated Data
Oana Inel, Khalid Khamkham, Tatiana Cristea, Anca Dumitrache, Arne Rutjes, Jelle van der Ploeg, Lukasz Romaszko, Lora Aroyo, and Robert-Jan Sips. 2014 · 2014
Later among the works it cites.
Combining information extraction and human computing for crowdsourced knowledge acquisition. In 30th International Conference on Data Engineering
Sarath Kumar Kondreddi, Peter Triantafillou, and Gerhard Weikum. 2014 · 2014
Later among the works it cites.
Measuring gradience in speakers’ grammaticality judgements. In Proceedings of the 36th Annual Conference of the Cognitive Science Society
Jey Han Lau, Alexander Clark, and Shalom Lappin. 2014 · 2014
Later among the works it cites.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Lora Aroyo and Chris Welty. 2013b · 2013
Cited alongside, same era.
Hybrid entity clustering using crowds and data
Jongwuk Lee, Hyunsouk Cho, Jin-Woo Park, Young-rok Cha, Seung-won Hwang, Zaiqing Nie, and Ji-Rong Wen. 2013 · 2013
Cited alongside, same era.
Crowdsourcing the verification of relationships in biomedical ontologies. In AMIA Annual Symposium Proceedings
Jonathan M Mortensen, Mark A Musen, and Natalya F Noy. 2013 · 2013
Cited alongside, same era.
Haijun Zhai, Todd Lingren, Louise Deleger, Qi Li, Megan Kaiser, Laura Stoutenborough, and Imre Solti. 2013 · 2013
Cited alongside, same era.
The Three Sides of CrowdTruth
Lora Aroyo and Chris Welty. 2014 · 2014
Cited alongside, same era.
Conference v2. 0: An uncertain version of the OAEI Conference benchmark
Michelle Cheatham and Pascal Hitzler. 2014 · 2014
Cited alongside, same era.
Linguistically debatable or just plain wrong?. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
Barbara Plank, Dirk Hovy, and Anders Søgaard. 2014 · 2014
Later among the works it cites.
Medical Relation Extraction with Manifold Models. In 52nd Annual Meeting of the ACL, vol. 1
Chang Wang and James Fan. 2014 · 2014
Later among the works it cites.
Achieving Expert-Level Annotation Quality with CrowdTruth: the Case of Medical Relation Extraction. In Proceedings of Biomedical Data Mining, Modeling, and Semantic Integration (BDM2I) Workshop, International Semantic Web Conference (ISWC) 2015
Anca Dumitrache, Lora Aroyo, and Chris Welty. 2015 · 2015
Later among the works it cites.
Exposing ambiguities in a relation-extraction gold standard with crowdsourcing
Tong Shu Li, Benjamin M Good, and Andrew I Su. 2015 · 2015
Later among the works it cites.
Resolvable vs. Irresolvable Ambiguity: A New Hybrid Framework for Dealing with Uncertain Ground Truth. In 1st Workshop on Human-Centered Machine Learning at SIGCHI 2016
Mike Schaekermann, Edith Law, Alex C. Williams, and William Callaghan. 2016 · 2016
Later among the works it cites.
Whose Vote Should Count More: Optimal Integration of Labels from Labelers of Unknown Expertise
Jacob Whitehill, Ting fan Wu, Jacob Bergsma, Javier R. Movellan, and Paul L. Ruvolo. 2009 · 2043
Closest in time.