2019

Question Answering for Privacy Policies: Combining Computational and Legal Perspectives

Ravichander, Abhilasha, Black, Alan W, Wilson, Shomir et al.

Understand

Privacy policies are long and complex documents that are difficult for users to read and understand, and yet, they have legal effects on how user data is collected, managed and used.

Ideally, we would like to empower users to inform themselves about issues that matter to them, and enable them to selectively explore those issues.
We present PrivacyQA, a corpus consisting of 1750 questions about the privacy policies of mobile applications, and over 3500 expert annotations of relevant answers.
We observe that a strong neural baseline underperforms human performance by almost 0.3 F1 on PrivacyQA, suggesting considerable room for improvement for future systems.

Built on

Privacy restrictions on transnational data flows: A comparison of the council of europe draft convention and oecd guidelines
P Howard Patrick. 1980 · 1980
Earlier work this paper cites.
Privacy online: A report to congress
Federal Trade Commission. 1998 · 1998
Earlier work this paper cites.
P3p: Making privacy policies more useful
Lorrie Faith Cranor. 2003 · 2003
Earlier work this paper cites.
The oecd principles of corporate governance
OCDE OECD. 2004 · 2004
Earlier work this paper cites.
The molecular basis of variation in human color vision
SS Deeb. 2005 · 2005
Earlier work this paper cites.
A question answer system for legal information retrieval
Paulo Quaresma and Irene Pimenta Rodrigues. 2005 · 2005
Earlier work this paper cites.
Question answering from structured knowledge sources
Anette Frank, Hans-Ulrich Krieger, Feiyu Xu, Hans Uszkoreit, Berthold Crysmann, Brigitte Jörg, and Ulrich Schäfer. 2007 · 2007
Earlier work this paper cites.
Question answering in restricted domains: An overview
Diego Mollá and José Luis Vicedo. 2007 · 2007
Earlier work this paper cites.
The cost of reading privacy policies
Aleecia M McDonald and Lorrie Faith Cranor. 2008 · 2008
Earlier work this paper cites.
A nutrition label for privacy
Patrick Gage Kelley, Joanna Bresee, Lorrie Faith Cranor, and Robert W Reeder. 2009 · 2009
Earlier work this paper cites.
Nlp for shallow question answering of legal documents using graphs
Alfredo Monroy, Hiram Calvo, and Alexander Gelbukh. 2009 · 2009
Earlier work this paper cites.
The limits of notice and choice
Fred H Cate. 2010 · 2010
Earlier work this paper cites.
Fixing broken doors: Strategies for drafting privacy policies young people can understand
Anca Micheti, Jacquelyn Burkell, and Valerie Steeves. 2010 · 2010
Earlier work this paper cites.
Necessary but not sufficient: Standardized mechanisms for privacy notice and choice
Lorrie Faith Cranor. 2012 · 2012
Earlier work this paper cites.
Protecting consumer privacy in an era of rapid change: Recommendations for businesses and policymakers
FTC US Federal Trade Commission et al. 2012 · 2012
Earlier work this paper cites.
Mctest: A challenge dataset for the open-domain machine comprehension of text
Matthew Richardson, Christopher JC Burges, and Erin Renshaw. 2013 · 2013
Earlier work this paper cites.
Glove: Global vectors for word representation
Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014 · 2014
Earlier work this paper cites.
Applying deep learning to answer selection: A study and an open task
Minwei Feng, Bing Xiang, Michael R Glass, Lidan Wang, and Bowen Zhou. 2015 · 2015
Earlier work this paper cites.
Teaching machines to read and comprehend
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015 · 2015
Earlier work this paper cites.

Similar

Applying a convolutional neural network to legal question answering
Mi-Young Kim, Ying Xu, and Randy Goebel. 2015 · 2015
Cited alongside, same era.
Privacy and behavioral advertising: Towards meeting users’ preferences
Pedro Giovanni Leon, Ashwini Rao, Florian Schaub, Abigail Marsh, Lorrie Faith Cranor, and Norman Sadeh. 2015 · 2015
Cited alongside, same era.
Predicting associated statutes for legal problems
Yi-Hung Liu, Yen-Liang Chen, and Wu-Liang Ho. 2015 · 2015
Cited alongside, same era.
Disagreeable privacy policies: Mismatches between meaning and users’ understanding
Joel R Reidenberg, Travis Breaux, Lorrie Faith Cranor, Brian French, Amanda Grannis, James T Graves, Fei Liu, Aleecia McDonald, Thomas B Norton, and Rohan Ramanath. 2015 · 2015
Cited alongside, same era.
A design space for effective privacy notices
Florian Schaub, Rebecca Balebako, Adam L Durity, and Lorrie Faith Cranor. 2015 · 2015
Cited alongside, same era.
How short is too short? implications of length and framing on the effectiveness of privacy notices
Joshua Gluck, Florian Schaub, Amy Friedman, Hana Habib, Norman Sadeh, Lorrie Faith Cranor, and Yuvraj Agarwal. 2016 · 2016
Cited alongside, same era.
Big data privacy: a technological perspective and review
Priyank Jain, Manasi Gyanchandani, and Nilay Khare. 2016 · 2016
Cited alongside, same era.
Analyzing vocabulary intersections of expert annotations and topic models for data practices in privacy policies
Frederick Liu, Shomir Wilson, Florian Schaub, and Norman Sadeh. 2016 · 2016
Cited alongside, same era.
Ms marco: A human generated machine reading comprehension dataset
Original
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016 · 2016
Cited alongside, same era.
Who did what: A large-scale person-centered cloze dataset
Takeshi Onishi, Hai Wang, Mohit Bansal, Kevin Gimpel, and David McAllester. 2016 · 2016
Cited alongside, same era.
SQuAD: 100,000+ questions for machine comprehension of text
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016 · 2016
Cited alongside, same era.
Improved representation learning for question answer matching
Ming Tan, Cicero dos Santos, Bing Xiang, and Bowen Zhou. 2016 · 2016
Cited alongside, same era.

Then

Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017 · 2017
Later among the works it cites.
The eu general data protection regulation (gdpr)
Paul Voigt and Axel Von dem Bussche. 2017 · 2017
Later among the works it cites.
Making neural QA as simple as possible but not simpler
Dirk Weissenborn, Georg Wiese, and Laura Seiffe. 2017 · 2017
Later among the works it cites.
Analyzing privacy policies at scale: From crowdsourcing to automated annotations
S. Wilson, F. Schaub, F. Liu, K.M. Sathyendra, S. Zimmeck, R. Ramanath, F. Liu, N. Sadeh, and N.A. Smith. 2017 · 2017
Later among the works it cites.
Revealed: 50 million facebook profiles harvested for cambridge analytica in major data breach
Carole Cadwalladr and Emma Graham-Harrison. 2018 · 2018
Later among the works it cites.
QuAC: Question answering in context
Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018 · 2018
Later among the works it cites.
Annotation artifacts in natural language inference data
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018 · 2018
Later among the works it cites.
Polisis: Automated analysis and presentation of privacy policies using deep learning
Original
Hamza Harkous, Kassem Fawaz, Rémi Lebret, Florian Schaub, Kang G Shin, and Karl Aberer. 2018 · 2018
Later among the works it cites.
How much reading does reading comprehension require? a critical investigation of popular benchmarks
Divyansh Kaushik and Zachary C. Lipton. 2018 · 2018
Later among the works it cites.
The narrativeqa reading comprehension challenge
Tomáš Kočiskỳ, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gáabor Melis, and Edward Grefenstette. 2018 · 2018
Later among the works it cites.
Stress test evaluation for natural language inference
Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. 2018 · 2018
Later among the works it cites.
Hypothesis only baselines in natural language inference
Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018 · 2018
Later among the works it cites.
Know what you don’t know: Unanswerable questions for SQuAD
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018 · 2018
Later among the works it cites.
Which apps have privacy policies?
Peter Story, Sebastian Zimmeck, and Norman Sadeh. 2018 · 2018
Later among the works it cites.
Wikiqa: A challenge dataset for open-domain question answering
Yi Yang, Wen-tau Yih, and Christopher Meek. 2015 · 2018
Later among the works it cites.
BERT: Pre-training of deep bidirectional transformers for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 · 2019
Closest in time.
The case for a gdpr-specific annotated dataset of privacy policies
Matthias Gallé, Athena Christofi, and Hady Elsahar. 2019 · 2019
Closest in time.
Natural questions: A benchmark for question answering research
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019 · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…