2022

Measuring Data

Mitchell, Margaret, Luccioni, Alexandra Sasha, Lambert, Nathan et al.

Understand

We identify the task of measuring data to quantitatively characterize the composition of machine learning data and datasets.

Similar to an object's height, width, and volume, data measurements quantify different attributes of data along common dimensions that support comparison.
Several lines of research have proposed what we refer to as measurements, with differing terminology; we bring some of this work together, particularly in fields of computer vision and language, and build from it to motivate measuring data as a critical component of responsible AI development.
Measuring data aids in systematically building and analyzing machine learning (ML) data towards specific goals and gaining better control of what modern ML systems will learn.

Built on

Variabilità e Mutabilità
C. Gini · 1912
Earlier work this paper cites.
T he variable desert
J. A. Harris · 1916
Earlier work this paper cites.
An Account of the Principles of Measurement and Calculation
N. Campbell · 1928
Earlier work this paper cites.
On the theory of scales of measurement
S. S. Stevens · 1946
Earlier work this paper cites.
On information and sufficiency
S. Kullback and R. A. Leibler · 1951
Earlier work this paper cites.
On measures of entropy and information
A. Rényi et al · 1961
Earlier work this paper cites.
Двоичные коды с исправлением выпадений, вставок и замещений символов
V. Levenshtein · 1965
Earlier work this paper cites.
A nonparametric estimate of a multivariate density function
D. O. Loftsgaarden and C. P. Quesenberry · 1965
Earlier work this paper cites.
Computational analysis of present-day american english
H. Kucera, W. N. Francis, W. F. Twaddell, M. L. Marckworth, L. M. Bell, and J. B. Carroll · 1967
Earlier work this paper cites.
Singular value decomposition and least squares solutions
G. H. Golub and C. H. Reinsch · 1970
Earlier work this paper cites.
Diversity and evenness: a unifying notation and its consequences
M. O. Hill · 1973
Earlier work this paper cites.
Reading rate and retention as a function of the number of propositions in the base structure of sentences
W. Kintsch and J. Keenan · 1973
Earlier work this paper cites.
Intelligence tests may favour the majority groups in a population
G. M. Harrington · 1975
Earlier work this paper cites.
Concentration in modern industry: Theory, measurement and the UK experience
L. Hannah and J. A. Kay · 1977
Earlier work this paper cites.
How to write plain english
R. Flesch · 1979
Earlier work this paper cites.
Measurement theory
F. S. Roberts · 1985
Earlier work this paper cites.
Ecological diversity and its measurement
A. E. Magurran · 1988
Earlier work this paper cites.
Spoken and written language
M. A. K. Halliday · 1989
Earlier work this paper cites.
A unified approach to the change of resolution: Space and gray-level
S. Peleg, M. Werman, and H. Rom · 1989
Earlier work this paper cites.
Critical values for spearman’s rank order correlation
P. H. Ramsey · 1989
Earlier work this paper cites.
An estimate of an upper bound for the entropy of english
P. F. Brown, S. A. Della Pietra, V. J. Della Pietra, J. C. Lai, and R. L. Mercer · 1992
Earlier work this paper cites.
100 million words of English: The British National Corpus (BNC)
G. N. Leech · 1992
Earlier work this paper cites.
Natural image statistics and efficient coding
B. A. Olshausen and D. J. Field · 1996
Earlier work this paper cites.
Quantitative science and the definition of measurement in psychology
J. Michell · 1997
Earlier work this paper cites.
A neural probabilistic language model
Y. Bengio, R. Ducharme, and P. Vincent · 2000
Earlier work this paper cites.
CRC standard probability and statistics tables and formulae
S. Kokoska and D. Zwillinger · 2000
Earlier work this paper cites.
Machine learning from imbalanced data sets 101
F. Provost · 2000
Earlier work this paper cites.
The earth mover’s distance as a metric for image retrieval
Y. Rubner, C. Tomasi, and L. J. Guibas · 2000
Earlier work this paper cites.
Zipf and heaps laws’ coefficients depend on language
A. Gelbukh and G. Sidorov · 2001
Earlier work this paper cites.
Diversity and inclusion metrics in subset selection
Original
M. Mitchell, D. Baker, N. Moorosi, E. Denton, B. Hutchinson, A. Hanna, T. Gebru, and J. Morgenstern · 2002
Earlier work this paper cites.
Data quality assessment
L. L. Pipino, Y. W. Lee, and R. Y. Wang · 2002
Earlier work this paper cites.
Theoretical comparison between the gini index and information gain criteria
L. E. Raileanu and K. Stoffel · 2004
Earlier work this paper cites.
Encyclopedia of social measurement
J. Hox and H. Boeije · 2005
Earlier work this paper cites.
Measurement theory
J. Michell · 2005
Earlier work this paper cites.
Elements of information theory
M. Thomas and A. T. Joy · 2006
Earlier work this paper cites.
Education inputs, student performance and school finance reform in michigan
L. Chaudhary · 2007
Earlier work this paper cites.
Unbiased split selection for classification trees based on the gini index
C. Strobl, A.-L. Boulesteix, and T. Augustin · 2007
Earlier work this paper cites.
Statistics in a Nutshell: A Desktop Quick Reference
S. Boslaugh and D. P. A. Watters · 2008
Earlier work this paper cites.

Similar

Burstiness and memory in complex systems
K.-I. Goh and A.-L. Barabási · 2008
Cited alongside, same era.
Revisiting readability: A unified framework for predicting text quality
E. Pitler and A. Nenkova · 2008
Cited alongside, same era.
Active learning with sampling by uncertainty and density for word sense disambiguation and text classification
J. Zhu, H. Wang, T. Yao, and B. K. Tsou · 2008
Cited alongside, same era.
Pearson correlation coefficient
J. Benesty, J. Chen, Y. Huang, and I. Cohen · 2009
Cited alongside, same era.
Increasing diversity: Natural language measures for software fault prediction
D. Binkley, H. Feild, D. Lawrie, and M. Pighin · 2009
Cited alongside, same era.
Idea density — a potentially informative characteristic of retrieved documents
M. A. Covington · 2009
Cited alongside, same era.
Imagenet: A large-scale hierarchical image database
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei · 2009
Cited alongside, same era.
History of measurement theory
J. A. Díez · 2009
Cited alongside, same era.
Learning multiple layers of features from tiny images
A. Krizhevsky, G. Hinton, et al · 2009
Cited alongside, same era.
Data mining with imbalanced class distributions: concepts and methods
R. C. Prati, G. E. Batista, and M. C. Monard · 2009
Cited alongside, same era.
A survey of types of text noise and techniques to handle noisy text
L. V. Subramaniam, S. Roy, T. A. Faruquie, and S. Negi · 2009
Cited alongside, same era.
Distributions of the kullback–leibler divergence with applications
D. I. Belov and R. D. Armstrong · 2011
Cited alongside, same era.

Then

Intrinsic evaluation of summarization datasets
R. Bommasani and C. Cardie · 2020
Later among the works it cites.
Intrinsic evaluation of summarization datasets
R. Bommasani and C. Cardie · 2020
Later among the works it cites.
Lessons from archives: Strategies for collecting sociocultural data in machine learning
E. S. Jo and T. Gebru · 2020
Later among the works it cites.
Diversity, density, and homogeneity: Quantitative characteristic metrics for text collections
Original
Y.-A. Lai, X. Zhu, Y. Zhang, and M. Diab · 2020
Later among the works it cites.
How the french revolution created the metric system
V. López · 2020
Later among the works it cites.
The definition and measurement of heterogeneity
A. Nunes, T. Trappenberg, and M. Alda · 2020
Later among the works it cites.
Estimating training data influence by tracing gradient descent
G. Pruthi, F. Liu, S. Kale, and M. Sundararajan · 2020
Later among the works it cites.
Deep reinforcement learning at the edge of the statistical precipice
R. Agarwal, M. Schwarzer, P. S. Castro, A. C. Courville, and M. Bellemare · 2021
Later among the works it cites.
Measuring model biases in the absence of ground truth
O. Aka, K. Burke, A. Bauerle, C. Greer, and M. Mitchell · 2021
Later among the works it cites.
On the dangers of stochastic parrots: Can language models be too big?
E. Bender and Gebru, Timnit, A. McMillan-Major, and S. Shmitchell · 2021
Later among the works it cites.
Multimodal datasets: misogyny, pornography, and malignant stereotypes
Original
A. Birhane, V. U. Prabhu, and E. Kahembwe · 2021
Later among the works it cites.
Excavating ai: The politics of images in machine learning training sets
K. Crawford and T. Paglen · 2021
Later among the works it cites.
Data-centric ai resource hub, 2021
Data-centric AI · 2021
Later among the works it cites.
Datasheets for datasets
T. Gebru, J. Morgenstern, B. Vecchione, J. W. Vaughan, H. Wallach, H. D. Iii, and K. Crawford · 2021
Later among the works it cites.
Know your data, 2021
Google Research · 2021
Later among the works it cites.
Measurement and fairness
A. Z. Jacobs and H. Wallach · 2021
Later among the works it cites.
Deduplicating training data makes language models better
Original
K. Lee, D. Ippolito, A. Nystrom, C. Zhang, D. Eck, C. Callison-Burch, and N. Carlini · 2021
Later among the works it cites.
A data-centric approach for training deep neural networks with less data
Original
M. Motamedi, N. Sakharnykh, and T. Kaldewey · 2021
Later among the works it cites.
Data and its (dis) contents: A survey of dataset development and use in machine learning research
A. Paullada, I. D. Raji, E. M. Bender, E. Denton, and A. Hanna · 2021
Later among the works it cites.
“everyone wants to do the model work, not the data work”: Data cascades in high-stakes ai
N. Sambasivan, S. Kapania, H. Highfill, D. Akrong, P. Paritosh, and L. M. Aroyo · 2021
Later among the works it cites.
Do datasets have politics? disciplinary values in computer vision dataset development
M. K. Scheuerman, A. Hanna, and E. Denton · 2021
Later among the works it cites.
Laion-400m: Open dataset of clip-filtered 400 million image-text pairs
Original
C. Schuhmann, R. Vencu, R. Beaumont, R. Kaczmarczyk, C. Mullis, A. Katta, T. Coombes, J. Jitsev, and A. Komatsuzaki · 2021
Later among the works it cites.
Announcing the neurips 2021 datasets and benchmarks track, 2021
J. Vanschoren and S. Yeung · 2021
Later among the works it cites.
Summvis: Interactive visual analysis of models, data, and evaluation for text summarization
Original
J. Vig, W. Kryściński, K. Goel, and N. F. Rajani · 2021
Later among the works it cites.
Data collection and quality challenges in deep learning: A data-centric ai perspective
Original
S. E. Whang, Y. Roh, H. Song, and J.-G. Lee · 2021
Later among the works it cites.
Tracing knowledge in language models back to the training data
Original
E. Akyürek, T. Bolukbasi, F. Liu, B. Xiong, I. Tenney, J. Andreas, and K. Guu · 2022
Closest in time.
Clean lab, 2022
Clean Lab · 2022
Closest in time.
Bertin: Efficient pre-training of a spanish language model using perplexity sampling
Original
J. De la Rosa, E. G. Ponferrada, P. Villegas, P. G. d. P. Salas, M. Romero, and M. Grandury · 2022
Closest in time.
dcbench: a benchmark for data-centric ai systems
S. Eyuboglu, B. Karlaš, C. Ré, C. Zhang, and J. Zou · 2022
Closest in time.
The vendi score: A diversity evaluation metric for machine learning
Original
D. Friedman and A. B. Dieng · 2022
Closest in time.
Whose language counts as high quality? measuring language ideologies in text data selection
Original
S. Gururangan, D. Card, S. K. Drier, E. K. Gade, L. Z. Wang, Z. Wang, L. Zettlemoyer, and N. A. Smith · 2022
Closest in time.
Data quality for AI tool: Exploratory data analysis on IBM API
A. Jariwala, A. Chaudhari, C. Bhatt, and D.-N. Le · 2022
Closest in time.
Advances, challenges and opportunities in creating data for trustworthy ai
W. Liang, G. A. Tadesse, D. Ho, L. Fei-Fei, M. Zaharia, C. Zhang, and J. Zou · 2022
Closest in time.
Bugs in the data: How imagenet misrepresents biodiversity
Original
A. S. Luccioni and D. Rolnick · 2022
Closest in time.
Introducing the data measurements tool: an interactive tool for looking at datasets, 2022
S. Luccioni, Y. Jernite, and M. Mitchell · 2022
Closest in time.
Seal : Interactive tool for systematic error analysis and labeling
Original
N. Rajani, W. Liang, L. Chen, M. Mitchell, and J. Zou · 2022
Closest in time.
Beyond Measure: The Hidden History of Measurement
J. Vincent · 2022
Closest in time.
Datalab: A platform for data analysis and intervention
Original
Y. Xiao, J. Fu, W. Yuan, V. Viswanathan, Z. Liu, Y. Liu, G. Neubig, and P. Liu · 2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…