Fetching the paper…
Reading the bibliography…
Recently, it was shown that most popular IR measures are not interval-scaled, implying that decades of experimental IR research used potentially improper methods, which may have produced questionable results.
The Probable Error of a Mean
Student · 1908
Earlier work this paper cites.
Physics: The Elements
N. R. Campbell · 1920
Earlier work this paper cites.
Statistical Methods for Research Workers
R. A. Fisher · 1925
Earlier work this paper cites.
An account of the principles of measurement and calculation
N. R. Campbell · 1928
Earlier work this paper cites.
The Use of Ranks to Avoid the Assumption of Normality Implicit in the Analysis of Variance
M. Friedman · 1937
Earlier work this paper cites.
A Correction: The Use of Ranks to Avoid the Assumption of Normality Implicit in the Analysis of Variance
M. Friedman · 1939
Earlier work this paper cites.
Quantitative estimates of sensory events: final report of the committee appointed to consider and report upon the possibility of quantitative estimates of sensory events
A. Ferguson, C. S. Myers, R. J. Bartlett, H. Banister, F. C. Bartlett, W. Brown, N. R. Campbell, K. J. W. Craik, J. Drever, J. Guild, R. A. Houstoun, J. O. Irwin, G. W. C. Kaye, S. J. F. Philpott, L. F. Richardson, J. H. Shaxby, T. Smith, R. H. Thouless, and W. S. Tucker · 1940
Earlier work this paper cites.
Individual Comparisons by Ranking Methods
F. Wilcoxon · 1945
Earlier work this paper cites.
On the Theory of Scales of Measurement
S. S. Stevens · 1946
Earlier work this paper cites.
Rank correlation methods
M. G. Kendall · 1948
Earlier work this paper cites.
Comparing Individual Means in the Analysis of Variance
J. W. Tukey · 1949
Earlier work this paper cites.
Use of Ranks in One-Criterion Variance Analysis
W. H. Kruskal and W. A. Wallis · 1952
Earlier work this paper cites.
On the Statistical Treatment of Football Numbers
F. M. Lord · 1953
Earlier work this paper cites.
Nonparametric Statistics: For the Behavioral Science
S. Siegel · 1956
Earlier work this paper cites.
Measurement and statistics: a basic text emphasizing behavioral science applications
V. L. Senders · 1958
Earlier work this paper cites.
Efficient Conversion Of Non-Metric Information Into Metric Information
R. P. Abelson and J. W. Tukey · 1959
Earlier work this paper cites.
Non-Parametric Methods in Psychological Research
J. Gaito · 1959
Earlier work this paper cites.
Scales and Statistics: Parametric and Nonparametric
NN. H. Anderson · 1961
Earlier work this paper cites.
Simultaneous Conjoint Measurement: A New Type of Fundamental Measurement
R. D. Luce and J. W. Tukey · 1964
Earlier work this paper cites.
A theory of appropriate statistics
E. W. Adams, R. F. Fagot, and R. E. Robinson · 1965
Earlier work this paper cites.
Computer Evaluation of Indexing and Text Processing
G. Salton and M. E. Lesk · 1968
Earlier work this paper cites.
Foundations of Measurement. Additive and Polynomial Representations , volume 1
D. H. Krantz, R. D. Luce, P. Suppes, and A. Tversky · 1971
Earlier work this paper cites.
Distance between sets as an objective measure of retrieval effectiveness
M. H. Heine · 1973
Earlier work this paper cites.
Foundations of Evaluation
C. J. van Rijsbergen · 1974
Earlier work this paper cites.
Scales and Statistics
P. L. Gardner · 1975
Earlier work this paper cites.
Appropriate Statistics and Measurement Scales
W. B. Ware and J. Benson · 1975
Earlier work this paper cites.
Information Retrieval
C. J. van Rijsbergen · 1979
Earlier work this paper cites.
Measurement-theoretical investigation of the MZ-metric
P. Bollmann and V. S. Cherniavsky · 1980
Earlier work this paper cites.
Measurement Scales and Statistics: Resurgence of an Old Misconception
J. Gaito · 1980
Earlier work this paper cites.
Restricted Evaluation in Information Retrieval
P. Bollmann and V. S. Cherniavsky · 1981
Earlier work this paper cites.
Retrieval effectiveness
C. J. van Rijsbergen · 1981
Earlier work this paper cites.
Scales and Meaningfulness of Quantitative Laws
J. C. Falmagne and L. Narens · 1983
Earlier work this paper cites.
Two Axioms for Evaluation Measures in Information Retrieval
P. Bollmann · 1984
Earlier work this paper cites.
Measurement Scales and Statistics: The Misconception Misconceived
J. T. Townsend and F. G. Ashby · 1984
Earlier work this paper cites.
The Relationship Between Ordinal Measures and Their Underlying Values: Why All the Disagreement?
R. M. O’Brien · 1985
Earlier work this paper cites.
Applications of the Theory of Meaningfulness to Psychology
F. S. Roberts · 1985
Earlier work this paper cites.
Measurement Scales and Statistics: A Clash of Paradigms
J. Michel · 1986
Cited alongside, same era.
Multiple Comparison Procedures
Y. Hochberg and A. C. Tamhane · 1987
Cited alongside, same era.
Meaningless Statistics
H. M. Marcus-Roberts and F. S. Roberts · 1987
Cited alongside, same era.
Foundations of Measurement. Geometrical, Threshold, and Probabilistic Representations , volume 2
P. Suppes, D. H. Krantz, R. D. Luce, and A. Tversky · 1989
Cited alongside, same era.
Foundations of Measurement. Representation, Axiomatization, and Invariance , volume 3
R. D. Luce, D. H. Krantz, P. Suppes, and A. Tversky · 1990
Cited alongside, same era.
An Introduction to the Logic of Psychological Measurement
J. Michel · 1990
Cited alongside, same era.
System Effectiveness, User Models, and User Utility: A Conceptual Framework for Investigation
B. A. Carterette · 2011
Later among the works it cites.
Nonparametric Statistical Inference
J. D. Gibbons and S. Chakraborti · 2011
Later among the works it cites.
Multiple Testing in Statistical Analysis of Systems-Based Information Retrieval Experiments
B. A. Carterette · 2012
Later among the works it cites.
A General Evaluation Measure for Document Organization Tasks
E. Amigó, J. Gonzalo, and M. F. Verdejo · 2013
Later among the works it cites.
Axiometrics: An Axiomatic Approach to Information Retrieval Effectiveness Metrics
L. Busin and S. Mizzaro · 2013
Later among the works it cites.
Seven Numeric Properties of Effectiveness Metrics
A. Moffat · 2013
Later among the works it cites.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Using Statistical Testing in the Evaluation of Retrieval Experiments
D. A. Hull · 1993
Cited alongside, same era.
Nominal, Ordinal, Interval, and Ratio Typologies Are Misleading
P. F. Velleman and L. Wilkinson · 1993
Cited alongside, same era.
A Statistical Analysis of the TREC-3 Data
J. M. Tague-Sutcliffe and J. Blustein · 1994
Cited alongside, same era.
Statistics and the Theory of Measurement
D. J. Hand · 1996
Cited alongside, same era.
Statistical Inference in Retrieval Effectiveness Evaluation
J. Savoy · 1997
Cited alongside, same era.
AT&T at TREC-6: SDR Track
A. Singhal, J. Choi, D. Hindle, and F. C. N. Pereira · 1997
Cited alongside, same era.
Users Versus Models: What Observation Tells Us About Effectiveness Metrics
A. Moffat, P. Thomas, and F. Scholer · 2013
Later among the works it cites.
Axiometrics: Axioms of Information Retrieval Effectiveness Metrics
E. Maddalena and S. Mizzaro · 2014
Later among the works it cites.
Statistical Reform in Information Retrieval?
T. Sakai · 2014
Later among the works it cites.
Towards a Formal Framework for Utility-oriented Measurements of Retrieval Effectiveness
M. Ferrante, N. Ferro, and M. Maistro · 2015
Later among the works it cites.
Correlation Between System and User Metrics in a Session
J. Jiang and J. Allan · 2016
Later among the works it cites.
Feeling Lucky? Multi-armed Bandits for Ordering Judgements in Pooling-based Evaluation
D. E. Losada, J. Parapar, and A. Barreiro · 2016
Later among the works it cites.
A Simple and Effective Approach to Score Standardisation
T. Sakai · 2016
Later among the works it cites.
Statistical Significance, Power, and Sample Sizes: A Systematic Review of SIGIR and TOIS, 2006-2015
T. Sakai · 2016
Later among the works it cites.
Quantifying the User Experience: Practical Statistics for User Research
J. Sauro and J. R. Lewis · 2016
Later among the works it cites.
Are IR Evaluation Measures on an Interval Scale?
M. Ferrante, N. Ferro, and S. Pontarollo · 2017
Later among the works it cites.
What Does Affect the Correlation Among Evaluation Measures?
N. Ferro · 2017
Later among the works it cites.
Some Common Mistakes In IR Evaluation, And How They Can Be Avoided
N. Fuhr · 2017
Later among the works it cites.
Evaluating Web Search with a Bejeweled Player Model
F. Zhang, Y. Liu, X. Li, M. Zhang, Y. Xu, and S. Ma · 2017
Later among the works it cites.
TREC 2017 Common Core Track Overview
J. Allan, D. K. Harman, E. Kanoulas, D. Li, C. Van Gysel, and E. M. Voorhees · 2018
Later among the works it cites.
An Axiomatic Analysis of Diversity Evaluation Metrics: Introducing the Rank-Biased Utility Metric
E. Amigó, D. Spina, and J. Carrillo-de Albornoz · 2018
Later among the works it cites.
Methods in Behavioral Research
P. C. Cozby and S. C. Bates · 2018
Later among the works it cites.
On Building Fair and Reusable Test Collections using Bandit Techniques
E. M. Voorhees · 2018
Later among the works it cites.
TREC 2018 Common Core Track Overview
J. Allan, D. K. Harman, E. Kanoulas, and E. M. Voorhees · 2019
Later among the works it cites.
A comparison of filtering evaluation metrics based on formal constraints
E. Amigó, J. Gonzalo, M. F. Verdejo, and D. Spina · 2019
Later among the works it cites.
Commission Delegated Regulation (EC) No 2019/33 of 17 October 2018 supplementing Regulation (EU) No 1308/2013 of the European Parliament and of the Council as regards applications for protection of designations of origin, geographical indications and traditional terms in the wine sector, the objection procedure, restrictions of use, amendments to product specifications, cancellation of protection, and labelling and presentation
European Commission · 2019
Later among the works it cites.
A General Theory of IR Evaluation Measures
M. Ferrante, N. Ferro, and S. Pontarollo · 2019
Later among the works it cites.
Information Retrieval Evaluation in a Changing World – Lessons Learned from 20 Years of CLEF , volume 41 of The Information Retrieval Series , 2019. Springer International Publishing, Germany
N. Ferro and C. Peters, editors · 2019
Later among the works it cites.
A New Perspective on Score Standardization
J. Urbano, H. Lima, and A. Hanjalic · 2019
Later among the works it cites.
On the nature of information access evaluation metrics: a unifying framework
E. Amigó and S. Mizzaro · 2020
Later among the works it cites.
How do interval scales help us with better understanding IR evaluation measures?
M. Ferrante, N. Ferro, and E. Losiouk · 2020
Later among the works it cites.
On Fuhr’s Guideline for IR Evaluation
T. Sakai · 2020
Later among the works it cites.
Evaluation measures for quantification: an axiomatic approach
F. Sebastiani · 2020
Later among the works it cites.
Models Versus Satisfaction: Towards a Better Understanding of Evaluation Metrics
F. Zhang, J. Mao, Y. Liu, X. Xie, W. Ma, M. Zhang, and S. Ma · 2020
Later among the works it cites.
Evaluating Information Retrieval and Access Tasks – NTCIR’s Legacy of Research Impact , volume 43 of The Information Retrieval Series , 2021. Springer International Publishing, Germany
T. Sakai, D. W. Oard, and N. Kando, editors · 2021
Closest in time.