Fetching the paper…

Reading the bibliography…

2017

Climbing the Kaggle Leaderboard by Exploiting the Log-Loss Oracle

Whitehill, Jacob

Original
PDF
Post on X

Understand

In the context of data-mining competitions (e.g., Kaggle, KDDCup, ILSVRC Challenge), we show how access to an oracle that reports a contestant's log-loss score on the test set can be exploited to deduce the ground-truth of some of the test examples.

By applying this technique iteratively to batches of $m$ examples (for small $m$), all of the test labels can eventually be inferred.
In this paper, (1) We demonstrate this attack on the first stage of a recent Kaggle competition (Intel & MobileODT Cancer Screening) and use it to achieve a log-loss of $0.00000$ (and thus attain a rank of #4 out of 848 contestants), without ever training a classifier to solve the actual task.
(2) We prove an upper bound on the batch size $m$ as a function of the floating-point resolution of the probability estimates that the contestant submits for the labels.

Climbing the Kaggle Leaderboard by Exploiting the Log-Loss Oracle · Around

Built on

Linearly constrained minimax optimization
K. Madsen and H. Schjær-Jacobsen · 1978
Earlier work this paper cites.
A new algorithm for statistical circuit design based on quasi-newton methods and function splitting
R. Brayton, S. Director, G. Hachtel, and L. Vidigal · 1979
Earlier work this paper cites.
Signal detection theory in the 2AFC paradigm: attention, channel uncertainty and probability summation
C. Tyler and C.-C. Chen · 2000
Earlier work this paper cites.
Generalization bounds for the area under the ROC curve
S. Agarwal, T. Graepel, R. Herbrich, S. Har-Peled, and D. Roth · 2005
Earlier work this paper cites.

Similar

Preventing false discovery in interactive data analysis is hard
M. Hardt and J. Ullman · 2014
Cited alongside, same era.
The ladder: A reliable leaderboard for machine learning competitions
Original
A. Blum and M. Hardt · 2015
Cited alongside, same era.
Preserving statistical validity in adaptive data analysis
C. Dwork, V. Feldman, M. Hardt, T. Pitassi, O. Reingold, and A. L. Roth · 2015
Cited alongside, same era.

Then

ImageNet Large Scale Visual Recognition Challenge
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei · 2015
Later among the works it cites.
Toward a better understanding of leaderboard
Original
W. Zheng · 2015
Later among the works it cites.
Exploiting an oracle that reports AUC scores in machine learning contests
J. Whitehill · 2016
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…