Faster Neural Network Training with Data Echoing

Some methods of speeding up the convergence of iteration methods

Polyak, B. T · 1964

Earlier work this paper cites.

A method for unconstrained convex minimization problem with the rate of convergence O ⁡ ( 1 / k 2 ) {O}(1/k^{2})

Nesterov, Y · 1983

Earlier work this paper cites.

Learning representations by back-propagating errors

Rumelhart, D. E., Hinton, G. E., and Williams, R. J · 1986

Earlier work this paper cites.

Learning multiple layers of features from tiny images

Krizhevsky, A. and Hinton, G · 2009

Earlier work this paper cites.

Parallelized stochastic gradient descent

Zinkevich, M., Weimer, M., Li, L., and Smola, A. J · 2010

Earlier work this paper cites.

On the importance of initialization and momentum in deep learning

Sutskever, I., Martens, J., Dahl, G. E., and Hinton, G. E · 2013

Earlier work this paper cites.

One billion word benchmark for measuring progress in statistical language modeling

Chelba, C., Mikolov, T., Schuster, M., Ge, Q., Brants, T., Koehn, P., and Robinson, T · 2014

Earlier work this paper cites.

Microsoft COCO: Common objects in context

Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L · 2014

Earlier work this paper cites.

Batch normalization: Accelerating deep network training by reducing internal covariate shift

Ioffe, S. and Szegedy, C · 2015

Earlier work this paper cites.

Human-level control through deep reinforcement learning

Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al · 2015

Cited alongside, same era.

ImageNet large scale visual recognition challenge

Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al · 2015

Cited alongside, same era.

Tensorflow: A system for large-scale machine learning

Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al · 2016

Cited alongside, same era.

Deep residual learning for image recognition

He, K., Zhang, X., Ren, S., and Sun, J · 2016

Cited alongside, same era.

SSD: Single shot multibox detector

Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.-Y., and Berg, A. C · 2016

Cited alongside, same era.

Parallel SGD: When does averaging help?

Zhang, J., De Sa, C., Mitliagkas, I., and Ré, C · 2016

Characterizing deep-learning I/O workloads in TensorFlow

Original

Chien, S. W., Markidis, S., Sishtla, C. P., Santos, L., Herman, P., Narasimhamurthy, S., and Laure, E · 2018

Later among the works it cites.

Faster SGD training by minibatch persistency

Original

Fischetti, M., Mandatelli, I., and Salvagnin, D · 2018

Later among the works it cites.

Not all samples are created equal: Deep learning with importance sampling

Original

Katharopoulos, A. and Fleuret, F · 2018

Later among the works it cites.

Efficient training of convolutional neural nets on large distributed systems

Kumar, S., Sreedhar, D., Saxena, V., Sabharwal, Y., and Verma, A · 2018

Later among the works it cites.

Measuring the effects of data parallelism on neural network training

Original