2018

Bidirectional Learning for Robust Neural Networks

Pontes-Filho, Sidney, Liwicki, Marcus

Understand

A multilayer perceptron can behave as a generative classifier by applying bidirectional learning (BL).

It consists of training an undirected neural network to map input to output and vice-versa; therefore it can produce a classifier in one direction, and a generator in the opposite direction for the same data.
The learning process of BL tries to reproduce the neuroplasticity stated in Hebbian theory using only backward propagation of errors.
In this paper, two novel learning techniques are introduced which use BL for improving robustness to white noise static and adversarial examples.

Built on

D. O. Hebb, The organization of behavior: A neuropsychological theory . New York: Wiley, Jun. 1949
1949
Earlier work this paper cites.
P. Smolensky, “Parallel distributed processing: Explorations in the microstructure of cognition, vol. 1,” D. E. Rumelhart, J. L. McClelland, and C. PDP Research Group, Eds. Cambridge, MA, USA: MIT Press, 1986, ch. Information Processing in Dynamical Systems: Foundations of Harmony Theory, pp. 194–281. [Online]. Available: http://dl.acm.org/citation.cfm?id=104279.104290
1986
Earlier work this paper cites.
D. S. Broomhead and D. Lowe, “Radial basis functions, multi-variable functional interpolation and adaptive networks,” Royal Signals and Radar Establishment Malvern (United Kingdom), Tech. Rep., 1988
1988
Earlier work this paper cites.
K. Hornik, M. Stinchcombe, and H. White, “Multilayer feedforward networks are universal approximators,” Neural Networks , vol. 2, no. 5, pp. 359 – 366, 1989. [Online]. Available: http://www.sciencedirect.com/science/article/pii/0893608089900208
1989
Earlier work this paper cites.
J. Park and I. W. Sandberg, “Universal approximation using radial-basis-function networks,” Neural Computation , vol. 3, no. 2, pp. 246–257, 1991. [Online]. Available: https://doi.org/10.1162/neco.1991.3.2.246
1991
Earlier work this paper cites.
S. Lowel and W. Singer, “Selection of intrinsic horizontal connections in the visual cortex by correlated neuronal activity,” Science , vol. 255, no. 5041, pp. 209–212, 1992. [Online]. Available: http://science.sciencemag.org/content/255/5041/209
1992
Earlier work this paper cites.
L. Xu, “Least mean square error reconstruction principle for self-organizing neural-nets,” Neural networks , vol. 6, no. 5, pp. 627–648, 1993
1993
Earlier work this paper cites.
J. Han and C. Moraga, “The influence of the sigmoid function parameters on the speed of backpropagation learning,” in Proceedings of the International Workshop on Artificial Neural Networks: From Natural to Artificial Neural Computation , ser. IWANN ’96. London, UK, UK: Springer-Verlag, 1995, pp. 195–201. [Online]. Available: http://dl.acm.org/citation.cfm?id=646366.689307
1995
Earlier work this paper cites.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” in Proceedings of the IEEE , 1998, pp. 2278–2324
1998
Earlier work this paper cites.
R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction . MIT Press, 1998. [Online]. Available: http://www.cs.ualberta.ca/~sutton/book/the-book.html
1998
Earlier work this paper cites.
P. Viola and M. Jones, “Rapid object detection using a boosted cascade of simple features,” in Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001 , vol. 1, 2001, pp. I–511–I–518 vol.1
2001
Earlier work this paper cites.
G. E. Hinton, S. Osindero, and Y.-W. Teh, “A fast learning algorithm for deep belief nets,” Neural Comput. , vol. 18, no. 7, pp. 1527–1554, Jul. 2006. [Online]. Available: http://dx.doi.org/10.1162/neco.2006.18.7.1527
2006
Earlier work this paper cites.
G. E. Hinton and R. R. Salakhutdinov, “Reducing the dimensionality of data with neural networks,” science , vol. 313, no. 5786, pp. 504–507, 2006
2006
Earlier work this paper cites.
H. Grabner, P. M. Roth, and H. Bischof, “Eigenboosting: Combining discriminative and generative information,” in 2007 IEEE Conference on Computer Vision and Pattern Recognition , June 2007, pp. 1–8
2007
Earlier work this paper cites.
A. Krizhevsky and G. Hinton, “Learning multiple layers of features from tiny images,” Citeseer, Tech. Rep., 2009
2009
Earlier work this paper cites.

Similar

B. Grewe, A. Bonnan, and A. Frick, “Back-propagation of physiological action potential output in dendrites of slender-tufted l5a pyramidal neurons,” Frontiers in Cellular Neuroscience , vol. 4, p. 13, 2010. [Online]. Available: https://www.frontiersin.org/article/10.3389/fncel.2010.00013
2010
Cited alongside, same era.
A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Proceedings of the 25th International Conference on Neural Information Processing Systems - Volume 1 , ser. NIPS’12. USA: Curran Associates Inc., 2012, pp. 1097–1105. [Online]. Available: http://dl.acm.org/citation.cfm?id=2999134.2999257
2012
Cited alongside, same era.
R. C. O’Reilly, Y. Munakata, M. J. Frank, T. E. Hazy, and Contributors, Computational Cognitive Neuroscience . Wiki Book, 1st Edition, URL: http://ccnbook.colorado.edu , 2012. [Online]. Available: http://ccnbook.colorado.edu
2012
Cited alongside, same era.
C. Szegedy, A. Toshev, and D. Erhan, “Deep neural networks for object detection,” in Advances in Neural Information Processing Systems 26 , C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, Eds. Curran Associates, Inc., 2013, pp. 2553–2561. [Online]. Available: http://papers.nips.cc/paper/5207-deep-neural-networks-for-object-detection.pdf
2013
Cited alongside, same era.
C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. J. Goodfellow, and R. Fergus, “Intriguing properties of neural networks,” CoRR , vol. abs/1312.6199, 2013. [Online]. Available: http://arxiv.org/abs/1312.6199
Original
2013
Cited alongside, same era.
T. P. Vogels, R. C. Froemke, N. Doyon, M. Gilson, J. S. Haas, R. Liu, A. Maffei, P. Miller, C. J. Wierenga, M. A. Woodin, F. Zenke, and H. Sprekeler, “Inhibitory synaptic plasticity: spike timing-dependence and putative network function,” Front Neural Circuits , vol. 7, p. 119, Jul 2013, 23882186[pmid]. [Online]. Available: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3714539/
2013
Cited alongside, same era.
A. M. Nguyen, J. Yosinski, and J. Clune, “Deep neural networks are easily fooled: High confidence predictions for unrecognizable images,” CoRR , vol. abs/1412.1897, 2014. [Online]. Available: http://arxiv.org/abs/1412.1897
Original
2014
Cited alongside, same era.
I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in Advances in Neural Information Processing Systems 27 , Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, Eds. Curran Associates, Inc., 2014, pp. 2672–2680. [Online]. Available: http://papers.nips.cc/paper/5423-generative-adversarial-nets.pdf
2014
Cited alongside, same era.
S. Gu and L. Rigazio, “Towards deep neural network architectures robust to adversarial examples,” CoRR , vol. abs/1412.5068, 2014. [Online]. Available: http://arxiv.org/abs/1412.5068
Original
2014
Cited alongside, same era.
D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” CoRR , vol. abs/1412.6980, 2014. [Online]. Available: http://arxiv.org/abs/1412.6980
Original
2014
Cited alongside, same era.
I. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” in International Conference on Learning Representations , 2015. [Online]. Available: http://arxiv.org/abs/1412.6572
Original
2015
Cited alongside, same era.
R. Huang, B. Xu, D. Schuurmans, and C. Szepesvári, “Learning with a strong adversary,” CoRR , vol. abs/1511.03034, 2015. [Online]. Available: http://arxiv.org/abs/1511.03034
Original
2015
Cited alongside, same era.

Then

N. Papernot, N. Carlini, I. Goodfellow, R. Feinman, F. Faghri, A. Matyasko, K. Hambardzumyan, Y.-L. Juang, A. Kurakin, R. Sheatsley et al. , “cleverhans v2. 0.0: an adversarial machine learning library,” arXiv preprint arXiv:1610.00768 , 2016
Original
2016
Later among the works it cites.
N. Carlini and D. A. Wagner, “Defensive distillation is not robust to adversarial examples,” CoRR , vol. abs/1607.04311, 2016. [Online]. Available: http://arxiv.org/abs/1607.04311
Original
2016
Later among the works it cites.
X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever, and P. Abbeel, “Infogan: Interpretable representation learning by information maximizing generative adversarial nets,” CoRR , vol. abs/1606.03657, 2016. [Online]. Available: http://arxiv.org/abs/1606.03657
Original
2016
Later among the works it cites.
M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. J. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Józefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. G. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. A. Tucker, V. Vanhoucke, V. Vasudevan, F. B. Viégas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng, “Tensorflow: Large-scale machine learning on heterogeneous distributed systems,” CoRR , vol. abs/1603.04467, 2016. [Online]. Available: http://arxiv.org/abs/1603.04467
Original
2016
Later among the works it cites.
V. Kuleshov and S. Ermon, “Deep hybrid models: Bridging discriminative and generative approaches.” UAI, 2017
2017
Later among the works it cites.
X. Yuan, P. He, Q. Zhu, R. R. Bhat, and X. Li, “Adversarial examples: Attacks and defenses for deep learning,” CoRR , vol. abs/1712.07107, 2017. [Online]. Available: http://arxiv.org/abs/1712.07107
Original
2017
Later among the works it cites.
J. Lu, T. Issaranon, and D. A. Forsyth, “Safetynet: Detecting and rejecting adversarial examples robustly,” CoRR , vol. abs/1704.00103, 2017. [Online]. Available: http://arxiv.org/abs/1704.00103
Original
2017
Later among the works it cites.
J. H. Metzen, T. Genewein, V. Fischer, and B. Bischoff, “On detecting adversarial perturbations,” in Proceedings of 5th International Conference on Learning Representations (ICLR) , 2017. [Online]. Available: https://arxiv.org/abs/1702.04267
Original
2017
Later among the works it cites.
S. Shen, G. Jin, K. Gao, and Y. Zhang, “AE-GAN: adversarial eliminating with GAN,” CoRR , vol. abs/1707.05474, 2017. [Online]. Available: http://arxiv.org/abs/1707.05474
Original
2017
Later among the works it cites.
H. Lee, S. Han, and J. Lee, “Generative adversarial trainer: Defense to adversarial perturbations with GAN,” CoRR , vol. abs/1705.03387, 2017. [Online]. Available: http://arxiv.org/abs/1705.03387
Original
2017
Later among the works it cites.
J. Bradshaw, A. G. d. G. Matthews, and Z. Ghahramani, “Adversarial Examples, Uncertainty, and Transfer Testing Robustness in Gaussian Process Hybrid Deep Networks,” ArXiv e-prints , Jul. 2017
2017
Later among the works it cites.
A. Nayebi and S. Ganguli, “Biologically inspired protection of deep networks from adversarial attacks,” ArXiv e-prints , Mar. 2017
2017
Later among the works it cites.
Y. Li, “Are generative classifiers more robust to adversarial attacks?” CoRR , vol. abs/1802.06552, 2018. [Online]. Available: http://arxiv.org/abs/1802.06552
Original
2018
Closest in time.
N. Akhtar and A. Mian, “Threat of adversarial attacks on deep learning in computer vision: A survey,” CoRR , vol. abs/1801.00553, 2018. [Online]. Available: http://arxiv.org/abs/1801.00553
Original
2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…