Fetching the paper…

Reading the bibliography…

K-TanH: Efficient TanH For Deep Learning · Around

2019

K-TanH: Efficient TanH For Deep Learning

Kundu, Abhisek, Heinecke, Alex, Kalamkar, Dhiraj et al.

Original
PDF
Post on X

Understand

We propose K-TanH, a novel, highly accurate, hardware efficient approximation of popular activation function TanH for Deep Learning.

K-TanH consists of parameterized low-precision integer operations, such as, shift and add/subtract (no floating point operation needed) where parameters are stored in very small look-up tables that can fit in CPU registers.
K-TanH can work on various numerical formats, such as, Float32 and BFloat16.
High quality approximations to other activation functions, e.g., Sigmoid, Swish and GELU, can be derived from K-TanH.

Built on

Google’s neural machine translation system: Bridging the gap between human and machine translation
Original
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, and et al · 1909
Earlier work this paper cites.
Rectified linear units improve restricted boltzmann machines
Vinod Nair and Geoffrey E. Hinton · 2010
Earlier work this paper cites.
Tensorflow distributions
Original
Joshua V. Dillon, Ian Langmore, Dustin Tran, Eugene Brevdo, Srinivas Vasudevan, Dave Moore, Brian Patton, Alex Alemi, Matt Hoffman, and Rif A. Saurous · 2017
Earlier work this paper cites.

Similar

Searching for activation functions
Original
Prajit Ramachandran, Barret Zoph, and Quoc V. Le · 2017
Cited alongside, same era.
Efficient processing of deep neural networks: A tutorial and survey
Original
Vivienne Sze, Yu-Hsin CHen, Tien-Ju Yang, and Joel Emer · 2017
Cited alongside, same era.
Gaussian error linear units (gelus)
Original
Dan Hendrycks and Kevin Gimpel · 2018
Cited alongside, same era.

Then

Optimizing deep learning rnn topologies on intel architecture
Kunal Banerjee, Evangelos Georganas, Dhiraj D. Kalamkar, Barukh Ziv, Eden Segal, Cristina Anderson, and Alexander Heinecke · 2019
Closest in time.
Accelarating ai
Bill Dally · 2019
Closest in time.
A study of bfloat16 for deep learning training
Original
Dhiraj Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Natraj Jammalamadaka, Jianyu Huang, Hector Yuen, Jiyan Yang, Jongshoo Park, Alexander Heinecke, Evangelos Georganas, Sudarshan Srinivasan, Abhisek Kundu, Misha Smelyanskiy, Bharat Kaul, and Pradeep Dubey · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…