2019

Multi-Graph Transformer for Free-Hand Sketch Recognition

Xu, Peng, Joshi, Chaitanya K., Bresson, Xavier

Understand

Learning meaningful representations of free-hand sketches remains a challenging task given the signal sparsity and the high-level abstraction of sketches.

Existing techniques have focused on exploiting either the static nature of sketches with Convolutional Neural Networks (CNNs) or the temporal sequential property with Recurrent Neural Networks (RNNs).
In this work, we propose a new representation of sketches as multiple sparsely connected graphs.
We design a novel Graph Neural Network (GNN), the Multi-Graph Transformer (MGT), for learning representations of sketches from multiple graphs which simultaneously capture global and local geometric stroke structures, as well as temporal information.

Built on

S. Hochreiter and J. Schmidhuber, “Long short-term memory,”
1997
Earlier work this paper cites.
X. Glorot, A. Bordes, and Y. Bengio, “Deep sparse rectifier neural networks,” in
2011
Earlier work this paper cites.
M. Eitz, J. Hays, and M. Alexa, “How do humans sketch objects?”
2012
Earlier work this paper cites.
R. Pascanu, T. Mikolov, and Y. Bengio, “On the difficulty of training recurrent neural networks,” in
2013
Earlier work this paper cites.
J. Bruna, W. Zaremba, A. Szlam, and Y. Lecun, “Spectral networks and locally connected networks on graphs,” in
2014
Earlier work this paper cites.
D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,”
Original
2014
Earlier work this paper cites.
N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: A simple way to prevent neural networks from overfitting,”
2014
Earlier work this paper cites.
D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,”
Original
2014
Earlier work this paper cites.
K. Cho, B. Van Merriënboer, D. Bahdanau, and Y. Bengio, “On the properties of neural machine translation: Encoder-decoder approaches,”
Original
2014
Earlier work this paper cites.
K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,”
Original
2014
Earlier work this paper cites.
Q. Yu, Y. Yang, Y.-Z. Song, T. Xiang, and T. Hospedales, “Sketch-a-net that beats humans,”
Original
2015
Earlier work this paper cites.
S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,”
Original
2015
Earlier work this paper cites.
R. K. Sarvadevabhatla, J. Kundu, and V. Babu R, “Enabling my robot to play pictionary: Recurrent neural networks for sketch recognition,” in
2016
Earlier work this paper cites.
Y. Ye, Y. Lu, and H. Jiang, “Human’s scene sketch understanding,” in
2016
Earlier work this paper cites.
P. Sangkloy, N. Burnell, C. Ham, and J. Hays, “The sketchy database: learning to retrieve badly drawn bunnies,”
2016
Earlier work this paper cites.
M. Defferrard, X. Bresson, and P. Vandergheynst, “Convolutional neural networks on graphs with fast localized spectral filtering,”
2016
Earlier work this paper cites.
S. Sukhbaatar, R. Fergus
2016
Earlier work this paper cites.
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in
2016
Earlier work this paper cites.
C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in
2016
Earlier work this paper cites.
D. Ha and D. Eck, “A neural representation of sketch drawings,”
Original
2017
Earlier work this paper cites.
L. Liu, F. Shen, Y. Shen, X. Liu, and L. Shao, “Deep sketch hashing: Fast free-hand sketch-based image retrieval,” in
2017
Earlier work this paper cites.

Similar

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,”
2017
Cited alongside, same era.
A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,”
2017
Cited alongside, same era.
J. Song, Q. Yu, Y.-Z. Song, T. Xiang, and T. M. Hospedales, “Deep spatial-semantic attention for fine-grained sketch-based image retrieval,” in
2017
Cited alongside, same era.
Q. Jia, M. Yu, X. Fan, and H. Li, “Sequential dual deep learning with shape and texture features for sketch recognition,”
Original
2017
Cited alongside, same era.
T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” in
2017
Cited alongside, same era.
W. Hamilton, Z. Ying, and J. Leskovec, “Inductive representation learning on large graphs,” in
2017
Cited alongside, same era.
F. Monti, D. Boscaini, J. Masci, E. Rodola, J. Svoboda, and M. M. Bronstein, “Geometric deep learning on graphs and manifolds using mixture model cnns,” in
2017
Cited alongside, same era.
G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in
2017
Cited alongside, same era.
P. Xu, Y. Huang, T. Yuan, K. Pang, Y.-Z. Song, T. Xiang, T. M. Hospedales, Z. Ma, and J. Guo, “Sketchmate: Deep hashing for million-scale human sketch retrieval,” in
2018
Cited alongside, same era.
Y. Shen, L. Liu, F. Shen, and L. Shao, “Zero-shot sketch-image hashing,” in
2018
Cited alongside, same era.
W. Chen and J. Hays, “Sketchygan: Towards diverse and realistic sketch to image synthesis,” in
2018
Cited alongside, same era.
Y. Lu, S. Wu, Y.-W. Tai, and C.-K. Tang, “Image generation from sketch constraint using contextual gan,” in
2018
Cited alongside, same era.

Then

J. Collomosse, T. Bui, and H. Jin, “Livesketch: Query perturbations for guided sketch-based visual search,” in
2019
Closest in time.
A. Dutta and Z. Akata, “Semantically tied paired cycle consistency for zero-shot sketch-based image retrieval,” in
2019
Closest in time.
S. Dey, P. Riba, A. Dutta, J. Llados, and Y.-Z. Song, “Doodle to search: Practical zero-shot sketch-based image retrieval,” in
2019
Closest in time.
K. Li, K. Pang, Y.-Z. Song, T. Xiang, T. M. Hospedales, and H. Zhang, “Toward deep universal sketch perceptual grouper,”
2019
Closest in time.
Q. Wang, B. Li, T. Xiao, J. Zhu, C. Li, D. F. Wong, and L. S. Chao, “Learning deep transformer models for machine translation,”
Original
2019
Closest in time.
Z. Dai, Z. Yang, Y. Yang, W. W. Cohen, J. Carbonell, Q. V. Le, and R. Salakhutdinov, “Transformer-xl: Attentive language models beyond a fixed-length context,”
Original
2019
Closest in time.
Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. Salakhutdinov, and Q. V. Le, “Xlnet: Generalized autoregressive pretraining for language understanding,”
Original
2019
Closest in time.
Z. Ye, Q. Guo, Q. Gan, X. Qiu, and Z. Zhang, “Bp-transformer: Modelling long-range context via binary partitioning,”
Original
2019
Closest in time.
A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga
2019
Closest in time.
M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro, “Megatron-lm: Training multi-billion parameter language models using gpu model parallelism,”
Original
2019
Closest in time.
X. Guo, H. Zhang, H. Yang, L. Xu, and Z. Ye, “A single attention-based combination of cnn and rnn for relation classification,”
2019
Closest in time.
H. Wang, M. Tan, M. Yu, S. Chang, D. Wang, K. Xu, X. Guo, and S. Potdar, “Extracting multiple-relations in one-pass with pre-trained transformers,”
Original
2019
Closest in time.
I. Hendrickx, S. N. Kim, Z. Kozareva, P. Nakov, D. O. Séaghdha, S. Padó, M. Pennacchiotti, L. Romano, and S. Szpakowicz, “Semeval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals,”
Original
2019
Closest in time.
P. Xu, Z. Song, Q. Yin, Y. Song, and L. Wang, “Deep self-supervised representation learning for free-hand sketch,”
2020
Closest in time.
H. Lin, Y. Fu, X. Xue, and Y.-G. Jiang, “Sketch-bert: Learning sketch bidirectional encoder representation from transformers by self-supervised learning of sketch gestalt,” in
2020
Closest in time.
P. Xu, “Deep learning for free-hand sketch: A survey,”
Original
2020
Closest in time.
L. Li, C. Zou, Y. Zheng, Q. Su, H. Fu, and C. L. Tai, “Sketch-r2cnn: An rnn-rasterization-cnn architecture for vector sketch recognition,”
2020
Closest in time.
Z. Wu, S. Pan, F. Chen, G. Long, C. Zhang, and S. Y. Philip, “A comprehensive survey on graph neural networks,”
2020
Closest in time.
U. S. Shanthamallu, J. J. Thiagarajan, H. Song, and A. Spanias, “Gramme: Semisupervised learning using multilayered graph attention models,”
2020
Closest in time.
J.-J. Liu, Q. Hou, M.-M. Cheng, C. Wang, and J. Feng, “Improving convolutional networks with self-calibrated convolutions,” in
2020
Closest in time.
D. Haase and M. Amthor, “Rethinking depthwise separable convolutions: How intra-kernel correlations lead to improved mobilenets,” in
2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…