2018

COCO-CN for Cross-Lingual Image Tagging, Captioning and Retrieval

Li, Xirong, Xu, Chaoxi, Wang, Xiaoxu et al.

Understand

This paper contributes to cross-lingual image annotation and retrieval in terms of data and baseline methods.

We propose COCO-CN, a novel dataset enriching MS-COCO with manually written Chinese sentences and tags.
For more effective annotation acquisition, we develop a recommendation-assisted collective annotation system, automatically providing an annotator with several tags and sentences deemed to be relevant with respect to the pictorial content.
Having 20,342 images annotated with 27,218 Chinese sentences and 70,993 tags, COCO-CN is currently the largest Chinese-English dataset that provides a unified and challenging platform for cross-lingual image tagging, captioning and retrieval.

Built on

V. I. Levenshtein, “Binary Codes Capable of Correcting Deletions, Insertions and Reversals,” Soviet Physics Doklady , vol. 10, p. 707, Feb. 1966
1966
Earlier work this paper cites.
M. Grubinger, P. Clough, H. Müller, and T. Deselaers, “The IAPR benchmark: A new evaluation resource for visual information systems,” in ICLRE , 2006
2006
Earlier work this paper cites.
R. Hadsell, S. Chopra, and Y. LeCun, “Dimensionality reduction by learning an invariant mapping,” in CVPR , 2006
2006
Earlier work this paper cites.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “ImageNet: a large-scale hierarchical image database,” in CVPR , 2009
2009
Earlier work this paper cites.
T.-S. Chua, J. Tang, R. Hong, H. Li, Z. Luo, and Y. Zheng, “NUS-WIDE: A real-world web image database from National University of Singapore,” in CIVR , 2009
2009
Earlier work this paper cites.
C. Rashtchian, P. Young, M. Hodosh, and J. Hockenmaier, “Collecting image annotations using Amazon’s Mechanical Turk,” in NAACL-HLT Workshop , 2010
2010
Earlier work this paper cites.
Y. Peng, J. Qi, and Y. Yuan, “Modality-specific cross-modal similarity measurement with recurrent attention network,” IEEE Trans. Image Processing , vol. 27, no. 11, pp. 5585–5599, 2018
2012
Earlier work this paper cites.
B. Caputo, H. Müller, B. Thomee, M. Villegas, R. Paredes, D. Zellhofer, H. Goeau, A. Joly, P. Bonnet, J. Gomez, I. Varea, and M. Cazorla, “ImageCLEF 2013: The vision, the data and the open challenges,” in CLEF , 2013
2013
Earlier work this paper cites.
M. Hodosh, P. Young, and J. Hockenmaier, “Framing image description as a ranking task: Data, models and evaluation metrics,” JAIR , vol. 47, pp. 853–899, 2013
2013
Earlier work this paper cites.
P. Young, A. Lai, M. Hodosh, and J. Hockenmaier, “From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions,” TACL , vol. 2, pp. 67–78, 2014
2014
Earlier work this paper cites.
Z. Wang, P. Cui, L. Xie, W. Zhu, Y. Rui, and S. Yang, “Bilateral correspondence model for words-and-pictures association in multimedia-rich microblogs,” ACM Trans. Multimedia Comput. Commun. Appl. , vol. 10, no. 4, pp. 34:1–34:21, 2014
2014
Earlier work this paper cites.
P. Cui, S.-W. Liu, W.-W. Zhu, H.-B. Luan, T.-S. Chua, and S.-Q. Yang, “Social-sensed image search,” ACM Trans. Inf. Syst. , vol. 32, no. 2, pp. 8:1–8:23, 2014
2014
Earlier work this paper cites.
X. Chen, H. Fang, T.-Y. Lin, R. Vedantam, S. Gupta, P. Dollár, and L. Zitnick, “Microsoft COCO captions: Data collection and evaluation server,” CoRR , vol. abs/1504.00325, 2015
Original
2015
Earlier work this paper cites.
R. Funaki and H. Nakayama, “Image-mediated learning for zero-shot cross-lingual document retrieval.” in EMNLP , 2015
2015
Earlier work this paper cites.
K. Min, C. Ma, T. Zhao, and H. Li, “BosonNLP: An ensemble approach for word segmentation and POS tagging,” in NLPCC , 2015
2015
Earlier work this paper cites.
O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, “Show and tell: A neural image caption generator,” in CVPR , 2015
2015
Earlier work this paper cites.
D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” in ICLR , 2015
2015
Earlier work this paper cites.

Similar

D. Elliott, S. Frank, K. Sima’an, and L. Specia, “Multi30k: Multilingual English-German image descriptions,” in ACL Workshop on Vision and Language , 2016
2016
Cited alongside, same era.
T. Miyazaki and N. Shimizu, “Cross-lingual image caption generation,” in ACL , 2016
2016
Cited alongside, same era.
J. Hitschler, S. Schamoni, and S. Riezler, “Multimodal pivots for image caption translation,” in ACL , 2016
2016
Cited alongside, same era.
J. Rajendran, M. Khapra, S. Chandar, and B. Ravindran, “Bridge correlational neural networks for multilingual multimodal representation learning,” in NAACL-HLT , 2016
2016
Cited alongside, same era.
X. Li, W. Lan, J. Dong, and H. Liu, “Adding Chinese captions to images,” in ICMR , 2016
2016
Cited alongside, same era.
X. Li, T. Uricchio, L. Ballan, M. Bertini, C. Snoek, and A. Del Bimbo, “Socializing the semantic gap: A comparative survey on image tag assignment, refinement and retrieval,” CSUR , vol. 49, no. 1, pp. 14:1–14:39, 2016
2016
Cited alongside, same era.
A. Koochali, S. Kalkowski, A. Dengel, D. Borth, and C. Schulze, “Which languages do people speak on Flickr? a language and geo-location study of the YFCC100m dataset,” in MMCommons , 2016
2016
Cited alongside, same era.
P. Mettes, D. Koelma, and C. Snoek, “The ImageNet shuffle: Reorganized pre-training for video event detection,” in ICMR , 2016
2016
Cited alongside, same era.
J. Song, L. Gao, F. Nie, H. T. Shen, Y. Yan, and N. Sebe, “Optimized graph learning using partial tags and multiple features for image and video annotation,” IEEE Trans. Image Processing , vol. 25, no. 11, pp. 4999–5011, 2016
2016
Cited alongside, same era.
L. Gao, Z. Guo, H. Zhang, X. Xu, and H. T. Shen, “Video captioning with attention-based lstm and semantic consistency,” IEEE Trans. Multimedia , vol. 19, no. 9, pp. 2045–2055, 2017
2017
Cited alongside, same era.
S. Tsutsui and D. Crandall, “Using artificial tokens to control languages for multilingual image caption generation,” in CVPR Language and Vision Workshop , 2017
2017
Cited alongside, same era.
Q. Wei, X. Wang, and X. Li, “Harvesting deep models for cross-lingual image annotation,” in CBMI , 2017, pp. 36:1–36:5
2017
Cited alongside, same era.

Then

S. Gella, R. Sennrich, F. Keller, and M. Lapata, “Image pivoting for learning multilingual multimodal representations,” in EMNLP , 2017
2017
Later among the works it cites.
J. Fu and Y. Rui, “Advances in deep learning approaches for image tagging,” APSIPA Transactions on Signal and Information Processing , vol. 6, 2017
2017
Later among the works it cites.
Y. Peng, X. Huang, and Y. Zhao, “An overview of cross-media retrieval: Concepts, methodologies, benchmarks and challenges,” TCSVT , 2017, in press
2017
Later among the works it cites.
S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He, “Aggregated residual transformations for deep neural networks,” in CVPR , 2017
2017
Later among the works it cites.
P. Cui, S. Liu, and W. Zhu, “General knowledge embedded image representation learning,” IEEE Trans. Multimedia , vol. 20, no. 1, pp. 198–207, 2018
2018
Closest in time.
Y. Hu, L. Zheng, Y. Yang, and Y. Huang, “Twitter100k: A real-world dataset for weakly supervised cross-media retrieval,” IEEE Trans. Multimedia , vol. 20, no. 4, pp. 927–938, 2018
2018
Closest in time.
Clarifai, “Clarifai image and video recognition API,” https://clarifai.com/demo
2018
Closest in time.
X. Li, B. Shen, B.-D. Liu, and Y.-J. Zhang, “Ranking-preserving low-rank factorization for image annotation with missing labels,” IEEE Trans. Multimedia , vol. 20, no. 5, pp. 1169–1178, 2018
2018
Closest in time.
L. Zhang, B. Ma, G. Li, Q. Huang, and Q. Tian, “Generalized semi-supervised and structured subspace learning for cross-modal retrieval,” IEEE Trans. Multimedia , vol. 20, no. 1, pp. 128–141, 2018
2018
Closest in time.
J. Dong, X. Li, and C. Snoek, “Predicting visual features from text for image and video caption retrieval,” IEEE Trans. Multimedia , vol. 20, no. 12, pp. 3377–3388, 2018
2018
Closest in time.
J. Yao, Y. Wang, Y. Zhang, J. Sun, and J. Zhou, “Joint latent dirichlet allocation for social tags,” IEEE Trans. Multimedia , vol. 20, no. 1, pp. 224–237, 2018
2018
Closest in time.
X. Wang, L. Gao, P. Wang, X. Sun, and X. Liu, “Two-stream 3-d convnet fusion for action recognition in videos with arbitrary size and length,” IEEE Trans. Multimedia , vol. 20, no. 3, pp. 634–644, 2018
2018
Closest in time.
Z. Shen, P. Cui, K. Kuang, B. Li, and P. Chen, “Causally regularized learning with agnostic data selection bias,” in ACMMM , 2018, pp. 441–419
2018
Closest in time.
J. Song, H. Zhang, X. Li, L. Gao, M. Wang, and R. Hong, “From deterministic to generative: Multi-modal stochastic rnns for video captioning,” IEEE Trans. Neural Networks and Learning Systems , 2018
2018
Closest in time.
J. Song, H. Zhang, X. Li, L. Gao, M. Wang, and R. Hong, “Self-supervised video hashing with hierarchical binary auto-encoder,” IEEE Trans. Image Processing , vol. 27, no. 7, pp. 3210–3221, 2018
2018
Closest in time.
Y. Peng, J. Qi, X. Huang, and Y. Yuan, “CCL: Cross-modal correlation learning with multigrained fusion by hierarchical network,” IEEE Trans. Multimedia , vol. 20, no. 2, pp. 405–420, 2018
2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…