2018

Engaging Image Captioning Via Personality

Shuster, Kurt, Humeau, Samuel, Hu, Hexiang et al.

Understand

Standard image captioning tasks such as COCO and Flickr30k are factual, neutral in tone and (to a human) state the obvious (e.g., "a man playing a guitar").

While such tasks are useful to verify that a machine understands the content of an image, they are not engaging to humans as captions.
With this in mind we define a new task, Personality-Captions, where the goal is to be as engaging to humans as possible by incorporating controllable style and personality traits.
We collect and release a large dataset of 201,858 of such captions conditioned over 215 possible traits.

Built on

Bleu: a method for automatic evaluation of machine translation
K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu · 2002
Earlier work this paper cites.
Rouge: A package for automatic evaluation of summaries
C.-Y. Lin · 2004
Earlier work this paper cites.
Automatic image captioning
J.-Y. Pan, H.-J. Yang, P. Duygulu, and C. Faloutsos · 2004
Earlier work this paper cites.
The utility of affect expression in natural language interactions in joint human-robot tasks
M. Scheutz, P. Schermerhorn, and J. Kramer · 2006
Earlier work this paper cites.
Agency and communion from the perspective of self versus others
A. E. Abele and B. Wojciszke · 2007
Earlier work this paper cites.
Filling the emotion gap in linguistic theory: Commentary on potts’ expressive dimension
T. Jay and K. Janschewitz · 2007
Earlier work this paper cites.
The sixteen personality factor questionnaire (16pf)
H. E. Cattell and A. D. Mead · 2008
Earlier work this paper cites.
Torchvision the machine-vision package of torch
S. Marcel and Y. Rodriguez · 2010
Earlier work this paper cites.
What we instagram: A first analysis of instagram photo content and user types
Y. Hu, L. Manikonda, and S. Kambhampati · 2014
Earlier work this paper cites.
Unifying visual-semantic embeddings with multimodal neural language models
Original
R. Kiros, R. Salakhutdinov, and R. S. Zemel · 2014
Earlier work this paper cites.
Microsoft coco: Common objects in context
T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick · 2014
Earlier work this paper cites.
Imagenet large scale visual recognition challenge
Original
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. S. Bernstein, A. C. Berg, and F. Li · 2014
Earlier work this paper cites.
From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions
P. Young, A. Lai, M. Hodosh, and J. Hockenmaier · 2014
Earlier work this paper cites.
Vqa: Visual question answering
S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. Lawrence Zitnick, and D. Parikh · 2015
Earlier work this paper cites.
Microsoft coco captions: Data collection and evaluation server
Original
X. Chen, H. Fang, T.-Y. Lin, R. Vedantam, S. Gupta, P. Dollár, and C. L. Zitnick · 2015
Earlier work this paper cites.
User conditional hashtag prediction for images
E. Denton, J. Weston, M. Paluri, L. Bourdev, and R. Fergus · 2015
Earlier work this paper cites.
Deep residual learning for image recognition
Original
K. He, X. Zhang, S. Ren, and J. Sun · 2015
Earlier work this paper cites.
Deep visual-semantic alignments for generating image descriptions
A. Karpathy and L. Fei-Fei · 2015
Earlier work this paper cites.
Multimodal convolutional neural networks for matching image and sentence
L. Ma, Z. Lu, L. Shang, and H. Li · 2015
Earlier work this paper cites.
Cider: Consensus-based image description evaluation
R. Vedantam, C. Lawrence Zitnick, and D. Parikh · 2015
Earlier work this paper cites.

Similar

Order-embeddings of images and language
Original
I. Vendrov, R. Kiros, S. Fidler, and R. Urtasun · 2015
Cited alongside, same era.
Show and tell: A neural image caption generator
O. Vinyals, A. Toshev, S. Bengio, and D. Erhan · 2015
Cited alongside, same era.
Show, attend and tell: Neural image caption generation with visual attention
K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhudinov, R. Zemel, and Y. Bengio · 2015
Cited alongside, same era.
Spice: Semantic propositional image caption evaluation
P. Anderson, B. Fernando, M. Johnson, and S. Gould · 2016
Cited alongside, same era.
Enriching word vectors with subword information
Original
P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov · 2016
Cited alongside, same era.
Capturing deep correlations with 2-way nets
Original
A. Eisenschtat and L. Wolf · 2016
Cited alongside, same era.
Instance-aware image and sentence matching with selective multimodal LSTM
Original
Y. Huang, W. Wang, and L. Wang · 2016
Cited alongside, same era.
Affect-language interactions in native and non-native English speakers
Jonczyk and R. Jończyk · 2016
Cited alongside, same era.
How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation
Original
C.-W. Liu, R. Lowe, I. V. Serban, M. Noseworthy, L. Charlin, and J. Pineau · 2016
Cited alongside, same era.
Senticap: Generating image descriptions with sentiments
A. P. Mathews, L. Xie, and X. He · 2016
Cited alongside, same era.
Dual attention networks for multimodal reasoning and matching
Original
H. Nam, J. Ha, and J. Kim · 2016
Cited alongside, same era.
Yfcc100m: The new data in multimedia research
B. Thomee, D. A. Shamma, G. Friedland, B. Elizalde, K. Ni, D. Poland, D. Borth, and L.-J. Li · 2016
Cited alongside, same era.

Then

Knowing when to look: Adaptive attention via a visual sentinel for image captioning
J. Lu, C. Xiong, D. Parikh, and R. Socher · 2017
Later among the works it cites.
Image-grounded conversations: Multimodal context for natural question and response generation
Original
N. Mostafazadeh, C. Brockett, B. Dolan, M. Galley, J. Gao, G. P. Spithourakis, and L. Vanderwende · 2017
Later among the works it cites.
Hierarchical multimodal lstm for dense visual-semantic embedding
Z. Niu, M. Zhou, L. Wang, X. Gao, and G. Hua · 2017
Later among the works it cites.
Attend to you: Personalized image captioning with context sequence memory networks
C. C. Park, B. Kim, and G. Kim · 2017
Later among the works it cites.
Self-critical sequence training for image captioning
S. J. Rennie, E. Marcheret, Y. Mroueh, J. Ross, and V. Goel · 2017
Later among the works it cites.
Attention is all you need
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin · 2017
Later among the works it cites.
Bottom-up and top-down attention for image captioning and vqa
P. Anderson, X. He, C. Buehler, D. Teney, M. Johnson, S. Gould, and L. Zhang · 2018
Closest in time.
Finding beans in burgers: Deep semantic-visual embedding with localization
M. Engilberge, L. Chevallier, P. Pérez, and M. Cord · 2018
Closest in time.
Vizwiz grand challenge: Answering visual questions from blind people
Original
D. Gurari, Q. Li, A. J. Stangl, A. Guo, C. Lin, K. Grauman, J. Luo, and J. P. Bigham · 2018
Closest in time.
First impressions: A survey on computer vision-based apparent personality trait analysis
Original
J. C. S. Jacques, Y. Güçlütürk, M. Pérez, U. Güçlü, C. Andújar, X. Baró, H. J. Escalante, I. Guyon, M. van Gerven, R. van Lier, and S. Escalera · 2018
Closest in time.
Neural baby talk
J. Lu, J. Yang, D. Batra, and D. Parikh · 2018
Closest in time.
Exploring the limits of weakly supervised pretraining
Original
D. Mahajan, R. B. Girshick, V. Ramanathan, K. He, M. Paluri, Y. Li, A. Bharambe, and L. van der Maaten · 2018
Closest in time.
Semstyle: Learning to generate stylised image captions using unaligned text
A. Mathews, L. Xie, and X. He · 2018
Closest in time.
Training Millions of Personalized Dialogue Agents
P.-E. Mazaré, S. Humeau, M. Raison, and A. Bordes · 2018
Closest in time.
Learning two-branch neural networks for image-text matching tasks
L. Wang, Y. Li, J. Huang, and S. Lazebnik · 2018
Closest in time.
Neural joking machine: Humorous image captioning
Original
K. Yoshida, M. Minoguchi, K. Wani, A. Nakamura, and H. Kataoka · 2018
Closest in time.
Image captioning at will: A versatile scheme for effectively injecting sentiments into image descriptions
Original
Q. You, H. Jin, and J. Luo · 2018
Closest in time.
Personalizing dialogue agents: I have a dog, do you have pets too?
Original
S. Zhang, E. Dinan, J. Urbanek, A. Szlam, D. Kiela, and J. Weston · 2018
Closest in time.
Adapting a virtual agent to user personality
O. Kampman, F. B. Siddique, Y. Yang, and P. Fung · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…