Fetching the paper…

Reading the bibliography…

2021

Multi-Modal Semantic Inconsistency Detection in Social Media News Posts

McCrae, Scott, Wang, Kehan, Zakhor, Avideh

Original
PDF
Post on X

Understand

As computer-generated content and deepfakes make steady improvements, semantic approaches to multimedia forensics will become more important.

In this paper, we introduce a novel classification architecture for identifying semantic inconsistencies between video appearance and text caption in social media news posts.
We develop a multi-modal fusion framework to identify mismatches between videos and captions in social media posts by leveraging an ensemble method based on textual analysis of the caption, automatic audio transcription, semantic video analysis, object detection, named entity consistency, and facial verification.
To train and test our approach, we curate a new video-based dataset of 4,000 real-world Facebook news posts for analysis.

Multi-Modal Semantic Inconsistency Detection in Social Media News Posts · Around

Built on

“RoBERTa: A Robustly Optimized BERT Pretraining Approach”, 2019
Original
Yinhan Liu et al · 1907
Earlier work this paper cites.
“Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training”, 2019
Original
Gen Li et al · 1908
Earlier work this paper cites.
“ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks”, 2019
Original
Jiasen Lu, Dhruv Batra, Devi Parikh and Stefan Lee · 1908
Earlier work this paper cites.
“VL-BERT: Pre-training of Generic Visual-Linguistic Representations”, 2020
Original
Weijie Su et al · 1908
Earlier work this paper cites.
“Long Short-Term Memory”
Sepp Hochreiter and Jürgen Schmidhuber · 1997
Earlier work this paper cites.
“Language Models are Few-Shot Learners”, 2020
Original
Tom. Brown et al · 2005
Earlier work this paper cites.
“Deep Speech: Scaling up end-to-end speech recognition”, 2014
Original
Awni Hannun et al · 2014
Earlier work this paper cites.
“Deep Residual Learning for Image Recognition”, 2015
Original
Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun · 2015
Earlier work this paper cites.
“FaceNet: A unified embedding for face recognition and clustering”
Florian Schroff, Dmitry Kalenichenko and James Philbin · 2015
Earlier work this paper cites.
“Video2vec Embeddings Recognize Events When Examples Are Scarce”
Amirhossein Habibian, Thomas Mensink and Cees.. Snoek · 2016
Earlier work this paper cites.
“FOIL it! Find One mismatch between Image and Language caption”
Ravi Shekhar et al · 2017
Earlier work this paper cites.
“Attention Is All You Need”, 2017
Original
Ashish Vaswani et al · 2017
Earlier work this paper cites.

Similar

“3D Inception Convolutional Neural Networks for Automatic Lung Nodule Detection”
Chen Zhao, Jungang Han and Yang Jia · 2017
Cited alongside, same era.
“Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?”
Kensho Hara, Hirokatsu Kataoka and Yutaka Satoh · 2018
Cited alongside, same era.
“Phoebe Bridgers (41599189180) (cropped).jpg” [Online; accessed May 7, 2021], 2018
David Lee · 2018
Cited alongside, same era.
“Paul McCartney in October 2018.jpg” [Online; accessed May 7, 2021], 2018
Raph · 2018
Cited alongside, same era.
“Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification”, 2018
Original
Saining Xie et al · 2018
Cited alongside, same era.
“BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”, 2019

Then

“spaCy: Industrial-strength Natural Language Processing in Python”
Matthew Honnibal, Ines Montani, Sofie Van and Adriane Boyd · 2020
Later among the works it cites.
“HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training”
Linjie Li et al · 2020
Later among the works it cites.
“UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation”
Original
Huaishao Luo et al · 2020
Later among the works it cites.
“End-to-End Learning of Visual Representations from Uncurated Instructional Videos”
Antoine Miech et al · 2020
Later among the works it cites.
“Detecting Cross-Modal Inconsistency to Defend Against Neural Fake News”
Reuben Tan, Bryan. Plummer and Kate Saenko · 2020
Later among the works it cites.
“CNN-generated images are surprisingly easy to spot…for now”

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Original

Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova · 2019

Cited alongside, same era.

“HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips”

Antoine Miech et al · 2019

Cited alongside, same era.

“The Rise of the Robot Reporter”

Jaclyn Peiser · 2019

Cited alongside, same era.

“Language Models are Unsupervised Multitask Learners”, 2019

Alec Radford et al · 2019

Cited alongside, same era.

“LXMERT: Learning Cross-Modality Encoder Representations from Transformers”

Hao Tan and Mohit Bansal · 2019

Cited alongside, same era.

“Defending Against Neural Fake News”

Rowan Zellers et al · 2019

Cited alongside, same era.

URL: http://ffmpeg.org/

FFmpeg Developers, 2020 · 2020

Cited alongside, same era.

Sheng-Yu Wang et al · 2020

Later among the works it cites.

“Transformers: State-of-the-Art Natural Language Processing”

Thomas Wolf et al · 2020

Later among the works it cites.

“Is Space-Time Attention All You Need for Video Understanding?”, 2021

Gedas Bertasius, Heng Wang and Lorenzo Torresani · 2021

Closest in time.

“CrowdTangle”, 2021

CrowdTangle Team · 2021

Closest in time.

“youtube-dl”, 2021

youtube-dl Developers · 2021

Closest in time.

“NewsCLIPpings: Automatic Generation of Out-of-Context Multimodal Media”, 2021

Grace Luo, Trevor Darrell and Anna Rohrbach · 2021

Closest in time.