2023

GPT-4V-AD: Exploring Grounding Potential of VQA-oriented GPT-4V for Zero-shot Anomaly Detection

Zhang, Jiangning, He, Haoyang, Chen, Xuhai et al.

Understand

Large Multimodal Model (LMM) GPT-4V(ision) endows GPT-4 with visual grounding capabilities, making it possible to handle certain tasks through the Visual Question Answering (VQA) paradigm.

This paper explores the potential of VQA-oriented GPT-4V in the recently popular visual Anomaly Detection (AD) and is the first to conduct qualitative and quantitative evaluations on the popular MVTec AD and VisA datasets.
Considering that this task requires both image-/pixel-level evaluations, the proposed GPT-4V-AD framework contains three components: \textbf{\textit{1)}} Granular Region Division, \textbf{\textit{2)}} Prompt Designing, \textbf{\textit{3)}} Text2Segmentation for easy quantitative evaluation, and have made some different attempts for comparative analysis.
The results show that GPT-4V can achieve certain results in the zero-shot AD task through a VQA paradigm, such as achieving image-level 77.1/88.0 and pixel-level 68.0/76.6 AU-ROCs on MVTec AD and VisA datasets, respectively.

Built on

Slic superpixels compared to state-of-the-art superpixel methods
Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Süsstrunk · 2012
Earlier work this paper cites.
Mvtec ad–a comprehensive real-world dataset for unsupervised anomaly detection
Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger · 2019
Earlier work this paper cites.
Uninformed students: Student-teacher anomaly detection with discriminative latent embeddings
Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger · 2020
Earlier work this paper cites.
Sub-image anomaly detection with deep pyramid correspondences
Original
Niv Cohen and Yedid Hoshen · 2020
Earlier work this paper cites.
Padim: a patch distribution modeling framework for anomaly detection and localization
Thomas Defard, Aleksandr Setkov, Angelique Loesch, and Romaric Audigier · 2021
Earlier work this paper cites.
An image is worth 16x16 words: Transformers for image recognition at scale
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby · 2021
Earlier work this paper cites.
Cutpaste: Self-supervised learning for anomaly detection and localization
Chun-Liang Li, Kihyuk Sohn, Jinsung Yoon, and Tomas Pfister · 2021
Earlier work this paper cites.
Learning transferable visual models from natural language supervision
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al · 2021
Earlier work this paper cites.
Learning transferable visual models from natural language supervision
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al · 2021
Earlier work this paper cites.
Same same but differnet: Semi-supervised defect detection with normalizing flows
Marco Rudolph, Bastian Wandt, and Bodo Rosenhahn · 2021
Earlier work this paper cites.
Draem-a discriminatively trained reconstruction embedding for surface anomaly detection
Vitjan Zavrtanik, Matej Kristan, and Danijel Skočaj · 2021
Earlier work this paper cites.
Anomaly detection via reverse distillation from one-class embedding
Hanqiu Deng and Xingyu Li · 2022
Earlier work this paper cites.
Towards total recall in industrial anomaly detection
Karsten Roth, Latha Pemula, Joaquin Zepeda, Bernhard Schölkopf, Thomas Brox, and Peter Gehler · 2022
Earlier work this paper cites.
Natural synthetic anomalies for self-supervised anomaly detection and localization
Hannah M Schlüter, Jeremy Tan, Benjamin Hou, and Bernhard Kainz · 2022
Earlier work this paper cites.
A unified model for multi-class anomaly detection
Zhiyuan You, Lei Cui, Yujun Shen, Kai Yang, Xin Lu, Yu Zheng, and Xinyi Le · 2022
Earlier work this paper cites.
Spot-the-difference self-supervised pre-training for anomaly detection and segmentation
Yang Zou, Jongheon Jeong, Latha Pemula, Dongqing Zhang, and Onkar Dabeer · 2022
Earlier work this paper cites.

Similar

Segment any anomaly without training via hybrid prompt regularization
Original
Yunkang Cao, Xiaohao Xu, Chen Sun, Yuqi Cheng, Zongwei Du, Liang Gao, and Weiming Shen · 2023
Cited alongside, same era.
Segment any anomaly without training via hybrid prompt regularization
Original
Yunkang Cao, Xiaohao Xu, Chen Sun, Yuqi Cheng, Zongwei Du, Liang Gao, and Weiming Shen · 2023
Cited alongside, same era.
Towards generic anomaly detection and understanding: Large-scale visual-linguistic model (gpt-4v) takes the lead
Original
Yunkang Cao, Xiaohao Xu, Chen Sun, Xiaonan Huang, and Weiming Shen · 2023
Cited alongside, same era.
A zero-/few-shot anomaly classification and segmentation method for cvpr 2023 vand workshop challenge tracks 1&2: 1st place on zero-shot ad and 4th place on few-shot ad
Original
Xuhai Chen, Yue Han, and Jiangning Zhang · 2023
Cited alongside, same era.
Clip-ad: A language-guided staged dual-path model for zero-shot anomaly detection
Original
Xuhai Chen, Jiangning Zhang, Guanzhong Tian, Haoyang He, Wuhao Zhang, Yabiao Wang, Chengjie Wang, Yunsheng Wu, and Yong Liu · 2023
Cited alongside, same era.
Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al · 2023
Cited alongside, same era.
Anomalygpt: Detecting industrial anomalies using large vision-language models
Original
Zhaopeng Gu, Bingke Zhu, Guibo Zhu, Yingying Chen, Ming Tang, and Jinqiao Wang · 2023
Cited alongside, same era.
Remembering normality: Memory-guided knowledge distillation for unsupervised anomaly detection
Zhihao Gu, Liang Liu, Xu Chen, Ran Yi, Jiangning Zhang, Yabiao Wang, Chengjie Wang, Annan Shu, Guannan Jiang, and Lizhuang Ma · 2023
Cited alongside, same era.
Winclip: Zero-/few-shot anomaly classification and segmentation
Jongheon Jeong, Yang Zou, Taewan Kim, Dongqing Zhang, Avinash Ravichandran, and Onkar Dabeer · 2023
Cited alongside, same era.
Segment anything
Original
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al · 2023
Cited alongside, same era.
Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi · 2023
Cited alongside, same era.
Omni-frequency channel-selection representations for unsupervised anomaly detection
Yufei Liang, Jiangning Zhang, Shiwei Zhao, Runze Wu, Yong Liu, and Shuwen Pan · 2023
Cited alongside, same era.

Then

Exploring ocr capabilities of gpt-4v (ision): A quantitative and in-depth evaluation
Original
Yongxin Shi, Dezhi Peng, Wenhui Liao, Zening Lin, Xinhong Chen, Chongyu Liu, Yuyi Zhang, and Lianwen Jin · 2023
Closest in time.
Llama: Open and efficient foundation language models
Original
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al · 2023
Closest in time.
An early evaluation of gpt-4v (ision)
Original
Yang Wu, Shilong Wang, Hao Yang, Tian Zheng, Hongbo Zhang, Yanyan Zhao, and Bing Qin · 2023
Closest in time.
Pushing the limits of fewshot anomaly detection in industry vision: Graphcore
Guoyang Xie, Jinbao Wang, Jiaqi Liu, Yaochu Jin, and Feng Zheng · 2023
Closest in time.
Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v
Original
Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao · 2023
Closest in time.
Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v
Original
Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao · 2023
Closest in time.
The dawn of lmms: Preliminary explorations with gpt-4v (ision)
Original
Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang · 2023
Closest in time.
Omnial: A unified cnn framework for unsupervised anomaly localization
Ying Zhao · 2023
Closest in time.
Anomalyclip: Object-agnostic prompt learning for zero-shot anomaly detection
Original
Qihang Zhou, Guansong Pang, Yu Tian, Shibo He, and Jiming Chen · 2023
Closest in time.
Minigpt-4: Enhancing vision-language understanding with advanced large language models
Original
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny · 2023
Closest in time.
Instructblip: Towards general-purpose vision-language models with instruction tuning
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi · 2024
Closest in time.
A diffusion-based framework for multi-class anomaly detection
Haoyang He, Jiangning Zhang, Hongxu Chen, Xuhai Chen, Zhishan Li, Xu Chen, Yabiao Wang, Chengjie Wang, and Lei Xie · 2024
Closest in time.
Anomalydiffusion: Few-shot anomaly image generation with diffusion model
Teng Hu, Jiangning Zhang, Ran Yi, Yuzhen Du, Xu Chen, Liang Liu, Yabiao Wang, and Chengjie Wang · 2024
Closest in time.
Real-iad: A real-world multi-view dataset for benchmarking versatile industrial anomaly detection
Chengjie Wang, Wenbing Zhu, Bin-Bin Gao, Zhenye Gan, Jianning Zhang, Zhihao Gu, Shuguang Qian, Mingang Chen, and Lizhuang Ma · 2024
Closest in time.
Learning feature inversion for multi-class unsupervised anomaly detection under general-purpose coco-ad benchmark
Jiangning Zhang, Xiangtai Li, Guanzhong Tian, Zhucun Xue, Yong Liu, Guansong Pang, and Dacheng Tao · 2024
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…