2022

Webly Supervised Concept Expansion for General Purpose Vision Models

Kamath, Amita, Clark, Christopher, Gupta, Tanmay et al.

Understand

General Purpose Vision (GPV) systems are models that are designed to solve a wide array of visual tasks without requiring architectural changes.

Today, GPVs primarily learn both skills and concepts from large fully supervised datasets.
Scaling GPVs to tens of thousands of concepts by acquiring data to learn each concept for every skill quickly becomes prohibitive.
This work presents an effective and inexpensive alternative: learn skills from supervised datasets, learn concepts from web image search, and leverage a key characteristic of GPVs: the ability to transfer visual knowledge across skills.

Built on

Fergus, R., Fei-Fei, L., Perona, P., Zisserman, A.: Learning object categories from google’s image search. Tenth IEEE International Conference on Computer Vision (ICCV’05) Volume 1
2005
Earlier work this paper cites.
Li, L.J., Fei-Fei, L.: Optimol: Automatic online picture collection via incremental model learning. International Journal of Computer Vision
2007
Earlier work this paper cites.
van der Maaten, L., Hinton, G.E.: Visualizing data using t-sne. Journal of Machine Learning Research
2008
Earlier work this paper cites.
Vijayanarasimhan, S., Grauman, K.: Keywords to visual categories: Multiple-instance learning forweakly supervised object categorization. 2008 IEEE Conference on Computer Vision and Pattern Recognition pp. 1–8 (2008)
2008
Earlier work this paper cites.
Wang, X.J., Zhang, L., Li, X., Ma, W.Y.: Annotating images by mining image search results. IEEE Transactions on Pattern Analysis and Machine Intelligence
2008
Earlier work this paper cites.
Dong, W., Socher, R., Li-Jia, L., Li, K., Fei-Fei, L.: ImageNet: A large-scale hierarchical image database. In: CVPR (2009)
2009
Earlier work this paper cites.
Farhadi, A., Endres, I., Hoiem, D., Forsyth, D.A.: Describing objects by their attributes. 2009 IEEE Conference on Computer Vision and Pattern Recognition pp. 1778–1785 (2009)
2009
Earlier work this paper cites.
Kumar, N., Berg, A.C., Belhumeur, P.N., Nayar, S.K.: Attribute and simile classifiers for face verification. 2009 IEEE 12th International Conference on Computer Vision pp. 365–372 (2009)
2009
Earlier work this paper cites.
Lampert, C.H., Nickisch, H., Harmeling, S.: Learning to detect unseen object classes by between-class attribute transfer. 2009 IEEE Conference on Computer Vision and Pattern Recognition pp. 951–958 (2009)
2009
Earlier work this paper cites.
Parikh, D., Grauman, K.: Relative attributes. 2011 International Conference on Computer Vision pp. 503–510 (2011)
2011
Earlier work this paper cites.
Chen, H., Gallagher, A.C., Girod, B.: Describing clothing by semantic attributes. In: ECCV (2012)
2012
Earlier work this paper cites.
Patterson, G., Hays, J.: Sun attribute database: Discovering, annotating, and recognizing scene attributes. 2012 IEEE Conference on Computer Vision and Pattern Recognition pp. 2751–2758 (2012)
2012
Earlier work this paper cites.
Brysbaert, M., Warriner, A., Kuperman, V.: Concreteness ratings for 40 thousand generally known english word lemmas. Behavior research methods
2013
Earlier work this paper cites.
Chen, X., Shrivastava, A., Gupta, A.: NEIL: Extracting visual knowledge from web data. In: ICCV (2013)
2013
Earlier work this paper cites.
Li, Q., Wu, J., Tu, Z.: Harvesting mid-level visual concepts from large-scale internet images. 2013 IEEE Conference on Computer Vision and Pattern Recognition pp. 851–858 (2013)
2013
Earlier work this paper cites.
Wang, S., Joo, J., Wang, Y., Zhu, S.C.: Weakly supervised learning for attribute localization in outdoor scenes. 2013 IEEE Conference on Computer Vision and Pattern Recognition pp. 3111–3118 (2013)
2013
Earlier work this paper cites.
Divvala, S., Farhadi, A., Guestrin, C.: Learning everything about anything: Webly-supervised visual concept learning. In: CVPR (2014)
2014
Earlier work this paper cites.
Golge, E., Sahin, P.D.: Conceptmap: Mining noisy web data for concept learning. In: ECCV (2014)
2014
Earlier work this paper cites.
Hoffman, J., Guadarrama, S., Tzeng, E., Hu, R., Donahue, J., Girshick, R.B., Darrell, T., Saenko, K.: Lsda: Large scale detection through adaptation. In: NIPS (2014)
2014
Earlier work this paper cites.
Kazemzadeh, S., Ordonez, V., Matten, M., Berg, T.: ReferItGame: Referring to objects in photographs of natural scenes. In: EMNLP (2014)
2014
Earlier work this paper cites.
Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Original
2014
Earlier work this paper cites.
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Lawrence Zitnick, C., Parikh, D.: VQA: Visual question answering. In: ICCV (2015)
2015
Earlier work this paper cites.
Chao, Y.W., Wang, Z., He, Y., Wang, J., Deng, J.: HICO: A benchmark for recognizing human-object interactions in images. In: Proceedings of the IEEE International Conference on Computer Vision (2015)
2015
Earlier work this paper cites.
Chen, X., Gupta, A.: Webly supervised learning of convolutional networks. In: ICCV (2015)
2015
Earlier work this paper cites.
Kay, M., Matuszek, C., Munson, S.A.: Unequal representation and gender stereotypes in image search results for occupations. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. pp. 3819–3828 (2015)
2015
Earlier work this paper cites.
Vedantam, R., Zitnick, C.L., Parikh, D.: Cider: Consensus-based image description evaluation. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) pp. 4566–4575 (2015)
2015
Earlier work this paper cites.
Krause, J., Sapp, B., Howard, A.G., Zhou, H., Toshev, A., Duerig, T., Philbin, J., Fei-Fei, L.: The unreasonable effectiveness of noisy data for fine-grained recognition. ArXiv
Original
2016
Earlier work this paper cites.
Yatskar, M., Zettlemoyer, L., Farhadi, A.: Situation recognition: Visual semantic role labeling for image understanding. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) pp. 5534–5542 (2016)
2016
Earlier work this paper cites.
Yatskar, M., Zettlemoyer, L., Farhadi, A.: Situation recognition: Visual semantic role labeling for image understanding. In: Conference on Computer Vision and Pattern Recognition (2016)
2016
Earlier work this paper cites.
Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., Parikh, D.: Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. In: CVPR (2017)
2017
Earlier work this paper cites.
He, K., Gkioxari, G., Dollar, P., Girshick, R.: Mask R-CNN. In: ICCV (2017)
2017
Earlier work this paper cites.

Similar

Jin, B., Segovia, M.V.O., Süsstrunk, S.: Webly supervised semantic segmentation. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) pp. 1705–1714 (2017)
2017
Cited alongside, same era.
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.J., Shamma, D.A., Bernstein, M., Fei-Fei, L.: Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV
2017
Cited alongside, same era.
Otterbacher, J., Bates, J., Clough, P.: Competent men and warm women: Gender stereotypes and backlash in image search results. In: Proceedings of the 2017 chi conference on human factors in computing systems. pp. 6620–6631 (2017)
2017
Cited alongside, same era.
Redmon, J., Farhadi, A.: Yolo9000: Better, faster, stronger. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) pp. 6517–6525 (2017)
2017
Cited alongside, same era.
Zhao, J., Wang, T., Yatskar, M., Ordonez, V., Chang, K.W.: Men also like shopping: Reducing gender bias amplification using corpus-level constraints. In: EMNLP (2017). https://doi.org/10.18653/v1/D17-1323,
2017
Cited alongside, same era.
Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., Zhang, L.: Bottom-up and top-down attention for image captioning and visual question answering. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition pp. 6077–6086 (2018)
2018
Cited alongside, same era.
Chao, Y.W., Liu, Y., Liu, X., Zeng, H., Deng, J.: Learning to detect human-object interactions. In: Proceedings of the IEEE Winter Conference on Applications of Computer Vision (2018)
2018
Cited alongside, same era.
Guo, S., Huang, W., Zhang, H., Zhuang, C., Dong, D., Scott, M.R., Huang, D.: Curriculumnet: Weakly supervised learning from large-scale web images. ArXiv
Original
2018
Cited alongside, same era.
Kuznetsova, A., Rom, H., Alldrin, N., Uijlings, J.R.R., Krasin, I., Pont-Tuset, J., Kamali, S., Popov, S., Malloci, M., Duerig, T., Ferrari, V.: The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale. arXiv:1811.00982 (2018)
Original
2018
Cited alongside, same era.
Liang, K., Guo, Y., Chang, H., Chen, X.: Visual relationship detection with deep structural ranking. In: AAAI (2018)
2018
Cited alongside, same era.
McCann, B., Keskar, N., Xiong, C., Socher, R.: The natural language decathlon: Multitask learning as question answering. ArXiv
Original
2018
Cited alongside, same era.
Niu, L., Tang, Q., Veeraraghavan, A., Sabharwal, A.: Learning from noisy web data with category-level supervision. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition pp. 7689–7698 (2018)
2018
Cited alongside, same era.

Then

Shen, Y., Ji, R., Chen, Z., Hong, X., Zheng, F., Liu, J., Xu, M., Tian, Q.: Noise-aware fully webly supervised object detection. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 11323–11332 (2020)
2020
Later among the works it cites.
Sun, G., Wang, W., Dai, J., Gool, L.V.: Mining cross-image semantics for weakly supervised semantic segmentation. In: ECCV (2020)
2020
Later among the works it cites.
Ulutan, O., Iftekhar, A.S.M., Manjunath, B.S.: Vsgnet: Spatial attention network for detecting human object interactions using graph convolutions. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 13614–13623 (2020)
2020
Later among the works it cites.
Wu, Z., Tao, Q., Lin, G., Cai, J.: Exploring bottom-up and top-down cues with attentive learning for webly supervised object detection. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 12933–12942 (2020)
2020
Later among the works it cites.
Yang, J., Feng, L., Chen, W., Yan, X., Zheng, H., Luo, P., Zhang, W.: Webly supervised image classification with self-contained confidence. In: ECCV (2020)
2020
Later among the works it cites.
Zheng, W., Yan, L., Gou, C., Wang, F.: Webly supervised knowledge embedding model for visual reasoning. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 12442–12451 (2020)
2020
Later among the works it cites.
Zhong, X., Ding, C., Qu, X., Tao, D.: Polysemy deciphering network for human-object interaction detection. In: ECCV (2020)
2020
Later among the works it cites.
Cho, J., Lei, J., Tan, H., Bansal, M.: Unifying vision-and-language tasks via text generation. arXiv preprint arXiv:2102.02779 (2021)
Original
2021
Later among the works it cites.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. ICLR
Original
2021
Later among the works it cites.
Fang, H., Xie, Y., Shao, D., Lu, C.: Dirv: Dense interaction region voting for end-to-end human-object interaction detection. In: AAAI (2021)
2021
Later among the works it cites.
Gu, X., Lin, T.Y., Kuo, W., Cui, Y.: Open-vocabulary object detection via vision and language knowledge distillation (2021)
2021
Later among the works it cites.
Jaegle, A., Borgeaud, S., Alayrac, J.B., Doersch, C., Ionescu, C., Ding, D., Koppula, S., Brock, A., Shelhamer, E., H’enaff, O.J., Botvinick, M.M., Zisserman, A., Vinyals, O., Carreira, J.: Perceiver io: A general architecture for structured inputs & outputs. ArXiv
Original
2021
Later among the works it cites.
Jaegle, A., Gimeno, F., Brock, A., Zisserman, A., Vinyals, O., Carreira, J.: Perceiver: General perception with iterative attention. In: ICML (2021)
2021
Later among the works it cites.
Jia, C., Yang, Y., Xia, Y., Chen, Y.T., Parekh, Z., Pham, H., Le, Q.V., Sung, Y.H., Li, Z., Duerig, T.: Scaling up visual and vision-language representation learning with noisy text supervision. In: ICML (2021)
2021
Later among the works it cites.
Kim, B., Lee, J., Kang, J., Kim, E.S., Kim, H.J.: Hotr: End-to-end human-object interaction detection with transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2021)
2021
Later among the works it cites.
Kim, W., Son, B., Kim, I.: Vilt: Vision-and-language transformer without convolution or region supervision. ArXiv
Original
2021
Later among the works it cites.
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S.C.F., Guo, B.: Swin transformer: Hierarchical vision transformer using shifted windows. ICCV
Original
2021
Later among the works it cites.
Pham, K., Kafle, K., Lin, Z., Ding, Z., Cohen, S., Tran, Q., Shrivastava, A.: Learning to predict visual attributes in the wild. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 13018–13028 (June 2021)
2021
Later among the works it cites.
Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, I.: Learning transferable visual models from natural language supervision (2021)
2021
Later among the works it cites.
Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, I.: Learning transferable visual models from natural language supervision. In: ICML (2021)
2021
Later among the works it cites.
Shen, S., Li, L.H., Tan, H., Bansal, M., Rohrbach, A., Chang, K.W., Yao, Z., Keutzer, K.: How much can clip benefit vision-and-language tasks? ArXiv
Original
2021
Later among the works it cites.
Wang, S., Thompson, L., Iyyer, M.: Phrase-bert: Improved phrase embeddings from bert with an application to corpus exploration. In: EMNLP (2021)
2021
Later among the works it cites.
Whitehead, S., Wu, H., Ji, H., Feris, R.S., Saenko, K., MIT-IBM, U.: Separating skills and concepts for novel visual question answering. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 5628–5637 (2021)
2021
Later among the works it cites.
Xu, H., Yan, M., Li, C., Bi, B., Huang, S., Xiao, W., Huang, F.: E2e-vlp: End-to-end vision-language pre-training enhanced by visual learning (2021)
2021
Later among the works it cites.
Zareian, A., Rosa, K.D., Hu, D.H., Chang, S.F.: Open-vocabulary object detection using captions. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 14388–14397 (2021)
2021
Later among the works it cites.
Zhang, A., Liao, Y., Liu, S., Lu, M., Wang, Y., Gao, C., Li, X.: Mining the benefits of two-stage and one-stage hoi detection. arXiv preprint arXiv:2108.05077 (2021)
Original
2021
Later among the works it cites.
Zhang, P., Li, X., Hu, X., Yang, J., Zhang, L., Wang, L.J., Choi, Y., Gao, J.: Vinvl: Making visual representations matter in vision-language models. ArXiv
Original
2021
Later among the works it cites.
Zou, C., Wang, B., Hu, Y., Liu, J., Wu, Q., Zhao, Y., Li, B., Zhang, C., Zhang, C., Wei, Y., et al.: End-to-end human object interaction detection with hoi transformer. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2021)
2021
Later among the works it cites.
Gupta, T., Kamath, A., Kembhavi, A., Hoiem, D.: Towards general purpose vision systems: An end-to-end task-agnostic vision-language architecture. In: CVPR (2022)
2022
Closest in time.
Gupta, T., Marten, R., Kembhavi, A., Hoiem, D.: Grit: General robust image task benchmark. arXiv preprint arXiv:2204.13653 (2022)
Original
2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…