Survey of Large Multimodal Model Datasets, Application Categories and Taxonomy

Caesar, H., Bankiti, V., Lang, A.H., Vora, S., Liong, V.E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G., Beijbom, O.: nuscenes: A multimodal dataset for autonomous driving (2020), https://arxiv.org/abs/1903.11027

Original

1903

Earlier work this paper cites.

Miech, A., Zhukov, D., Alayrac, J.B., Tapaswi, M., Laptev, I., Sivic, J.: Howto100m: Learning a text-video embedding by watching hundred million narrated video clips (2019), https://arxiv.org/abs/1906.03327

Original

1906

Earlier work this paper cites.

Girdhar, R., Ramanan, D.: Cater: A diagnostic dataset for compositional actions and temporal reasoning. CoRR (2019), https://arxiv.org/abs/1910.04744

Original

1910

Earlier work this paper cites.

Shridhar, M., Thomason, J., Gordon, D., Bisk, Y., Han, W., Mottaghi, R., Zettlemoyer, L., Fox, D.: Alfred: A benchmark for interpreting grounded instructions for everyday tasks (2020), https://arxiv.org/abs/1912.01734

Original

1912

Earlier work this paper cites.

Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Li, F.: Imagenet: A large-scale hierarchical image database. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 248–255 (2009)

2009

Earlier work this paper cites.

Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? the kitti vision benchmark suite. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2012)

2012

Earlier work this paper cites.

Young, P., Lai, A., Hodosh, M., Hockenmaier, J.: From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics 2

2014

Earlier work this paper cites.

Chang, A., Funkhouser, T., Guibas, L., Hanrahan, P., Huang, Q., Li, Z., Savarese, S., Savva, M., Song, S., Su, H., Xiao, J.: Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012 (2015)

Original

2015

Earlier work this paper cites.

Chen, X., Fang, H., Lin, T.Y., Vedantam, R., Gupta, S., Dollar, P., Zitnick, C.L.: Microsoft coco captions: Data collection and evaluation server (2015), https://arxiv.org/abs/1504.00325

Original

2015

Earlier work this paper cites.

Panayotov, V., Chen, G., Povey, D., Khudanpur, S.: Librispeech: an asr corpus based on public domain audio books. In: 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP). pp. 5206–5210 (2015)

2015

Earlier work this paper cites.

Aytar, Y., Vondrick, C., Torralba, A.: See, hear, and read: Deep aligned representations. arXiv preprint arXiv:1706.00932 (2017)

Original

2017

Earlier work this paper cites.

Gemmeke, J., Ellis, D., Freedman, D., Jansen, A., Lawrence, W., Moore, R., Plakal, M., Ritter, M.: Audio set: An ontology and human-labeled dataset for audio events. In: 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP). pp. 776–780 (2017)

2017

Earlier work this paper cites.

Nagrani, A., Chung, J., Zisserman, A.: Voxceleb: a large-scale speaker identification dataset. arXiv preprint arXiv:1706.08612 (2017)

Original

2017

Earlier work this paper cites.

Baltrušaitis, T., Ahuja, C., Morency, L.: Multimodal machine learning: A survey and taxonomy. IEEE transactions on pattern analysis and machine intelligence 41

2018

Earlier work this paper cites.

Du, J., Na, X., Liu, X., Bu, H.: Aishell-2: Transforming mandarin asr research into industrial scale (2018), https://arxiv.org/abs/1808.10583

Original

2018

Earlier work this paper cites.

Lei, J., Yu, L., Bansal, M., Berg, T.: Tvqa: Localized, compositional video question answering. arXiv preprint arXiv:1809.01696 (2018)

Original

2018

Earlier work this paper cites.

Poria, S., Hazarika, D., Majumder, N., Naik, G., Cambria, E., Mihalcea, R.: Meld: A multimodal multi-party dataset for emotion recognition in conversations. arXiv preprint arXiv:1810.02508 (2018)

Original

2018

Earlier work this paper cites.

Zadeh, A., Liang, P., Poria, S., Cambria, E., Morency, L.P.: Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. pp. 2236–2246 (01 2018). https://doi.org/10.18653/v1/P18-1208

2018

Earlier work this paper cites.

Ardila, R., Branson, M., Davis, K., Henretty, M., Kohler, M., Meyer, J., Morais, R., Saunders, L., Tyers, F., Weber, G.: Common voice: A massively-multilingual speech corpus. arXiv preprint arXiv:1912.06670 (2019)

Original

2019

Earlier work this paper cites.

Etten, A.V., Lindenbaum, D., Bacastow, T.M.: Spacenet: A remote sensing dataset and challenge series (2019), https://arxiv.org/abs/1807.01232

Original

2019

Earlier work this paper cites.

Helber, P., Bischke, B., Dengel, A., Borth, D.: Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification (2019)

2019

Earlier work this paper cites.

Johnson, A.E.W., Pollard, T.J., Greenbaum, N.R., Lungren, M.P., ying Deng, C., Peng, Y., Lu, Z., Mark, R.G., Berkowitz, S.J., Horng, S.: Mimic-cxr-jpg, a large publicly available database of labeled chest radiographs (2019)

2019

Cited alongside, same era.

Kim, C., Kim, B., Lee, H., Kim, G.: Audiocaps: Generating captions for audios in the wild. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics. pp. 119–132 (2019)

2019

Cited alongside, same era.

Zhang, C., Yang, Z., He, X., Deng, L.: Multimodal intelligence: Representation learning, information fusion, and applications. IEEE Journal of Selected Topics in Signal Processing 14

2020

Cited alongside, same era.

Changpinyo, S., Sharma, P., Ding, N., Soricut, R.: Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts (2021), https://arxiv.org/abs/2102.08981

Original

2021

Cited alongside, same era.

Yin, Z., Wang, J., Cao, J., Shi, Z., Liu, D., Li, M., Sheng, L., Bai, L., Huang, X., Wang, Z., Shao, J., Ouyang, W.: Lamm: Language-assisted multi-modal instruction-tuning dataset, framework, and benchmark (2023), https://arxiv.org/abs/2306.06687

Original

2023

Later among the works it cites.

Zhao, B., Wu, B., He, M., Huang, T.: Svit: Scaling up visual instruction tuning (2023), https://arxiv.org/abs/2307.04087

Original

2023

Later among the works it cites.

Zhao, R., Chen, H., Wang, W., Jiao, F., Do, X., Qin, C., Ding, B., Guo, X., Li, M., Li, X., Joty, S.: Retrieving multimodal information for augmented generation: A survey. arXiv preprint arXiv:2303.10868 (2023)

Original

2023

Later among the works it cites.

Alwajih, F., Nagoudi, E., Bhatia, G., Mohamed, A., Abdul-Mageed, M.: Peacock: A family of arabic multimodal large language models and benchmarks. arXiv preprint arXiv:2403.01031 (2024)

Original

2024

Closest in time.