VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Models

Understand

We present VLMEvalKit: an open-source toolkit for evaluating large multi-modality models based on PyTorch.

• The toolkit aims to provide a user-friendly and comprehensive framework for researchers and developers to evaluate existing multi-modality models and publish \textbf{reproducible} evaluation results.
• In VLMEvalKit, we implement over 450+ large multi-modality model configurations, including both proprietary APIs and open-source models, and support 330+ benchmarks across diverse multi-modal benchmarks.
• By implementing a single interface, new models can be easily added to the toolkit, while the toolkit automatically handles the remaining workloads, including data preparation, distributed inference, prediction post-processing, and metric calculation.