Abstract

Benchmarking video–language models has largely focused on short clips and single-sentence metrics, leaving open whether current systems can generate accurate long-form, paragraph-level descriptions. We introduce CLIP-CC-Bench, an evaluation suite for long-form video description built from 5 hours of movie content segmented into 90-second clips, each paired with an expert-written paragraph-style reference. The suite employs an ensemble of five state-of-the-art LLM-based embedding models to mitigate single-model bias, applying two complementary methodologies — coarse-grained and fine-grained semantic matching — to compare model-generated descriptions against the references. Using this framework we evaluate 17 state-of-the-art video–language models and report both their Borda-aggregated rankings and average scores, and we quantify the protocol's internal reliability through inter-judge agreement and bootstrap ranking stability.

Approach

CLIP-CC-Bench scores a candidate paragraph against an expert reference using an ensemble of five long-context embedding judges. Each judge combines a coarse-grained, paragraph-level cosine similarity with a fine-grained, sentence-level F1 (greedy best-match), merged into a harmonic mean (HM-CF). Per-judge rankings are aggregated with Borda count to produce the leaderboard, and the protocol's reliability is assessed via inter-judge agreement and bootstrap ranking stability.

1

Long-form benchmark200 movie clips of ~90 seconds (~5 hours) with expert paragraph references that deliberately exclude proper nouns, so scoring targets visual understanding over memorization.

2

Ensemble embedding protocolFive embedding judges, each fusing coarse-grained (paragraph) and fine-grained (sentence-level F1) similarity into a harmonic mean, aggregated by Borda count.

3

17-VLM study with reliabilityRankings and mean scores for 17 video–language models, with inter-judge agreement and bootstrap ranking stability to quantify confidence.

Results

17

video–language models (benchmarked)

200

movie clips (~5 hours, expert paragraphs)

5

embedding judges (ensemble, Borda-aggregated)

Final ranking of the 17 VLMs. Borda = Borda count across the five embedding judges; Mean HM-CF = average harmonic mean of coarse- and fine-grained similarity across judges.

Rank	VLM	Borda	Mean HM-CF
1	VideoLLaMA3	80	0.67
2	mPLUG-Owl3	75	0.66
3	LLaVA-OneVision	67	0.64
4	ViLAMP	67	0.64
5	LongVU	61	0.63
6	Qwen2.5-72B	55	0.62
7	Qwen2.5-32B	48	0.61
8	VideoChat-Flash	42	0.60
9	MiniCPM-V	42	0.60
10	Video-XL	36	0.59
11	ShareGPT4Video	29	0.58
12	InternVL2	27	0.58
13	TimeChat	20	0.56
14	LLaVA-NeXT-Video	16	0.55
15	TS-LLaVA	9	0.53
16	Oryx	6	0.52
17	LongVA	0	0.48

BibTeX

@misc{ali2026clipccbench,
  title         = {{CLIP-CC-Bench}: Evaluating Paragraph-Level Video Descriptions in Video--Language Models},
  author        = {Ali, Mukhtiar and Dubey, Harsh and Mishra, Sugam and Pack, Chulwoo},
  year          = {2026},
  eprint        = {2608.04302},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  note          = {Presented at the 2nd Workshop on Evaluation for Multimodal Generation (EvalMG), ACM SIGIR 2026},
  url           = {https://arxiv.org/abs/2608.04302}
}