A video model can answer a question correctly while pointing to the wrong moment or object. The answer may come from a general understanding of the clip rather than the specific evidence requested.
OmniCompass, posted October 13, pairs temporal grounding, spatial grounding and question answering across visual and audio-visual tracks. Its evaluations also vary whether the target is specified through text alone or text with an image. Read the research paper.
A convincing video summary can be poorly grounded
A model may describe the broad story of a clip correctly while failing to identify the precise interval or location supporting its answer. That difference matters when video understanding becomes infrastructure for search, annotation or physical-AI training. A plausible description is insufficient if another system needs the exact event boundary or object region. Evaluation that asks only what happened can leave this weakness hidden.
OmniCompass combines understanding with grounding. It tests when a referenced target appears, where it is and what happens, while separating visual-only evidence from tasks requiring joint audio and visual information. The goal is diagnosis across related abilities rather than one blended notion of video comprehension.
Three tasks test the same target in different ways
Each of the two tracks includes temporal grounding, spatial grounding and question answering. OmniCompass-VL contains eight hundred and twenty videos and 2,460 instances across short, medium and long durations. An audio-rich subset supplies the audio-visual track, with two hundred and forty videos and seven hundred and twenty instances. Annotations receive multiple rounds of manual quality checks.
Each instance also has paired target specifications: text alone and text plus an image referring to the same target. This asks whether a model's answer remains correct and consistent when guidance changes form without changing the intended subject. It can expose a system that knows the scene broadly but loses track of which object or event the question identifies.
Leading systems still struggle to point to the evidence
The authors evaluate fifty multimodal language models. They report that open models trail proprietary models, particularly in audio-visual understanding, while precise spatial and temporal grounding remains difficult even for leading systems. Inconsistencies across paired target specifications reveal another limitation that an unpaired question-answering score would miss.
The input setup deserves attention when reading those comparisons. The paper samples different frame counts according to model capacity and interface constraints, timestamps frames, and uses native audio in the audio-visual track. These are evaluations of models through particular input pipelines, not identical full-resolution access to every moment of every clip. Sampling and interface limits are part of the demonstrated performance.
Consistency requires both answers to stay correct
OmniCompass’s cross-specification scores jointly consider correctness and consistency across text and text-plus-image references. Merely repeating the same wrong answer is not success. The paper reports that visual target specifications improve the single-setting average for only eleven of thirty-six models in the relevant comparison, making additional visual guidance an empirical question rather than an assumed benefit.
The input analysis also finds substantial benefits from timestamps, and audio helps question answering and temporal grounding in the audio-visual track. These interventions locate some of the performance sensitivity in how the evidence is supplied. They do not eliminate the grounding challenge, but help distinguish insufficient access to temporal cues from a model that receives those cues and still cannot locate the target reliably.
Training data needs the right interval, not just the right story
For a video-to-action pipeline, an answer about the correct activity can still be attached to the wrong frames. Repeated similar events and long recordings make that error especially consequential. Temporal and spatial checks should therefore precede using a model's output as supervision for another learner. An image-guided reference is also worth testing because added guidance can expose inconsistency rather than reliably improve it.
Multimodal fluency and evidence localisation are separable capabilities. A benchmark should let them disagree. OmniCompass provides a structured way to find that disagreement, helping teams decide whether a model is ready to explain a video, search within it or produce the precise grounded labels that downstream automation requires.
References
Questions, news tips or corrections? Contact [email protected].
