Are We Using the Right Benchmark:
An Evaluation Framework for Visual Token Compression Methods

Some recent MLLMs, such as Qwen2-VL and Qwen2.5-VL, natively support inputs of varying resolutions. A trivial yet efficient method to handle high-resolution images is to simply downsample them to a lower resolution. However, most token compression methods for MLLMs choose to adaptively drop useless tokens or merge similar tokens instead of directly downsampling the original image, which theoretically should be more intelligent.

Surprisingly, we find that image downsampling consistently exceeds other sophisticated methods under some settings. Based on comprehensive experiments, we propose a bold hypothesis:

Some data in the existing benchmarks is overly simplistic and irrelevant to evaluating visual token compression methods, leading to the unreasonable phenomenon that even the downsampling method is sufficient to deal with the visual token compression task.

To validate this, we design a data-centric analysis using downsampling as a discriminator. We identify two crucial findings:

1. Current benchmarks are noisy for the visual token compression task. Many samples can be answered correctly even with significant downsampling, indicating they do not test fine-grained visual understanding.
2. Downsampling can serve as a data filter. By separating samples into "simple" (Group B) and "difficult" (Group A) based on whether downsampling succeeds, we can effectively distinguish samples that truly require advanced compression.

Based on these findings, we propose VTC-Bench, a new evaluation framework specifically designed to optimize and denoise current existing benchmarks. By explicitly distinguishing between “simple” and “difficult” samples through downsampling, VTC-Bench adaptively selects "difficult" samples that satisfy the requirements of evaluating visual token compression methods.

The pipeline consists of three critical steps:

Step 1: Inference & Compression. Given a sample and a target token compression ratio, we run two inference pipelines: (1) a downsampling baseline (the filter) and (2) advanced visual token compression methods (e.g., FastV, VisionZip, DART) evaluated directly on the target MLLM.

Step 2: Grouping. We use the performance of the downsampling method as a binary discriminator to categorize samples:

• Group A (Difficult Samples): Samples that are answered incorrectly by the downsampling method.
• Group B (Simple Samples): Samples that are answered correctly by the downsampling method.

This step filters the existing benchmarks and removes noisy data that is not applicable for evaluating the visual token compression methods.

Step 3: Result Aggregation. We perform a statistical analysis on the accuracy of the "difficult" samples to obtain an indicator that truly reflects the capability of visual compression methods.

When evaluated using VTC-Bench (focusing on "Difficult Samples"), the landscape of visual token compression changes completely. Advanced methods prove their worth where it matters most.

Is downsampling all you need? Across many benchmarks, simple image downsampling often beats more advanced compression methods. VTC-Bench overturns this impression: when we restrict evaluation to the compression-relevant difficult samples (Group A), the trend reverses. By filtering out easy samples, VTC-Bench reveals that for truly challenging instances, advanced visual token compression methods are not only effective but necessary.

This paper systematically analyzes the task mismatch problem presented in current MLLM benchmarks when evaluating visual token compression methods. Based on a surprising and counterintuitive finding: simple image downsampling consistently outperforms many advanced compression methods across multiple widely used benchmarks, we conduct a comprehensive empirical study across several advanced visual token compression methods.

Thus, two crucial findings are concluded based on the empirical study:

1. Current benchmarks are noisy for the visual token compression task.
2. Downsampling can serve as a data filter to evaluate the difficulty of samples upon the visual token compression task.

Furthermore, we propose VTC-Bench, a new evaluation framework specifically designed to optimize and denoise current existing benchmarks by explicitly distinguishing between “simple” and “difficult” samples through downsampling. Through this work, we hope to not only advance the field of visual token compression but also inspire more discussions within the community on "how to properly evaluate efficient MLLMs."

Dependence on a Single Base Model: The construction of the sample filtering mechanism is entirely dependent on Qwen2-VL. While chosen for its technical necessity, this reliance limits generalizability, as different models may define "difficult" samples slightly differently. In the future, this framework will be extended to more base models that have great dynamic resolution support.

Single Criterion for Filtering: We use only downsampling as the criterion for identifying "difficult" samples. Although effective, employing a more diverse set of baseline methods for ensemble filtering could yield a more robust definition. We will explore a more robust filtering mechanism for the subsequent version of VTC-Bench in future work.

Lack of a Formal Theoretical Definition: Our identification of "difficult" and "simple" samples is primarily based on the experimental results. A formal theoretical framework and mathematical proof for a more explainable sample filtering mechanism is yet to be established.

Limited Coverage of Methods: Due to the rapidly evolving field and adaptation constraints, our experimental comparison doesn't include all visual token compression methods. However, the selected methods are sufficient to support our core claims, and the framework remains open and extensible.

Concurrent Similar Research

VisionThink: VisionThink identifies that downsampling shows surprising effectiveness on most general VQA tasks except for OCR and detail-sensitive benchmarks. It employs a reinforcement learning framework to dynamically decide if high-resolution images are needed. Our work further investigates this observation through extensive experiments, clarifying the task mismatch in current benchmarks and proving downsampling's potential as a data filter.

EffiVLM-Bench: EffiVLM-Bench offers a unified evaluation framework for assessing training-free acceleration techniques. Unlike EffiVLM-Bench, which aims to provide a toolkit, VTC-Bench specifically targets the task mismatch problem in visual token compression evaluation.

Experiment Details

All experiments are conducted on one A800 GPU. Code environment: Python=3.10, torch=2.6.0, torchvision=0.21.0. Bicubic interpolation is used for downsampling.

Benchmark Details

• GQA: 22M reasoning questions with strict distribution control.
• MMBench: 3,217 multiple-choice questions across 20 dimensions.
• MME: Evaluates perceptual and cognitive abilities across 14 subtasks.
• POPE: Evaluates object hallucination under different sampling strategies.
• MMStar: 1,500 samples covering six core abilities.
• OCRBench: 1,000 manually verified samples for OCR capabilities.
• ChartQA: Evaluates visual and logical reasoning over charts.

Visualization between Groups

We visualize "difficult" and "simple" samples to observe their characteristics. "Difficult" samples (Group A) tend to require multiple detail perceptions and comparisons (e.g., finding extreme values in charts or complex counting). "Simple" samples (Group B) usually rely on medium/large-sized patterns or simple comparisons.

Statistical Analysis of Image Properties

We investigated whether the difficulty of samples could be attributed to basic low-level visual properties (entropy, brightness, contrast, colorfulness, size). Results show no statistically significant difference in these properties between Group A and Group B, validating that "difficulty" cannot be simply predicted by these elementary features and justifying the use of downsampling as a filter.