A model can lead an overall ranking while performing poorly on the particular work an organization needs. Aggregating tasks makes comparison easier, but it also embeds a decision about which tasks matter most. Understanding that decision is necessary before using a benchmark to choose a deployed system.

Vals AI’s current Index page, updated October 26 and labeled version 2.2, combines six private and two public benchmarks across professional sectors using GDP-related weights. It publishes capability, latency and cost views. Those measures describe the evaluation suite; a score is not a measured percentage of economic activity that can be automated. Read the original source.

An average embeds a weighting choice

Weighting changes the question answered by an average. An equal-task mean treats every benchmark unit similarly, while a sector-weighted index gives some components more influence. Neither automatically matches a customer’s workload. A team should examine the component scores before concluding that a small overall lead matters for its own application.

Versioning is equally important. If a task set, scoring rule or weight changes, movement in a headline score can reflect the measurement rather than a new model capability. A comparison over time should pin the index version and identify which earlier results were rerun. Otherwise a line chart can imply continuity that the underlying experiment does not have.

Pin the version and configuration

Private benchmarks can reduce direct exposure of evaluation items, but they also limit what an outside reader can independently inspect. That trade-off does not invalidate the measurement. It means the published methodology and the evaluator’s account of task conditions carry more responsibility, while local acceptance tests remain necessary.

Cost and latency should travel with the configuration used to obtain the score. Reasoning settings, tools and retry policies can change the effort spent on a task. A capability result from an expensive configuration should not be paired with the price of a cheaper configuration as though they describe the same service.

Test the receiving workflow

In the near term, the index can help teams shortlist models for a closer task-specific trial. Its broader value would be a stable, interpretable record of professional-task progress. That depends on transparent revisions and a clear distinction between benchmark success, accepted work and the institutional changes needed to use that work.

The next evidence for a deployment decision is a held-out comparison on the receiving organization’s tasks, with failures and review effort included. The aggregate supplies a reference point, while the actual workflow determines whether the capability becomes useful economic value.

Source index snapshot

Vals Index source snapshot with October 26 date
Original Vals Index source snapshot, accessed October 27. It is retained as evidence rather than used as a cover.
References
  1. Vals AI primary source · 2026-10-26

Questions, news tips or corrections? Contact [email protected].

Share this article

Post to XShare on LinkedInShare on WhatsAppEmail