What Computer Use Benchmarks Measure, and What They Miss

If you have been sizing up computer-use agents from published benchmark results, you have been reading a number that says whether the task finished and nothing about what finishing cost. An agent that opens the wrong menu, works out that it is lost, backtracks and eventually saves the record scores exactly the same as one that goes straight there. Both completed the task. Only one of them is something you can put in a pipeline.

That is not something anyone hid. It is what the benchmarks were built to measure. Computer use benchmarks measure what an agent can do, and until recently almost nothing about how long it takes or what it spends getting there. When you are shipping into a customer's legacy desktop system, that second number is the one that decides whether you have an integration or a demo.

What the Benchmarks Actually Scored

Agent evaluations have been built around task success rate: did the agent reach the goal state, yes or no. The point is to determine whether an agent can get to a specific final state, like updating a field in a spreadsheet or filing a record in a desktop application. The benchmark environment is usually a clean sandbox where the agent has infinite time and a narrow set of possible actions. If the agent eventually reaches the goal, it gets a binary score of one. If it fails or crashes, it gets a zero.

This binary scoring treats all successes as equal. It does not matter whether one agent goes straight to the goal while another wanders, backtracks and corrects itself at length first. On the leaderboards they look identical. These tests measure the model's ability to plan a path through a UI and say nothing about the cost of a wandering one. For an engineer choosing a component for a live pipeline, that is a false sense of security.

These benchmarks also tend to use a gold path approach. They compare the agent's actions against a recorded sequence of human actions. If the agent deviates but still completes the task, it might get penalized in some scoring systems, or rewarded for a success that's actually a symptom of instability. The benchmarks measure the ability to solve a puzzle, not the ability to run a reliable business process. They tell you a model can figure out how to use a computer, which is a real research milestone, but they don't tell you if that model can handle the high-volume demands of an enterprise ERP or a medical billing system.

Why That Left a Hole: Efficiency Was Never a Scored Dimension

When efficiency is not a scored dimension, efficiency is what gets optimised away. Models were tuned for reasoning depth, because depth is what the score rewarded, and nothing in the score pushed back on how long that reasoning took.

Wandering has a second cost the score never shows. OSWorld-Human found that across the sixteen agents it evaluated, even the best took 2.7 to 4.3x more steps than necessary. Every one of those extra steps is another interaction with a legacy UI whose loading behaviour is its own business, and a sandbox never reproduces that. An agent that waits too long or moves too early because it is still working out where it is will hit conditions the leaderboard cannot see.

Then there is unit cost. Every action a model decides is a model call somebody pays for, so an agent that takes more steps than it needs costs more than one that does not, on every run, forever. A benchmark score does not penalise that, because spend was never one of the columns either. The hole is the absence of a time-per-task and a cost-per-task figure sitting next to the success rate, and those are the two numbers an engineering budget actually turns on. Where the time itself goes is covered separately in computer use agent speed.

What OSWorld-Human Found When It Finally Measured Latency

The gap between research and reality became undeniable with the release of the OSWorld-Human study (arxiv.org/abs/2506.16042). This research compared agent performance against a human baseline for the same set of desktop tasks. The findings were stark: while humans completed complex tasks in a few minutes, agents often took tens of minutes to reach the same result. This is the latency wall many developers hit only after committing to a specific model architecture.

It also located the time rather than only recording it: large model calls for planning, reflection and judging account for most of the overall latency. The cost is in the deciding, not the clicking. That finding only exists because somebody finally instrumented the clock, and it stayed invisible for as long as the only column was pass or fail.

For technical founders the practical consequence is narrow and it is worth stating plainly. A high benchmark score tells you an agent can do the task. It tells you nothing about whether your customer will wait for it, or whether you can afford it on every record.

Why the Gap Persisted in Practice

The reason this gap stayed invisible for so long is that most early agents were built as standalone demos. In a demo a long wait is acceptable, and nobody is holding a stopwatch anyway. The gap only surfaces when an automation step has something downstream depending on it. In a live pipeline, an automation is rarely an island. It's usually part of a chain: an AI reads a document, extracts data, a desktop agent writes that data into a legacy system, and a third system triggers a notification to a customer.

When the desktop step is the slowest and least predictable stage, the whole chain inherits it. That creates queuing and timeout problems most teams have not sized for, because nothing else in their stack behaves that way. If you are managing automations at scale, the infrastructure consequences of an unbounded step arrive long before anything on a leaderboard warns you about them.

Benchmarks also don't test for environmental drift. In a benchmark, the UI is static. In a customer's system, a Windows update might change a font, or a network lag might delay a popup. A benchmark scores an agent on how it handles the task today, but not how it handles the task when the environment is slightly broken. Because benchmarks focused on the reasoning capability of the LLM, they ignored the infrastructure required to make that reasoning fast and reliable in a messy, live environment. The focus was on the intelligence of the agent, not the industrialization of the process.

What This Means When You Are Shipping Into a Customer's System

If you are shipping into a legacy desktop system, 100% computer use is the version that does not scale, because the time and the spend are both charged on every action a model decides. Minicor separates the deciding from the running. Agents author the automation through Minicor's MCP, and what executes is deterministic Python.

Minicor reports one to two seconds per step, and a run is step count times per-step time, so the total is something you can work out for your own workflow instead of discovering it afterwards. When a UI change breaks an automation, agents repair it between runs. Minicor's own internal tests report 96-99% click accuracy.

For engineers, this means you can give your customers an API for their legacy systems that actually feels like an API. You get structured JSON back upon task completion. You get full observability with video replays of every run and Slack alerts if a job needs attention. The shift worth making is to stop reading a completion score as a readiness signal, and start asking what a step costs in seconds and in spend before you build anything on top of it.

Conclusion

Computer use benchmarks are good at measuring research progress and were never built to tell you what a step costs. A high completion score does not make a viable component if the time and the spend on every action are more than your workflow can carry. The awareness gap here is a measurement gap: buyers did not know because, for years, nobody was counting.

So count it yourself before you commit. Time a full run on the system you actually have to drive rather than a sandbox, run it several times because the variance is the finding, and time the same task performed by the person who does it today. Those two numbers are the comparison that matters, and nobody else can produce them for you.