An AI agent that operates a computer scored better than a person on an easy test this year. On a harder, more realistic test released in June, that same agent's score fell by two-thirds. Two months later, a newer model had already climbed most of the way back. The real question was never which way an agent looks at a screen. It is whether a fast-climbing score means the agent works, or just that it passed this particular test.
Eighteen months of headlines told you that AI agents which operate a computer, clicking buttons and filling in forms the way a person would, had solved the reliability problem. That claim got a hard check on June 26. That was the day the team behind OSWorld, the standard test for these agents, released a tougher version called OSWorld 2.0. The new test uses real workflows that take a person about 1.6 hours and roughly 318 separate actions to finish, compared to about 30 actions in the original test. Anthropic's leading model at the time completed only 20.6 percent of the new tasks, even though that same model had scored in the low 80s on the older, easier version (OSWorld 2.0, 2026).
This is the part that should make you more careful. By late August, a newer model had already pushed the OSWorld 2.0 score from 20.6 percent up to roughly 70 percent, a two-month recovery of most of the gap (BenchLM, 2026). That climb is proof a benchmark score moves fast once vendors start optimizing for it, which is exactly what happened to the original OSWorld before June. For two years, OSWorld was the test everyone pointed to as proof these agents were ready for real work. Scores climbed from 12 percent in April 2024 past a human baseline of roughly 72 percent by late 2025, then into the low-to-mid 80s by mid-2026 (OSWorld 2.0, 2026). Vendors read that climb as proof the hard problem was solved, and the conversation moved on to price. A fast recovery on a new test tells you the same climb is starting again. It tells you nothing about whether your own multi-hour workflow is one of the tasks these systems have learned to handle.
Vision and Structure Share the Same Agent Now
Self-driving cars went through the same argument, and it is a useful way to picture what is happening here. Tesla built its system mostly on cameras: the car looks at the road the way a person looks at it, and a model figures out what those images mean. Waymo built its system on cameras plus lidar, radar, and detailed maps of the road itself, so the car does not have to guess what a stop sign is from a picture. It already knows a stop sign is there because the map says so. Neither approach won outright. Every serious self-driving system today uses some mix of both: cameras for the things maps cannot capture, and structured data for the things a camera can get wrong in bad light or bad weather.
AI agents that operate a computer face the identical choice. A vision agent takes a picture of your screen, the same picture you would see, and figures out where to click based on what it looks like. A structure agent skips the picture entirely and reads the software's own internal map of the page, the invisible list of buttons, fields, and labels that the software already knows about, then uses that list to act. Vision works like the camera in that comparison. Structure works from the map the software already carries of itself.
Each has a weak spot the other one does not. Vision struggles with dense text, with zoomed-in or oddly sized screens, and with anything drawn as an image rather than described in the software's code. Structure struggles with older, custom-built systems that were never set up to describe themselves that way, which is a large share of what a real company still runs on. So the systems shipping in 2026 do what self-driving cars did: they use both. Anthropic built its Claude computer use tools around a strict order of operations, try a direct plug-in connection first, fall back to browsing in Chrome if no connection exists, and only take over the whole screen, picture by picture, if neither of those works (Anthropic, 2026). Apple's research team built something called UltraCUA from the opposite direction: one model trained to switch back and forth between clicking on things it sees and calling the software's own commands, whichever gets the job done. That combination beat vision-only agents by 22 percent on OSWorld while taking 11 percent fewer steps (UltraCUA, 2025). Both companies treat the camera as the fallback option inside a system built to try the faster, safer route first.
The same logic shows up in a self-driving car. The map is faster and more certain than the camera, so you use it whenever you can and save the camera for what the map cannot tell you. A direct connection resolves in seconds because the software hands over exactly what the agent needs. Browsing takes longer. Taking over the whole screen takes longest of all, because every single step needs a fresh picture, a guess about what that picture means, and a click that can miss. Ask a vendor which of the three modes their agent uses by default on your highest-volume task, not which one looked best in the demo.
The Math Only Works if the Job Is Short
On a short, well-defined task, an AI agent already beats a human worker on both price and results. One 2026 estimate puts the cost of running a computer-use agent at roughly $6 to $8 an hour, once you count everything, against about $10 an hour for an outsourced worker in India and $10 to $12 in the Philippines, with a Western back-office worker running $30 to $45 an hour (Andreessen Horowitz, 2026). For quick, repeatable tasks, that same estimate shows the agent already winning on both counts.
The math breaks down once the job runs long. A separate study built a version of OSWorld that measures how many steps an agent needs, not just whether it gets the answer right, and found that even the best agents take 2.7 to 4.3 times more steps than a person needs for the same job, and each extra step in a long task takes longer than the one before it (OSWorld-Human, 2025). Every extra step is another picture taken, another guess, another chance to click the wrong thing and have to try again. That is the real reason the OSWorld 2.0 score fell so far. An agent that needs four times as many steps also burns four times the money trying to recover from its own mistakes.
Ask a vendor what a completed task costs on your actual workflow, not what an hour costs on a demo task that ends after thirty steps.
A Screenshot Sees More Than the Agent Needs
A structure agent's traffic looks like a normal software request, the kind your security tools already know how to read. A vision agent's traffic is a picture of whatever is on the screen at that moment, which might include a customer record, a password field, or an open email that has nothing to do with the task at hand. That difference changes what your security team has to watch for. It is also why a well-known attack, hiding an instruction inside a webpage or a document, works differently against these agents than it does against a chatbot. If an agent takes a picture of a page that contains hidden, hostile text, that text becomes an instruction the agent might follow. Anthropic built a defense into its computer use tooling: an automatic check that scans each picture for signs of this kind of hidden instruction and pauses to ask a person before continuing (Anthropic, 2026).
The connector-first order doubles as a safety rule. A direct connection runs through a narrow, monitored channel built for one purpose. A browser session can see whatever is open in that browser. Full screen control can see whatever else is open on that machine. Ask any vendor which mode handles which kind of task by default, and require a person to approve anything that touches a payment, exports data, or changes a setting.
An agent that needs four times as many steps also burns four times the money trying to recover from its own mistakes.
A picture of the screen shows everything on it, including things the agent has no reason to see. That is why the connector-first order also works as a safety rule. Ask which mode handles which kind of task before you ask how accurate it is.
What We Still Don't Know
Self-driving cars ran into the same wall on their way to today's systems. A car could rack up a million safe miles on ordinary streets and still get confused by something it had never seen: a strange piece of road construction, an unusual hand signal from a police officer. The easy miles said nothing about the rare, hard ones, and it took years of driving into those rare cases to close the gap. Two questions decide whether AI agents follow the same path. The first is whether the fast climb from 20.6 percent to roughly 70 percent on OSWorld 2.0 came from agents getting better at the underlying long-horizon problem, or from vendors training on the kind of tasks this specific test uses, the same way OSWorld 1.0 scores climbed and then stopped meaning much. The second is whether OSWorld 2.0's hour-and-a-half tasks match what your company asks these agents to do, or whether the test has jumped ahead of what most companies are attempting today. Neither question has a clear answer yet.
Before your next vendor briefing on agentic automation, ask for the OSWorld 2.0 score as of this week, not the score from whenever the vendor's slide deck was made. This number is moving fast enough that a two-month-old figure is close to meaningless. Then ask what the agent does the moment its usual connection fails partway through a task. If the answer is a shrug toward taking over the whole screen with no person checking in along the way, you are looking at a demo wearing a production label.
Anthropic. "Let Claude use your computer in Cowork." Anthropic Help Center, 2026, support.claude.com/en/articles/14128542-let-claude-use-your-computer-in-cowork.
Anthropic. "Computer use tool." Claude Platform Docs, 2026, platform.claude.com/docs/en/agents-and-tools/tool-use/computer-use-tool.
Yuan, Mengqi, et al. "OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks." arXiv, 26 June 2026, arxiv.org/abs/2606.29537.
Abhyankar, Reyna, et al. "OSWorld-Human: Benchmarking the Efficiency of Computer-Use Agents." arXiv, 19 June 2025, arxiv.org/abs/2506.16042.
Yang, Yuhao, et al. "UltraCUA: A Foundation Model for Computer Use Agents with Hybrid Action." arXiv, 20 Oct. 2025, arxiv.org/abs/2510.17790.
Andreessen Horowitz. "Can Agents Use a Computer Yet? We've Got the Data." a16z, Aug. 2026, a16z.com/can-agents-use-a-computer-yet-weve-got-the-data/.
BenchLM.ai. "OSWorld 2.0 Leaderboard & Scores, August 2026." BenchLM, Aug. 2026, benchlm.ai/benchmarks/osworld2.
