Auto Browser on WebVoyager-mini: 82.8% at 2.6 cents a task.
With DeepSeek V4.1 Flash, Auto Browser completed 77 of 93 graded tasks on a 105-task WebVoyager subset for $2.75 in model calls. Claude Sonnet 5.5 on the same agent scored within noise of that, for 33 times the cost.
With DeepSeek V4.1 Flash as its model, Auto Browser completed 77 of 93 graded tasks (82.8%) on wv-mini, a 105-task subset of the WebVoyager benchmark. The model calls for all 105 tasks cost $2.75, which is 2.6 cents a task.
We ran the same agent on the same tasks with Claude Sonnet 5.5, graded by the same judge. It completed 73 of 92 graded tasks (79.3%), and its model calls cost $91.55. On this task set the two scores are within noise of each other, and DeepSeek costs 33 times less.
Every task’s answer, judge verdict, action log and final screenshot is in the results repository.
Setup
Agent. Auto Browser runs a language model in a loop inside Chrome. At each step the model reads the page as a text snapshot of its elements or its text, then acts through tools such as click, fill, navigate and take_screenshot. Both runs used the same development build: Auto Browser 1.5.1 with five agent fixes that are not released yet. Only the model changed. Both models were called through OpenRouter.
Tasks. WebVoyager (He et al., 2024) has 643 tasks on 15 live websites. wv-mini takes the first seven tasks from each site, in WebVoyager’s order, and applies the task fixes that Alumnium published. We left out four tasks, two of them because finishing would make a real booking or purchase, and took the next task from the same site in their place. We changed five more: three no longer matched their site, and two had dates in the past, which we replaced with dates relative to the day of the run. The patch file lists every change with its reason.
Grading. An LLM judge grades each task: GPT-5.5 (snapshot gpt-5.5-2026-04-23, medium reasoning effort) with our own prompt. The judge reads the task, the date of the run, the agent’s final answer, the agent’s actions with the page text it read, the final page’s URL and text, and a screenshot of the final page when one was captured. A task passes only when every condition in it is met and the answer comes from the website. An answer the model gave from its own knowledge fails, even when it is correct. WebVoyager’s original judge model, GPT-4V, has been retired, so recent entries also grade with newer models and adjusted prompts. Ours differs from the original prompt by showing the judge the pages the agent read, which lets it check where an answer came from.
Bot checks. When a site shows a bot check, the test harness stops the task and leaves it out of the pass rate. The agent is never allowed past one. In both runs this removed all seven Cambridge Dictionary tasks (Cloudflare) and five of the seven Google Search tasks. It also removed one Hugging Face task from the Sonnet run.
Protocol. Each task had one attempt and a ten-minute limit. We ran from Kuala Lumpur, Malaysia: Sonnet on 31 October 2026 and DeepSeek on 1 and 2 November 2026. Sites served their Malaysian versions, so some prices appear in ringgit.
Cost. Cost is the spend OpenRouter recorded for the agent’s model calls during the run. It does not include the judge.
Results
| DeepSeek V4.1 Flash | Claude Sonnet 5.5 | |
|---|---|---|
| Passed / graded | 77 / 93 | 73 / 92 |
| Pass rate (95% Wilson interval) | 82.8% (73.9 to 89.1) | 79.3% (70.0 to 86.4) |
| Tasks stopped by a bot check | 12 | 13 |
| Model cost, 105 tasks | $2.75 | $91.55 |
| Cost per task | $0.026 | $0.87 |
| Cost per passed task | $0.036 | $1.25 |
| Median cost per task | $0.010 | $0.54 |
| Median actions per graded task | 10 | 5 |
| Median time per graded task | 54 s | 38 s |
Both runs graded the same 92 tasks. 69 passed in both runs and 11 failed in both. DeepSeek passed 8 tasks that Sonnet failed, and Sonnet passed 4 that DeepSeek failed. An exact McNemar test on that split gives p = 0.39, so this task set cannot separate the two models on accuracy.
| Site | DeepSeek V4.1 Flash | Claude Sonnet 5.5 |
|---|---|---|
| Allrecipes | 6 / 7 | 6 / 7 |
| Amazon | 5 / 7 | 4 / 7 |
| Apple | 6 / 7 | 6 / 7 |
| ArXiv | 6 / 7 | 6 / 7 |
| BBC News | 6 / 7 | 5 / 7 |
| Booking | 6 / 7 | 7 / 7 |
| Cambridge Dictionary | all 7 stopped by a bot check | all 7 stopped by a bot check |
| Coursera | 7 / 7 | 6 / 7 |
| ESPN | 6 / 7 | 6 / 7 |
| GitHub | 7 / 7 | 7 / 7 |
| Google Flights | 5 / 7 | 5 / 7 |
| Google Map | 6 / 7 | 5 / 7 |
| Google Search | 1 / 2 (5 stopped) | 2 / 2 (5 stopped) |
| Hugging Face | 6 / 7 | 6 / 6 (1 stopped) |
| Wolfram Alpha | 4 / 7 | 2 / 7 |
Cost
OpenRouter lists DeepSeek V4.1 Flash at $0.15 per million input tokens and $0.60 per million output tokens, and Claude Sonnet 5.5 at $2 and $10 (prices on 2 November 2026). Per token, DeepSeek is 13 to 17 times cheaper. Over the run the measured gap was 33 times, although DeepSeek took about twice as many actions per task. We did not record token counts, so we cannot say where the rest of the gap comes from.
Half of DeepSeek’s tasks cost about a cent or less, and its most expensive task cost 15 cents. Sonnet’s median task cost 54 cents and its most expensive cost $5.72. At these rates, 1,000 tasks like these would cost about $26 in model calls with DeepSeek and about $870 with Sonnet.
The saving costs some speed. DeepSeek’s median task took 54 seconds against Sonnet’s 38.
With DeepSeek, grading costs more than the agent. The judge’s GPT-5.5 calls for the DeepSeek run cost about $6, more than twice the agent’s $2.75.
Where it fails
DeepSeek failed 16 graded tasks. By our reading of the judge’s reasons, seven were answers that missed a condition of the task, such as a recipe with 59 reviews where the task asked for more than 60, or a parking lot reported as a garage. Three were filters or dates that never took effect on the page. In two, the answer did not come from the site: the agent gave an integral without taking a single action on Wolfram Alpha, and one news answer cited an article the agent had not opened. One misread a price graph. Three ended without a result: a login wall, an item that was never added to the cart, and a run that stopped with no final answer. Sonnet failed 11 of the same tasks.
Limitations
- Sample. wv-mini is 105 of WebVoyager’s 643 tasks, and each model ran once. One run’s standard error is about 4 points, and live sites change between runs.
- Judge. The grade comes from an LLM judge with our own prompt. The prompt, and every verdict with its reason, are in the results repository. A final screenshot was captured for 57 of DeepSeek’s 93 graded tasks and 88 of Sonnet’s 92. For the rest, the judge had the page text and the action log.
- Bot checks. 12 to 13 tasks were stopped by bot checks and left out, so each pass rate covers the tasks that remained.
- Region. All runs came from Malaysia, and sites served local versions.
- Comparability. Top entries on Steel’s WebVoyager board report 97% to 99%, on larger task sets, each with its own judge and setup. Our score is not directly comparable to theirs. All results here are self-reported.
The evidence
The results repository holds both runs: a summary and one row per task for each model, a record per task with the final answer, the judge’s verdict and reason, the final URL and every action the agent took, the final-page screenshots the judge saw, the 105 task files with their patch file, and the judge’s prompt and settings. Tool results and page text are left out of the action logs. The judge saw them.
Try it
Auto Browser is on the Chrome Web Store, and DeepSeek V4.1 Flash is its default model on OpenRouter.