Auto Browser on WebVoyager-mini, final runs: 92.7% at 1.3 cents a task.

With DeepSeek V4.1 Flash as its model, Auto Browser passed 82 of 89 graded tasks (92.1%) in one run and 84 of 90 (93.3%) in a second, on wv-mini, a 103-task subset of the WebVoyager benchmark. The mean is 92.7%. The model calls for all 103 tasks cost $1.33 and $1.40, about 1.3 cents a task.

Claude Sonnet 5.5 on the same build passed 79 of 91 graded tasks (86.8%), and its model calls cost $97.19, about 71 times as much. DeepSeek passed more tasks in both of its runs, but the gap is too small for this task set to separate the two models.

Our first post, on 2 November, reported 82.8% for DeepSeek. Since then we changed the agent and our test setup, and retired two tasks the live sites can no longer satisfy. What changed covers each. These are our last runs on WebVoyager: its tasks were written in early 2024, and more of them go stale as the sites change.

Every task’s answer, judge verdict, action log and final screenshot is in the results repository.

Setup

We tested the same way as in the first post. The tasks come from the same source, and the same judge grades them: GPT-5.5 (snapshot gpt-5.5-2026-04-23) with our own prompt. Each task gets one attempt and ten minutes. Tasks stopped by a bot check don’t count toward the pass rate, and cost is what OpenRouter charged for the agent’s model calls, without the judge. Five things changed.

• Build. All three runs used Auto Browser 1.6.0 with the agent changes described below, and only the model changed between runs. Three small fixes went into the release after the runs: the side panel puts the space back between paragraphs, the agent stops cleanly after three replies in a row it can’t read, and with the thinking switch off, the model reasons by its own default. We haven’t rerun the benchmark on the release itself.
• Tasks. 103 tasks instead of 105, after we retired two.
• Reasoning. For these runs we asked the model to reason on every call, with two deliberate exceptions: the last answer when the agent is about to hit its step or time limit, and the retry after a reply came back empty because it ran out of length. In a quick test, DeepSeek V4.1 Flash reasoned the same with or without the request, so this shouldn’t change its runs. On default settings, 1.6.0 sends no request either way, so DeepSeek reasons by its own default there too. Claude Sonnet 5.5 doesn’t reason unless asked, so its run here reasons and its run in the first post didn’t.
• Timing. We ran all three on 7 November 2026, between 16:07 and 22:14 Kuala Lumpur time, which is 00:07 to 06:14 US Pacific. The date was the same in Malaysia and the contiguous US the whole time, which matters for tasks that ask for the latest result or the last two days.
• Test setup. We fixed a fault in our test setup after the first post’s run.

Retired tasks. We drop a task when the live site can no longer satisfy it. We decide that by checking the task against the site, not by how the agent did. We don’t replace dropped tasks, so older runs can be rescored on the tasks that are left.

• allrecipes-3 asks for vegan chocolate chip cookies with over 60 reviews and a rating of at least 4.5 stars. Alumnium and Magnitude, the two published sets of task fixes we follow, both removed it, and our rule is to drop a task both removed unless we can show it’s still doable. We had kept this one because the recipe had 72 ratings, but the task asks for reviews, and it has only 59.
• huggingface-1 asks for a short story from the Hugging Face Inference API, which now needs a signed-in account. An anonymous request returned HTTP 401 on 7 November. omxyz’s task set drops it for the same reason.

The first post’s DeepSeek run failed both, so on the remaining 103 tasks it scores 84.6% (77 of 91). We also checked google-map-5, which none of our runs had ever passed. A separate agent that hadn’t seen our runs found a garage on Maps that fits the task, so it stays in. The second DeepSeek run passed it.

Results

Every task graded in the first DeepSeek run was also graded in Sonnet’s run, and 89 of the second run’s 90 were. On those shared tasks, the first DeepSeek run passed 6 that Sonnet failed, and Sonnet passed 3 that it failed. For the second run it was 7 and 1. Exact McNemar tests give p = 0.51 and p = 0.07, so neither run shows a difference between the models at the usual 5% level.

The two DeepSeek runs graded 88 tasks in common. 80 passed in both and 4 failed in both. One passed only in the first run and three only in the second.

What changed

Retiring two tasks accounts for 1.8 points of the difference: the first post’s run scores 84.6% on the final 103 tasks. The test setup also changed after that run, so its number is not a clean baseline for the agent changes, and we don’t try to split the rest of the gain between the agent and the setup.

Since 4 November. We ran the whole task set with DeepSeek five times on the current test setup. In build order the runs scored 83.7%, 85.6%, 88.9%, 92.1% and 93.3%. The first of them came after a change that let the agent click the same control again and again, which the next build stopped. Each step is within about one standard error, and the two final runs are the highest.

The agent. The main changes since the first post fall into six groups. None of them names a site, a task or an answer.

• Reading pages. The agent reads the page after each action on it instead of waiting to be asked. Screenshots it takes now reach the model on the default agent loop, which had dropped them. Page text keeps table cells apart, gives images their alt text instead of their data, and shortens very long link addresses. Text-only pages, such as JSON responses, come back as text, and find searches the whole page.
• Loops. Repeated clicks on one control, and short cycles of two to four actions, are stopped. A click that opens a new tab says so, and page snapshots show whether a control is checked or expanded.
• Instructions. The prompt asks the agent to pick the latest item by the page’s own dates, to copy names exactly, and to take answers from the site even when the model already knows them.
• Finishing. Before it answers, the agent checks its draft against the task. Near its step or time limit it is asked for its best answer, so a long task still ends with one. When it rereads a page it has already read in full, it is told to try another route on the same site.
• Model replies. After a reply is cut off for length, the allowance for later replies grows for the rest of the task. The parser now accepts replies with raw line breaks inside a string, or with a reminder copied after the decision.
• Cost. The prompt puts its fixed part first, where providers can cache it.

Cost

DeepSeek’s cost per task halved between the posts, from 2.6 cents to 1.3. Most of the drop came when we reordered the prompt so providers can cache it: our full runs before that cost 2.3 to 4.2 cents a task, and the runs from 4 November on cost 1.2 to 1.7. Sonnet cost 94 cents a task, against 87 in the first post, and its run now reasons.

DeepSeek’s median graded task cost about a cent, and its most expensive task cost 9 cents. Sonnet’s median graded task cost 71 cents and its most expensive cost $4.40. At these rates, 1,000 tasks like these would cost about $13 in model calls with DeepSeek and about $940 with Sonnet.

The per-task figures are approximate, as in the first post. We measure each task’s cost as the change in our OpenRouter key’s running total between the start of that task and the start of the next, and OpenRouter records a call’s cost about 30 seconds late. In these runs 4, 3 and 3 graded tasks show a cost of zero or less. The run totals are not affected.

DeepSeek is slower. It took a median of 13 to 13.5 actions per graded task against Sonnet’s 7, and 55 to 65 seconds against 45.

The judge costs more than DeepSeek. Each run sent GPT-5.5 between 1.1 and 1.3 million input tokens, about $6 to $7 at its list price, roughly five times what DeepSeek’s model calls cost.

Where it fails

DeepSeek failed four tasks in both runs.

• amazon-5 asks for a Blue iPhone 12 Pro 128GB in the cart. Amazon says the item cannot ship to the delivery location, which is in Malaysia, and offers only Add to List. Sonnet failed it the same way.
• google-search-6 asks for the monthly trending searches in Columbus. Google Trends’ trending page offers periods from 4 hours to 7 days. The first run answered with the past 24 hours, and the second reported that it found no monthly list. Sonnet failed it too, although its run for the first post found a monthly view through Trends’ Explore page.
• huggingface-2 asks for three new, popular open-source translation models released in the past month. Both answers included a model with no open-source licence shown, and the first also picked a model with a single like.
• wolfram-alpha-1 asks for a constraint on the inequalities for the inner region of a pentagram. Wolfram Alpha answered the agent’s queries with general facts about pentagons. One answer was not supported by the pages the agent read, and the other misstated the constraint.

Five more failed in one run and passed in the other, or were stopped by a bot check there. In the first run, the apple-3 answer left out a price the page showed, google-flights-3 never sorted the flights by emissions, and google-map-5 cited garages whose hours it could not confirm. In the second, amazon-1 found the item but did not save it, and amazon-4 found no listing in “Used - Good” condition.

Sonnet failed 12 tasks. Five of them passed in both DeepSeek runs. Three of those five were answers that missed one condition of the task: a repository with no licence for an open-source request, an arXiv search limited to physics where the task said all archives, and a course result not clearly about Python data science. On a fourth, allrecipes-5, the judge was not consistent: it failed Sonnet’s recipe for having 843 reviews where the task asked for more than 1,000, and passed the same recipe in DeepSeek’s second run.

Limitations

• Sample. wv-mini is 103 of WebVoyager’s 643 tasks. DeepSeek ran twice and Sonnet once. One run’s standard error is about 3 to 4 points, and live sites change between runs.
• Tuning. We made the agent changes while studying failures on these same tasks. Each change is general, and we checked each against tasks that already passed, but an agent usually scores higher on the tasks it was tuned against than on new ones. A held-out set of WebVoyager tasks, frozen before most of these changes, has not been run.
• Judge. The grade comes from an LLM judge with our own prompt. The prompt, and every verdict with its reason, are in the results repository. The judge is not always consistent: it passed DeepSeek’s second run on allrecipes-5 with a recipe it failed for Sonnet. Graded like Sonnet’s, that run scores 92.2% instead of 93.3%, and its split against Sonnet is 6 and 1 (p = 0.125).
• Bot checks. 12 to 14 tasks per run were stopped by a bot check and left out: all seven Cambridge Dictionary tasks and four Google Search tasks in every run, and one to three more depending on the run. Each pass rate covers the tasks that remained.
• Region. All runs came from Malaysia, and sites served their Malaysian versions. amazon-5 fails on that.
• Comparability. Top entries on Steel’s WebVoyager board report 97% to 99%, on larger task sets, each with its own judge and setup. Our score is not directly comparable to theirs. All results here are self-reported.

The evidence

The results repository holds these three runs: a summary and one row per task for each run, a record per task with the final answer, the judge’s verdict and reason, the final URL and every action the agent took, the final-page screenshots the judge saw, the 103 task files with their patch file, and the judge’s prompt and settings. Tool results and page text are left out of the action logs. The judge saw them. The first post’s runs are kept at the tag first-post-2026-11-02.

Try it

Auto Browser 1.6.0, with the agent changes tested here, is on the Chrome Web Store, and DeepSeek V4.1 Flash is its default model on OpenRouter.

Related: Auto Browser on WebVoyager-mini: 82.8% at 2.6 cents a task and How to tell if a browser agent is any good.