Domain: Finance
Benchmark: SheetBench-50 (50 spreadsheet tasks in finance and accounting) Metric: avg@3

*Opus results will be slightly updated after the SDK fix lands.

†Muse token figures are rounded observed lower bounds: usage is available for 22,144 responses versus 22,155 gateway calls. The remaining 11 calls have unverified token usage.

Method

• Each model runs all 50 tasks × 3 attempts, with up to 200 model turns per attempt. A turn can include several browser actions.
• The agent works only through a browser, in a fresh Google Sheet built from each task's Excel file. The score is the share of expected answer cells that exactly match; the CI is a task-level bootstrap.
• All models were called through HUD's inference gateway using HUD's default settings for each model. The jobs submitted no model-parameter overrides.
• Scores average graded attempts. Attempts lost to infrastructure errors (provider timeouts, quota limits) are excluded rather than scored as 0. Cost covers HUD inference and environment, duration is the median, and tokens are averaged over all 150 attempts.
• Muse is complete at 50 tasks × 3 graded attempts: 61 original grades and 89 replacements for failed attempts. Its score, cost, duration and tokens use the same selected 150 traces. All 66 attempts that reached the 200-turn cap retain their grades. Cost excludes discarded retries and separate Anchor/Google charges; duration excludes queue time. Its CI is a task-level percentile bootstrap with 1,000,000 resamples.

Longer budgets (400 turns)

Some models often ran out of turns at 200, before finishing the task. Grok stopped at the cap on 112 of 144 graded attempts, Gemini on 48, and DeepSeek on 28. We reran these models with 400 turns to give them more room:

• Grok 4.7: full 50-task × 3 run, 33.5% (vs 14.3% at 200 turns).
• Gemini 3.8 Flash: 5-task pilot, 24.2% on those tasks (vs 15.9% at 200 turns).
• DeepSeek V4.1 Flash: some of its reruns in the table above used 400 turns.

HUD jobs

For tasks with failed attempts, we ran targeted follow-up batches in separate HUD jobs, sometimes only a few tasks at a time. Grok's 400-turn job is a fully separate run. Some of the reported results include targeted runs we launched locally with harness fixes ahead of the platform release. The taskset and grading stayed the same. We aim to reconcile the run record around Opus 5.5 by Monday, 12 oct EOD. Muse Spark 1.3 is complete; its selected-attempt audit below links every contributing trace and job. GLM-5.3 remains unresolved.

• Muse Spark 1.3 — selected 150-attempt audit