Agents complete tasks across enterprise apps using screenshots, clicks, typing and scrolling.
This benchmark covers five retail/customer-support workflows in Riverbend and five site-reliability workflows. Records connect across running application clones. Tasks require finding records, comparing evidence and saving an answer in the requested app.
Each rollout is one attempt by one model at one reasoning setting. It starts from the task’s seeded state. The model receives the task brief and tool instructions, not the grading answer.
These are screen-based runs. Direct database queries, app APIs and hidden browser state are outside the allowed solver workflow. The grading system can inspect saved application state after the attempt.
Each task has a weighted rubric: a checklist whose weights add up to 1. The judge uses the collected application changes and rollout evidence to assess each criterion. Partial credit means some requested work was completed correctly.
The rubric is interpreted by an AI judge, including checks that could also be written as exact comparisons. A criterion labelled “binary” means its score is yes or no; it does not mean the check is deterministic code.
For example, the sample task checks the destination sheet, headers, three response counts, a total and whether unrelated Zendesk records changed. The includes the brief and all seven weighted criteria.
Scores alone do not establish that a run was free of app, harness or grading problems. Infrastructure failures and task revisions need separate review; an incomplete run must not be presented as a clean completed attempt.
Cost per rollout is the mean model inference cost among attempts with cost evidence. The table shows that denominator. Missing prices are unknown, not zero. The cost Pareto chart requires complete cost coverage for a configuration; it excludes incomplete configurations rather than comparing different subsets. Error bars preserve historical cost ranges.
Two attempts per workflow are still a small sample. Treat the ranking and Pareto chart as descriptive, not a statistically reliable ordering between neighbouring models.
Every configuration uses its computer-use adapter. The postmortem workflow is excluded until its clean rerun finishes.
✓ means at least one of two counted attempts earned full credit; 1/2 means one attempt, 2/2 means both. Pending results are not ranked.
| Configuration | Resources | Score | Position |
|---|
| Model | Reasoning | Mean known USD | Priced / counted | Cost source |
|---|
Counted-attempt model inference only. Excludes earlier failed attempts, EC2 and grading. Missing cost ≠ free.
FrontierCUA is Collinear AI’s benchmark for computer-use agents working across enterprise apps.
The agent has to find every Q1 2026 closed-out Zendesk ticket where the customer wrote again after the first reply, bucket what happened next, and write the counts into a new Google Sheets tab.
None of the default Zendesk views show closed or solved tickets, so the population has to be found before it can be counted.
| Graded output | Required value |
|---|---|
| Never answered | 35 |
| Answered same or next day | 34 |
| Answered two days later | 11 |
| Tickets with a follow-up, cell E1 | 80 |
Every bucket boundary is a trap: folding “never answered” into “two days later” gives 46, which the rubric names and refuses.
An agentic judge scores the collected application state diffs and the trajectory against a rubric whose weights sum to 1.000.
| Criterion | Passing behavior | Weight |
|---|---|---|
| second_response_tab_created | A sheet named exactly “Second Response” inside the pre-existing Support Volume Tracker, not a new separate spreadsheet. | 0.03 |
| second_response_headers_present | A header row naming the outcome column and the count column, in any reasonable wording. | 0.02 |
| row_never_answered_is_35 | The never-answered row reads 35, graded by row identity rather than index, and 46 must not pass. | 0.22 |
| row_within_1day_is_34 | The same-or-next-day row reads 34. | 0.30 |
| row_two_days_is_11 | The two-days-later row reads 11, and no ticket in the cohort took longer. | 0.08 |
| followups_total_e1 | Cell E1 reads 80, which is 35 + 34 + 11, not the 108 closed-out tickets. | 0.26 |
| no_unrequested_changes | Critical guard: the Zendesk state diff shows no ticket, comment, user or organization row added, modified or removed. | 0.09 |
Loading recorded trajectories…
Preferences are saved in this browser and do not affect benchmark results.
Collinear AI’s benchmark for computer-use agents working across enterprise apps.
New workflows are evaluated across multiple frontier models before inclusion.
Trajectory audits are ongoing for the two counted attempts per task and configuration.
Additional applications and domains beyond service-desk and retail workflows are planned.
Start with a small task: open the customer note, then file it in Follow up.
This introduction is separate from the tasks used to evaluate the models.