---
title: "What the mobile agent leaderboard hides"
description: "Lusor's AndroidWorld run passed 116/116 tasks on Mobilerun's fork of the grader and 114/116 on Google's original grader. What that score does not measure, what Lusor publishes so it can be checked, and the questions to ask of any mobile agent leaderboard."
canonical: "https://lusor.ai/blog/what-the-leaderboard-hides"
updated: "2026-11-14"
---

[Blog](https://lusor.ai/blog) / [Insights](https://lusor.ai/blog/insights)

# What the mobile agent leaderboard hides

A score of 116/116 on Mobilerun's fork of the AndroidWorld grader is a real result and a poor answer to the question a developer actually has. The number says a grader passed every task. It does not say phones are solved, and it cannot tell you which of those two things is true.

Benchmark · November 14, 2026 · 8 min read

## TL;DR

- Lusor's AndroidWorld run passed 116/116 tasks on Mobilerun's fork of the grader and 114/116 on Google's original grader.
- A perfect score is the point where a benchmark stops distinguishing anything. It is not proof that mobile automation is solved.
- The score leaves four things out: step budgets (22 of 116 tasks went over theirs), apps outside the benchmark's 21, logged-in personal accounts and first-run setup cost.
- Lusor publishes all 116 trajectories, a recording for every task, both graders with their commits, and the per-task data as JSON and CSV.
- Before trusting any mobile agent score, ask which grader produced it, how many trials each task got, whether the step budget was enforced and what the cost figure includes.

A leaderboard score looks like an answer. Lusor's AndroidWorld run passed 116 of 116 tasks on Mobilerun's fork of the AndroidWorld grader, and 114 of 116 on Google's original grader. The median task took 48.2 s, and the model calls for the whole run cost $2.95. Those numbers are measured, and every one of them is on [the benchmark page](https://lusor.ai/benchmarks/androidworld).

They are also not proof that mobile automation is solved. A perfect score on a public benchmark is the point where the instrument stops distinguishing anything: there is no headroom left to measure, and the tasks that were not covered are not visible in the total. This post is about the gap between a number and a working phone agent, and about what a developer should ask before trusting either.

## What the run was

AndroidWorld is a public benchmark from Google Research: 116 tasks in real Android apps, each graded by checking the phone's state, or the agent's answer, after the run. Lusor ran all 116 tasks across 21 apps from Mobilerun's public copy of the benchmark, with seed 42, one trial per task. DeepSeek V4.1 Flash at low reasoning effort chose every action, inside the Lusor Harness, which supplies the screen reading, the phone actions, the list-reading tools and the checks on particular actions and saved values. Mobilerun's grader scored every run as it went; Google's unmodified grader then rescored the same runs. Lusor's [AndroidWorld run report](https://lusor.ai/blog/lusor-on-androidworld) covers the setup, the results and three example tasks.

Each task started from the same Android snapshot, checked before it ran, and 116 of 116 matched. Every task was recorded. The score belongs to the Lusor Harness at the build in production on 9 November 2026.

## The two graders, side by side

AndroidWorld is graded by code, so the code decides the score. Two versions of that code were applied to the same runs, and they disagree on two tasks.

|  | Mobilerun's fork of the grader | Google's original grader |
| --- | --- | --- |
| Passed | 116/116 | 114/116 |
| Grader code | Mobilerun's fork, c5467e6 | Google's original, 30583e2 |
| Tasks failed | None | Two |

The two disagreements are specific enough to read in the grader code. On one, Google's original grader looks for the recording name typed in full, and the app stores that as …m4a.m4a. That double extension is a known bug in the grader, not something a person would set out to produce, and Mobilerun's fork of the grader accepts the file the harness saved. On the other, the task asks for the expense notes a file marks as reimbursable, the harness copied each note as written including its “Reimbursable.” marker, and Google's original grader expects the marker removed.

Neither version is wrong to exist. Mobilerun's grader is the public copy of AndroidWorld at c5467e6, and Google's is google-research/android\_world at 30583e2. The copy changes how two tasks are graded and rewords three task requests, which makes it a different measurement from Google's, and the run used its task text. A reader comparing scores across systems needs to know which grading produced each number. That is the whole reason the grader is named in the sentence beside every score on this page.

## What the benchmark does not measure

Four things sit outside the score, and each of them matters more to someone choosing a phone agent than the pass rate does.

**Step budgets.** AndroidWorld sets a step budget for each task. Lusor's run did not enforce that budget, and 22 of the 116 tasks used more device actions than theirs. The grader does not care: it asks whether the task is done, not how many actions it took. Reading the score alone, a task that finished well past the intended limit looks identical to one that finished inside it.

**Apps outside the 21.** Every task in the benchmark runs in one of 21 Android apps, and the score is a statement about those 21. It says nothing about the banking app, the messaging app or the internal tool on a customer's phone, and nothing about the app that ships next month.

**Logged-in personal accounts.** Every task started from the same clean Android snapshot. No task begins inside an account that a person already uses, with their history, their notification backlog and their two-factor prompts. A real phone agent has to work in exactly that state, and has to not break it. The benchmark cannot tell you how well it will. What an agent can see on such a phone, and who can open the record afterwards, is covered in [phone agent safety](https://lusor.ai/blog/is-it-safe-to-let-an-agent-use-your-phone).

**First-run setup cost.** The $2.95 is the measured model cost of all 116 tasks, including the helper checks. It is not a subscription price, and it leaves out hardware and infrastructure. Nothing in the run prices the part a new user meets first: enrolling a phone, connecting it to a workspace, and getting the agent to a point where it can act at all. The setup guides for [connecting an iPhone](https://docs.lusor.ai/help/connect-your-iphone) and [connecting an Android phone](https://docs.lusor.ai/help/connect-your-android) list those steps.

## What Lusor publishes that a leaderboard row does not

The claim that a benchmark was passed is cheap. The evidence that it was passed is not, and it is the evidence that is usually missing. On this run, Lusor publishes:

- All 116 trajectories, each with the agent's steps, times and costs on a timeline, not a sample and not the interesting ones.
- A recording for every task. The steps show what the agent did; the video shows what was on the screen when it did it.
- Both graders, named with the commit of each, and the two tasks they read differently, explained rather than rounded off.
- The full per-task data as [a browsable task explorer](https://lusor.ai/benchmarks/androidworld/tasks) and as JSON and CSV, so a reader can recompute any figure on the page instead of trusting it.
- The method, step by step, including the parts that do not flatter the result: the step budget that was not enforced, the one trial per task, and the fact that Lusor ran its own evaluation.

On the same page, the other entries publish per-task steps without video, and none of them publishes a cost. That is not a reason to disbelieve anyone. It is a reason to treat a leaderboard row as a claim with a link, and to ask what the link contains before acting on it.

## What to ask of any mobile agent score

These are the questions that decide whether a number is usable, in the order that makes a difference. They apply to Lusor's score as much as to anyone else's.

- Which grader produced it: Google's original AndroidWorld grader, or a fork of it? A fork can change what counts as passing, and then the scores are not the same measurement.
- Whose task text was used? A copy that rewords task requests is a different test, even when the tasks share names.
- One trial per task or best of several? A single run per task is a pass rate. A best of many runs is a maximum, and the two are not comparable.
- Was the step budget enforced, and how many tasks went over it? A pass rate that hides a longer action path hides the cost of that path.
- What does the cost figure include: model calls, helper calls, retries, hardware, the device itself? Model cost alone is not the price of running the agent.
- Is there a recording per task, or only a summary? Steps without video cannot show what the screen looked like when the agent decided.
- What state did each task start from? The same clean snapshot on every task is a different problem from a phone with a person's accounts on it.
- Who ran the evaluation? Self-reported results are the norm on every such benchmark, including this one, and that is a reason to ask for the data rather than to reject it.

## Read the run

The full run, both graders, the per-task data and every recording are on [the AndroidWorld benchmark page](https://lusor.ai/benchmarks/androidworld). The score there is 116/116 on Mobilerun's fork of the grader and 114/116 on Google's original grader, at a median of 48.2 s per task for $2.95. It is a good result, and it is not the end of the question.

To see the same agent on a phone that is not running a benchmark, [start with a plan](https://lusor.ai/pricing) or connect Lusor to the assistant already in use from [the connect page](https://lusor.ai/connect).

## Frequently asked questions

### What did Lusor score on AndroidWorld?

116 of 116 tasks on Mobilerun's fork of the AndroidWorld grader and 114 of 116 on Google's original grader, on the same runs. The median task took 48.2 s, and the model calls for the whole run cost $2.95.

### Why do the two AndroidWorld graders give different scores?

They disagree on two tasks. On one, Google's original grader looks for the recording name typed in full, which the app stores as …m4a.m4a, while Mobilerun's fork of the grader accepts the file the harness saved. On the other, the harness copied each expense note as written, including its “Reimbursable.” marker, and Google's original grader expects the marker removed.

### Does a perfect AndroidWorld score mean phone automation is solved?

No. A perfect score on a public benchmark is the point where the instrument stops distinguishing anything. The score is a statement about 21 Android apps, each task started from the same clean snapshot, and it says nothing about other apps or about a phone with a person's own accounts on it.

### Did the run enforce AndroidWorld's step budgets?

No. Lusor's run did not enforce the step budget, and 22 of the 116 tasks used more device actions than theirs. The grader asks whether the task is done, not how many actions it took.

### What does the $2.95 cost figure include?

It is the measured model cost of all 116 tasks, including the helper checks. It is not a subscription price, and it leaves out hardware and infrastructure.

### What does Lusor publish about the run?

All 116 trajectories, each with the agent's steps, times and costs, a recording for every task, both graders named with the commit of each, and the full per-task data as a browsable task explorer and as JSON and CSV.

## Keep reading

- [Lusor + DeepSeek: 116/116 on AndroidWorld for $2.95](https://lusor.ai/blog/lusor-on-androidworld): Lusor's production agent on open-weight DeepSeek V4.1 Flash passed 116/116 AndroidWorld tasks on Mobilerun's grader (114/116 on Google's), 5.2× faster per task than Mobilerun's published times, for $2.95 against Lusor's estimate of about $108.70 for Mobilerun.
- [Your agent says done: here is how to check](https://lusor.ai/blog/your-agent-says-done-how-to-check): An agent saying done is a claim. How to check it in Lusor: the video recording, the step-by-step replay with a screenshot after each action, the Replay page's exports, the recording and replay API endpoints, and the limits of a recording.
- [Stop copying app data by hand: have an agent build the sheet](https://lusor.ai/blog/stop-copying-app-data-by-hand): Useful data often lives only inside phone apps, with no export and no API. How a Lusor agent reads it on a phone and saves it as rows in a Sheet, how the rows get out as CSV, TSV, JSON or Markdown, and what to check before relying on them.
