Lusor harness + DeepSeek: 116/116 on AndroidWorld for $2.95
Compare the Lusor harness to other mobile app and phone-use harnesses on how well they perform on the AndroidWorld benchmark.
- Updated Nov 10
- Trials: 1 per task (pass@1)
- 116 tasks
Results
What is the best agent for Android tasks?
On AndroidWorld, 4 systems publish enough to compare. The Lusor harness, with DeepSeek V4.1 Flash choosing its actions, passes all 116 tasks on Mobilerun’s fork of the grader and 114 on Google’s original grader, with a median of 48.2 s per task and $2.95 for the whole run.
- Passed
- 116/116
- 114/116 on Google’s original grader
- Time per task
- 48.2 s
- 5.2× faster than Mobilerun
- Cost, all 116 tasks
- $2.95
- ~37× cheaper than Mobilerun
| Use case | Best pick | Measured result |
|---|---|---|
| Highest success rate |  Lusor harness Mobilerun agent-qa(tie) | Lusor harness 116/116 on Mobilerun’s fork of the grader; Mobilerun and agent-qa report 100%; Artemis (Google) reports 99.1%, grader not stated. |
| Graded by both graders | Lusor harness | 116/116 on Mobilerun’s fork of the grader, 114/116 on Google’s original grader, which reads two instructions more literally than the harness did ¹. |
| Fastest per task | Lusor harness | Median 48.2 s, against 120.5 s for agent-qa and 216.9 s for Mobilerun. |
| Lowest cost | Lusor harness | $2.95, ~37× cheaper than Mobilerun (~$108.70, our estimate); no other entry publishes enough to estimate ². |
| Video of every run | Lusor harness | Video and steps for all 116 tasks; Mobilerun and agent-qa publish per-task steps without video. |
Other systems’ rates are from the AndroidWorld community sheet (exported Nov 10, 2026); see the note under Full results.
Leaderboard
Share of the 116 tasks passed, per grader. Each system appears under the grader it reported on.
Success rate
AndroidWorld, 116 tasks · Mobilerun’s fork of the grader
- Our run
- Reported by its team
- 95% confidence interval
- Lusor harnessDeepSeek V4.1 Flash, low effort1.000 (116/116)95% CI [0.968, 1.000]
- MobilerunGPT-6 Astra · reference1.00
Success rate over 116 tasks. The Lusor harness’s interval is a 95% Wilson interval; the references are drawn as their teams report them, with no interval. The axis starts at 0.90. Every figure is as its team reports it (see the note under Full results).
AndroidWorld, 116 tasks · Google’s original grader
- Our run
- Reported by its team
- 95% confidence interval
- Lusor harnessDeepSeek V4.1 Flash, low effort0.983 (114/116)95% CI [0.939, 0.995]
- agent-qa (Vostride AI)gpt-6-sol · reference · same Google grader code as ours1.00
The Lusor harness’s two misses: Google’s grader reads those instructions more literally than the harness did (see ¹). agent-qa ran Google’s original task text, we ran the fork’s, which rewords three task requests.
Artemis (Google) reports 99.1%; grader not stated.
Success against cost and speed
Higher and further left is better on both charts. The Lusor harness is the only entry with a measured cost and a confidence interval.
By cost per run
Cost of all 116 tasks, log scale
By median time
Median seconds per task
Artemis (Google) publishes neither a cost nor per-task times, so it isn’t plotted. agent-qa publishes no cost, so it appears only on the time chart.
Full results
Performance of systems side by side.
Measured performance
JSON| System | Mobilerun’s fork of the grader ↑ | Google’s original grader ↑ | Median time ↓ | Cost, 116 tasks ↓ |
|---|---|---|---|---|
| Lusor harnessDeepSeek V4.1 Flash, low effort | 116/116[96.8, 100] | 114/116 ¹[93.9, 99.5] | 48.2 s | $2.95~37× cheaper than Mobilerun |
| agent-qa (Vostride AI)gpt-6-sol · reference | — | 100% | 120.5 s ³ | — ² |
| MobilerunGPT-6 Astra · reference | 100% | — | 216.9 s | ~$108.70our estimate |
 Artemis (Google)Gemini 3.7 Flash + Gemini Robotics ER 2 · reference · 99.1%, grader not stated | — | — | — | — |
Ranked by reported success rate, ties broken by median time. Each rate sits under the grader it was reported on. Highlighted: the best value in each column, among the values in that column; tied values share it. Brackets are 95% Wilson intervals, in percent.
¹ Google’s original grader failed two tasks the Lusor harness completed: it looks for the recording name typed in full (….m4a.m4a), and for the expense notes without the “Reimbursable.” marker. Mobilerun’s fork of the grader accepts the harness’s reading on both.
² agent-qa publishes per-task action counts but not its reasoning effort or how many model calls each step uses (a separate verifier checks every step), so we don’t estimate its cost.
³ agent-qa’s published “Time” has no documented method, so this comparison is indicative.
FluizAI, Kirk Engine and AGI-0 also report 100% on the AndroidWorld community sheet (exported Nov 10, 2026). They publish no per-task data, so they aren’t compared.
Every figure here, ours included, is reported by the team that ran it; the AndroidWorld community sheet does not verify entries. We publish all 116 of our runs so ours can be checked.
The Method
DeepSeek + the Lusor harness
DeepSeek is the decision-maker, but it does not work alone. The Lusor harness gives it tools for reading and navigating apps, plus small, focused checks around particular actions and intent.
Read screen
Choose action
Check
Act
Verify
Repeat
| Role | Who | What it does |
|---|---|---|
| Decides | DeepSeek V4.1 Flash (open-weight, low effort) | Interprets the goal and chooses each tool action. |
| Supplies and checks | Lusor harness | Screen information, phone actions and list-reading tools that gather information across multiple screens. Focused checks compare actions and saved values with the request and what the agent has read, so the model does not simply declare success. |
The reported cost includes the harness’s helper calls. This is a result for that complete system, not a claim about DeepSeek running without a harness.
How long does each agent take per task?
The Lusor harness (DeepSeek V4.1 Flash, open-weight, low effort) against Mobilerun (GPT-6 Astra, as published) and agent-qa (gpt-6-sol), the two we could compare per task. FluizAI also publishes step timings; we have not compared them.
Median time per task, by system
116 tasks each.
Seconds · lower is better · scale 0–240 s
Data for figure 1
| System | Median per task | Total, 116 tasks | Source |
|---|---|---|---|
| Lusor harness · DeepSeek V4.1 Flash | 48.2 s | 8,617.8 s | This run |
| agent-qa · gpt-6-sol | 120.5 s | 20,729 s | vostride.com |
| Mobilerun · GPT-6 Astra | 216.9 s | 30,768.3 s | Task explorer |
What does a task cost?
The Lusor harness’s actual cost per task with DeepSeek V4.1 Flash, including its helper checks, against our estimate of Mobilerun’s with GPT-6 Astra. Mobilerun does not publish token use or a bill.
Median cost per task
116 tasks each.
US dollars · lower is better · scale $0–0.75
Data for figure 2
| System | Median per task | All 116 tasks | Basis |
|---|---|---|---|
| Lusor harness · DeepSeek V4.1 Flash | $0.017 | $2.95 | Actual |
| Mobilerun · GPT-6 Astra (low) | ~$0.689 | ~$108.70 | Our estimate |
Is the Lusor harness faster on every task?
No. Each task’s time divided by Mobilerun’s published time for the same task: below 1.0, the Lusor harness was faster. It was slower on 5 of 116.
The Lusor harness’s time as a share of Mobilerun’s, per task
One bar per task, sorted · lower is better.
Data for figure 3
| Band | Ratio | Tasks | Share |
|---|---|---|---|
| 5× faster or more | 0.20 or less | 62 | 53% |
| 2–5× faster | 0.20–0.50 | 41 | 35% |
| 1–2× faster | 0.50–1.00 | 8 | 7% |
| Slower | above 1.00 | 5 | 4% |
| Quartiles | 0.150 · 0.193 · 0.321 | 116 | 100% |
Which apps take longest?
The median time per task in each of the 21 apps. The Lusor harness was slower in one app: Tasks, where 4 of its 6 tasks took longer than Mobilerun’s published times.
Median seconds per task, by app
App · number of tasks. Shorter is faster.
- Lusor harness
- Mobilerun (published)
- Simple Calendar · 17Lusor harness: 42.2 sMobilerun: 170.3 s
- Settings · 15Lusor harness: 24.2 sMobilerun: 125.6 s
- Markor · 14Lusor harness: 40.7 sMobilerun: 220.9 s
- Broccoli · 13Lusor harness: 108.2 sMobilerun: 429.7 s
- Pro Expense · 9Lusor harness: 72.2 sMobilerun: 366.7 s
- Tasks · 6Lusor harness: 175.8 sMobilerun: 146.9 s
- OpenTracks · 6Lusor harness: 108.2 sMobilerun: 263.3 s
- Simple SMS · 6Lusor harness: 24.5 sMobilerun: 173.1 s
- Joplin · 4Lusor harness: 22.7 sMobilerun: 142.3 s
- Retro Music · 4Lusor harness: 70.9 sMobilerun: 633.5 s
- Clock · 3Lusor harness: 21.1 sMobilerun: 122.1 s
- Browser · 3Lusor harness: 90.2 sMobilerun: 362.9 s
- OsmAnd · 3Lusor harness: 117.2 sMobilerun: 212.6 s
- VLC · 2Lusor harness: 94.7 sMobilerun: 375.8 s
- Audio Recorder · 2Lusor harness: 34.8 sMobilerun: 199.0 s
- Camera · 2Lusor harness: 49.7 sMobilerun: 196.1 s
- Files · 2Lusor harness: 37.7 sMobilerun: 292.6 s
- Contacts · 2Lusor harness: 43.7 sMobilerun: 312.2 s
- Gallery · 1Lusor harness: 51.2 sMobilerun: 266.3 s
- Simple Draw Pro · 1Lusor harness: 42.2 sMobilerun: 275.3 s
- Launcher · 1Lusor harness: 24.2 sMobilerun: 178.4 s
Data for figure 4
| App | Tasks | Lusor harness, median | Mobilerun, median | Median ratio |
|---|---|---|---|---|
| Simple Calendar | 17 | 42.2 s | 170.3 s | 0.193 |
| Settings | 15 | 24.2 s | 125.6 s | 0.217 |
| Markor | 14 | 40.7 s | 220.9 s | 0.177 |
| Broccoli | 13 | 108.2 s | 429.7 s | 0.381 |
| Pro Expense | 9 | 72.2 s | 366.7 s | 0.153 |
| Tasks | 6 | 175.8 s | 146.9 s | 1.198 |
| OpenTracks | 6 | 108.2 s | 263.3 s | 0.358 |
| Simple SMS | 6 | 24.5 s | 173.1 s | 0.148 |
| Joplin | 4 | 22.7 s | 142.3 s | 0.171 |
| Retro Music | 4 | 70.9 s | 633.5 s | 0.113 |
| Clock | 3 | 21.1 s | 122.1 s | 0.149 |
| Browser | 3 | 90.2 s | 362.9 s | 0.232 |
| OsmAnd | 3 | 117.2 s | 212.6 s | 0.173 |
| VLC | 2 | 94.7 s | 375.8 s | 0.250 |
| Audio Recorder | 2 | 34.8 s | 199.0 s | 0.177 |
| Camera | 2 | 49.7 s | 196.1 s | 0.256 |
| Files | 2 | 37.7 s | 292.6 s | 0.126 |
| Contacts | 2 | 43.7 s | 312.2 s | 0.160 |
| Gallery | 1 | 51.2 s | 266.3 s | 0.192 |
| Simple Draw Pro | 1 | 42.2 s | 275.3 s | 0.153 |
| Launcher | 1 | 24.2 s | 178.4 s | 0.136 |
How we ran it
Use the benchmark’s own tasks
All 116 AndroidWorld tasks from Mobilerun’s public copy (droidrun/android-world c5467e6), seed 42.
Reset the phone before every task
Each task started from the same Android snapshot, checked before it ran: 116 of 116 matched.
Let DeepSeek decide, with the Lusor harness’s tools and checks
DeepSeek V4.1 Flash, at low reasoning effort, chose every action. The Lusor harness supplied the screen information, phone actions, list-reading tools and focused checks. The agent also had access to the app-skill catalog shipped with that build.
Run the agent on pi
The agent runs on pi, an open-source (MIT) agent toolkit. Lusor adds the mobile layer: device tools, screen reading, the agent guidance and the checks that decide when a task is done.
Run each task once
One trial per task (pass@1) in the reported evaluation. That single run is the score.
Record everything
A video of every task, with the agent’s steps, times and costs on a timeline.
Grade with the benchmark’s code
Mobilerun’s grader scored every run; Google’s unmodified grader (google-research/android_world 30583e2) rescored the same runs.
Compare like with like
Speed against Mobilerun and agent-qa, the two we could compare per task, as the median of per-task ratios. Cost against Mobilerun only, as our estimate.
Questions
What is AndroidWorld?
A public benchmark from Google Research: 116 tasks in real Android apps, each graded by checking the phone’s state, or the agent’s answer, after the run.
Why are there two graders?
Mobilerun’s public copy of AndroidWorld (c5467e6) changes how two tasks are graded and rewords three task requests. We ran its task text and report both graders. On two tasks Google’s original grader takes a more literal reading of the instruction than Lusor did; Mobilerun’s fork of the grader accepts the Lusor harness’s reading on both. Each is explained under Full results.
What does “1 trial per task (pass@1)” mean?
In the reported evaluation, each task ran once, and that single run is the score. No retries, and no best-of-several.
How is “5.2× faster” calculated?
For each task we divide the Lusor harness’s time by Mobilerun’s published time, then take the median of the 116 ratios (0.19). It is not a ratio of medians, and the harness was slower on 5 tasks.
Why compare speed only with Mobilerun and agent-qa?
They are the two we could compare per task: both publish a time for each of the 116 tasks. FluizAI also publishes step timings; we have not compared them. We make no speed claim against the others.
Is the Mobilerun cost their bill?
No. It is our estimate: each task’s published action count × $0.0766, one GPT-6 Astra (low) call per action, at the per-call cost we measured running Astra in our harness.
Who decides, and who checks?
DeepSeek V4.1 Flash decides: it interprets the goal and chooses each tool action. The Lusor harness supplies screen information, phone actions and list-reading tools, and runs focused checks on particular actions and saved values. The benchmark’s grader judges the outcome on its own, from the phone’s state, or the agent’s answer.
What does the $2.95 include?
The measured model cost of all 116 tasks, including the helper checks’ calls. It is not a subscription price, and it does not include hardware or infrastructure.
What does “open-weight” mean?
DeepSeek publishes the model’s weights, so anyone can download and run DeepSeek V4.1 Flash. It describes the model, not the Lusor harness, which is not open-sourced yet.
Is the Lusor harness available?
Benchmark evidence comes first: task results, recordings, steps and methodology. We’re preparing a cleaned-up open-source version of the Lusor harness, lusor-harness. It is coming; we are not announcing a release date yet. The score above is from the Lusor harness at the build in production on 9 November 2026. The version we are open-sourcing is a separate release and has not been benchmarked.
Can I get the raw data?
Yes. Download every task’s grades, times, costs and steps as JSON or CSV from the task directory, or one task’s as Markdown, plain text or JSON from its task page; the data schema describes each field.
Where are the evidence and the details to rerun it?
Each task’s recording, steps and grades are on its task page, linked from the task directory (/benchmarks/androidworld/tasks). How we ran it lists the task source (c5467e6), seed 42, the snapshot check and both graders, including Google’s at 30583e2.