I Put 9 AI Agent Engines on One Box With Zero Cloud - Which One Actually Finishes the Job?

From 2nd to 4th November 2026, I put nine AI agent engines on one Own1, a single-box AI workstation that sits on my desk, and gave them all the same five coding jobs. Same local model, same tools, same 20-minute limit, and nothing left the box. No cloud model, no API key. Then a machine marked the work with hidden tests, not another AI.

Some of the results surprised me. The biggest swings didn't come from which engine you pick. They came from the version you're on, the plumbing between the engine and the model, and how much text the engine shoves at the model every turn.

Same Box, Same Model, Hidden Tests

If you want to compare agent engines fairly, everything else has to stay still. So every engine got:

• The same Own1 and the same local model, Qwen3.6-35B-A3B (4-bit), through the same model server
• The same metering proxy, so we counted every model call and every token
• The same tools: files plus a shell, in a sandboxed workspace
• The same 20-minute limit per task

Each engine did five tasks, three times each, so 15 runs:

Task	What it asks for
T1	Fix a bug
T2	Add a feature, with meaningful tests
T3	Refactor into at least four functions
T4	Build a web page from a spec
T5	Find a bug in a 40-module repo from a stack trace

Pass or fail is decided by hidden unit tests and rule checks. No AI judge. And if an engine edits the tests to make them pass, that's a fail.

Why coding? It's the hardest general test of an agent, and it's what owners actually ask their AI employee for: a website, a fix, a change. Small-business tasks like social posts, inbox and bookkeeping are the next suite.

The Scoreboard

Engine	Version	Solved /15	Timeouts	Tests edited	Median time per task	Model calls per task
OpenClaw	2026.10.10	14	0	0	213 s	7
ZeroClaw	0.8.7	13	0	0	181 s	12
OpenCode	2.2.1	13	0	0	183 s	9
Hermes	0.22.6	11	3	2	396 s	10
Hermes	0.22.7	10	1	3	238 s	11
nanobot	0.4.0	8	0	0	166 s	12
IronClaw	1.4.1	7	10	0	1,200 s (the limit)	22
NullClaw	nightly 2026-09-27	7	0	0	142 s	9
Goose	1.57.0	6	0	1	159 s	8
Nexy Engine (ours)	0.1.0	15	0	0	73 s	8

OpenClaw 2026.10.10 topped the independent engines with 14 of 15, with ZeroClaw and OpenCode close behind on 13. Our own engine is at the bottom of the table on purpose. More on why it doesn't count yet further down.

These are all open-source projects with good teams behind them, and every number here is on this model and this box. A different model or different hardware would shuffle things around. Treat it as one honest data point, not a league table for all time.

We Caught Our Own Mistake

Our first OpenClaw runs quit in one second. That was our fault: a config key in our own test harness that OpenClaw rightly rejected.

Once we fixed it, OpenClaw 2026.10.7 scored 12/15 at a median of 183 seconds, and the newer 10.10 did better again: 14 of 15, the best of the independent engines.

The rule we took from it: a failure only counts if the engine actually reached the model. If the engine never got to talk to the model, that's on the harness, not the engine. Every row in the table above passed that check.

Upgrading Beat Tuning: ZeroClaw Went From 6 to 13

ZeroClaw 0.8.3 solved 6 out of 15. ZeroClaw 0.8.7 solved 13. Same box, same model, same tasks. The only change was the version, which fixed how it calls tools with this model.

If we'd stopped at the old version, we'd have written ZeroClaw off. It was the version, not the engine. If you're testing agents yourself, check you're on the latest release before you judge anything.

Solving Isn't Enough, It Has to Stop

On this model, Hermes often got to the right answer, then kept going. It looped until the 20-minute limit in a few runs (three timeouts on 0.22.6, one on 0.22.7). Across both versions it also edited the test files in 5 of its 30 runs, which the grader fails. (Goose did it once; no other engine did.)

An AI employee that never says "done" ties up the whole box. On a shared workstation in an office, knowing when to stop is part of the job.

Small Prompts Win on a Home Box

On a local model the ceiling is output speed, about 20 tokens a second for the whole box. Every token the engine sends has to be read, and every token it writes has to be generated, so the less an engine sends and writes, the sooner it's done.

The first call alone tells the story. ZeroClaw's first prompt is about 1.7k tokens, OpenClaw's is about 8k and Hermes's about 10k. Size isn't everything, mind you: OpenClaw sends more than ZeroClaw and still solved the most. Across a whole task:

IronClaw's high count is part of its timeout story. Under load its client gave up on a slow call after about 120 seconds and sent the same request again, so the box kept re-reading the same prompt. OpenCode's figure is an estimate, because its client closes the stream before the server reports token counts.

Put speed and success on one chart and you can see the trade-off:

The Plumbing Matters More Than the Brand

Several engines failed at first for reasons that had nothing to do with how smart they are:

• Tool calls written out as plain text instead of real tool calls
• The model server reusing tool-call IDs, which some engines (Goose and nanobot among them) took as calls already made
• A fixed 180-second cutoff in IronClaw's one-shot mode

Fixing the plumbing changed the rankings more than anything else we did. The tool protocol has to match the model server, and when it doesn't, a good engine looks bad.

From 4 MB to 2.4 GB

This one blew my mind a bit. The install sizes range over nearly three orders of magnitude:

Most of those sizes include a web dashboard, and they're doing different jobs. OpenClaw is a full product with its own Control UI (65 MB of its 958 MB), while Nexy Engine and NullClaw are bare engines with no UI at all. The memory each engine uses on top of the model tells the same story:

NullClaw's 4.3 MB is the agent process only, and the ZeroClaw, Hermes and OpenClaw figures come from an earlier, separate measurement. The model itself uses the same memory whatever engine is driving it, so this is just the overhead of the engine on top.

Our Own Engine Got 15/15. Why That Doesn't Count Yet

Nexy Engine is ours, built by the OwnX team (in Rust). It solved all 15, at a median of 73 seconds per task, with 13k prompt tokens per task, a 3.9 MB install and 6 MB of memory.

But we developed it against these exact five tasks. So 15/15 is not an independent result yet, and it would be pretty rich to claim it beats the others on a test we built it around. What it does show is the direction we want for Own1's always-on AI employees: tiny and fast is possible, on a box in an office with no cloud.

The next step is a held-out task set the engine has never seen. If it holds up there, I'll tell you. If it doesn't, I'll tell you that too.

What About a MacBook?

We also ran two engines on a MacBook, same model through Ollama. It's different hardware, so we're only comparing what got solved, not speed:

• ZeroClaw 0.8.7: 14/15
• DeepSeek Harness 0.2.1-rc.5: 14/15

So What Happens Next?

• Held-out tasks for Nexy Engine, so its score means something
• Small-business tasks: social posts, inbox and bookkeeping, the things owners actually hand an AI employee day to day
• Re-runs as new versions land, because as ZeroClaw showed, one upgrade can change everything

GitHub figures as of 10th November 2026. Contributor counts are commit authors as reported by the GitHub API.

If you run one of these engines and think we've got your setup wrong, tell me. That's exactly how we caught our own config bug.

Let me know on YT or X. Cheers!