Benchmarks
Same model, same tasks, hidden tests: Naseem is on par with Claude Code and Hermes on speed and tokens. Here is the whole picture, including the run it lost.
Setup
Six coding tasks, each a small real repository with a user-style request and hidden tests. All three harnesses used the same model, kimi-k2.6:cloud through Ollama. Naseem 1.14.0 ran at Medium reasoning; Claude Code and Hermes ran from the command line at their defaults. Time is each harness's own run time; tokens are input tokens including cache reads. Sep 29–30, 2026.
Five tasks, one run each
These five are the tasks Naseem's own release check uses, so Naseem has been developed against them. Read them as a best case for Naseem.
| Task | Naseem | Claude Code | Hermes |
|---|---|---|---|
| t02 · Python bug fix | pass · 96 s · 120K | pass · 91 s · 179K | pass · 89 s · 112K |
| t12 · TypeScript crash | pass · 53 s · 104K | pass · 86 s · 139K | pass · 93 s · 121K |
| t13 · Swift package setup | pass (2nd run) · 128 s · 240K | pass · 154 s · 255K | fail · 264 s · 363K |
| t08 · Swift build repair | pass · 298 s · 230K | pass · 320 s · 230K | pass · 310 s · 199K |
| t11 · Swift behaviour change | pass · 520 s · 441K | pass · 487 s · 707K | pass · 431 s · 241K |
| Total | 5/5 · 1,095 s · 1.14M | 5/5 · 1,138 s · 1.51M | 4/5 · 1,187 s · 1.04M |
Time in seconds, then input tokens. Naseem's t13 failed on its first run (it edited the tests instead of the package) and passed on a rerun; Hermes made the same mistake on its one run.
A task Naseem was never tuned on, two runs each
t05 asks for a new feature in a TypeScript app: a Slack notification channel with a given message format. Naseem had never been developed against it.
| Harness | Results | Average time | Average calls | Average input tokens |
|---|---|---|---|---|
| Naseem | fail, pass | 281 s | 19 | 288K |
| Claude Code | pass, pass | 201 s | 18 | 350K |
| Hermes | pass, pass | 361 s | 34 | 725K |
Naseem's failed run passed 9 of the 10 hidden tests; the one it missed wanted the Slack link on the line right after the text, with no blank line, which the request did not say.
What it shows
On par, not ahead
On tasks it was tuned for, Naseem is fastest on three of five and sends the fewest tokens on four. On the fresh task it sits between the two CLIs on time and tokens and dropped one run of two. A week before these runs it was about twice as slow as both CLIs; that gap is gone.
Correctness comes from checking
The failures on both sides came from the model taking a shortcut, such as editing tests to make them pass, or guessing an unstated format. The harness that checks the work against the request catches the first kind; nothing catches the second every time.
The remaining gap is call count
On the hard tasks Naseem makes more model calls (27 on t11 against 22 and 13). Each is short, so time stays close, but fewer, better-planned calls would cut time and cost. That is the next thing to fix.
Limitations
- One run per task on the five, two on the sixth: differences under about 20% are noise.
- Naseem's author wrote the tasks and the benchmark, and Naseem was developed against five of the six.
- One model only. Claude Code is built around Claude models and ran a third-party model here; every harness may do better on its own.
- Settings differ slightly: Naseem at Medium reasoning, the CLIs at their defaults; Claude Code ran with a scoped tool permission list.
Raw run logs, hidden-test output and per-run metrics are kept with the benchmark. Ask on the community forum if you want them.