Built for the Arm AI Optimization Challenge · Track 2 — Cloud AI

Armsmith

The agent that forges your repo for Arm.

Every other submission runs a model on Arm. Armsmith makes everyone else's repo fast on Arm — and refuses to lie about the numbers.

armsmith — zsh
$ armsmith diagnose --replay fixtures/replays/scenario_ragserve → 6 proposed · 4 proven · 2 refused · report signed (ed25519)
submitted on Devpost as Armsmith · built as the CLI armsmith MIT · github.com/edycutjong/armsmith

02the number that matters

It deletes its own fixes.

On the ragserve seed, Armsmith proposed six performance fixes — and deleted the two it couldn't prove, reasons attached. That refusal is the product.

03the problem

The Graviton lore nobody has.

a solo founder, 2 a.m.

Her AWS bill doubled overnight when the RAG service outgrew its instance. The Arm migration guide is 40 pages of compiler flags. The next invoice lands in 9 days.

Arm's Learning Paths teach a human every fix by hand. Nothing automates the loop — so teams skip the migration, or do it blind.

the lore — 6 of 13 aarch64 anti-patterns
  • amd64 image pins → QEMU emulationR1
  • -mcpu / -march left at defaultsR2
  • NumPy on reference BLASR3
  • silent float64 coercionR4
  • GGUF quant ↔ ISA repack mismatchR5
  • threads × workers > vCPUsR6
20–40%

cheaper per unit of AI throughput on Arm cloud — Graviton · Ampere · Axion — going unclaimed.

2–3×

left on the table after a blind migration: QEMU emulation, float64 coercion, kernels built without dotprod/i8mm.

04the solution

Only proven fixes ship.

One command profiles the repo, runs the 13-rule aarch64 pack, and gates every proposed fix through a reproduce benchmark. What survives becomes a signed report and a PR.

armsmith diagnose — replay mode
$ armsmith diagnose --replay fixtures/replays/scenario_ragserve ▌ replay bundle · synthetic: true — no hardware numbers claimed fingerprint dotprod ✓ · i8mm ✓ · SVE ✓ (recorded fixture → kernel routing) scan 13-rule pack → 6 candidate fixes on ragserve R1 amd64-pinned image → QEMU emulation Dockerfile R6 threads × workers > vCPUs env R3 NumPy on reference BLAS show_config() R8 pip sdist fallback pip log R4 silent float64 coercion Python AST R11 THP/allocator untuned sysfs planner 6 fixes ordered (deterministic fallback · Claude loop = TODO(S1)) REPRODUCE GATE median-of-N · MAD noise band (k=3) · output-hash equality kept 4 Δ outside the noise band and outputs hash-equal drop 1 inside noise band → reported, not hidden drop 1 output-hash mismatch → reported, not hidden report signed ed25519 · canonical-JSON sha256 → report.json ✔

05how it works

The trust inversion.

The LLM planner may only propose. The only component allowed to claim a result is the gate — and armsmith verify re-checks the gate from raw samples.

your repo Dockerfile · CI pyproject · GGUF armsmith diagnose ISA: dotprod · i8mm · SVE 13-rule scan — R1…R13 planner — proposes deterministic fallback today Claude tool-use: TODO(S1) REPRODUCE GATE median-of-N · ABAB runs MAD noise band · k = 3 output-hash equality refuses claims inside band signed report ed25519 + sha256 raw samples embedded bot PR evidence table (dry-run) drop log — refused fixes inside noise band · hash mismatch reasons shipped, never hidden armsmith verify recomputes stats from raw samples tamper one digit → VERIFY FAILED proposes 6 claims 4 refuses 2 re-checked

06live demo

Four proofs. Zero hardware.

the harness — offline
$ python -m pytest -q 219 passed — fully offline · ~93% line coverage $ armsmith scan fixtures/replays/scenario_ragserve static rules on a real directory — zero hardware: R1 amd64-pinned image R4 float64 coercion R12 amd64-only CI
the gate — refusal on record
$ armsmith diagnose --replay fixtures/replays/scenario_ragserve ▌ replay · synthetic: true gate: 6 candidates → 4 kept · 2 dropped drop — inside noise band reported, not hidden drop — output-hash mismatch reported, not hidden
the witness — ISA proof
$ armsmith witness objdump_before.txt objdump_after.txt counting SDOT · UDOT · SMMLA · USMMLA in the disassembly… sdot 0 → 4 ✔ Arm dot-product kernels emitted wall-clock can be argued with; instructions cannot.
the tamper test
$ armsmith verify …/scenario_ragserve/report.json VERIFY OK — every statistic recomputed from raw samples # now edit ONE digit in a samples array and re-run: $ armsmith verify …/scenario_ragserve/report.json VERIFY FAILED — sample hash mismatch

every command above runs on any laptop — no Graviton, no network · ~2 minutes end-to-end

07what's under the hammer

Built to refuse.

The reproduce gate

band = k·√(smad_a² + smad_b²) , k = 3
  • median-of-7 timed runs · 2 warmups discarded · ABAB interleave against drift
  • output-hash equality — a faster wrong answer is a dropped fix
  • refuse-to-claim inside the band — in-band deltas report as no change, never wins
  • drops ship with reasons — machine-readable, in the report and the PR
R1–R13

13-rule aarch64 pack

YAML descriptors + fix generators; 10 rules cite an Arm Learning Path.

SDOT 0→4

ISA witness

Before/after objdump counts of SDOT/UDOT/SMMLA/USMMLA — deterministic, noise-immune.

ed25519

Tamper-evident reports

Raw samples embedded beside every stat; canonical-JSON sha256; verify recomputes it all.

ubuntu-24.04-arm

Drop-in arm64 CI gate

Composite GitHub Action runs the same gate as an exit-code check on free arm64 runners.

08why now · why us

Cheap silicon, missing proof.

the money

Arm cloud runs AI inference 20–40% cheaper per unit of throughput — Graviton, Ampere, Axion — and most teams still haven't collected.

the rails

GitHub's native arm64 runners (ubuntu-24.04-arm) are free — an Arm perf gate in CI now costs nothing to adopt.

the lore

Arm's Learning Paths teach every fix by hand — 10 of our 13 rules cite one. Armsmith is that curriculum, automated.

Anyone can wrap an LLM around perf advice. The moat is refusal — a gate that deletes unproven fixes, and a verify that catches a single edited digit.

09shipped · hardware-free phase 1

Counted real. Labeled pending.

■ real today — runs offline on any machine

219✓

pytest tests passing · ~93% line coverage · zero network

13+13

aarch64 rules → 13 exported x86→Arm migration-template cards

10/13

rules citing an Arm Learning Path — the judges' own curriculum

7cmd

scan · diagnose · witness · verify · pr · ci · rules export

3×2

CI matrix: 2 native arm64 runners + 1 x86 runner × Python 3.11 / 3.12 (6 jobs)

MIT

public signed-report JSON schema + importable benchstats / gate / report / witness modules

■ not yet claimed — and the tool says so

  • [PENDING]Live Graviton throughput multiplier — not yet measured; no hardware number appears anywhere.
  • syntheticReplay bundles are labeled synthetic: true at every layer; loaders refuse unlabeled data.
  • TODO(S1)Live perf/PMU · llama-bench capture on target hardware.
  • TODO(S1)Claude planner tool-use loop — deterministic fallback planner today.
  • dry-runPR posting renders exactly what would ship; never touches the network.

Every number on the left is re-derivable by a judge. Everything on the right is labeled in the code itself.

10the ask

Run the two-minute proof.

judge quickstart — any laptop, no Arm hardware, no network
$ git clone https://github.com/edycutjong/armsmith && cd armsmith $ python -m venv .venv && source .venv/bin/activate $ pip install -e '.[dev]' $ python -m pytest -q # 219 passed · offline $ armsmith scan fixtures/replays/scenario_ragserve # static rules, zero hw $ armsmith diagnose --replay fixtures/replays/scenario_ragserve # 4 kept · 2 dropped $ armsmith witness fixtures/witness/objdump_before.txt \ fixtures/witness/objdump_after.txt # ISA proof: 0 → 4 $ armsmith verify fixtures/replays/scenario_ragserve/report.json VERIFY OK # tamper one digit in report.json → re-run verify → VERIFY FAILED
github.com/edycutjong/armsmith

The agent proposes. The silicon disposes.

presenter notes — 01