ai-engineering

AI Harness Engineering — 别换模型，先装 Harness

URL: https://mp.weixin.qq.com/s/nGHqwQyjjp0qHEqMsqcLDg Date: 2025-05-24 5|Source: 新智元 (Xinzhiyuan)

  • Tags: ai-engineering

Summary

Anthropic and OpenAI both independently concluded: AI coding agents fail not because of model limitations, but because of missing “Harness” — the engineering infrastructure around the agent. DeepSeek is now hiring Harness engineers too.

Key Data Points

  • Anthropic experiment: Claude Opus 4.5 bare = $9, total failure. With Harness = $200, dramatically better results. The extra $191 went to verification loops (run tests after every code write, fix until passing).
  • OpenAI experiment (Codex): Adding a single AGENTS.md file (<100 lines of markdown) to the repo root significantly improved success on real million-line codebases.

The 5 Harness Subsystems

  1. Instructions — AGENTS.md (OpenAI) or CLAUDE.md (Anthropic). Auto-injected as system prompt on startup. Defines project conventions, forbidden operations, coding style.

  2. Tools — Permission controls on what commands the agent can run. .claude/settings.json or ~/.codex/config.toml. Prevents rm -rf, force push, unauthorized API calls.

  3. Environment — Locked dependencies and runtime config. setup.sh / Dockerfile / devcontainer.json. Key line: pnpm install --frozen-lockfile — agent cannot upgrade dependencies.

  4. State — PROGRESS.md for cross-session persistence. Tracks completed tasks, in-progress, TODOs, known issues. New session’s first action: read PROGRESS.md.

  5. Feedback — Machine-executable verification (test, lint, type-check, build). Exit code ≠ 0 means task ≠ complete. This is the most critical — without it, all other subsystems are useless.

3 Fatal Failure Modes

  1. Premature Victory Declaration — Agent says “Done!” but code doesn’t compile/pass tests. Fix: Feedback subsystem (exit code is the judge, not self-assessment).

  2. Context Anxiety — At 70% context window usage, agent rushes to finish: skips tests, writes stubs, drops edge cases. Fix: State subsystem + proactive restart. Write checkpoint to PROGRESS.md when tokens > 70%, start new session.

  3. Cross-Session Amnesia — Second session rewrites what first session already built, with conflicting interfaces. Fix: State + Instructions combo. PROGRESS.md maintains completed feature list; AGENTS.md mandates reading it first.

5 Steps to Build a Harness (<200 lines total)

  1. Create AGENTS.md — project description, forbidden ops, completion definition
  2. Configure permissions — .claude/settings.json or config.toml
  3. Write setup.sh — lock all dependency versions (—frozen-lockfile)
  4. Create PROGRESS.md — completed / in-progress / TODO / known issues
  5. Define completion criteria in AGENTS.md — type-check, test, lint, build must all pass

Core Takeaway

Model capability determines the ceiling. Harness determines how much of that ceiling you actually reach.

Without Harness, even Opus 4.5 produces code that won’t compile. With Harness, a smaller model can deliver reliably. Don’t wait for the next model — install Harness now.

Reference: https://walkinglabs.github.io/learn-harness-engineering/en/