Harnessed agentic RL: train the agent without rewriting the harness

The important release is a boundary change, not another RL wrapper

Microsoft released Agent Lightning v1.0.0 on August 17 and v1.0.1 one week later. The accompanying paper names the pattern harnessed agentic reinforcement learning: the deploy-time agent harness owns environment interaction, while a separate trainer observes sequences of model requests and responses and turns them into learning updates.

That differs from a training loop that owns a toy environment and calls a policy at each step. A production harness already decides which context to assemble, which tools to expose, how to recover from failure, when to ask for approval, and whether to delegate to another agent. Reimplementing all of that inside an RL framework produces a second system that looks like production but behaves differently. Training against the real harness closes that gap.

The official v1.0 paper reports that reinforcement learning raised Qwen3.5-9B from 41.8% to 56.4% on SWE-bench Verified using 6,000 training examples, a 14.6-point absolute gain. The repository was also a substantial developer signal: 17,796 stars, 153 open issues, an MIT license, and code updated on August 25 when checked. A focused Hacker News release thread reached 53 points and eight comments. That is enough to justify technical attention, not a claim that the architecture has become universal.

The paper is valuable because it rejects the easy story. Separating trainer and harness does not make training trivial. It exposes new failure surfaces in retokenization, sample merging, advantage calculation, loss normalization, and backend scheduling. The interface becomes simpler at the HTTP boundary while the evidence contract becomes more demanding underneath.

How Agent Lightning v1.0 divides the system

The v1.0 documentation describes three primary components. The API Gateway stores rollouts, model endpoints, resources, and events and exposes an OpenAI-compatible proxy. The Rollout Controller launches agent executions as local processes or Kubernetes jobs. The Customized Trainer serves model inference and optimization on the GPU side, consumes the recorded rollout data, and produces policy updates.

1. The trainer enqueues a task with a specific resource version and rollout identifier.
2. The controller starts the existing agent harness in an isolated attempt.
3. The harness sends its normal chat-completion requests through the gateway.
4. The gateway attaches rollout and attempt identity, sequences events, and stores request-response evidence.
5. The harness executes tools and emits trace spans, annotations, and rewards.
6. An adapter reconstructs training samples from calls, responses, trace relationships, and rewards.
7. The trainer calculates updates, produces a candidate model or resource, and sends it to an evaluation gate.

This architecture preserves framework choice. A LangGraph agent, Microsoft Agent Framework application, custom Python loop, or coding harness can retain its own control flow. The proxy is a narrow waist: the harness asks for model output using a familiar interface, and the trainer does not have to import the harness's private orchestration code.

The narrow waist is also lossy. A model request does not reveal why a particular context item was selected, whether a tool result came from a stable fixture or a changing production API, whether a retry repeated an external effect, or whether an approval was genuine. OpenTelemetry spans and Agent Lightning annotations can carry that evidence, but only if the harness emits it and the adapter interprets it consistently.

Give every rollout a complete identity

A trustworthy training sample needs more than prompt, completion, and scalar reward. It needs the versioned state that made those values meaningful. Without that identity, two samples can look mergeable while representing different tasks, tool permissions, tokenizers, or reward policies.

rollout_id: swe-fix-004812
attempt_id: 03
task:
  dataset: swebench-verified
  revision: 2026-08-15
  item_digest: sha256:91d...
harness:
  image: agent-runner@sha256:7ab...
  config_digest: sha256:4f0...
resources:
  system_prompt: prompt-v18
  skill_bundle: skills-2026-08-25
tools:
  manifest_digest: sha256:31c...
  network_policy: fixtures-only-v4
policy:
  model: qwen3.5-9b-candidate-12
  tokenizer_digest: sha256:aa2...
  sampling: {temperature: 0.7, top_p: 0.95}
reward:
  definition: patch-consequence-v6
  evaluator_image: verifier@sha256:72e...
  weights: {tests: 0.7, scope: 0.2, efficiency: 0.1}

The task digest prevents a mutated fixture from keeping the same label. The harness image and config identify control flow. Resource versions capture prompts, skills, and model-facing instructions. Tool and network manifests record the environment the policy could affect. Tokenizer identity matters because the trainer may retokenize text that the serving stack generated under different assumptions. Reward definition and evaluator image make the target reproducible.

Store lifecycle events as well. A rollout may have multiple attempts after infrastructure failure, invalid output, or budget exhaustion. Do not concatenate those attempts blindly. A second attempt may see changed repository state or a partially completed side effect. Agent Lightning's store models tasks, rollouts, attempts, spans, and versioned resources separately; preserve those boundaries when exporting data.

Sequence identifiers should be the canonical ordering signal across processes. The official adapter documentation warns that timestamps from different machines can be unsynchronized. Parent-child span relationships can also be incomplete or duplicated when multiple instrumentation libraries observe the same call. Repairing the trace tree is a useful best effort, but a repaired hierarchy should carry a flag so training audits can distinguish observed from inferred relationships.

A reward is a policy decision disguised as a number

Agent Lightning can record a final float, intermediate rewards, structured annotations, exceptions, and custom objects. That flexibility is essential for long agent trajectories. It also makes reward governance the center of the system. If the reward credits a patch because tests pass, the policy may learn to narrow tests, exploit stale state, or produce a correct output through an unacceptable side effect.

import agentlightning as agl

@agl.rollout
def repair_task(task, resources):
    patch = coding_agent(task, resources)
    checks = run_isolated_verifier(task, patch)

    agl.emit_annotation({
        "type": "verification",
        "task_digest": task.digest,
        "test_digest": checks.test_digest,
        "unauthorized_effects": checks.unauthorized_effects,
    })
    agl.emit_reward({
        "task_completion": float(checks.accepted),
        "scope_control": checks.scope_score,
        "efficiency": checks.budget_score,
    }, primary_key="task_completion")
    return patch

Keep the primary optimization target narrow and auditable. Auxiliary values may help analysis, but a composite reward can hide a failed hard constraint behind enough points elsewhere. Enforce authorization, secret exposure, destructive side effects, and fixture integrity as gates before reward aggregation. A rollout that violates them should not become a merely low-scoring example; it should be quarantined and reviewed.

Credit assignment must match causal structure. A final test reward after thirty model calls does not reveal which calls mattered. Intermediate rewards can improve density but may teach the agent to satisfy proxies. For tool-using agents, prefer consequence-based checkpoints tied to environment state: a reproduced bug, a minimal diff, a newly failing regression test before the fix, a passing test after it, and no unauthorized files or network calls.

Independent evaluation matters because the harness and trainer can share blind spots. If the same model writes the patch, summarizes the trace, and grades success, agreement is not strong evidence. Use deterministic checks where possible, then a separately versioned grader for residual judgment. Preserve raw verifier output rather than only the reward it produced.

Build a promotion pipeline, not an endless learning loop

Continuous data collection does not require continuous production updates. Treat training as a staged release process. Collect bounded rollouts, validate their identity, quarantine ambiguous attempts, train a candidate, replay a frozen suite, run adversarial tasks, compare cost and latency, and require an explicit promotion decision.

candidate = train(accepted_rollouts)
evidence = replay(
    model=candidate,
    suites=[frozen_core, adversarial_tools, side_effect_drills],
    harness_digest=approved_harness,
)

if evidence.task_acceptance < baseline.task_acceptance:
    reject("task regression")
if evidence.unauthorized_effects > 0:
    reject("authority regression")
if evidence.p95_cost > budget.max_p95_cost:
    limit("cost regression")
if evidence.trace_coverage < 0.98:
    hold("insufficient evidence")
promote(candidate, receipt=evidence.digest, approver="agent-release-owner")

Task acceptance should be stricter than a benchmark headline. Define it as the share of tasks whose required consequence checks pass with valid evidence and no hard-policy violation. Report task success, acceptance, cost, latency, retries, trace coverage, and side-effect exceptions separately. A candidate can improve raw success while becoming too expensive or less controllable.

Keep training and evaluation populations distinct. Near-duplicate repository issues, regenerated tests, or leaked golden patches can inflate improvement. Freeze evaluation digests before training begins, retain a never-trained incident set, and rotate a small secret holdout controlled by a different owner. If external tools or web services remain live, record their response digests and accept that exact replay may be impossible.

Read the 41.8% to 56.4% result as a system result

The v1.0 paper's SWE-bench Verified improvement is material: 14.6 absolute points on the same named base model after RL. It supports the claim that a real coding harness can produce useful post-training trajectories. It does not isolate the contribution of every layer or prove that the resulting policy transfers to a different harness, repository mix, tokenizer, tool policy, or reward.

Ask five questions before using the number in an adoption decision. Was the baseline run under the same harness and budget? Were training examples disjoint from evaluation repositories and issue variants? Which verifier produced the reward? Did acceptance require newly added tests to demonstrate a before-fix failure? Were retries, tool calls, cost, and unauthorized effects included? A benchmark can answer capability while leaving operational equivalence open.

Reproduce a smaller experiment first. Choose fifty to two hundred tasks from your own distribution, freeze the environment, run the base policy multiple times to estimate variance, train on a separate population, and replay with the identical evaluation harness. If the gain disappears under your task mix, that is useful evidence. The architecture still may be valuable for prompt or resource optimization, but the public result should not be treated as your forecast.

Choose the training pattern that matches the environment

Harnessed RL is most attractive when the production control loop is costly to reproduce and the model is the component you actually intend to change. It is a poor first step when task success is subjective, tool side effects cannot be isolated, trace coverage is low, or the team cannot define a stable reward. In those cases, invest in behavior specifications, execution receipts, and deterministic evaluation before optimizing weights.

Failure modes to test before the first GPU run

A 30-day adoption checklist

1. Choose one task family with an objective success oracle and reversible or sandboxed effects.
2. Freeze task, harness, tool, resource, tokenizer, reward, and evaluator version schemes.
3. Instrument rollout, attempt, model call, tool call, environment consequence, exception, and reward spans.
4. Measure missing, duplicate, late, and repaired spans before using traces for training.
5. Run a base-policy repeat study to quantify task and sampling variance.
6. Define hard rejection gates separately from scalar optimization rewards.
7. Train on a bounded population and retain a never-trained incident and adversarial suite.
8. Replay candidates with the exact approved evaluation harness and independent verifier.
9. Compare accepted tasks, hard failures, cost, latency, retries, and trace coverage.
10. Issue a signed promotion receipt and keep rollback to the prior model and resources tested.

Start with evidence plumbing even if the first optimization is only a prompt. The same rollout identity, trace validation, and release receipt will support future RL, fine-tuning, resource search, or model routing. Training should become one consumer of trustworthy agent evidence, not the reason the evidence exists.

Frequently asked questions

What is harnessed agentic RL?

It is reinforcement learning where the real agent harness owns environment interaction and a disaggregated trainer observes versioned model calls, traces, resources, and rewards. The trainer updates the policy without reimplementing the harness's full control loop.

Can Agent Lightning train any existing agent with zero changes?

The compatible proxy can reduce model-client changes, but meaningful training still needs task identity, trace coverage, reward emission, resource versioning, isolation, and reproducible evaluation. “No framework rewrite” is not “no integration work.”

Should teams train directly from production traffic?

Usually not at first. Production data can contain private inputs, changing tools, uncontrolled effects, biased selection, and weak rewards. Begin with approved datasets and sandboxed or replayable environments, then add carefully governed production evidence.

Does the SWE-bench result prove the method will improve my coding agent?

No. It is compelling evidence for the reported model and pipeline. Your result depends on task distribution, harness, tools, budgets, reward, verifier, data separation, and sampling variance.

What should block model promotion?

Any hard authority or safety regression, invalid rollout identity, insufficient trace coverage, task-acceptance regression, unacceptable cost or latency, unreconciled side effects, or evaluation contamination should block or limit promotion.