GitHub outage resilience starts with recoverable agent state

The outage was a coupled-control-plane failure

On August 17, GitHub recorded elevated errors and latency for 7 hours and 47 minutes across Issues, Pull Requests, APIs, Actions, and Copilot. At peak, web and API errors were about 20%, while archive and raw-content downloads reached about 50%. SAML and OIDC authentication, SCIM, Team Sync, and some Actions workflows also failed. This was not one missing web page. It was a shared control plane degrading across source, automation, identity, event, and agent services.

The mechanism matters more than the outage headline. GitHub's status record says an Istio sidecar hit concurrency limits while an autoscaling policy watched the host service rather than the sidecar. Failures cascaded until four HAProxy nodes exhausted their flow limits. Optimistic retry logic then increased pressure on already saturated load balancers. A latent client retry defect amplified Copilot Token Service traffic from a normal 7,000 to 9,000 requests per second to roughly 70,000 to 100,000 requests per second.

That official record does not prove the popular claim that AI-generated repositories or agent traffic caused the incident. Reddit threads collected during this run repeatedly proposed that theory, but inference is not an RCA. The source-backed lesson is narrower and more useful: agent clients can magnify a partial failure when they coordinate through one control plane, retry without a shared budget, or lose durable state between attempts.

The August 17 event was also not isolated context. GitHub's July availability report lists eight incidents, calls the separate August 6 Actions outage unacceptable, and says the company is accelerating isolation, resiliency, scale work, and the migration of Actions components to Azure. A continuity design should therefore assume intermittent degradation, partial recovery, and service-specific lag, not only a clean all-up or all-down state.

Model GitHub as six dependency planes

Teams often say, “We can keep working because Git is distributed.” That is true for source history and incomplete for modern delivery. An agent may read an issue, obtain a token, fetch a workflow action, start a hosted job, post check results, open a pull request, wait for reviews, and call a model through the same platform. Each step has different failure behavior.

The dependency graph explains two reports that otherwise look surprising. One r/GithubCopilot user said a GitHub outage blocked an independently chosen local model because the Copilot integration still controlled access. Another reported lost AI credits after long-running tasks stopped and had to be rerun. The model can be local while authentication, orchestration, progress visibility, or the tool gateway remains remote.

Do not promise full offline equivalence unless you have tested it. A degraded mode may support reading, editing, local tests, and a signed patch bundle while disabling remote issue updates, package publication, merges, deployments, and external messages. That is still valuable if the boundary is explicit.

Write a continuity contract for each agent workflow

A continuity contract makes hidden coupling reviewable before an incident. It names the provider dependency, evidence of success, local or alternate path, write behavior during uncertainty, retry budget, and recovery owner. Store it beside the workflow, then test it like code.

workflow: dependency-update-agent
owner: developer-platform
source_of_truth:
  primary: github.com/acme/payments
  mirror: forge.internal/acme/payments
state_store: postgres://agent-ledger/task_runs
planes:
  source:
    degraded_mode: mirror_read_patch_bundle
  work_state:
    degraded_mode: local_task_manifest
  execution:
    degraded_mode: make verify
  events:
    recovery: reconcile_delivery_cursor
writes:
  during_provider_degradation: deny
  idempotency_required: true
retry_budget:
  max_attempts: 4
  max_elapsed_seconds: 180
  jitter: full
recovery_gate:
  require: [source_digest, side_effect_receipt, human_release]
drill_frequency_days: 30

The contract is intentionally consequence-based. “Use GitLab if GitHub is down” is not a plan unless repositories, identities, secrets, rules, packages, events, and build definitions are actually portable. A narrower plan that produces a verified patch bundle and queues the remote changes can be more credible than an untested multi-provider promise.

Keep the mirror one-way during normal operations unless you have designed conflict resolution. Bidirectional synchronization during an outage can split history or replay automation on both sides. Source custody can be redundant while release authority remains single-writer.

Checkpoint before effects, not after a whole agent turn

A chat transcript is poor recovery state. It mixes reasoning, tool results, untrusted text, and side effects without a stable boundary. Persist a small state machine around every consequential action: planned, authorized, executing, observed, verified, or ambiguous.

type Step = {
  runId: string;
  taskVersion: string;
  sourceSha: string;
  actionDigest: string;
  status: "planned" | "authorized" | "executing" |
          "observed" | "verified" | "ambiguous";
  idempotencyKey: string;
  approvalReceipt?: string;
  providerReceipt?: string;
  retryRemaining: number;
  leaseExpiresAt: string;
};

async function execute(step: Step) {
  await ledger.persist({...step, status: "executing"});
  const result = await provider.call(step, {
    idempotencyKey: step.idempotencyKey
  });
  await ledger.persist({
    ...step,
    status: "observed",
    providerReceipt: result.receipt
  });
  await verifyEffect(step, result);
  await ledger.persist({...step, status: "verified"});
}

If the connection fails after the remote service accepted a write but before the response arrives, the state is ambiguous, not failed. Retrying immediately can create a duplicate comment, branch, deployment, or release. First query by idempotency key, object digest, or expected state. If the provider offers no idempotency primitive, construct one in your integration and make duplicate detection part of verification.

Checkpointing also changes cost accounting. A failed run should resume from the last verified transition, not replay model calls, repository reads, test runs, and human approvals. Measure recovery tokens and review minutes alongside availability.

A thousand polite agents can still form a retry storm

GitHub's REST guidance says to avoid polling, avoid concurrent requests, pause between mutative requests, honor Retry-After and rate-limit reset headers, use conditional requests, and increase wait time exponentially. Those rules need coordination above the individual agent. If each worker independently retries four times, a 500-agent fleet can produce 2,000 additional calls at the worst possible moment.

Use a retry budget shared by provider, tenant, and operation class. When status indicates provider degradation, open a circuit breaker and preserve work locally. Add full jitter so workers do not wake together. Give writes a smaller budget than reads. Stop when the response is ambiguous, an approval has expired, source state has changed, or the next attempt would exceed the workflow's cost or time ceiling.

Recovery is a controlled release. Start read-only health checks, then a small cohort of low-consequence tasks, then bounded writes. Track error rate, latency, retry consumption, duplicate suppression, and queue age. Do not release every paused agent when the status page turns green.

Reconcile events because webhooks are not a durable queue

GitHub documents that failed webhook deliveries are not automatically redelivered. Consumers must inspect deliveries and request redelivery themselves. Webhooks may also arrive out of order or be throttled during a surge. An agent workflow that advances only when one webhook appears can therefore stall or skip work silently.

Persist every delivery ID, event type, resource version, received timestamp, processing result, and resulting state transition. Maintain a high-water mark per repository or organization. After an incident, compare the ledger with provider state: open pull requests, check runs, workflow runs, review decisions, and failed deliveries. Redeliver or synthesize only the missing transition, then record the reconciliation receipt.

{
  "reconciliation_id": "rec_2026_08_23_0042",
  "repository": "acme/payments",
  "cursor_before": "delivery_91822",
  "provider_window": ["2026-08-17T13:28:00Z", "2026-08-17T21:15:00Z"],
  "missing": ["pull_request_review:pr-482"],
  "duplicate_suppressed": ["check_run:cr-991"],
  "redelivered": ["delivery_91907"],
  "final_state_digest": "sha256:...",
  "reviewer": "platform-oncall"
}

Reconciliation is safer than blind replay because it asks what happened, not what the agent remembers trying to do.

Choose degraded modes before the incident

For package and workflow dependencies, keep an allowlisted cache with digests and provenance. GitHub's Actions cache is useful for performance but remains part of the same platform; it is not an independent continuity store. Cache critical toolchains, action bundles, lockfile-resolved packages, and build images in infrastructure you can access during a GitHub incident. Pin versions and verify hashes so resilience does not become a supply-chain bypass.

Failure modes that look safe until recovery

Run one 30-day continuity drill

1. Days 1-3: choose one real long-running coding-agent workflow and inventory all six dependency planes.
2. Days 4-7: add an external task ledger, task version, source digest, lease, retry budget, and idempotency key.
3. Days 8-10: define provider-uncertain, read-only recovery, bounded replay, and manual-release modes.
4. Days 11-14: mirror the repository and cache the exact toolchain, packages, actions, and build images needed for verification.
5. Days 15-18: simulate API 5xx responses, webhook loss, auth failure, raw-content failure, and a timeout after a remote write.
6. Days 19-21: prove that the agent stops within its retry budget, marks ambiguous effects, and resumes from the last verified step.
7. Days 22-24: reconcile missed events and perform a cohort-based recovery without duplicate comments, branches, checks, or releases.
8. Days 25-27: execute the manual patch-bundle and release path with a named human approver.
9. Days 28-30: close gaps and record recovery time, duplicate suppression, lost work, token cost, review cost, and accepted output.

The pass condition is not “the agent kept running.” It is that no unverified remote effect was repeated, critical work remained inspectable, recovery cost was bounded, and a human could explain exactly what was queued, replayed, or stopped.

Frequently asked questions

Did AI agents cause the August 17 GitHub outage?

The official incident record does not support that broad claim. It identifies a new traffic peak, a sidecar autoscaling policy error, load-balancer saturation, optimistic gateway retries, and a client retry defect that amplified Copilot token requests during recovery. Community discussion about broader AI-generated load is a hypothesis, not the incident RCA.

Is a local Git clone enough?

No. It preserves source objects you already fetched. It does not preserve pull-request state, issue assignments, reviews, hosted checks, webhook deliveries, current identities, cloud-agent sessions, packages, release assets, or model access.

Should every failed request retry?

No. Reads and writes need different budgets. Honor provider headers, add jitter, coordinate retries across the fleet, stop after a bounded ceiling, and reconcile ambiguous writes before another attempt.

Can GitHub Actions cache provide the offline dependency store?

It can improve normal performance, but it is hosted on the same platform and should not be treated as an independent outage store. Keep critical verified artifacts in a separately reachable cache.

What should the first drill test?

Start with a timeout after a write. It forces the system to distinguish failed from ambiguous, prove idempotency, query provider state, and resume without duplicating the effect.