Jev gives an agent typed decisions, not safe decisions

The interesting change is the interface, not the launch claim

Most agent loops ask a generative model to answer a narrow operational question: which queue owns this ticket, whether a draft violates a rule, which tool should run next, or whether a case needs a person. The model emits text or JSON, the application parses it, and the output often carries more freedom than the decision requires. Jev makes a narrower contract explicit: provide state, define typed questions, receive typed judgments.

That design moved from product announcement to usable developer surface in September. Pydantic AI releases 2.45 and 2.46 added a TypeSafe provider, typed output support, configurable boolean thresholds, union handling, and tool-routing behavior. TypeSafe publishes Python and JavaScript SDKs plus an MIT-licensed agent skill. The focused 30-day scan found 25 Reddit threads, 13 Hacker News items, and two specified GitHub projects. TypeSafe's skill repository carried roughly 1,200 stars during the scan, while the Pydantic AI repository supplied the live integration path.

The community reaction is useful because it cuts through the category label. One high-voted comment called the launch “the industry rediscovering classification models,” while another described it as classification with broader language understanding. Both are partly right. Jev occupies ground between deterministic rules, a task-specific classifier, and a generative LLM. It can evaluate flexible natural-language state against questions chosen at runtime, but it still returns a bounded decision rather than an open-ended completion.

The caution arrived just as quickly. Developers testing Jev for model routing asked how often it disagreed with a regular LLM, whether reported confidence was calibrated, and what happened on ambiguous or out-of-distribution inputs. Those are not peripheral benchmark questions. They determine whether the model should automatically route a low-stakes support ticket, merely flag content for review, or remain outside any action path.

Separate the state from the judgment

A generative prompt usually mixes material and instruction: “Read this ticket and decide whether it is urgent.” Pydantic AI's TypeSafe documentation describes the opposite habit for Jev. The prompt carries the material to judge. The output type and field descriptions carry the questions. Each supported field becomes a separate typed judgment over the shared state.

The state/question split improves inspectability. A reviewer can see the exact input, each atomic question, the allowed options, the selected answer, and provider details. It also exposes design errors. If a field asks three judgments at once, a typed response can still hide which clause drove the result. If the question is accidentally placed inside the state, the model evaluates it as content rather than treating it as the judgment contract.

Ask one operationally meaningful thing per field. “Is this a good account?” is too vague. “Is the customer asking to cancel?”, “Does the message contain a charge dispute?”, and “Does policy require review within one hour?” are separable. Combine their results in code that the team owns and tests. The model supplies evidence-bearing judgments; the application owns the policy.

Put Jev in a six-stage router, not directly on the actuator

authoritative event or agent state
  -> deterministic schema, size, identity, and policy prechecks
  -> minimum necessary state projection
  -> pinned Jev version + versioned typed questions
  -> action-specific confidence and validation gate
       | accepted advisory route
       | fallback model / human review
       | reject as out of scope
  -> deterministic authorization and side-effect controls
  -> outcome, disagreement, override, and drift evidence

Prechecks reject missing identity, malformed fields, forbidden content, unsupported media, oversized state, and actions the caller can never perform. A model should not decide whether a request is structurally valid or whether a principal possesses a permission that an access-control system can answer exactly.

Projection sends only the state needed for the question. Pydantic AI documents that conversation history, tool arguments, and tool results can be sent to the TypeSafe API when they are part of the judged history. That is a data-boundary decision, not a convenience flag. Remove secrets, unrelated turns, and provider-irrelevant details before the call.

Judgment binds a pinned model version to a versioned question schema. Moving aliases are convenient during exploration; they are hostile to a calibrated production threshold. The response should record the resolved model version, schema version, input population tag, answer, confidence details, latency, and provider status.

Validation applies a threshold chosen for the action, not for the model globally. A low-cost label that only sorts a queue can tolerate more uncertainty than a route that suppresses an alert. Missing facts, unfamiliar segments, or a failed adversarial check should send the case to fallback even when the model sounds confident.

Authorization remains deterministic. A router may recommend refund; it should not decide whether the caller can refund $8,000, whether the order is eligible, or whether dual approval is required. Those are policy and state-machine questions. Keep them outside the model.

Outcome evidence closes the loop. Record whether the human or downstream model agreed, which route ultimately succeeded, the cost of fallback, and the consequence of errors. Without outcomes, the team can measure latency and confidence but not decision quality.

Compile the decision contract from code

Pydantic AI lets a normal output model become the question contract. The example below keeps the task advisory: it classifies a support request, reports an urgency flag, and exposes confidence for a routing gate. Authentication and actual queue permissions remain outside the model.

from enum import Enum
from pydantic import BaseModel, Field
from pydantic_ai import Agent

class Queue(str, Enum):
    billing = "billing"
    security = "security"
    account = "account"
    other = "other"

class Triage(BaseModel):
    """Classify an inbound support request; do not authorize an action."""
    queue: Queue = Field(description="Which team owns the request?")
    urgent: bool = Field(description="Does policy require review within one hour?")
    contains_secret: bool = Field(description="Does the text expose a credential or secret?")

router = Agent("typesafe:jev-1.13.0", output_type=Triage)
result = router.run_sync("My production API token was posted in ticket #4821.")

details = result.response.provider_details or {}
confidence = details.get("confidence", {})
minimum = min(confidence.values(), default=0.0)

if result.output.contains_secret:
    route = "security-quarantine"
elif minimum < 0.85:
    route = "human-triage"
else:
    route = result.output.queue.value

# A separate policy engine now checks caller identity, permissions,
# data handling, queue availability, rate limits, and audit requirements.

This example is intentionally incomplete as a production security control. It needs labelled tests, an approved data boundary, failure handling, timeout and retry rules, segment-aware thresholds, and a deterministic path for secret detection. Its value is structural: input, questions, answer, confidence, and downstream policy are separate artifacts.

Store the contract in a versioned manifest so a runtime change is reviewable:

apiVersion: typed-decision-router/v1
router: support-triage
provider: typesafe
model: jev-1.13.0
schema_commit: "<git-sha>"
population: english-support-tickets-v3
actions:
  queue_only:
    min_confidence: 0.78
    fallback: human-triage
  urgent_alert:
    min_confidence: 0.90
    fallback: deterministic-rule-plus-human
prohibited_authority:
  - close_ticket
  - disclose_customer_data
  - issue_refund
validation:
  labelled_set: triage-gold-2026-09
  adversarial_set: triage-injection-2026-09
  max_segment_error_rate: 0.03
  max_fallback_rate: 0.35

Calibrate the action, not the confidence field

Pydantic AI warns that Jev confidence is not one uniform probability that the answer is correct. For yes/no outputs, the documented value reflects distance from a coin flip. For pick-one fields it comes from the model's distribution. For a list it may reflect the least certain option. A bounded float can itself be the judgment, with no second confidence number. A single threshold copied across these shapes has no coherent meaning.

Build a labelled evaluation set from the population the router will actually see. Freeze model and schema versions, run every case, and preserve the full response. Report confusion matrices per field, not only aggregate accuracy. For a queue selector, measure per-class precision and recall. For a boolean gate, measure false negatives at the intended threshold. For multi-label output, measure each option and co-occurrence pattern. Slice results by language, source, customer tier, message length, ambiguity, and any segment that changes the decision cost.

1. Define the action loss: write the operational cost of a wrong route, missed flag, unnecessary fallback, and delayed case.
2. Freeze the corpus: deduplicate, label independently, adjudicate disagreements, and keep a future-like holdout.
3. Measure calibration: bucket predictions by the relevant score and compare predicted strength with observed outcomes.
4. Choose thresholds by action: automated routing, review flagging, and destructive action cannot share one bar.
5. Run a challenger: compare Jev with rules, a simple classifier, the current LLM, and human decisions on the same cases.
6. Inspect disagreements: disagreement cases teach more than a blended win rate. Classify missing context, label ambiguity, model error, and policy mismatch.
7. Shadow before acting: log recommendations without changing the live path; measure fallback, override, error, and segment drift.

Vendor latency and cost claims are useful hypotheses, not acceptance evidence. Measure end-to-end latency including serialization, network, retries, fallback, and downstream work. Measure cost per completed business task, not per decision call. A cheap router that hands most cases to a full model can cost nearly as much as the baseline while adding complexity.

A decision model is not a permission kernel

TypeSafe's narrow output surface can reduce arbitrary text and parsing risk, but the judged state is still untrusted input. Pydantic AI explicitly documents adversarial text as a weakness: content designed to steer a judgment can move the answer. Option order can also influence results. A schema can be valid while the selected option is attacker-controlled.

Separate four questions that agent frameworks often collapse:

Tool selection deserves particular care. Pydantic AI can let Jev choose between output and available tools, and newer integration work can fill arguments where Jev's supported typed decisions express them. That is routing, not consent. A no-argument tool can still send mail, rotate a credential, or open a production incident. Wrap every effectful function with independent identity, scope, approval, rate, environment, and idempotency checks.

Data handling is another boundary. Do not pass a full conversation when the judgment needs one ticket field. Tool outputs may contain credentials, private documents, or injected instructions. Project the minimum state, redact secrets before provider transmission, record what categories were sent, and give the caller a clear retention and residency policy.

Failure modes a polished demo will miss

Also test provider failure. Timeouts, rate limits, version removal, malformed responses, and changed SDK behavior should produce a known state. “Use the first option” is not a fallback. Decide whether to queue, reject, use deterministic rules, call a challenger, or ask a person. Preserve the original case and the reason for the fallback.

Use a 30-day acceptance plan

Week 1: choose one bounded, reversible decision such as queue assignment. Define the population, action loss, prohibited authority, source fields, data boundary, model version, question schema, baseline, labels, reviewers, and stop conditions.

Week 2: build a gold set and adversarial set. Compare Jev with deterministic rules, the current generative model, and human labels. Measure field-level error, calibration, segment performance, latency, cost, fallback, and disagreement. Rewrite compound questions rather than tuning around them.

Week 3: shadow the live stream. The router makes recommendations but does not change the operational path. Review every high-impact disagreement, every out-of-scope case, and a risk-based sample of agreements. Verify that minimum-state projection and provider logging match policy.

Week 4: enable one advisory route for a small cohort. Keep deterministic authorization and a human recovery queue. Publish a versioned acceptance record with thresholds, known gaps, rollback, monitoring, expiry, and the exact change triggers that reopen evaluation.

Track outcome error, not only model output. Useful metrics include false route rate, missed urgent cases, human override, fallback rate, unresolved queue age, decision latency, provider failure, cost per resolved case, segment drift, option-order instability, and incidents prevented by the deterministic policy layer.

For choosing among full generative models, use the task-based model routing guide. For enforcing authority after a route is selected, use the permission-kernel architecture. Jev belongs between those layers: a narrow judgment engine whose output is measured and constrained.

FAQ

Is Jev just a classifier?

It serves classifier-like operational jobs, but the questions and option sets can be supplied dynamically over natural-language state. That flexibility is useful. It also means each deployed question contract needs its own validation rather than inheriting reliability from the product label.

Should I replace structured-output LLM calls with Jev?

Compare them on your labelled task. Jev is attractive when the output is a bounded judgment and open-ended generation adds no value. Keep a generative model for writing, multi-step reasoning, missing-argument recovery, or tasks outside Jev's supported output forms.

Can I use jev-latest in production?

Use a moving alias for exploration. Once a threshold and acceptance result depend on model behavior, pin the resolved version and move through a deliberate revalidation process.

What is the safest first use case?

Start with advisory classification that is reversible, labelled, high-volume, and easy for a person to correct. Avoid secrets, payments, access grants, destructive tools, or decisions whose error cannot be repaired.