Skip to content
Prompt Management

Take the prompt out of the code

Prompts are versioned in a registry, published by label and distributed to the gateway as a signed bundle. Your application sends a prompt_id and variables; the template is filled in the gateway.

  • Immutable versions, movable labels
  • Gradual rollout through canaries
  • A version that fails the eval gate cannot ship
Request a demoCompare plans

< 30sp99 rollout

A prompt is configuration, not code

Embed a 4 KB instruction in application code and the only person who can change it is an engineer — and every change is a release. A prompt owner needs to be able to cut a version themselves, and rolling back a prompt that broke on a Friday evening should not require waking the release pipeline. The registry provides that; signed distribution keeps it from costing you any security.

What it does

Versions and labels

Versions are immutable, labels are movable. Dev, Staging and Production labels travel between versions; rollback is one call.

The variable contract

An unknown variable returns 400; a missing required variable returns 400. The contract between template and caller never breaks silently.

Canary

Give a new version 10% of traffic. The bucket is deterministic and pinned by conversationId, so a version never changes mid-conversation.

Eval gate

An unsigned eval report is rejected. If a metric the gate asks for is missing from the report, the result counts as a failure — an absent measurement is not a pass.

Prompt-aware policy

Thanks to the input.prompt block, policy can say "this prompt is regulated, pin it to a local model". The prompt is no longer the invisible part of the decision.

Cost per prompt

The decision trace carries prompt identity and version, so cost and cache savings can be broken down per prompt.

Request lifecycle

The 8 stages a request passes through

The order is not arbitrary. Prompt hydration, for instance, runs before policy evaluation: PII arrives inside the variables, so filling the template afterwards would mean scanning the wrong string.

  1. 01

    Auth & quota

    The virtual key is verified, organization and region context resolved, rate limits applied.

    mk_live_… · X-Modelion-Organization
  2. 02

    Signal hydration (PIP)

    Guardrail scanners, budget state and provider health are written into the input the policy will see.

    signals.pii · signals.injection · runtime.health
  3. 03

    Prompt hydration

    If the request arrived with a prompt_id and variables, the template is filled here — before policy, because the PII is inside those variables.

    prompt registry · signed bundle
  4. 04

    Policy decision (PDP)

    Platform baseline, organization governance and custom rules evaluate into one pool; constraints intersect, obligations union.

    Rego → WASM · p99 222µs
  5. 05

    Enforcement (PEP)

    This is where the decision becomes action: deny, mask the prompt, narrow the model pool, flag the obligations.

    deny · redact · constrain
  6. 06

    Semantic cache

    If policy allows it, the answer to a similar prompt is served. When PII is detected, cache writes are switched off.

    pgvector · org scope · threshold 0.85
  7. 07

    Routing & failover

    The combo resolves and candidates are tried in order; failover is possible right up to the first chunk. The circuit breaker takes an unhealthy route out of rotation.

    combo/production-chat · 20 candidates
  8. 08

    Budget, execution & telemetry

    Spend is reserved, the request goes upstream, and the decision trace plus usage record are written.

    ledger · decision_traces · Modelion-Decision

How it works

Four steps from authoring to production — and when a prompt version changes, so does the cache namespace.

  1. 1

    Create the version

    Write the template and the variable schema. Once saved, a version never changes.

    v1.5.0 · immutable
  2. 2

    Put it through eval

    A signed eval report must meet the metrics the gate asks for. The judge model cannot be the candidate model.

    signed eval report
  3. 3

    Publish by label

    Move the staging label, then production. The production label cannot be granted without promotion approval.

    promotion.decision
  4. 4

    Roll out by canary

    Start at 10% of traffic. If the canary is unhealthy, policy stops it itself; an explicit version pin overrides the canary.

    10% → 100%

Technical summary

Call paths
/v1/prompts/{id}/completions · /v1/chat/completions + modelion_prompt
Hydration latency
p99 ≤ 0.4 ms
Versions
Immutable · semver
Labels
Dev · Staging · Production (movable)
Distribution
Signed bundle, same path as the policy bundle · rollout p99 < 30 s
Canary
Deterministic bucket, pinned by conversationId
Rollback
One call
Cache effect
A new prompt version changes the cache namespace

The one ordering decision that matters

Prompt hydration runs BEFORE policy evaluation; prompt rendering runs AFTER routing. Reversing that breaks correctness first and latency second: PII arrives inside the variables, so any design that scans before substitution scans the wrong string — and filling the template after policy would force the decision point to run twice on every request.

The rest of the control plane

This capability works on its own, but most of its value comes from sharing one decision contract with the others.

Policy EngineTake the decision out of the code
Routing & FailoverBind traffic to a decision, not to a model
GuardrailsThe layer that looks inside the prompt
ObservabilityMake the decision explainable

See it on your own traffic

In a 30-minute session we run your own rule set in shadow mode.

Request a demoCompare plans