LLMOps and AI observability

Use when an AI feature has grown beyond a prototype and ownership is becoming unclear.

Map the lifecycle for this AI feature from data and prompt changes through evaluation, deployment, monitoring, incident response, and retirement. Name the artifact, owner, approval, version, environment, and rollback path at each stage. Include the minimum release gate for a small pilot and the additional gates needed for customer-facing or high-consequence use.

AI feature:
[PASTE FEATURE]
Team and environments:
[PASTE TEAM]
Risk profile:
[PASTE RISK]

Output: An owned lifecycle with release and rollback gates.

Guardrails: Version prompts, models, data, and tools · Do not ship without a rollback owner · Scale gates with risk

Use before promoting a prompt, model, retrieval, or tool change to production.

Create a release evidence pack for this AI change. Include change summary, affected users and workflows, evaluation set and results, known regressions, latency and cost impact, security review, data changes, human approval, rollout plan, monitoring dashboard, and rollback command or procedure. Mark each item complete, missing, or not applicable, and identify the release blocker.

Change:
[PASTE CHANGE]
Current baseline:
[PASTE BASELINE]
Risk and compliance requirements:
[PASTE REQUIREMENTS]

Output: A reviewable release decision with explicit blockers.

Guardrails: Do not treat a demo as evidence · Keep known limitations visible · Require a rollback path

Use when a team can see that an AI workflow is wrong but cannot locate where it failed.

Design an observability map for this AI workflow. Define metrics, logs, and traces for request identity, prompt and model version, retrieved context, tool calls, latency, tokens, cost, output quality, human override, escalation, safety events, and business outcome. For each signal state source, retention, access, dashboard, alert, and owner. Include the trace fields needed to reconstruct one run without storing unnecessary sensitive content.

Workflow:
[PASTE WORKFLOW]
Systems:
[PASTE SYSTEMS]
Privacy constraints:
[PASTE CONSTRAINTS]

Output: An AI observability schema and operating dashboard plan.

Guardrails: Minimise sensitive telemetry · Correlate model behavior with application events · Prefer actionable signals

Use when quality needs to be reviewed consistently across models, prompts, retrieval, or agents.

Create a human and automated evaluation rubric for this AI feature. Separate task success, groundedness, instruction following, safety, citation quality, tone, latency, cost, and user outcome. Define a five-point scale, examples of pass and fail, sampling method, evaluator instructions, disagreement handling, and release thresholds. Make clear which metrics are diagnostic and which block release.

Feature:
[PASTE FEATURE]
Representative outputs:
[PASTE OUTPUTS]
Business outcome:
[PASTE OUTCOME]

Output: A repeatable rubric with thresholds and evaluator guidance.

Guardrails: Do not equate fluent writing with task success · Keep safety and business outcomes separate · Document evaluator disagreement

Use when an AI workflow needs a reliable quality, latency, and cost envelope.

Set a service budget for this AI workflow. Define target and maximum latency, error rate, quality score, token or inference cost, concurrency, data freshness, and recovery time. Map each budget to the design lever that can change it: model, prompt, context, caching, retrieval, batching, routing, or human fallback. Include alerts, an owner, and the action when a budget is exceeded.

Workflow:
[PASTE WORKFLOW]
Usage forecast:
[PASTE FORECAST]
Quality requirement:
[PASTE QUALITY]

Output: An actionable service budget with tuning levers.

Guardrails: Do not trade away safety for latency · Measure outcome quality alongside cost · Set a maximum before scaling

Use before exposing an LLM workflow to customer data, internal systems, or regulated decisions.

Review this AI system for security and compliance. Cover identity and access, data minimisation, encryption, provider and model boundaries, prompt injection, tool abuse, retention, secrets, auditability, human oversight, incident response, and user disclosure. Return blockers, mitigations, evidence needed, and the approval owner. Do not assume a vendor feature satisfies a requirement without verification.

System:
[PASTE SYSTEM]
Data and identities:
[PASTE DATA]
Applicable requirements:
[PASTE REQUIREMENTS]

Output: An AI security review with blockers and owners.

Guardrails: Least privilege by default · Treat model output as untrusted · Escalate legal or regulatory uncertainty

Use when the team needs to respond consistently to a quality, safety, cost, or availability incident.

Write an incident runbook for this AI workflow. Cover unsafe output, data leakage, prompt injection, tool misuse, provider outage, latency spike, cost anomaly, stale context, and silent quality regression. For each incident define detection signal, severity, immediate containment, user communication, rollback or fallback, evidence to preserve, owner, and resume criteria. Include a short post-incident review agenda.

Workflow:
[PASTE WORKFLOW]
Dependencies:
[PASTE DEPENDENCIES]
Incident roles:
[PASTE ROLES]

Output: A practical response, containment, and resume policy.

Guardrails: Contain before retrying · Preserve a human recovery path · Communicate uncertainty clearly

Use when a system is intermittently slow, wrong, or inconsistent without a clean outage.

Investigate this AI system's gray failure using its metrics, logs, traces, evaluations, deploy history, and user reports. Form a short list of hypotheses, rank them by evidence, identify missing signals, and propose the smallest safe experiment for each. Distinguish symptom, likely cause, confirmed cause, and remediation. Do not invent a root cause from correlation alone.

Incident:
[PASTE INCIDENT]
Telemetry:
[PASTE TELEMETRY]
Recent changes:
[PASTE CHANGES]

Output: An evidence-ranked investigation and safe next tests.

Guardrails: Label hypotheses as unconfirmed · Do not roll back without a reason · Protect user data in evidence