An AI feature can produce an answer that is clear, relevant and apparently correct.

On that evidence alone, I still wouldn’t ship it.

A successful demonstration answers one question: can this work?

Production requires answers to several others:

  • Will it behave acceptably across the range of inputs we actually receive?
  • Can we show where its factual claims came from?
  • What is it permitted to read, recommend, prepare or change?
  • What happens when the model, retrieval process or a tool fails?
  • Can we detect, investigate and reverse a bad change?

A good model response is valuable. It is not yet a reliable system.

The engineering work sits in the gap between those two things.

The production path

A proposal earns authority one gate at a time

  1. 01 Model proposes A useful answer, classification or suggested action.
  2. 02 Evidence attaches Claims point back to acceptable retrieved material.
  3. 03 Rules validate Deterministic code checks state, ranges and invariants.
  4. 04 Authority allows The system or a person approves what may happen next.
  5. 05 Action proceeds The outcome is recorded, observable and reversible.

Any failed or uncertain gate Refuse · explain · log · send for human review

A plausible answer is the start of the path. Evidence, validation and authority decide whether it may influence a production action.

One successful response is not enough evidence

Traditional software can pass a demonstration and fail in production. AI makes the weakness of demonstration-driven testing more pronounced because its output is probabilistic and sensitive to context.

Running the same friendly prompt several times tells us very little about the feature’s operating boundary. I would rather build a scenario set that represents the work the system will encounter:

  • a normal request with complete information;
  • an ambiguous request that requires clarification;
  • missing or stale supporting material;
  • two approved sources that disagree;
  • an instruction outside the feature’s authority;
  • a dependency timeout or partial tool failure;
  • hostile content inside a retrieved document—such as an embedded instruction attempting to redirect the model.

Each scenario needs an expected behaviour. That does not always mean one exact sentence. It might mean that the response must include a fact, cite an approved source, ask for clarification, avoid a prohibited action or return control to a person.

The better question: not “Was that answer impressive?” but “How often does the system meet an explicit standard across representative and difficult cases?”

For .NET teams, this can fit into familiar test and delivery workflows. Microsoft’s AI evaluation libraries include evaluators for qualities such as relevance, completeness, groundedness, task adherence and tool-call accuracy, with support for ordinary test frameworks and CI/CD.

Those scores are signals, not proof. Some quality evaluators use another language model to score the response. Pair them with deterministic assertions, known ground truth and human review appropriate to the risk.

Evidence changes the value of an answer

Suppose the diagnostic assistant says:

“The most likely cause is a communication timeout during the previous operating state.”

That may be correct. But what supports it?

Did the system read an alarm record, an event trace and the relevant section of an approved guide? Did it infer the answer from a partial description? Is it repeating a pattern learned from unrelated material?

The wording alone does not tell us.

In production, important factual claims should connect to evidence the surrounding application can inspect: a document identifier, alarm code, database record, timestamp or exact section of a guide. A citation is useful only when the application verifies that it refers to retrieved material and that the material actually supports the claim.

If the evidence is absent or contradictory, the correct behaviour may be to say that the conclusion cannot be verified. That is less impressive in a demonstration and far more useful in a real system.

Structured output is a boundary, not proof

Asking a model to return structured data is valuable. A defined schema gives the application something it can parse and validate. But valid JSON is not the same as a valid decision.

The surrounding software still needs to check:

  • required fields and permitted values;
  • ranges, identifiers and relationships;
  • whether referenced records actually exist;
  • whether the response is consistent with current system state;
  • whether the proposed action is allowed in this context.

Deterministic business rules should remain deterministic. An AI component can classify, summarise, retrieve or propose. It should not quietly become the source of truth for an invariant ordinary code can enforce reliably.

The schema protects the software boundary. Semantic validation protects the operational boundary.

Authority should be introduced in stages

One of the most important design decisions is not which model to use. It is what the AI is permitted to do.

  1. ReadAccess approved information.
  2. ExplainSummarise or answer with evidence.
  3. RecommendSuggest a bounded next action.
  4. PrepareCreate an action for explicit approval.
  5. ExecuteAct only within validated authority.

Teams can move through these stages deliberately. They should not arrive at the final stage accidentally because a model has been given a powerful tool and a persuasive system prompt.

Model confidence is not permission. Confidence can inform a decision; it cannot grant authority, satisfy a safety rule or replace explicit approval.

The failure path is part of the feature

Any production dependency can fail. AI services are no exception.

The model may time out. Retrieval may return nothing useful. A tool call may fail after a partial operation. The response may violate its schema. A provider change may alter behaviour. The system may simply be uncertain.

Depending on the risk, an intentional degraded mode might mean:

  • retaining the established deterministic workflow;
  • showing the evidence without generating a conclusion;
  • marking the case for manual review;
  • retrying only operations that are safe to repeat;
  • refusing an action and recording the reason;
  • escalating to an operator or support team.

A feature is not production-ready if its useful path has been designed but its failure path is still an exception message. Failing safely is part of the functional behaviour.

If we cannot observe it, we cannot own it

When an ordinary service misbehaves, we inspect logs, traces, metrics and deployments. AI features require the same discipline, with some additional context.

For an investigation to be meaningful, we may need to know:

  • the application and prompt versions;
  • the model or deployment identifier and any version information the provider exposes;
  • which material was retrieved;
  • which tools were called and with what result;
  • latency, resource use and validation failures;
  • whether a person accepted, changed or rejected the suggestion.

Observability must respect privacy and security boundaries. But without enough traceability, a team cannot explain a failure, compare releases or know whether a change made the system safer.

Prompts, retrieval strategies, application code, configuration and evaluation sets should be versioned. We should be able to compare releases, roll back application, prompt and configuration changes, and replace a model safely when required.

My minimum production release gate

Before calling an AI-enabled feature production-ready, I would want clear answers to ten questions.

  1. RoleWhat precise task is the feature responsible for, and what is outside its role?
  2. ScenariosWhich representative and difficult cases are evaluated before release?
  3. RequirementsWhich behaviours are hard requirements rather than preferences?
  4. EvidenceCan important factual claims be traced to acceptable supporting material?
  5. ValidationWhich outputs and decisions are checked by deterministic code?
  6. AuthorityWhat can the AI read, explain, recommend, prepare and execute?
  7. ApprovalWhere is explicit human or system approval required?
  8. FailureWhat happens when the model, retrieval process or a tool fails?
  9. TraceabilityCan we investigate behaviour without exposing inappropriate data?
  10. ChangeCan we compare releases, roll back controlled changes and replace a model safely?

The answers will differ between a document assistant, a financial workflow and an industrial diagnostic system. That is exactly the point. “Using AI” is not a sufficient architecture. The surrounding risk and responsibility determine the engineering.

The correct answer is the beginning

A correct response proves that the model can succeed in one situation.

Engineering establishes whether the complete system can be trusted across many situations, including the uncomfortable ones. That work involves test cases, evidence, validation, permissions, fallback behaviour, telemetry and human ownership.

AI engineering is not only about making a model answer. It is about designing a system that knows what the answer may be used for, how it can be checked and what must happen when it is wrong.

Sources and further reading