jperezduerto.info
Security in depth · Unit 27 · Lesson 8 of 11
Generated output still needs validation
Review generated content separately for structure, factual support, and disclosure.
Helpful before thisAI on offense
After this lesson you can
- Identify which output checks passed and which remain unresolved in a supplied customer-message draft.
One idea. One situation. One reasoned decision.
How it works
Model output can be inaccurate, malformed, or unsuitable for its destination. Rendering text, constructing a database operation, sending a message, and invoking a tool have different safety requirements. Validate structure and policy at the destination and use appropriate safe representations. A well-formed response is not necessarily factually correct or authorized to cause an external effect.
Valid structure can carry an unsupported claim
A fictional assistant produces a customer update in the application’s required message format.
| Record | Review observation |
|---|---|
| O1 | Schema validation passes: required fields and types are present |
| O2 | Draft says the service is restored and the cause confirmed |
| O3 | Approved incident record says service checks are incomplete and cause remains unknown |
Worked decision: the structure passes, but the factual claim does not. Rewrite the update to match O3: restoration checks are incomplete and the cause has not been confirmed. A schema validator can establish expected shape without deciding whether those statements are true.
Before sending, check the actual recipient and permitted disclosure as separate requirements. Correct facts sent to the wrong audience can still cause harm. Likewise, reviewing the wording does not prove the final tool arguments contain the reviewed recipient and content.
At display boundaries, treat generated text according to the destination’s rendering rules instead of assuming it is trusted application code. At action boundaries, validate the concrete effect and authorization. Keep these checks named separately in the acceptance record so a green structure check cannot silently stand in for truth, safe presentation, or permission.
The key distinction: Syntax, truth, and permission are separate properties.
Check yourself
No timer. No penalties. Read the explanation and try again whenever you like.
This lesson’s questions have changed. Your reading progress is saved; review the updated questions.
-
What does O1 establish about this draft?
Show the answer
Correct answer: The required structure is present; the status claims still need correction and destination checks remain separate. O1 supports structure only. The content must match the approved incident evidence, with independent review of destination and disclosure.
Try it
- WriteRewrite the draft’s status claim using O1-O3. Then specify separate acceptance checks for factual support and the actual destination before the message can be sent.