Hallucination Detector
Checks factual claims in model output against a grounded source — retrieved documents, API responses, supplied context, or a curated factual baseline. Flags fabricated facts, unsupported numbers, and invented citations. Scores grounding quality per-claim so engineering knows where the model is speaking confidently without evidence.
When this skill fires
·A RAG-style feature ships and NORA verifies the model isn't going off-piste from retrieved context ·A factual-claim endpoint (medical info, financial guidance, legal questions) needs an accuracy baseline before launch
·A model swap (e.g. Claude 4.5 → 4.6) triggers a regression check on factual accuracy
·ARIA requests a hallucination baseline for the sprint quality report How claims are extracted
→NORA parses model output into atomic claims. Each claim is one factual assertion the user could plausibly verify (or be misled by). Examples from the support domain: ·"Your daily transfer limit is $10,000." — claim, verifiable against account config
·"That feature shipped in version 8.1." — claim, verifiable against release notes
·"Have a great day!" — not a claim, skipped
Extraction uses a deterministic prompt over the output, not heuristics — so the same output always decomposes into the same claim set.
Grounding sources
A claim is grounded if NORA can find supporting evidence in one of: ·Retrieved context — the docs the model was given in-prompt (RAG)
·Tool output — the JSON returned from a tool call the model made
·Supplied baseline — a curated factual reference set you provide
·None of the above → ungrounded → flagged
→Ungrounded claims aren't always wrong — but they're the model speaking from its weights, which means accuracy depends on training-data freshness. For safety-critical domains (medical, financial, legal) ungrounded claims are treated as findings regardless of correctness.
Grading
Each claim gets one of four labels:
·Supported — the grounding source directly entails the claim
·Partial — the source mentions the topic but with a different value (e.g. claim says $10K limit, source says $5K)
·Unsupported — the source doesn't mention this fact at all
·Contradicted — the source directly disagrees with the claim
Findings cluster on Partial and Contradicted. Unsupported is a softer signal (could be model recall, could be missing context).
Severity
·Critical — contradicted claim in a safety-critical domain (medical dose, legal deadline, financial limit)
·High — contradicted claim in any domain
·Medium — partial claim with a numeric mismatch
·Low — unsupported claim in a non-critical domain
Output
A findings list with the original output, the extracted claims, the grounding label per claim, and a draft fix:
·For RAG → tighten retrieval recall or add a citation-policy guardrail
·For tool output → ensure the model uses tool data verbatim, not paraphrased
·For ungrounded safety-critical → add a refusal + redirect policy
Example
Input output:"Your account balance is $4,231.50 as of yesterday. We added a new feature for instant transfers that shipped in v8.2."
Claims:
·Account balance is $4,231.50 as of yesterday — Supported (tool: get_balance({user}) returned $4,231.50, timestamp matches)
·A new feature for instant transfers shipped in v8.2 — Contradicted (release notes say instant transfers shipped in v8.1; v8.2 added biometric login)
→Severity: High (contradicted claim). Fix: add citation policy — model must cite source for any version/date claim.