Hallucination and overconfidence

Fluent prose can assert false specifics with high apparent confidence. Models optimize plausibility under the training distribution, not verified truth . Hallucination in ML usage means confident outputs weakly grounded in evidence or provided context .

Temperature sampling adjusts how peaked the softmax distribution is before drawing a token. Top-$p$ sampling keeps the smallest set of tokens whose cumulative probability exceeds $p$, adapting the cutoff to each step .

Engineering mitigations include retrieval-augmented generation (conditioning on fetched documents), tool APIs, and calibration monitoring, but none eliminate error. Users should treat unsupported claims as hypotheses, not citations .

Overconfidence persists partly because training rewards sounding authoritative on incomplete evidence .

Users should treat numeric claims, citations, and legal/medical advice from raw models as unverified unless tied to retrieved sources or human review workflows .

Calibration tools (reliability diagrams, temperature scaling on a held-out set) adjust reported probabilities but do not by themselves guarantee factual answers on open-domain questions .

Human review loops remain important for high-stakes domains: models can cite plausible sources that do not exist unless retrieval actually fetched them .

Lowering temperature alone does not fix factual errors; it mainly reduces sampling diversity so the model repeats its most likely continuation .