Week 6 Self-Paced Module: RLHF, Constitutional AI, and Calibration

Overview

This week's async lecture explained what the failure modes are — hallucination, sycophancy, sensitivity, over-refusal, bias. This module explains why they exist at the training level. It covers the RLHF pipeline, where sycophancy enters it, Constitutional AI as an alternative, and calibration. Understanding the mechanism behind a failure is what lets you design around it rather than just hoping it does not happen.

Why This Matters

Knowing that sycophancy exists tells you to watch for it. Knowing that it comes from RLHF reward modelling tells you why giving the model specific, checkable criteria works as a mitigation — you're designing around a trained preference, not asking the model to ignore one. Knowing that hallucination is a consequence of next-token prediction tells you why retrieval-augmented generation addresses it at the right level, rather than just asking the model to "be more careful." Mechanism understanding is what turns the failure modes and their mitigations into a system you can reason about, instead of a list of facts to memorise.

Format and Timing

• Framing video (~7 minutes, posted on E3) — link coming soon; check back before starting the module.
• Required reading — Anthropic's “Core Views on AI Safety” plus selected sections of Sharma et al. on sycophancy (~25 minutes combined). Linked below.
• Multiple choice questions (單一選擇題) — four questions on E3, about 10 minutes.
• Total estimated time: ~60 minutes
• Complete before Week 7 (October 19).

What You'll Learn

• The RLHF pipeline — three stages: supervised fine-tuning on (prompt, ideal response) pairs written by human contractors; reward model training, where humans rank multiple responses to the same prompt and a reward model learns to predict those preference scores; and RL fine-tuning, where the language model is optimised to maximise the reward model's score. The result is a model aligned with what human raters prefer — which is not always the same as what is most accurate or most helpful.
• Where sycophancy enters — human raters tend to score confident, agreeable, fluent responses higher, so the reward model learns to associate those surface properties with quality, and the RL stage optimises the language model to produce them — even when agreement is wrong. This isn't a design failure; it's an emergent consequence of optimising for human preference at scale, which is also why you can't just instruct a model to "be less sycophantic" — that instruction competes with a deeply trained preference.
• Constitutional AI as an alternative — Anthropic's approach replaces human raters with the model critiquing its own responses against a set of explicit written principles (the “constitution”): generate a response, critique it against each principle, then train on the self-critiqued, revised version. The constitution is explicit and auditable, so alignment reflects articulated principles rather than opaque rater preferences — though it does not eliminate sycophancy on its own.
• Calibration — a model is well-calibrated if its expressed confidence matches its actual accuracy. Current models are often miscalibrated and overconfident on facts they get wrong, which is why a hallucination delivered confidently is much harder to catch than one delivered with appropriate uncertainty. Calibration is distinct from hallucination: a model can hallucinate while still expressing appropriate uncertainty, or hallucinate while sounding completely sure.
• Measuring sycophancy — the pushback test (does the model maintain a correct answer under confident but unsupported disagreement?) and the consistency test (does it answer the same question consistently across framings that imply different answers?). Sharma et al. (2024) found sycophancy across five RLHF-trained assistants on free-form tasks — the practical implication is to treat any AI evaluation component in your project as likely to lean toward the user's apparent preference, and to test for it directly.

Required Materials

• Framing video: link coming soon — posted on E3 under Week 6 Self-paced Modules once available.
• Reading: Anthropic, “Core Views on AI Safety” — read the full post (~15 minutes).
• Reading: Sharma et al. (2023/2024), “Towards Understanding Sycophancy in Language Models” — read the abstract, the introduction, and the section on how human preference data and preference models relate to sycophancy (~10 minutes). You do not need the full experimental details.
• Multiple choice questions: link coming soon — posted on E3 under Week 6 Self-paced Modules once available.

Optional Materials

If you want to go deeper on any of these topics (not required, but helpful if you're curious):

• Anthropic, “Constitutional AI: Harmlessness from AI Feedback” (blog summary) — the accessible summary of the Constitutional AI paper, explaining the self-critique mechanism without requiring an ML background.
• Lilian Weng, “Extrinsic Hallucinations in LLMs” — a thorough technical treatment of hallucination types and mitigations. Longer than the required reading — read selectively: the “Types of Hallucination” and “Mitigation Methods” sections are most relevant.
• Perez et al. (2022), “Discovering Language Model Behaviors with Model-Written Evaluations” (abstract only) — includes early measurements of sycophancy as a model behaviour. One page. Shows that it can be measured, which makes it an engineering problem rather than a philosophical one.

Completion and Assessment

← Back to Calendar