Week 1 Self-Paced Module: History of Language Models
CSIC30216 · Fall 2026
Overview
In Monday's lecture, you heard about the engineer's new role in the agentic era: specifying intent, evaluating trajectories, and knowing when to take control. You would probably ask yourself: “where did these tools come from?” This is what this week's self-paced module is for. You don't need to understand the math — you will just need to get what the intuition is. By the end of the module, you'll know why these models work the way they do, which will make you a better prompter, evaluator, and engineer working with AI.
Why This Matters
This helps you, very briefly, understand how LLMs work, and how that shapes how you use them. If you know a model is predicting the next token based on probability distributions, you'll prompt differently than if you think it is just retrieving facts from a database. If you understand what embeddings do, you'll make better decisions about which tools to build. If you know the difference between pre-training and instruction tuning, you'll understand why models behave differently and what safety training actually costs.
Format and Timing
- Framing video (6 to 8 minutes, posted on E3) — a brief walkthrough of the big ideas: why embeddings matter, what transformers do at a high level, and how instruction tuning changed everything.
- Required reading — “The Illustrated Transformer” by Jay Alammar (~30 minutes). Linked below and posted on E3.
- Multiple choice questions (單一選擇題) — four questions on E3, about 5–10 minutes. Submit by the end of the week to mark the module complete.
Required Materials
- Framing video: Week 1 Self-Paced Intro — posted on E3 under Week 1 Self-paced Modules.
- Reading: The Illustrated Transformer, by Jay Alammar.
- Multiple choice questions: Question set posted on E3 under Week 1 Self-paced Modules.
Optional Materials
If you want to go deeper on any of these topics (not required, but helpful if you're curious):
- Jay Alammar, “The Illustrated Word2Vec,” 2019 — a great, illustrated article on word embeddings and Word2Vec, helpful if you want to go deeper into the topic.
- Anthropic, “Core Views on AI Safety,” 2023 — the least “technical” of this batch, but provides a great overview of alignment and why it is an important issue in the landscape of rapid model advancement today.
- Simon Willison, “Catching up on the weird world of LLMs,” 2023 — slightly outdated, but a good starting point to dive deeper into LLM capabilities and the logic behind them, including a brief history of LLM advancement up to when we were first startled by what it could do.
Completion and Assessment
Use these questions to check your understanding, not to test yourself. The goal is to make sure you're ready for next week.