Reinforcement Learning: An Introduction
A chapter-by-chapter guide to Sutton & Barto (2nd ed., 2020), Chapters 1–7 and 9: every algorithm with its pseudocode and salient features, side-by-side comparisons, all key formulas, the role of each hyperparameter, interactive simulations, and self-test questions.
Chapters
1 · Introduction
RL vs supervised/unsupervised · exploration vs exploitation · policy, reward, value, model · tic-tac-toe TD learner · history.
2 · Multi-armed Bandits
Action-value methods · ε-greedy · incremental update · constant vs sample-average step size · optimistic initial values · UCB · gradient bandit.
3 · Finite Markov Decision Processes
Agent–environment interface · dynamics p(s′,r|s,a) · returns & discounting · episodic vs continuing · Bellman equations · optimality.
4 · Dynamic Programming
Policy evaluation · policy improvement theorem · policy iteration · value iteration · asynchronous DP · GPI · efficiency.
5 · Monte Carlo Methods
First-visit vs every-visit · exploring starts · on-policy ε-soft · off-policy via importance sampling (ordinary vs weighted) · incremental MC.
6 · Temporal-Difference Learning
TD(0) · TD error · TD vs MC vs DP · batch updating · Sarsa · Q-learning · Expected Sarsa · maximisation bias & Double Q-learning.
7 · n-step Bootstrapping
n-step TD · n-step Sarsa · off-policy n-step with importance sampling · per-decision control variates · tree backup · Q(σ).
9 · On-policy Prediction with Approximation
Value error $\overline{\mathrm{VE}}$ · SGD & semi-gradient · linear methods & TD fixed point · coarse/tile coding · RBFs · step-size rule · ANNs · LSTD · kernels · interest & emphasis.
Chapter 8 (Planning and Learning with Tabular Methods) is intentionally out of scope.
Revision tools
Algorithm catalogue & comparisons
Every algorithm in one place: update target, on/off-policy, bootstrapping, model needs, and how each differs from its neighbours.
Hyperparameters & step sizes
Constant vs decreasing α, the Robbins–Monro conditions, ε, γ, c, n, Q₁, θ, feature resolution — what each controls and how to set it.
Formula sheet
All key equations from Chapters 2–9 on one printable page.
Mixed exam practice
Cross-chapter questions, worked numerical problems and "compare and contrast" answers.
The big picture: how the methods relate
Almost every method in Chapters 4–7 updates a value estimate toward a target. Methods differ along two dimensions (book §8.13, Figure 8.11):
- Width — does the update use a single sampled next step, or the expected value over all possible next steps (which needs a model)?
- Depth — does the target bootstrap from the current estimate after one step, or wait for the actual return at the end of the episode?
Bandits (Ch 2): target $=R$. Monte Carlo (Ch 5): target $=G_t$. TD(0) (Ch 6): target $=R_{t+1}+\gamma V(S_{t+1})$. n-step TD (Ch 7): target $=G_{t:t+n}$. Function approximation (Ch 9): the same targets, but the "estimate" is $\hat v(S_t,\mathbf{w})$ and the step is taken along $\nabla \hat v$.