Sem2 Week 10

Learning regularities between events: either a cue predicts an outcome (classical conditioning) or an action changes what happens next (operant conditioning).

Classical conditioning associates a stimulus with an involuntary response (Pavlov: bell → salivation). Operant conditioning associates a voluntary action with a consequence (Skinner: lever press → food).

Classical conditioning focuses on stimulus → response. RL shifts to stimulus → reward prediction, modelling behaviour as an internal expectation of reward.

Vt = Vt-1 + α(R − Vt-1), where V is associative strength, R is received reward, α is learning rate, and (R − Vt-1) is the prediction error δ.

δ > 0: outcome better than expected → increase V. δ = 0: outcome as expected → no change. δ < 0: outcome worse than expected → decrease V.

When reward is removed (R drops to 0 after being 1), prediction error becomes negative and V gradually decreases back toward 0.

Reward is only delivered on some trials (e.g. 50% of the time), so V settles at an intermediate value and oscillates around it rather than converging cleanly.

If stimulus v1 already fully predicts reward (V1 ≈ R), adding v2 produces near-zero prediction error (δ ≈ 0), so v2 learns nothing. Prior knowledge blocks new learning.

Dopamine neurons in the substantia nigra fire in a pattern matching the prediction error signal δ — strongly at first, then decreasing as the reward becomes fully predicted (Hollerman & Schultz, 1998).

At each timestep the agent receives state s_t and reward r_t, takes action a_t, which changes the environment, producing new state s_t+1 and reward r_t+1.

Because the prediction error δ = R − V shrinks as V gets closer to R, so each update moves V by a smaller amount — exactly like a capacitor whose charging current falls as voltage approaches its target.

V follows a discrete exponential curve: Vn = R(1 − (1−α)^n). This is the same shape as a capacitor charging (V = Vmax(1 − e^−t/RC)), with learning rate α playing the role of 1/RC — a small α means slow learning, just like a large RC means slow charging.

A stimulus (CS2) acquires predictive value by being paired with an already-conditioned stimulus (CS1), without ever being directly paired with the reward. E.g. a squeaky door → can opener → food means the squeaky door alone triggers salivation.

Because R-W only updates when real reward is present. When CS2 is paired with CS1 but no reward (R=0), the rule extinguishes any association rather than building one. It has no mechanism to pass predictive value along a chain.

The problem of figuring out which past action was responsible for a later reward. Rewards often arrive long after the actions that caused them, so it is unclear what to give credit to.

Instead of comparing your prediction to the actual reward, you compare your current prediction to your prediction one step later: δ(t) = r(t) + V(t+1) − V(t). V(t+1) acts as a proxy for future reward, allowing the signal to propagate backwards in time without waiting for the real reward.

If your prediction next step is higher than your current prediction (δ > 0), something good must be coming — increase your current estimate. If lower (δ < 0), you were too optimistic — decrease it. Reward information ripples backwards step by step across trials.

γ ∈ (0,1] scales the value of future rewards. γ near 1 = patient, values future rewards almost as much as immediate ones. γ near 0 = short-sighted, only cares about immediate reward. Captures the tradeoff: chocolate now or chocolate tomorrow?

V(s) is the value of being in a state — how much total future reward to expect from here. Q(s,a) is the value of taking a specific action a in state s. Q-values allow the agent to directly choose the best action, rather than just knowing how good a state is.

V(s) = max over actions of [R + γV(s’)]. The value of a state equals the best immediate reward plus the discounted value of the best reachable next state. It defines the optimal policy recursively.

An RL algorithm that learns Q(s,a) — expected future reward for each state-action pair — directly. The agent picks the action with the highest Q-value in the current state. It is more efficient than computing state values first.

Dopamine neurons in the VTA/substantia nigra fire in a pattern matching the TD prediction error δ. Early in training they fire at reward; after learning, firing shifts to the earliest predictor of reward and drops to zero at reward time (Schultz 1998).

Dopamine neurons (midbrain/VTA): signal TD prediction error δ. Striatum: stores value estimates V(s) and Q(s,a). Cerebral cortex: represents current state and actions. Pallidum: handles action selection.

AlphaGo Zero learned purely through self-play with no human game data at all. It rediscovered centuries of human Go strategy in ~3 days, and surpassed the original AlphaGo which had been trained on human expert games.