42. Activation Functions, Loss, and Gradient Descent (Plain English)
42.3 Gradient Descent Intuition
Why. Knowing the loss is not enough — you must move weights downhill on that loss surface. That walk is gradient descent.
What. Gradient ≈ “which way loss increases fastest.” Descent steps the opposite way: weight := weight − learning_rate × gradient.
How.
Hill story
- You stand on a foggy hill (loss).
- Feel the slope under your feet (gradient).
- Take a small step downhill (learning rate).
- Repeat many times (epochs / iterations).
# 1D toy: loss = (w - 3)**2 ; true best w is 3
w = 0.0
lr = 0.2
for step in range(8):
grad = 2 * (w - 3) # d/dw of (w-3)^2
w = w - lr * grad
print(step, "w=", round(w, 3), "loss=", round((w-3)**2, 3))
What you should see. w walks toward 3 and loss shrinks. Real nets have millions of weights — same idea, automated.