42. Activation Functions, Loss, and Gradient Descent (Plain English)
42.4 Learning Rate Simply
Why. Learning rate too big → overshoot. Too small → snail training.
What. Learning rate (lr) is the step size. Beginners start with values like 0.1, 0.01, 0.001 and watch the loss curve.
How.
Compare two lrs on the toy
- Rerun the Chapter 46.3 loop with lr=1.5 (often unstable).
- Rerun with lr=0.01 (slow but steady).
- Prefer steady drop of loss over wild jumps.
def walk(lr, steps=6):
w = 0.0
hist = []
for _ in range(steps):
w = w - lr * 2 * (w - 3)
hist.append(round(w, 3))
return hist
print("lr=0.2", walk(0.2))
print("lr=1.5", walk(1.5))
Ravindra Bagale's Tip
Loss graph madhe spike asel tar lr kami kara. Patience > huge steps.