42. Activation Functions, Loss, and Gradient Descent (Plain English)
42.1 Activation Functions Simply
Why. Without an activation, stacked layers collapse into one big linear formula. Activations add the bends that let nets learn curves.
What. Common starters: ReLU (max(0, z)) for hidden layers; sigmoid or softmax often at the end for probabilities.
How.
Try three activations on paper
- ReLU: negatives become 0; positives pass.
- Sigmoid: squishes any number into (0, 1).
- Softmax (multi-class later): turns a vector into probabilities that sum to 1.
import math
def relu(z): return max(0.0, z)
def sigmoid(z): return 1 / (1 + math.exp(-z))
for z in (-2, 0, 2):
print(z, "ReLU", relu(z), "sigmoid", round(sigmoid(z), 3))
Ravindra Bagale's Tip
Hidden layers madhe ReLU default shik. Output la problem type (number vs probability) bagha.