1. The Sigmoid Function: Definition, Properties, and Universality
Mathematical Definition
The standard logistic sigmoid function maps any real-valued number into a strict probability-like range between 0 and 1.
The Derivative
Using the quotient rule and the chain rule, the derivative simplifies into a remarkably elegant, self-referential expression where the slope is computed directly from the value of the sigmoid itself.
Why Euler's Number (e) is Used
While any exponential base (such as 2-x or 10-x) could theoretically be used, Euler's number (e ≈ 2.71828) is universally chosen for deep mathematical reasons.
The Natural Derivative
The derivative of ex is exactly ex. This unique property eliminates unwanted scaling constants such as ln(2) or ln(10), making the derivative beautifully simple and computationally efficient during backpropagation.
Information Theory & Physics
The base e naturally appears in the Boltzmann distribution in statistical physics and in log-odds (logits) in statistics. It represents the natural rate of continuous growth and maximum entropy, making it the mathematically preferred choice for probabilistic modeling.
Universal Significance of the Sigmoid Function
Why the Sigmoid Function Is So Widely Used
The sigmoid function is much more than just another mathematical equation. It possesses several unique properties that make it one of the most important functions in statistics, probability, machine learning, and deep learning.
-
Smooth Thresholding
It acts as a continuous, differentiable alternative to a hard step function. Instead of abruptly switching from 0 to 1, it transitions smoothly, making gradient-based optimization possible. -
Probability Mapping
It transforms unconstrained real numbers ranging from−∞ to +∞into valid probabilities between0 and 1making it ideal for probabilistic prediction.
2. Why the Derivative Is Not a True Parabola
The derivative of the sigmoid function is written as
At first glance, this expression resembles the quadratic form
Since this resembles the familiar equation x − x2, many beginners assume the derivative must be a parabola. Although the algebraic form appears similar, the geometric behavior is completely different.
The Input Variable Matters
A true parabola is defined directly as a polynomial in the horizontal variable.
Here, the independent variable x appears directly as a polynomial. Consequently, the graph follows the familiar parabolic shape.
In contrast, the sigmoid derivative is ultimately a function of x through the exponential term e−x. Therefore, the horizontal axis is not related polynomially to the output.
The Exponential Constraint
Inside the sigmoid function, the input variable appears inside an exponential:
Because of this exponential dependence, the rate of change grows and decays exponentially rather than polynomially.
As a result, the derivative cannot produce the infinitely expanding shape associated with a geometric parabola.
The Geometric Result
Instead of expanding outward forever, the exponential terms continuously bend the curve inward.
The curve gradually flattens on both sides, producing a smooth, symmetric, bell-shaped distribution.
Mathematically, this curve is a hyperbolic secant squared function rather than a parabola.
Visual Comparison: Parabola vs. Sigmoid Derivative
Although the derivative contains an algebraic expression that resembles a quadratic function, its dependence on the exponential term e−x causes the graph to become a bounded bell-shaped curve rather than an unbounded parabola.
3. Core Applications: Logistic Regression and Neural Networks
Why the Sigmoid Function Became So Important
The sigmoid function became one of the most influential mathematical functions in Machine Learning because it naturally converts unrestricted real-valued numbers into probabilities.
This single property makes it extremely useful in both statistical machine learning and artificial neural networks.
Logistic Regression (Statistical Machine Learning)
In binary classification problems, an algorithm must predict a probability between 0 and 1.
Examples include determining whether:
- Email → Spam or Not Spam
- Patient → Disease Present or Healthy
- Transaction → Fraudulent or Genuine
- Student → Pass or Fail
Step 1 — Linear Model Produces a Raw Score
A linear model first computes a raw numerical score known as the logit.
This value may lie anywhere between −∞ and +∞.
Step 2 — Apply the Sigmoid Function
The raw score is then passed through the sigmoid function.
The output is now guaranteed to lie between 0 and 1, allowing it to be interpreted as a probability.
Step 3 — Make the Final Prediction
A threshold is then applied.
- If the probability is ≥ 0.5, predict Class 1.
- If the probability is < 0.5, predict Class 0.
Logistic Regression Pipeline
Activation Functions in Neural Networks
Early neural networks adopted the sigmoid function because it provides a simple mathematical approximation of how biological neurons behave.
Biological Analogy
- A biological neuron receives electrical signals from many other neurons.
- It accumulates these incoming signals.
- When the accumulated signal exceeds a threshold, the neuron fires.
Mathematical Interpretation
The sigmoid function acts as a smooth mathematical gate that determines how much information should pass from one layer of neurons to the next, depending on the strength of the incoming signal.
Sigmoid as an Activation Function
4. The Vanishing Gradient Problem and the Evolution of Activation Functions
The Vanishing Gradient Problem
Although the sigmoid function is mathematically elegant, it introduces a major difficulty when training deep neural networks.
The maximum value of the sigmoid derivative occurs at x = 0, where
As the input moves away from zero in either direction, the sigmoid curve gradually flattens and its derivative rapidly approaches zero.
During backpropagation, gradients from deeper layers are multiplied together while moving toward earlier layers.
Since each sigmoid derivative is at most 0.25, repeatedly multiplying these small values causes the gradient to shrink exponentially.
By the time the gradient reaches the earliest layers of a deep network, it has effectively become zero. Consequently, those layers stop learning because their weights receive almost no updates.
Mathematical Intuition
Suppose a network contains several hidden layers. During backpropagation, the gradient reaching an early layer is roughly the product of all the derivatives encountered along the path.
If every derivative is smaller than 0.25, the product decreases exponentially as the number of layers increases.
Example of Gradient Shrinkage
Consider a simplified network in which each layer contributes a gradient of approximately 0.2.
| Layer | Approximate Gradient |
|---|---|
| Layer 4 | 0.20 |
| Layer 3 | 0.04 |
| Layer 2 | 0.008 |
| Layer 1 | Almost 0 (Vanished) |
Gradient Flow Through a Deep Neural Network
Why Learning Stops
Neural networks learn by updating their weights using gradients computed during backpropagation.
If the gradient reaching a layer becomes extremely small, the weight updates become almost zero.
As a result, the earliest layers stop learning useful features even though the later layers continue to update.
This inability to update the early layers severely limits the training of very deep neural networks and was one of the primary motivations for developing newer activation functions.
How Other Activation Functions Solve the Vanishing Gradient Problem
To overcome the vanishing gradient problem, modern deep learning relies on activation functions that preserve stronger gradients during backpropagation.
Instead of rapidly flattening like the sigmoid function, these activations maintain a larger derivative over a wider range of inputs, allowing information to propagate efficiently through very deep neural networks.
ReLU (Rectified Linear Unit)
The Rectified Linear Unit (ReLU) is defined as:
For any positive input, the derivative of ReLU is a constant 1.
Because a gradient of 1 does not shrink during multiplication, it can travel through hundreds of layers without vanishing, making ReLU the default activation function in many deep neural networks.
Tanh (Hyperbolic Tangent)
The hyperbolic tangent function scales inputs into the range:
Its maximum derivative is 1.0 (at x = 0), allowing gradients to flow more effectively than with the sigmoid function.
However, tanh still saturates for very large positive and negative inputs, so its gradients also become extremely small in those regions.
| Activation | Output Range | Maximum Derivative | Main Characteristic |
|---|---|---|---|
| Sigmoid | 0 to 1 | 0.25 | Good for probabilities but suffers from vanishing gradients. |
| Tanh | −1 to 1 | 1.0 | Improves gradient flow but still saturates. |
| ReLU | 0 to ∞ | 1.0 | Fast training and excellent gradient propagation. |
The Dying ReLU Problem
Although ReLU effectively prevents vanishing gradients for positive inputs, it introduces another limitation.
Whenever the input is negative, the output becomes 0 and the derivative is also exactly 0.
If a neuron continually receives negative inputs, it always produces zero output. Consequently, its gradient remains zero, its weights stop updating, and the neuron permanently stops contributing to the network.
The neuron becomes permanently inactive, a phenomenon commonly known as the Dying ReLU Problem.
Solutions to the Dying ReLU Problem
Leaky ReLU
Instead of forcing every negative input to zero, Leaky ReLU assigns a small non-zero slope.
This ensures that a small gradient (approximately 0.01) always flows backward, allowing inactive neurons to recover during training.
GELU (Gaussian Error Linear Unit)
GELU combines the behavior of ReLU with a probabilistic weighting based on a Gaussian distribution.
Instead of introducing a sharp transition at zero, GELU provides a smoother activation function that has become the dominant choice in modern Transformer architectures and Large Language Models (LLMs).
Summary
The mathematical elegance of the sigmoid function relies on Euler's number (e) to create clean probabilistic outputs.
Although its derivative has an algebraic expression that resembles a parabola, the exponential nature of the sigmoid function causes the derivative to form a bell-shaped curve rather than a true parabola.
As neural networks became deeper, the sigmoid function's rapidly shrinking derivative led to the vanishing gradient problem, motivating the development of improved activation functions such as ReLU, Leaky ReLU, and GELU that maintain stronger gradients during training.
No comments:
Post a Comment