Production-grade AI features (matches with constraints, commands, resources, rules and performance evaluation)
Technical, Analytical, & Deep Research
CLEAR
Context, Limits, Emphasis, Alignment, Review
Strategic research and highly regulated topics
4. Advanced "Reasoning" Frameworks
Advanced "Reasoning" Frameworks
Chain-of-Thought (CoT)
Architectural behavioral method
Forces the AI to show its step-by-step reasoning before outputting a final answer.
Advanced "Reasoning" Frameworks
Tree-of-Thought (ToT)
Architectural behavioral method
Guides the AI to evaluate multiple different solution paths simultaneously, self-correcting as it goes.
Advanced "Reasoning" Frameworks
ReAct
Reasoning + Acting
Prompts the AI to alternate between "thinking" about a problem and "acting" (e.g., searching the web, running code).
Prompt Components: Constraints, Commands, Resources, Rules, and Performance Evaluation
A prompt can consist of constraints, commands, resources, rules, and performance evaluation. While a basic prompt might only be a simple question, advanced prompt engineering frameworks frequently use these exact five components to obtain high-quality, predictable outputs from AI models.
Key Idea
Advanced prompt engineering frameworks extend beyond a simple instruction by defining what the AI should do, what limitations it must follow, what information it can use, what rules it must obey, and how the final output will be evaluated.
Component Breakdown
Component
Description
Example
Commands
The core instruction telling the AI what to do.
"Write," "Analyze," "Summarize"
Constraints
The boundaries or limits placed on the output.
"Under 500 words," "Do not use jargon," "Format as markdown"
Resources
The background data, context, or examples provided to help the AI understand the task.
"Based on the attached CSV file," "Use the following style guide"
Rules
The logical guidelines or conditional policies the AI must follow while processing the request.
"If the sentiment is negative, escalate to human tone," "Always verify facts before listing them"
Performance Evaluation
The criteria or rubric used to tell the AI how its output will be judged, often used in iterative prompts.
"Your output will be scored on clarity, accuracy, and conciseness from 1 to 5"
Commands:What to do
Constraints: Limits
Resources: Context
Rules: Policies
Performance Evaluation: Quality Criteria
Important Note
Although a simple prompt may contain only a question or instruction, advanced prompt engineering frameworks typically combine these five components to produce more accurate, consistent, and predictable AI responses.
1. The Sigmoid Function: Definition, Properties, and Universality
Mathematical Definition
The standard logistic sigmoid function maps any real-valued number
into a strict probability-like range between
0 and 1.
σ(x) =
1
1 + e-x
The Derivative
Using the quotient rule and the chain rule, the derivative simplifies
into a remarkably elegant, self-referential expression where the slope
is computed directly from the value of the sigmoid itself.
σ′(x) = σ(x) (1 − σ(x))
Why Euler's Number (e) is Used
While any exponential base (such as
2-x or 10-x)
could theoretically be used,
Euler's number
(e ≈ 2.71828)
is universally chosen for deep mathematical reasons.
The Natural Derivative
The derivative of
ex
is exactly
ex.
This unique property eliminates unwanted scaling constants such as
ln(2) or ln(10), making the derivative beautifully simple and
computationally efficient during backpropagation.
Information Theory & Physics
The base
e
naturally appears in the
Boltzmann distribution
in statistical physics and in
log-odds (logits)
in statistics. It represents the natural rate of continuous growth
and maximum entropy, making it the mathematically preferred choice
for probabilistic modeling.
Universal Significance of the Sigmoid Function
Why the Sigmoid Function Is So Widely Used
The sigmoid function is much more than just another mathematical equation.
It possesses several unique properties that make it one of the most important
functions in statistics, probability, machine learning, and deep learning.
Smooth Thresholding
It acts as a continuous, differentiable alternative to a hard
step function. Instead of abruptly switching from
0 to 1,
it transitions smoothly, making gradient-based optimization possible.
Probability Mapping
It transforms unconstrained real numbers ranging from
−∞ to +∞
into valid probabilities between
0 and 1
making it ideal for probabilistic prediction.
2. Why the Derivative Is Not a True Parabola
The derivative of the sigmoid function is written as
σ′(x) = σ(x) (1 − σ(x))
At first glance, this expression resembles the quadratic form
σ − σ2
Since this resembles the familiar equation
x − x2,
many beginners assume the derivative must be a parabola.
Although the algebraic form appears similar, the geometric behavior is completely different.
The Input Variable Matters
A true parabola is defined directly as a polynomial in the horizontal
variable.
f(x) = x − x2
Here, the independent variable
x
appears directly as a polynomial.
Consequently, the graph follows the familiar parabolic shape.
In contrast, the sigmoid derivative is ultimately a function of
x through the exponential term
e−x.
Therefore, the horizontal axis is not related polynomially to the output.
The Exponential Constraint
Inside the sigmoid function,
the input variable appears inside an exponential:
σ(x) =
1
1 + e−x
Because of this exponential dependence,
the rate of change grows and decays exponentially rather than polynomially.
As a result,
the derivative cannot produce the infinitely expanding shape associated
with a geometric parabola.
The Geometric Result
Instead of expanding outward forever,
the exponential terms continuously bend the curve inward.
The curve gradually flattens on both sides,
producing a smooth,
symmetric,
bell-shaped distribution.
Mathematically,
this curve is a
hyperbolic secant squared
function rather than a parabola.
Visual Comparison: Parabola vs. Sigmoid Derivative
Key Observation
Although the derivative contains an algebraic expression that resembles
a quadratic function, its dependence on the exponential term
e−x
causes the graph to become a bounded bell-shaped curve rather than
an unbounded parabola.
3. Core Applications: Logistic Regression and Neural Networks
Why the Sigmoid Function Became So Important
The sigmoid function became one of the most influential mathematical
functions in Machine Learning because it naturally converts unrestricted
real-valued numbers into probabilities.
This single property makes it extremely useful in both
statistical machine learning and
artificial neural networks.
In binary classification problems, an algorithm must predict a probability
between 0 and 1.
Examples include determining whether:
Email → Spam or Not Spam
Patient → Disease Present or Healthy
Transaction → Fraudulent or Genuine
Student → Pass or Fail
Step 1 — Linear Model Produces a Raw Score
A linear model first computes a raw numerical score known as the
logit.
z = β0 + β1x1 + β2x2 + ··· + βnxn
This value may lie anywhere between
−∞ and +∞.
Step 2 — Apply the Sigmoid Function
The raw score is then passed through the sigmoid function.
P =
1
1 + e−z
The output is now guaranteed to lie between
0 and 1,
allowing it to be interpreted as a probability.
Step 3 — Make the Final Prediction
A threshold is then applied.
If the probability is ≥ 0.5, predict Class 1.
If the probability is < 0.5, predict Class 0.
Logistic Regression Pipeline
Activation Functions in Neural Networks
Early neural networks adopted the sigmoid function because it provides
a simple mathematical approximation of how biological neurons behave.
Biological Analogy
A biological neuron receives electrical signals from many other neurons.
It accumulates these incoming signals.
When the accumulated signal exceeds a threshold, the neuron fires.
Mathematical Interpretation
The sigmoid function acts as a smooth mathematical gate that determines
how much information should pass from one layer of neurons to the next,
depending on the strength of the incoming signal.
Sigmoid as an Activation Function
4. The Vanishing Gradient Problem and the Evolution of Activation Functions
The Vanishing Gradient Problem
Although the sigmoid function is mathematically elegant, it introduces a
major difficulty when training deep neural networks.
The maximum value of the sigmoid derivative occurs at
x = 0, where
σ′(0) = 0.25
As the input moves away from zero in either direction, the sigmoid curve
gradually flattens and its derivative rapidly approaches zero.
During backpropagation, gradients from deeper layers are multiplied
together while moving toward earlier layers.
Since each sigmoid derivative is at most
0.25, repeatedly multiplying these small values causes
the gradient to shrink exponentially.
Result
By the time the gradient reaches the earliest layers of a deep
network, it has effectively become zero. Consequently, those layers
stop learning because their weights receive almost no updates.
Mathematical Intuition
Suppose a network contains several hidden layers.
During backpropagation, the gradient reaching an early layer is roughly
the product of all the derivatives encountered along the path.
Gradient =
σ′1
×
σ′2
×
σ′3
×
···
×
σ′n
If every derivative is smaller than
0.25,
the product decreases exponentially as the number of layers increases.
Example of Gradient Shrinkage
Consider a simplified network in which each layer contributes a gradient
of approximately
0.2.
Layer
Approximate Gradient
Layer 4
0.20
Layer 3
0.04
Layer 2
0.008
Layer 1
Almost 0 (Vanished)
Gradient Flow Through a Deep Neural Network
Why Learning Stops
Neural networks learn by updating their weights using gradients
computed during backpropagation.
If the gradient reaching a layer becomes extremely small, the weight
updates become almost zero.
As a result, the earliest layers stop learning useful features even
though the later layers continue to update.
Key Consequence
This inability to update the early layers severely limits the
training of very deep neural networks and was one of the primary
motivations for developing newer activation functions.
How Other Activation Functions Solve the Vanishing Gradient Problem
To overcome the vanishing gradient problem, modern deep learning relies on
activation functions that preserve stronger gradients during
backpropagation.
Instead of rapidly flattening like the sigmoid function, these
activations maintain a larger derivative over a wider range of inputs,
allowing information to propagate efficiently through very deep neural
networks.
ReLU (Rectified Linear Unit)
The Rectified Linear Unit (ReLU) is defined as:
f(x) = max(0, x)
For any positive input, the derivative of ReLU is a constant
1.
Because a gradient of 1 does not shrink during
multiplication, it can travel through hundreds of layers without
vanishing, making ReLU the default activation function in many deep
neural networks.
Tanh (Hyperbolic Tangent)
The hyperbolic tangent function scales inputs into the range:
−1 to +1
Its maximum derivative is 1.0 (at
x = 0), allowing gradients to flow more effectively
than with the sigmoid function.
However, tanh still saturates for very large positive and negative
inputs, so its gradients also become extremely small in those regions.
Activation
Output Range
Maximum Derivative
Main Characteristic
Sigmoid
0 to 1
0.25
Good for probabilities but suffers from vanishing gradients.
Tanh
−1 to 1
1.0
Improves gradient flow but still saturates.
ReLU
0 to ∞
1.0
Fast training and excellent gradient propagation.
The Dying ReLU Problem
Although ReLU effectively prevents vanishing gradients for positive
inputs, it introduces another limitation.
Whenever the input is negative, the output becomes
0 and the derivative is also exactly
0.
If a neuron continually receives negative inputs, it always produces
zero output. Consequently, its gradient remains zero, its weights stop
updating, and the neuron permanently stops contributing to the network.
Result
The neuron becomes permanently inactive, a phenomenon commonly known
as the Dying ReLU Problem.
Solutions to the Dying ReLU Problem
Leaky ReLU
Instead of forcing every negative input to zero, Leaky ReLU assigns a
small non-zero slope.
f(x) = 0.01x for x < 0
This ensures that a small gradient (approximately
0.01) always flows backward, allowing inactive neurons
to recover during training.
GELU (Gaussian Error Linear Unit)
GELU combines the behavior of ReLU with a probabilistic weighting based
on a Gaussian distribution.
Instead of introducing a sharp transition at zero, GELU provides a
smoother activation function that has become the dominant choice in
modern Transformer architectures and Large Language Models (LLMs).
Summary
The mathematical elegance of the sigmoid function relies on Euler's
number (e) to create clean probabilistic outputs.
Although its derivative has an algebraic expression that resembles a
parabola, the exponential nature of the sigmoid function causes the
derivative to form a bell-shaped curve rather than a true parabola.
As neural networks became deeper, the sigmoid function's rapidly
shrinking derivative led to the vanishing gradient problem, motivating
the development of improved activation functions such as ReLU, Leaky
ReLU, and GELU that maintain stronger gradients during training.
Relationship, Hierarchy, and Conditions for Non-Differentiability
Note
The discussion below explains the relationship between limits, continuity and differentiability, together with important examples illustrating when differentiability fails.
All mathematical expressions have been formatted using pure HTML so that the article remains fully compatible with Blogger without requiring MathJax, KaTeX or JavaScript.
1. Limits are not the same as derivatives
A derivative is defined using a limit, but the existence of a limit alone does not imply the existence of a derivative.
f′(a) =
limh→0f(a+h) − f(a)
h
Notice that this is a very specific limit (called the difference quotient).
A function may have an ordinary limit
limx→af(x)
while the derivative does not exist.
Example
Definition
f(x) = |x|
At x = 0,
limx→0 |x| = 0
exists.
Graph of f(x) = |x|
Observation
The graph is continuous, but there is a sharp corner at the origin. This geometric feature prevents the derivative from existing at x=0 even though the limit exists.
But
limh→0
|h| − 0
h
does not exist because
One-Sided Limit
Value
Left-hand limit
−1
Right-hand limit
1
Important
Left-hand derivative = −1
Right-hand derivative = 1
Since the two one-sided derivatives are different, the derivative does not exist.
Hence
Property
Result
Limit exists
✓
Derivative exists
✗
Conclusion
Your first statement is correct.
The existence of an ordinary limit does not guarantee the existence of a derivative.
A derivative requires the existence of a very specific limit—the difference quotient—which is a much stronger condition than the ordinary limit.
2. Order of Strictness
The proper hierarchy is
Every differentiable function is continuous.
Every continuous function has a limit equal to the function value.
The converse of none of these is true.
Key Observation
Differentiability is the strongest property.
Continuity is weaker than differentiability.
The existence of a limit is weaker than continuity.
Therefore,
Therefore
(i) Limit exists
↓
(ii) Function may still not be continuous because
limx→af(x)
may exist but
f(a)
≠
limx→af(x)
or
f(a)
may not even exist.
Example
f(x) =
{
1, x = 0
0, x ≠ 0
Here
limx→0f(x) = 0
exists,
but
f(0) = 1
Hence not continuous.
Graph of the Piecewise Function
Observation
Every point except the origin lies on the horizontal line
y = 0.
As x approaches 0 from either side, the function approaches 0.
However, the actual value of the function at the origin is
f(0)=1,
shown by the filled red point.
Therefore,
the limit exists,
but the function is not continuous.
↓
(iii) Even if continuous, the derivative may not exist.
Example
f(x) = |x|
Continuous everywhere.
Not differentiable at 0.
Geometric Interpretation
Conclusion
The function is perfectly continuous,
yet its graph contains a sharp corner.
The left-hand and right-hand slopes are different,
so the derivative does not exist at the origin.
3. When is a Function Not Differentiable?
(i) Not Continuous
Correct.
Differentiability always implies continuity.
Differentiable ⇒ Continuous
Therefore,
Logical Consequence
If a function is discontinuous, then it cannot be differentiable.
Summary
Discontinuous ⇒ Not Differentiable
(ii) Sharp Corner (Corner Point)
Correct.
Example
f(x) = |x|
Left slope
= −1
Right slope
= 1
Hence the derivative does not exist.
Corner Geometry
Observation
Although the graph is continuous, the slope changes abruptly at the corner.
Since the left-hand and right-hand derivatives are unequal, the derivative does not exist.
(iii) Vertical Tangent
Mostly correct.
Example
f(x) = x1/3
The derivative is
f′(x) =
1
3x2/3
At
x = 0
the derivative becomes infinite.
Important
Most elementary calculus books simply state that the derivative
does not exist at this point.
Geometrically, however, the tangent line is vertical.
Graph of f(x) = x1/3
Geometric Interpretation
Unlike a corner, the graph remains smooth.
However, the tangent becomes perfectly vertical at the origin.
Since the slope tends to positive infinity, the ordinary finite derivative does not exist.
(iv) Wild Oscillation
Correct.
Example
f(x) =
x sin
(
1
x)
At 0,
define
f(0)=0
The function is continuous.
But
f′(0)=
limh→0f(h)-f(0)
h
Substituting
f(h)=h sin(1/h)
and
f(0)=0,
f′(0)=
limh→0
sin
(
1
h
)
does not exist because
Observation
As h approaches zero,
the quantity
1/h
becomes arbitrarily large.
Consequently,
the value of
sin(1/h)
oscillates endlessly between
−1
and
1
without approaching any single value.
Therefore,
the required limit does not exist.
Graph of
f(x) =
x sin(1/x)
Conclusion
The function itself approaches the origin continuously.
However,
its slope oscillates infinitely rapidly near the origin.
Since the derivative limit fails to converge,
the derivative does not exist.
4. Examples
Let's examine each one.
(i) f(x) = |x|
Correct.
Continuous.
Not differentiable at 0.
Reason: corner.
Key Point
The function is perfectly continuous because there is no break in its graph.
However, the graph changes direction abruptly at the origin.
The left-hand derivative is −1 while the right-hand derivative is 1.
Since these one-sided derivatives are unequal, the derivative does not exist.
Visual Interpretation of the Corner
(ii) x sin(π/x)
(assuming f(0)=0).
Continuous.
Not differentiable at 0.
Reason: oscillation.
Observation
Although the function itself approaches the origin continuously,
its slope oscillates increasingly rapidly near the origin.
Therefore,
the derivative fails to converge even though the function remains continuous.
Graph of
f(x) =
x sin(π/x)
(iii) Weierstrass Function
This is one of the most famous examples.
Properties
Continuous everywhere.
Differentiable nowhere.
Illustration of a Weierstrass-Type Curve
Important Observation
The graph never contains any breaks or jumps, so it is continuous everywhere.
However, no matter how much the graph is magnified, it never becomes locally straight.
Consequently, no unique tangent exists at any point.
(iv) Blancmange Function
Also called the Takagi function.
Properties
Continuous everywhere.
Differentiable nowhere.
Illustration of the Blancmange (Takagi) Function
Observation
The Blancmange function exhibits self-similarity at different scales.
Although it is continuous everywhere, it has no well-defined derivative at any point.
(v) Koch Snowflake
This one needs a small correction.
Important Correction
The Koch snowflake is not a function.
Instead, it is a fractal curve.
Therefore, we do not normally discuss its differentiability as a function of the form
y = f(x).
More precisely,
The boundary of the Koch snowflake is a continuous curve.
It has no well-defined tangent at any point.
Consequently, it is regarded as a nowhere-differentiable curve.
Illustration of the Koch Snowflake Boundary
Why it is Different
Unlike the previous examples,
the Koch snowflake does not represent a single-valued function.
Instead,
it is studied as a geometric curve.
Its boundary is continuous,
but nowhere smooth,
making it an important example in fractal geometry.
One More Important Example
There is another classic example.
f(x) = x2/3
At
x = 0,
the graph has a cusp.
The derivative approaches
+∞ from one side,
−∞ from the other side.
Therefore,
the derivative does not exist.
Key Observation
A cusp is fundamentally different from a corner.
The graph remains continuous,
but the tangent direction changes infinitely rapidly,
producing infinite slopes with opposite signs.
Corner vs Cusp vs Vertical Tangent
Although all three situations result in the derivative not existing, they are geometrically very different.
Understanding these differences is essential in elementary calculus.
Summary
Corner: Finite one-sided derivatives exist but are unequal.
Cusp: One-sided derivatives become infinite with opposite signs.
Vertical Tangent: Both one-sided derivatives approach the same infinite value.
Visual Comparison
Important Distinction
Although all three graphs are continuous, the derivative fails for different geometric reasons.
A corner has two different finite slopes.
A cusp has two infinite slopes with opposite signs.
A vertical tangent has infinite slopes of the same sign.
Recognizing these differences is extremely important when studying differentiability.
Summary Hierarchy
The relationship between limits, continuity and differentiability can now be summarized as follows.
Differentiable → Continuous → Limit Exists
Key Implication
Every differentiable function is automatically continuous.
Likewise,
every continuous function automatically has a limit equal to the function value.
However,
the converse implications are not true.
Important Note
A function may possess a limit without being continuous.
Similarly,
a function may be continuous without being differentiable.
Therefore,
each property is stronger than the one below it in the hierarchy.
Converse Implications are False
The hierarchy
Differentiable ⇒ Continuous ⇒ Limit Exists
does not work in the reverse direction.
Common Misconception
Many students mistakenly assume that if a function is continuous, it must also be differentiable.
The absolute value function,
f(x) = |x|,
is an immediate counterexample.
Likewise,
the existence of a limit alone does not guarantee continuity.
Complete Summary
A function may fail to be differentiable because of the following reasons.
Discontinuity ✓
Corner (sharp turn) ✓
Cusp ✓
Vertical tangent ✓
Wild oscillation ✓
Fractal behavior (for example, the Weierstrass or Blancmange function) ✓
Appendix A — Common Misconceptions
Misconception
Reality
If the limit exists, the function is continuous.
False.
The function value must also equal the limit.
If a function is continuous, it is differentiable.
False.
The graph may contain a corner, cusp or vertical tangent.
Infinite derivative means derivative exists.
False.
Elementary calculus requires a finite derivative.
Every continuous curve represents a function.
False.
The Koch snowflake is a continuous curve but not a function y=f(x).
Appendix B — How to Test Differentiability
Revision Cheat Sheet
Derivative is defined using a limit.
Every differentiable function is continuous.
Every continuous function has a limit equal to its function value.
The converse of both statements is false.
Discontinuity always destroys differentiability.
Corner → unequal finite slopes.
Cusp → opposite infinite slopes.
Vertical tangent → same infinite slope.
Oscillation → derivative limit fails to converge.
Weierstrass and Blancmange functions are continuous everywhere but differentiable nowhere.
Koch snowflake is a fractal curve, not a function.
Practice Questions
Give an example of a function whose limit exists but which is not continuous.
Give an example of a function that is continuous but not differentiable.
Why does |x| fail to be differentiable at x = 0?
Differentiate between a corner and a cusp.
What is meant by a vertical tangent?
Explain why x sin(1/x) is continuous but not differentiable at x = 0.
State the hierarchy relating limits, continuity and differentiability.
Why is the Koch snowflake not considered a function?
Name two famous functions that are continuous everywhere but differentiable nowhere.
Give one real-life application where differentiability is important.
