Understanding Regression

1. Linear Regression

Linear Regression is the simplest and one of the most fundamental algorithms in Machine Learning. It is used to predict a continuous numerical value by learning the relationship between one or more input variables and an output variable.

What is Linear Regression?

The objective of Linear Regression is to find the best-fitting straight line that represents the relationship between the input variables and the target value.

It is commonly used for predicting values such as:

• 🏠 Predicting house prices
• 💼 Predicting salaries
• 🌡 Predicting tomorrow's temperature
• 📈 Predicting next month's sales

Key Assumption

Linear Regression assumes that the relationship between the input and output is approximately a straight line.

Example Dataset

Consider the following relationship between years of experience and salary:

Experience (Years)	Salary
1	30,000
2	35,000
3	40,000
4	46,000
5	52,000

If these points are plotted on a graph, they form a pattern that is approximately a straight line.

Mathematical Equation

The equation of a straight line is:

y = m × x + c

where

• y = Predicted value
• x = Input feature
• m = Slope of the line
• c = Intercept (where the line crosses the Y-axis)

Machine Learning Goal

The learning algorithm automatically discovers the best values of m and c so that the prediction error becomes as small as possible.

Worked Example

Suppose the relationship between salary and experience is:

Salary = 20,000 + (5,000 × Experience)

If someone has 6 years of experience:

Salary = 20,000 + (5,000 × 6)

Salary = 20,000 + 30,000

Salary = 50,000

This is exactly how Linear Regression predicts continuous values.

Key Takeaways

• Linear Regression predicts continuous numerical values.
• It assumes an approximately linear relationship between input and output.
• Its objective is to find the best-fitting straight line.
• The equation of the model is y = m × x + c.
• The model learns the optimal values of m (slope) and c (intercept) from the training data.

2. Normal Equation

The Normal Equation is a mathematical technique used to calculate the best-fitting line for Linear Regression directly, without repeatedly adjusting the model parameters.

What is the Normal Equation?

Many beginners confuse the Normal Equation with the Normal Distribution (Bell Curve).

These are completely different concepts.

Important Note

In Machine Learning, when someone refers to the Normal Function, they almost always mean the Normal Equation, which is used to compute the optimal regression coefficients.

Why Do We Need It?

Recall that Linear Regression tries to find the best values of the model parameters so that the prediction error becomes as small as possible.

One approach is to improve the parameters gradually using an optimization algorithm such as Gradient Descent.

However, mathematics also provides a direct solution that computes the best parameters in a single calculation.

Instead of repeatedly improving the line...

The Normal Equation calculates the optimal regression line immediately using matrix algebra.

Mathematical Formula

The Normal Equation is:

θ = (X^TX)⁻¹ X^Ty

Although the equation may appear complicated at first, each symbol has a simple meaning.

Meaning of the Symbols

Symbol	Meaning
X	Input feature matrix
XT	Transpose of the feature matrix
(XTX)−1	Inverse of the matrix XTX
y	Observed output values
θ	Regression coefficients (model parameters)

How Does It Work?

Instead of repeatedly adjusting the model parameters, the Normal Equation performs a sequence of matrix operations:

1. Construct the feature matrix X.
2. Compute the transpose X^T.
3. Multiply X^T and X.
4. Compute the inverse of the resulting matrix.
5. Multiply the result by X^Ty.
6. Obtain the optimal parameter vector θ.

The resulting coefficients define the regression line that minimises the prediction error.

Advantages

• ✔ No iterative optimisation is required.
• ✔ Produces the exact optimal solution for Linear Regression.
• ✔ Very accurate for small datasets.
• ✔ Easy to implement when the number of features is limited.

Disadvantages

The biggest limitation is the computation of the matrix inverse.

As the dataset grows larger, matrix inversion becomes computationally expensive.

For datasets containing hundreds of thousands or millions of records, the Normal Equation becomes slow and memory intensive.

Practical Recommendation

In real-world Machine Learning:

• Small datasets → Normal Equation works very well.
• Large datasets → Gradient Descent is usually preferred.

Why?

Gradient Descent avoids expensive matrix inversion and scales efficiently to very large datasets containing millions of training examples.

Key Takeaways

• The Normal Equation computes the optimal Linear Regression model directly.
• It uses matrix algebra rather than iterative optimisation.
• It provides an exact solution.
• It is excellent for small and medium-sized datasets.
• For very large datasets, Gradient Descent is generally the preferred optimisation technique.

3. Gradient Descent

Gradient Descent is one of the most important optimisation algorithms in Machine Learning. Instead of directly calculating the best model parameters, it gradually improves them through many small steps until the prediction error becomes as small as possible.

Understanding Gradient Descent with an Analogy

Imagine you are standing on top of a mountain during a very foggy day.

Your goal is to reach the lowest point of the valley, but you cannot see the entire mountain because of the dense fog.

What would you do?

• Look around carefully.
• Find the direction that slopes downward the most.
• Take one small step.
• Stop and look around again.
• Repeat the process until you reach the bottom.

This is exactly how Gradient Descent works. Instead of searching for the lowest point on a mountain, it searches for the smallest prediction error.

Why is Gradient Descent Needed?

Machine Learning models usually begin with random parameter values.

These random values rarely produce accurate predictions.

Gradient Descent repeatedly adjusts the parameters so that every iteration produces a model with lower prediction error.

Objective

Find the values of the model parameters that minimise the prediction error.

Cost Function

The Cost Function measures how far the model's predictions are from the actual values.

A large cost means poor predictions, while a small cost means better predictions.

Cost = Prediction Error

How the Error Reduces

Suppose the model initially makes poor predictions.

Iteration	Cost
Initial Model	500
After Update 1	350
After Update 2	120
After Update 3	18
After Update 4	2
Final Model	≈ 0

Every iteration makes the model slightly better by reducing the prediction error.

Gradient Descent Update Rule

The model parameters are updated using the following equation:

θ = θ − α × ( ∂J / ∂θ )

Each symbol has an important meaning:

Symbol	Meaning
θ	Model parameters (coefficients)
α	Learning Rate
J	Cost Function
∂J / ∂θ	Gradient (direction of steepest increase)

Learning Rate (α)

The learning rate determines how large each update step should be.

Learning Rate	Effect
Too Small	Training becomes very slow.
Too Large	The algorithm may overshoot the optimum and fail to converge.
Well Chosen	The model converges efficiently to the minimum cost.

Gradient Descent Algorithm

1. Initialise the model parameters with random values.
2. Compute predictions using the current parameters.
3. Calculate the prediction error using the cost function.
4. Compute the gradient.
5. Update the parameters using the Gradient Descent update rule.
6. Repeat until the cost stops decreasing significantly.

Key Takeaways

• Gradient Descent is an optimisation algorithm, not a prediction algorithm.
• Its objective is to minimise the prediction error.
• It repeatedly updates the model parameters until the cost reaches its minimum.
• The learning rate (α) controls the size of each update step.
• Gradient Descent is widely used for training Linear Regression, Logistic Regression, Neural Networks, and many other Machine Learning models.

4. Logistic Regression

Despite its name, Logistic Regression is not a regression algorithm. It is one of the most widely used classification algorithms in Machine Learning.

What is Logistic Regression?

Unlike Linear Regression, which predicts a continuous numerical value, Logistic Regression predicts the probability that an input belongs to a particular class.

Key Idea

Logistic Regression answers questions that have a limited number of possible outcomes, such as Yes/No, True/False, or Spam/Not Spam.

Common Applications

Logistic Regression is widely used in binary classification problems.

• 📧 Spam or Not Spam
• 🩺 Cancer or Healthy
• 💳 Fraudulent or Genuine Transaction
• 🎓 Pass or Fail
• 🏦 Loan Approved or Rejected
• 😊 Positive or Negative Sentiment

Linear Regression vs Logistic Regression

Linear Regression	Logistic Regression
Predicts continuous numerical values.	Predicts probabilities and class labels.
Output can be any real number.	Output is always between 0 and 1.
Used for regression problems.	Used for classification problems.

Predicting Probabilities

Instead of directly predicting a class, Logistic Regression first predicts a probability.

Prediction	Meaning
0.91	91% probability that the patient has cancer.
0.02	Only 2% probability that the email is spam.
0.76	76% probability of rain.

Converting Probability into a Class

A probability alone is usually not enough. The model must decide which class to predict.

A threshold is therefore chosen. The most common threshold is 0.5.

If Probability ≥ 0.50 → Positive Class

---

If Probability < 0.50 → Negative Class

The Sigmoid Function

Logistic Regression converts any numerical value into a probability using the Sigmoid Function.

P = 1 / (1 + e^−z)

where:

• P = Predicted probability
• e = Euler's mathematical constant (approximately 2.718)
• z = Linear combination of the input features

Why Use the Sigmoid Function?

A Linear Regression model can predict any number, including negative values or numbers much greater than one.

However, probabilities must always lie between 0 and 1.

The Sigmoid Function solves this problem by:

• Converting every input into a value between 0 and 1.
• Producing smooth probability estimates.
• Making Logistic Regression suitable for classification tasks.

Key Takeaways

• Despite its name, Logistic Regression is a classification algorithm.
• It predicts the probability that an input belongs to a particular class.
• The predicted probability always lies between 0 and 1.
• The Sigmoid Function converts any real-valued input into a valid probability.
• A probability threshold (typically 0.5) is used to determine the final predicted class.

5. Polynomial Regression

Polynomial Regression is an extension of Linear Regression that can model curved (non-linear) relationships between the input features and the target variable.

Why Do We Need Polynomial Regression?

Linear Regression works well only when the relationship between the input and output is approximately a straight line.

However, many real-world datasets follow a curved pattern. In such situations, a straight line cannot accurately represent the data.

Important Idea

When the relationship between the variables is curved rather than linear, Polynomial Regression usually produces much better predictions.

Example

Consider predicting the price of a house based on its age.

The relationship may look something like this:

• Initially, prices decrease slowly.
• After several years, prices begin to fall more rapidly.
• Eventually, the prices stabilise.

Observation

This behaviour is clearly not a straight line. Therefore, a Linear Regression model would not fit the data well.

Linear vs Polynomial Regression

Linear Regression	Polynomial Regression
Fits a straight line.	Fits a smooth curve.
Best for linear relationships.	Best for non-linear relationships.
Uses only the original feature.	Uses transformed polynomial features.

Polynomial Equations

A second-degree (quadratic) polynomial is written as:

y = ax² + bx + c

A third-degree (cubic) polynomial is:

y = ax³ + bx² + cx + d

How Does Polynomial Regression Work?

An interesting fact is that Polynomial Regression still uses Linear Regression internally.

Instead of changing the learning algorithm, it transforms the input features into higher-degree polynomial features.

Original Feature

x

Transformed Features

x     x²     x³     x⁴     ...

These transformed features allow the Linear Regression algorithm to fit a curved relationship while still solving a linear optimisation problem.

Advantages

• ✔ Models curved relationships effectively.
• ✔ Simple extension of Linear Regression.
• ✔ Easy to implement using polynomial feature transformation.
• ✔ Can significantly improve prediction accuracy when the relationship is non-linear.

Limitations

• Very high-degree polynomials may overfit the training data.
• The model can become unstable for extremely large polynomial degrees.
• Choosing the appropriate polynomial degree is important for achieving good performance.

Key Takeaways

• Polynomial Regression is designed for non-linear relationships.
• It extends Linear Regression by creating polynomial features such as x², x³, and x⁴.
• Despite fitting a curve, it still relies on the Linear Regression algorithm internally.
• It is useful when a straight line cannot adequately represent the data.
• Choosing an excessively high polynomial degree may lead to overfitting.

6. Ridge Regression

Ridge Regression is an improved version of Linear Regression that helps prevent overfitting by discouraging the model from assigning excessively large values to its coefficients.

Why Do We Need Ridge Regression?

In many real-world machine learning problems, datasets contain a large number of input features.

Some of these features are highly useful, while others contribute very little to the prediction.

Example

Suppose a dataset contains 100 features. Many of them may have little or no influence on the target variable, yet a standard Linear Regression model may still assign them very large coefficients.

Example of Large Coefficients

A Linear Regression model may produce coefficients like these:

Feature	Coefficient
Age	500
Height	1200
Weight	−2000

Extremely large coefficients often indicate that the model is fitting not only the true pattern in the data but also random noise.

What is Overfitting?

Overfitting occurs when a model learns the training data too closely, including random variations and noise.

As a result, the model performs extremely well on the training data but performs poorly on new, unseen data.

Key Idea

Large coefficient values often make a model unnecessarily sensitive to small changes in the input data, increasing the risk of overfitting.

How Ridge Regression Solves This Problem

Ridge Regression modifies the cost function by adding an additional penalty whenever the coefficients become too large.

New Cost Function

Prediction Error

+

Penalty

This penalty discourages the model from assigning excessively large values to the coefficients.

Ridge Penalty

The penalty added by Ridge Regression is:

λ Σ θ²

where:

• λ (lambda) = Regularisation parameter.
• Σ = Sum of all coefficients.
• θ = Model coefficients.
• θ² = Square of each coefficient.

The Role of λ (Lambda)

Lambda controls how strongly Ridge Regression penalises large coefficient values.

Lambda Value	Effect
Very Small	Behaves almost like ordinary Linear Regression.
Moderate	Produces a balanced model with good generalisation.
Very Large	Shrinks coefficients aggressively and may cause underfitting.

Effect of Ridge Regression

Feature	Before Ridge	After Ridge
Age	500	320
Height	1200	610
Weight	−2000	−950

Notice that the coefficients become smaller, but none of them become exactly zero.

Benefits of Ridge Regression

• ✔ Reduces overfitting.
• ✔ Produces more stable models.
• ✔ Handles multicollinearity effectively.
• ✔ Keeps every feature in the model.
• ✔ Improves generalisation on unseen data.

Key Takeaways

• Ridge Regression is Linear Regression with L2 regularisation.
• It reduces the size of model coefficients by adding the penalty λ Σ θ².
• It helps reduce overfitting and improves model stability.
• Unlike Lasso Regression, Ridge Regression never removes features completely; it only reduces their influence.
• It is particularly useful when many input features are correlated.

7. Lasso Regression

Lasso Regression is another regularised version of Linear Regression. Like Ridge Regression, it helps prevent overfitting, but it has one unique characteristic— it can completely eliminate unimportant features from the model.

Why Do We Need Lasso Regression?

In many datasets, not every feature contributes equally to the prediction.

Some features contain valuable information, while others contribute very little or simply introduce noise into the model.

Key Idea

Instead of merely reducing large coefficients, Lasso Regression can reduce some coefficients all the way to zero, effectively removing those features from the model.

How is Lasso Different from Ridge Regression?

Ridge Regression and Lasso Regression have the same overall goal: reducing overfitting.

However, the way they apply the penalty is different.

Ridge Regression	Lasso Regression
Shrinks coefficients.	Shrinks coefficients.
Keeps every feature.	Can remove unnecessary features.
Uses L2 Regularisation.	Uses L1 Regularisation.

Lasso Penalty

Instead of squaring the coefficients, Lasso Regression uses the absolute values of the coefficients.

λ Σ |θ|

where:

• λ (lambda) = Regularisation parameter.
• Σ = Sum of all coefficients.
• |θ| = Absolute value of each coefficient.

Example

Suppose a Linear Regression model produces the following coefficients:

Feature	Coefficient
Age	400
Height	120
Weight	600
Hair Colour	20
Pet Name	15

After Applying Lasso Regression

Feature	New Coefficient
Age	350
Height	100
Weight	450
Hair Colour	0
Pet Name	0

Observation

The coefficients for Hair Colour and Pet Name become exactly 0, meaning these features are completely removed from the model.

Automatic Feature Selection

One of the biggest advantages of Lasso Regression is its ability to perform automatic feature selection.

By forcing some coefficients to become exactly zero, Lasso automatically identifies the most important features while removing irrelevant ones.

This provides several benefits:

• Produces simpler models.
• Reduces overfitting.
• Improves model interpretability.
• Reduces computational complexity.
• Removes noisy or irrelevant features automatically.

Ridge Regression vs Lasso Regression

Feature	Ridge	Lasso
Regularisation	L2 (λ Σ θ²)	L1 (λ Σ |θ|)
Coefficient Size	Reduced	Reduced
Feature Removal	No	Yes
Best Use Case	Many useful correlated features	Feature selection and simpler models

Key Takeaways

• Lasso Regression is Linear Regression with L1 regularisation.
• It adds the penalty λ Σ |θ| to the cost function.
• It reduces overfitting while simplifying the model.
• Some coefficients become exactly 0, automatically removing unimportant features.
• Lasso Regression is widely used for automatic feature selection and building interpretable machine learning models.

8. Elastic Net Regression

Elastic Net Regression combines the strengths of both Ridge Regression and Lasso Regression. Instead of choosing between the two techniques, Elastic Net applies both L1 and L2 regularisation simultaneously.

Why Do We Need Elastic Net?

Ridge Regression performs well when most features are useful, while Lasso Regression is excellent for removing irrelevant features.

However, in many real-world datasets, we want both behaviours at the same time.

Example

Suppose a dataset contains hundreds of highly correlated features. Some should remain in the model, while others should be removed. Elastic Net handles this situation very effectively.

Basic Idea

Elastic Net adds two penalties to the cost function:

Prediction Error

+

Ridge (L2) Penalty

+

Lasso (L1) Penalty

Advantages

• ✔ Shrinks excessively large coefficients.
• ✔ Removes unimportant features automatically.
• ✔ Handles highly correlated features effectively.
• ✔ Reduces overfitting.
• ✔ Produces models that generalise well to unseen data.

Ridge vs Lasso vs Elastic Net

Technique	Shrinks Coefficients	Removes Features	Best For
Ridge	✔ Yes	✖ No	Many correlated useful features
Lasso	✔ Yes	✔ Yes	Automatic feature selection
Elastic Net	✔ Yes	✔ Yes	Large datasets with correlated features

Because it combines the strengths of Ridge and Lasso, Elastic Net is often one of the best choices for practical machine learning problems.

Key Takeaways

• Elastic Net combines L1 and L2 regularisation.
• It shrinks coefficients and can also remove unnecessary features.
• It works particularly well when many features are highly correlated.
• It often produces better generalisation than Ridge or Lasso alone.

9. Stepwise Regression

Stepwise Regression is a traditional feature selection technique that attempts to build a simpler model by adding or removing input features one step at a time.

Understanding the Idea

Imagine interviewing candidates for a job.

Initially, you have 100 applicants.

After each interview round, you eliminate the least suitable candidates.

Eventually, only the 10 best candidates remain.

Machine Learning Analogy

Instead of selecting candidates, Stepwise Regression selects the most useful input features.

Three Types of Stepwise Regression

Method	How It Works
Forward Selection	Starts with no features and adds one feature at a time.
Backward Elimination	Starts with every feature and removes the least useful feature repeatedly.
Bidirectional Selection	Simultaneously adds useful features and removes unnecessary ones.

Forward Selection Example

Forward Selection starts with an empty model.

Features are added one by one until adding another feature no longer improves the model.

Salary
↓
Experience
↓
Education
↓
Age

Once additional features stop improving the model, the algorithm terminates.

Backward Elimination

1. Begin with every available feature.
2. Evaluate their statistical significance.
3. Remove the least useful feature.
4. Repeat until only important features remain.

Bidirectional Selection

Bidirectional Selection combines the ideas of Forward Selection and Backward Elimination.

During each iteration, the algorithm may:

• Add an important feature.
• Remove a feature that has become unnecessary.

Modern Perspective

Stepwise Regression was once one of the most popular feature selection techniques.

Today, it is used less frequently because it can produce unstable models and may not always find the best combination of features.

Current Practice

Modern machine learning often prefers Lasso Regression or Elastic Net Regression, as they perform feature selection automatically while reducing overfitting.

10. Decision Tree Regression

Decision Tree Regression is a completely different approach to regression. Instead of fitting a mathematical equation such as a straight line or a curve, it learns a series of decision rules that divide the data into smaller and smaller groups.

Basic Idea

Imagine that you are trying to predict the price of a house.

Rather than finding one equation that works for every house, the algorithm keeps asking questions about the house.

• Is the house larger than 2,000 square feet?
• Is it located in a premium neighbourhood?
• Does it have a garden?
• Does it have a garage?

Based on the answers, the tree eventually reaches a final prediction.

Example Decision Tree

Suppose we want to predict house prices.

House Size > 2000 sq ft ? │ ┌────┴────┐ │ │ Yes No │ │ Location Price Premium? = 3,000,000 │ ┌───┴───┐ │ │ Yes No │ │ Price Price =15,000,000 =9,000,000

Every question divides the dataset into smaller groups until the algorithm reaches a final prediction.

What is a Leaf Node?

The final box of a Decision Tree is called a leaf node.

For regression problems, every leaf node stores the average target value of the training samples that reach that leaf.

Example

If every training house reaching one leaf has an average price of 15,000,000, then every future house reaching that leaf will also be predicted as 15,000,000.

Linear Regression vs Decision Tree Regression

Linear Regression	Decision Tree Regression
Fits one mathematical equation.	Learns a sequence of decision rules.
Best for mostly linear relationships.	Handles highly nonlinear relationships.
Requires feature scaling in many cases.	Usually does not require feature scaling.
Limited interaction modelling.	Automatically captures feature interactions.

Advantages

• ✔ Handles nonlinear relationships without requiring polynomial features.
• ✔ Easy to understand and interpret.
• ✔ No feature scaling is normally required.
• ✔ Automatically captures interactions between variables.
• ✔ Works well with both numerical and categorical features.

Weaknesses

• A single Decision Tree can easily overfit the training data.
• Small changes in the data may produce a completely different tree.
• Deep trees can become very complex and unstable.
• The prediction accuracy of one tree is often lower than that of ensemble methods.

Tree Ensemble Methods

Because a single Decision Tree can overfit easily, modern machine learning usually combines many trees together.

These combinations are known as ensemble methods.

Algorithm	Main Idea
Random Forest Regression	Builds many independent decision trees and averages their predictions.
Gradient Boosting Regression	Builds trees sequentially so that each new tree corrects the mistakes made by previous trees.
XGBoost	Highly optimised implementation of Gradient Boosting with excellent speed and accuracy.
LightGBM	Fast boosting algorithm designed for very large datasets.
CatBoost	Boosting algorithm that handles categorical variables particularly well.

Key Takeaways

• Decision Tree Regression predicts values using a sequence of decision rules instead of a mathematical equation.
• Each internal node asks a question, while each leaf node stores the predicted value.
• Decision Trees naturally model complex nonlinear relationships.
• A single Decision Tree is easy to interpret but can overfit the training data.
• In real-world machine learning, ensemble methods such as Random Forest, Gradient Boosting, XGBoost, LightGBM, and CatBoost are generally preferred because they provide much higher prediction accuracy.

Summary of Regression Algorithms

The following table summarises the purpose, main idea, and best use case for each of the regression algorithms discussed in this article.

Algorithm	Predicts	Main Idea	Best Used For
Linear Regression	Continuous values	Fits a straight-line relationship	Simple linear trends
Normal Equation	Continuous values	Direct mathematical solution	Small to medium datasets
Gradient Descent	Model optimiser	Iteratively minimises prediction error	Large datasets and many ML models
Logistic Regression	Classes / Probabilities	Uses the Sigmoid function for classification	Binary and multiclass classification
Polynomial Regression	Continuous values	Fits curved relationships	Nonlinear trends
Ridge Regression	Continuous values	L2 regularisation shrinks coefficients	Reducing overfitting while keeping all features
Lasso Regression	Continuous values	L1 regularisation removes some features	Automatic feature selection
Elastic Net Regression	Continuous values	Combines L1 and L2 penalties	Many correlated features
Stepwise Regression	Continuous values	Adds or removes features statistically	Traditional feature selection
Decision Tree Regression	Continuous values	Learns decision rules instead of equations	Complex nonlinear relationships

How These Concepts Build on Each Other

One of the easiest ways to understand these algorithms is to view them as a learning progression, where each concept extends or improves upon the previous one.

1. Linear Regression introduces the idea of fitting a straight line to model the relationship between input variables and the target.
2. Normal Equation provides a direct mathematical method for calculating the best regression line without iterative optimisation.
3. Gradient Descent offers an iterative optimisation technique that efficiently finds the best parameters for large datasets.
4. Logistic Regression builds upon similar optimisation ideas, but predicts probabilities for classification instead of continuous values.
5. Polynomial Regression extends Linear Regression by modelling curved relationships through polynomial features.
6. Ridge Regression, Lasso Regression, and Elastic Net Regression improve regression models by reducing overfitting and, in the case of Lasso and Elastic Net, automatically selecting useful features.
7. Stepwise Regression focuses specifically on selecting the most informative input features using statistical procedures.
8. Decision Tree Regression introduces an entirely different approach by replacing mathematical equations with a hierarchy of decision rules.

Concept Map

Linear Regression │ ├────────► Normal Equation │ ├────────► Gradient Descent │ ├────────► Logistic Regression │ ├────────► Polynomial Regression │ ├────────► Ridge Regression │ │ │ └────────► Lasso Regression │ │ │ └────────► Elastic Net │ ├────────► Stepwise Regression │ └────────► Decision Tree Regression │ ├──► Random Forest ├──► Gradient Boosting ├──► XGBoost ├──► LightGBM └──► CatBoost

Conclusion

Although these algorithms differ significantly in their mathematical foundations and modelling approaches, they collectively form the core building blocks of supervised machine learning.

Understanding these algorithms provides a solid foundation for learning more advanced machine learning techniques.

After Mastering These Algorithms, You Will Be Ready to Learn:

• Support Vector Machines (SVMs)
• Random Forests
• Gradient Boosting Machines
• XGBoost
• LightGBM
• CatBoost
• Artificial Neural Networks
• Deep Learning

In summary: Learning these regression algorithms is not just about understanding individual techniques—it is about developing the intuition behind optimisation, feature engineering, regularisation, model selection, and decision-based learning. These concepts underpin many of today's most powerful machine learning algorithms and serve as an essential stepping stone toward mastering modern Artificial Intelligence.