Feature Importace

Feature Importance

Feature importance is a measure of how much each input feature contributes to a machine learning model's predictions. It helps answer questions such as:

• Which features are most influential?
• Which features can be removed without significantly affecting performance?
• Which features should receive more attention during feature engineering?

For example, suppose you are predicting house prices using the following features:

Feature	Importance
Area	0.48
Location	0.29
Number of Bedrooms	0.12
Age of House	0.08
Parking Spaces	0.03

This tells us that Area has the greatest influence on the predicted price, while Parking Spaces contributes very little.

Why is Feature Importance Useful?

Feature importance helps in:

• ✅ Understanding the model
• ✅ Removing irrelevant features
• ✅ Reducing overfitting
• ✅ Faster training
• ✅ Better feature engineering
• ✅ Explaining predictions to stakeholders

Methods for Determining Feature Importance

There is no single method. The appropriate method depends on the machine learning algorithm being used.

The methods can be broadly grouped into six categories.

1. Model Coefficients
2. Decision Tree Feature Importance
3. Permutation Feature Importance
4. SHAP (SHapley Additive exPlanations)
5. LIME (Local Interpretable Model-agnostic Explanations)
6. Mutual Information

Note: Different algorithms require different interpretation techniques. For example, coefficients naturally explain Linear Regression models, while SHAP is far more suitable for Neural Networks and Gradient Boosting models.

1. Model Coefficients (Linear Models)

Applicable to:

• Linear Regression
• Logistic Regression
• Ridge Regression
• Lasso Regression

A linear regression model is

y = β₀ + β₁x₁ + β₂x₂ + … + β_nx_n

Each coefficient represents the influence of its feature.

Example:

Price=5000+ 120 x Area +25000 x Bedrooms-800 x  Age 

Interpretation:

• Area coefficient = 120
• Bedrooms coefficient = 25000
• Age coefficient = −800

A larger absolute coefficient generally indicates greater importance.

Advantages

• Very easy to interpret
• Fast

Limitations

• Features should be scaled before comparing coefficients.
• Assumes linear relationships.

2. Decision Tree Feature Importance

Applicable to:

• Decision Trees
• Random Forests
• Extra Trees

Decision trees split data using the feature that produces the largest reduction in impurity.

Common impurity measures include:

• Gini Index
• Entropy
• Variance Reduction (for regression)

The importance of a feature is calculated as the total impurity reduction contributed by all splits using that feature.

Example

Income
├── Age
│    ├── Salary
│    └── Experience
└── Loan Amount

If Income is selected near the root and appears frequently, it is usually more important than the other features.

Advantages

• Built into tree-based models
• Very fast

Limitations

• Biased toward features with many unique values
• Can overestimate importance of continuous features

3. Permutation Feature Importance

Important

This is one of the most popular model-agnostic methods.

Idea

1. Train the model.
2. Measure baseline accuracy.
3. Shuffle one feature.
4. Measure accuracy again.
5. The larger the performance drop, the more important the feature.

Example

Original Accuracy:

95%

Shuffle Age:

Accuracy → 80%
Drop = 15%

Shuffle Income:

Accuracy → 94%
Drop = 1%

Therefore,

Age is much more important than Income.

Advantages

• Works with any model
• Easy to understand

Limitations

• Computationally expensive
• Can underestimate importance when features are highly correlated

4. SHAP (SHapley Additive exPlanations)

SHAP is one of the most powerful methods for interpreting machine learning models.

It comes from game theory, where each feature is viewed as a "player" contributing to the prediction.

For every prediction,

Prediction = Base Value + ∑(SHAP Values)

Each feature receives a contribution value.

Example

Feature	SHAP Value
Income	+12
Age	−4
Experience	+8

Interpretation:

• Income increases the prediction by 12.
• Age decreases it by 4.
• Experience increases it by 8.

Advantages

• Very accurate
• Explains individual predictions and global feature importance
• Works with almost all ML models

Limitations

• Computationally intensive for large datasets
• More complex to understand than simpler methods

5. LIME (Local Interpretable Model-agnostic Explanations)

LIME explains one prediction at a time.

Instead of analysing the whole model, it:

1. Chooses one observation.
2. Creates many similar samples around it.
3. Fits a simple interpretable model locally.
4. Uses that local model to estimate feature importance.

Example

Suppose a loan application is rejected.

LIME might report:

Income          +35%
Credit Score    +25%
Existing Loans  -40%
Age             +5%

This explains why that specific applicant received the prediction.

Advantages

• Easy to understand
• Works with any model

Limitations

• Explains only local behaviour
• Results may vary depending on the sampled neighbourhood

6. Mutual Information

Mutual Information measures how much knowing one variable reduces uncertainty about another.

If a feature provides a lot of information about the target, it has high mutual information.

Example

Suppose we are predicting whether a customer buys a product.

Income → strongly related
Age → weakly related
Favourite Colour → unrelated

The mutual information scores might be:

Feature	MI Score
Income	0.81
Age	0.22
Favourite Colour	0.01

Advantages

• Captures non-linear relationships
• Does not assume linearity

Limitations

• Does not indicate whether the relationship is positive or negative
• Does not explain interactions between features

Comparison of Feature Importance Methods

Method	Model-Specific	Works with Any Model	Explains Individual Predictions	Speed	Best For
Linear Coefficients	✅	❌	❌	Very Fast	Linear models
Tree Importance	✅	❌	❌	Fast	Decision Trees, Random Forests
Permutation Importance	❌	✅	❌	Medium	General-purpose feature ranking
SHAP	Mostly model-agnostic (specialized versions exist)	✅	✅	Slow	Detailed global and local explanations
LIME	❌	✅	✅	Medium	Explaining individual predictions
Mutual Information	❌	✅	❌	Fast	Feature selection before training

Which Method Should You Use?

If you're using...	Recommended Method
Linear Regression	Coefficients (after feature scaling)
Logistic Regression	Coefficients (after feature scaling)
Decision Tree	Built-in tree feature importance
Random Forest	Built-in importance + Permutation Importance
XGBoost / LightGBM / CatBoost	Built-in importance + SHAP
Neural Networks	SHAP or Permutation Importance
Any black-box model	SHAP or LIME
Feature selection before modelling	Mutual Information or Permutation Importance

Key Takeaways

• Feature importance quantifies how much each input feature influences a model's predictions.
• Different models require different interpretation techniques; there is no universally best method.
• Model coefficients are simple and effective for linear models.
• Tree-based importance is efficient but can be biased toward high-cardinality features.
• Permutation importance is model-agnostic and measures the effect of disrupting a feature on model performance.
• SHAP provides the most comprehensive explanations by attributing contributions to each feature for both global and individual predictions.
• LIME focuses on explaining a single prediction using a local surrogate model.
• Mutual Information is primarily a feature-selection technique that detects both linear and non-linear dependencies before or during model development.

---

There are a few additional methods that are also used in practice: 

•  Drop-Column Feature Importance – Retrain the model after removing one feature at a time and measure the decrease in performance. This is accurate but computationally expensive. 
• Information Gain – Often discussed for decision trees and closely related to impurity reduction (especially when entropy is used instead of Gini impurity). 
• Gain, Cover, and Frequency Importance – Built-in importance measures used by gradient boosting libraries such as XGBoost and LightGBM. 
• Recursive Feature Elimination (RFE) – An iterative feature selection method that removes the least important features based on a model. 
• Boruta Algorithm – A wrapper method built around Random Forests that identifies all relevant features by comparing them with randomized "shadow" features. 

These methods are generally considered more specialized, whereas the six methods from my previous response are the ones most commonly taught in machine learning courses and most frequently used in practice.