The "Penalty" and "C" values in scikit-learn's Logistic Regression

In scikit-learn, the penalty and C parameters modify the standard cost function (loss function) of the LogisticRegression class during the optimization process. They do not change the core predictive sigmoid formula itself, but they directly dictate how the model weights (w) are calculated during training.

The core math breaks down as follows:

1. The Standard Cost Function (No Regularization)

Without any penalty (penalty=None), a binary logistic regression model optimizes the standard log-loss (binary cross-entropy) cost function, denoted as L(w):

L(w) = − Σ_i=1ⁿ  [ y_i log(ŷ_i) + (1 − y_i) log(1 − ŷ_i) ]

Where:

• y_i is the true label (0 or 1).
• ŷ_i is the predicted probability generated by the sigmoid function:

1

1 + e^−wTx_i

2. How penalty Alters the Formula

The penalty parameter adds a regularization term to the cost function to restrict the size of the coefficients and prevent overfitting. It determines the shape of this constraint:

penalty='l2' (Ridge, Default)

Adds the squared magnitude of coefficients (L₂ norm).

Penalty Term = ½ ‖w‖₂² = ½ Σ_j=1^m w_j²

penalty='l1' (Lasso)

Adds the absolute magnitude of coefficients (L₁ norm), which drives unimportant weights to exactly zero to create a sparse model.

Penalty Term = ‖w‖₁ = Σ_j=1^m |w_j|

3. How C Alters the Formula

In traditional statistics, regularization strength is controlled by a parameter named λ (lambda), where a larger λ means more aggressive penalty enforcement:

Traditional Cost = L(w) + λ × (Penalty Term)

Instead of λ, scikit-learn uses C, which is defined as the inverse of regularization strength:

C = 1/λ

Crucially, scikit-learn multiplies C by the loss term rather than the penalty term. The objective function that scikit-learn minimizes looks like this:

Important

Scikit-Learn Cost = C × L(w) + Penalty Term

Directly Comparing the Behavior of C

Value of C	Regularization Strength	Effect on the Objective Function	Model Behavior
Small C (e.g., 0.01)	High	The penalty term dominates; the data loss L(w) is largely ignored.	Coefficients shrink heavily toward zero; protects against overfitting.
Large C (e.g., 1000)	Low	The data loss L(w) dominates; the penalty term is ignored.	Model prioritizes fitting the training data exactly; risks overfitting.

Key Observation

Although C is defined as the inverse of the regularization strength (C = 1 / λ), scikit-learn's optimization objective is implemented as:

Scikit-Learn Cost = C × L(w) + Penalty Term

As a result:

• A smaller C places relatively less emphasis on fitting the training data and therefore results in stronger regularization.
• A larger C places greater emphasis on minimizing the data loss, resulting in weaker regularization.