Regularisation

Regularization is a crucial technique to prevent overfitting in machine learning models. When models become too complex, they can memorize training data rather than learning generalizable patterns.

In civil engineering applications, regularization helps create more robust models for tasks like predicting structural responses, material behavior, or system performance under varying conditions.

• Adjust polynomial degree and L1/L2 regularization sliders—L1 (Lasso) creates sparse coefficients, L2 (Ridge) smooths the curve
• Click the plot to add data points, or use "Generate New Data" and "Clear All Data" buttons
• Try high polynomial degrees with and without regularization to observe overfitting prevention

What polynomial degree causes overfitting without regularization? How does adding L1 or L2 regularization help?

• High polynomial degree: Creates complex curves that may overfit without regularization
• L1 regularization: Watch coefficients become exactly zero - automatic feature selection
• L2 regularization: Smooths the curve and reduces oscillations
• Bias-variance tradeoff: Higher regularization increases bias but reduces variance
• Optimal balance: Find the sweet spot between underfitting and overfitting

y = - 6.476 - 0.228 x

Squared Errors

L1 Norm

L2 Norm

Polynomial Degree (Max Power)

L1 Regularisation (

λ_{1}

)

0.0

L2 Regularisation (

λ_{2}

)

0.0

MSE = 54

Mean Squared Error

MAE = 6.28

Mean Absolute Error

Coefficients & Regularization Effects

Variable	$1$	$x$
Unregularized $a_{i}^{0}$	-6.476	-0.228
L1 Penalty $λ_{1} \| a_{i}^{0} \|$	—	—
L2 Penalty $λ_{2} (a_{i}^{0})^{2}$	—	—
Effective $a_{i}$	-6.476	-0.228
$Δ \| a_{i} \|$	—	—

Mathematical Foundations

The regularized cost function adds a penalty term to the standard mean squared error:
$J (\boldsymbol β) = \frac{1}{2 m} \sum_{i = 1}^{m} {(h_{\boldsymbol β} (x^{(i)}) - y^{(i)})}^{2} + λ \cdot penalty (\boldsymbol β)$
where $λ \geq 0$ is the regularization parameter controlling penalty strength. When $λ = 0$ , we recover ordinary least squares (no regularization). As $λ$ increases, we constrain parameter magnitudes, favoring simpler models.

L2 Regularization (Ridge) adds the sum of squared parameter values: $λ \sum_{j = 1}^{n} β_{j}^{2}$ . Ridge regression shrinks all coefficients towards zero proportionally but never eliminates features entirely. This is particularly effective when many features are relevant but should have modest contributions. Ridge tends to perform better when features are correlated, as it distributes weight among correlated features rather than arbitrarily selecting one.

L1 Regularization (Lasso) adds the sum of absolute parameter values: $λ \sum_{j = 1}^{n} | β_{j} |$ . The L1 penalty drives some parameters to exactly zero, performing automatic feature selection. Lasso is preferred when we believe only a sparse subset of features are truly relevant, producing interpretable models by eliminating irrelevant features. However, when many features are correlated, Lasso tends to arbitrarily select one and zero out the others.

Practical guidance: Use Ridge when most features are relevant and should contribute modestly. Use Lasso when you need automatic feature selection and interpretable models with sparse coefficients. Both methods require tuning the regularization strength $λ$ (typically via cross-validation). Feature scaling is essential before applying regularization to ensure penalties apply fairly across all features.

Developed by Kevin Yu & Panagiotis Angeloudis