Mathematical Foundations

Polynomial Regression models the relationship between temperature $x$ and bike rentals $y$ using a polynomial of degree $d$: $$\hat{y}={\beta }_0+{\beta }_{1}x+{\beta }_2{x}^2+\cdots +{\beta }_d{x}^d$$
Increasing $d$ allows more flexible curves but risks overfitting to noise.

Bias-Variance Decomposition breaks prediction error into three components: $$\text{Expected Error}={\text{Bias}}^2+\text{Variance}+\text{Irreducible Error}$$
Bias² measures error from wrong model assumptions. High-bias models (low degree polynomials) are too simple to capture the true pattern—they systematically underpredict or overpredict. As model complexity increases, bias decreases.

Variance measures sensitivity to specific training data. High-variance models (high degree polynomials) fit training noise and vary wildly when trained on different samples. As model complexity increases, variance increases.

Irreducible Error is noise inherent in the data that no model can eliminate (weather variations, individual preferences, measurement errors).

The Tradeoff: Optimal model complexity balances bias and variance. Too simple = high bias (underfitting). Too complex = high variance (overfitting). The sweet spot minimizes total error on unseen test data.

Training vs Test Error: Training error always decreases with complexity (more parameters = better fit to training data). Test error follows a U-shaped curve: decreases initially (reducing bias), reaches minimum at optimal complexity, then increases (variance dominates). The gap between train and test error indicates overfitting—larger gap means the model memorized training-specific patterns.