Regression means predicting a numeric value: a price, a duration, a consumption. The simplest model, and often an excellent starting point, is linear regression:
ŷ = b + w₁·x₁ + w₂·x₂ + … + wₚ·xₚ
Each feature xⱼ receives a weight wⱼ, and b is the intercept. Training looks for the weights that minimize the sum of squared errors (ordinary least squares).
Training and reading the model
The built-in load_diabetes dataset describes 442 patients with 10 measurements (age, BMI, blood pressure…) already centered and scaled. The target is a measure of disease progression one year later.
from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
X, y = load_diabetes(return_X_y=True, as_frame=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0)
reg = LinearRegression().fit(X_train, y_train)
print(reg.intercept_) # constant b
print(dict(zip(X.columns, reg.coef_.round(1)))) # one weight per featureA positive coefficient means that, with other variables held constant, the target increases when the feature increases. Comparing coefficients with one another only makes sense if the features are on the same scale, which is the case here.
Measuring the error
from sklearn.metrics import mean_absolute_error, root_mean_squared_error, r2_score
y_pred = reg.predict(X_test)
print("MAE :", mean_absolute_error(y_test, y_pred)) # about 45
print("RMSE:", root_mean_squared_error(y_test, y_pred)) # about 56
print("R² :", r2_score(y_test, y_pred)) # about 0.36- MAE (mean absolute error): the average gap, in the unit of the target. Easy to explain.
- RMSE (root mean squared error): also in the unit of the target, but large errors weigh more. Always greater than or equal to the MAE.
- R²: share of the variance of y explained by the model. 1 = perfect; 0 = no better than predicting the mean; negative = worse than the mean. This is the value returned by
reg.score.
Note: in recent versions of scikit-learn, you use root_mean_squared_error; the old squared=False argument of mean_squared_error has been removed.
Always compare to a baseline
Is an R² of 0.36 good? To find out, compare with a naive model:
from sklearn.dummy import DummyRegressor
baseline = DummyRegressor(strategy="mean").fit(X_train, y_train)
print(mean_absolute_error(y_test, baseline.predict(X_test))) # about 58The linear model reduces the average error from about 58 to 45: it brings real information, even though a good part of the variability remains unexplained. In a business context, it is often this comparison, expressed in the unit of the target, that convinces.
Limits
Linear regression assumes an additive, linear relationship. If reality is curved or involves interactions, you will need to create suitable features (DS5 course) or use nonlinear models (lessons 4 and 5). It is also sensitive to extreme values and to highly correlated features, which brings us to regularization.
Exercises
Solutions and tests stay on the server. In the student space, your code is run and checked, and the tutor gives hints before the solution.
Training and evaluating a linear regression
Train a
LinearRegressiononX_train, y_train, predict the test set, and store the MAE inmaeand the R² inr2(usesklearn.metrics).from sklearn.datasets import load_diabetes from sklearn.model_selection import train_test_split from sklearn.linear_model import LinearRegression from sklearn.metrics import mean_absolute_error, r2_score X, y = load_diabetes(return_X_y=True) X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0) reg = ... mae = ... r2 = ...Beating the baseline
Compute the test RMSE of a
DummyRegressor(strategy='mean')(rmse_baseline) and of aLinearRegression(rmse_modele), then the relative gaingain = 1 - rmse_modele / rmse_baseline.from sklearn.datasets import load_diabetes from sklearn.model_selection import train_test_split from sklearn.linear_model import LinearRegression from sklearn.dummy import DummyRegressor from sklearn.metrics import root_mean_squared_error X, y = load_diabetes(return_X_y=True) X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0) rmse_baseline = ... rmse_modele = ... gain = ...Interpreting a negative R²
On the test set, a model gets R² = -0.2. What can you conclude?
- The model is perfect
- The model does worse than predicting the mean
- There is a calculation error, R² is always positive
- The model has a 20% error
End-of-lesson quiz
Which metric penalizes large errors the most?
- MAE
- RMSE
- R²
- Accuracy
What is the R² of a model that always predicts the mean of the evaluated set?
- 1
- 0
- -1
- 0.5
Where are the learned weights of a LinearRegression?
- reg.weights
- reg.coef_ and reg.intercept_
- reg.params()
- reg.score_
The rest of the course
- Regularization: Ridge and Lasso15 min
- Classification with Logistic Regression15 min
- Nonlinear Models: Decision Trees and k-Nearest Neighbors15 min
- Ensemble Methods: Random Forests and Gradient Boosting20 min