Group Data with K-means

In unsupervised learning, there is no target to predict. You look for hidden structure in the data. Clustering groups observations that resemble each other: customer segments, product families, behavior profiles.

How does K-means work?

K-means looks for k groups in which every point is close to the center (centroid) of its group:

1. place k initial centers (by default, scikit-learn uses k-means++ initialization, which spreads the centers apart from each other);
2. assign each point to the nearest center;
3. recompute each center as the mean of the points in its group;
4. repeat steps 2 and 3 until the centers stop moving.

The algorithm minimizes the inertia: the sum of squared distances between each point and its center.

First clustering

from sklearn.datasets import make_blobs
from sklearn.cluster import KMeans

# 500 points spread over 4 blobs (the true membership y is not used)
X, _ = make_blobs(n_samples=500, centers=4, cluster_std=1.0, random_state=42)

km = KMeans(n_clusters=4, random_state=0)
labels = km.fit_predict(X)         # one cluster number per point: 0, 1, 2 or 3

print(km.cluster_centers_)          # coordinates of the 4 centers
print(km.inertia_)                  # final inertia (about 949)
print(km.predict([[0, 0]]))         # assign a new point

Notice that fit does not receive y. Cluster numbers are arbitrary: “cluster 2” is not bigger or better than “cluster 0”, and rerunning with a different seed may permute the numbers.

Initialization and reproducibility

K-means can converge to a poor solution depending on the starting centers. The n_init parameter runs several initializations and keeps the one with the lowest inertia. Since scikit-learn 1.4, its default value is "auto" (a single initialization with k-means++). Set random_state to get reproducible results.

Feature scale matters

K-means computes Euclidean distances: a feature expressed in the thousands dominates those expressed in units. In the Wine dataset, proline (hundreds up to over 1,600) overwhelms the other measurements.

from sklearn.datasets import load_wine
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import adjusted_rand_score

X, y = load_wine(return_X_y=True)     # y is only used to judge the result afterwards
brut = KMeans(n_clusters=3, random_state=0).fit_predict(X)
norme = make_pipeline(StandardScaler(), KMeans(n_clusters=3, random_state=0)).fit_predict(X)

print(adjusted_rand_score(y, brut))    # about 0.37
print(adjusted_rand_score(y, norme))   # about 0.90

The adjusted Rand index (ARI) compares two partitions: 1 = identical, about 0 = agreement due to chance. It is only useful when you have true labels available as a check; in real situations, you generally don’t have any.

Limits of K-means

• k must be chosen in advance (next lesson);
• it assumes compact groups, roughly spherical in shape and of comparable sizes;
• it is sensitive to extreme values, which shift the centers;
• every point is assigned to a cluster, even if it is isolated.

For groups with complex shapes or in the presence of noise, use DBSCAN or hierarchical clustering (lesson 3).