The Dimensions of Data Quality

First lesson of the Data Quality and Governance course, free to read: about 16 minutes, 3 exercises and 4 quiz questions. This is the catalogue's base lesson; in the student space, it is rewritten for your profile.

Objectives

  • Name and define the main dimensions of data quality
  • Turn a business requirement into a measurable quality rule
  • Compute quality indicators on a dataset

Quality dimensionsQuality rulesQuality indicators

“The numbers are wrong”: this is the sentence that ruins trust in a dashboard. Behind it, the causes vary: a customer counted twice, yesterday’s sales not yet loaded, a mistyped postal code. To act, you first need to name the problem and measure it. The dimensions of data quality give this common vocabulary.

Quality depends on the use

Data is not good or bad in absolute terms: it is fit or unfit for a use. A customer file without phone numbers suits an emailing campaign, not a calling campaign. Before measuring, you therefore ask: who uses this data, for what, and what error would have consequences?

The six usual dimensions

Frameworks do not all list exactly the same dimensions, but these six come up everywhere:

Dimension Question Example of a defect
Completeness Is the value filled in? empty email
Validity Does it respect the expected format or domain? 4-digit postal code, email without @
Accuracy Does it match reality? wrong address, well formed
Consistency Is it compatible with other data? “France” here, “FR” elsewhere; delivery date before the order
Uniqueness Is each real-world object present only once? same customer entered twice
Freshness (timeliness) Is it recent enough for the use? yesterday’s sales missing at 9 AM

Two frequent confusions:

  • validity and accuracy: [email protected] is a valid email, but if it belongs to someone else, it is inaccurate. Validity is checked with a rule; accuracy often requires a reference source or human verification;
  • uniqueness and technical duplicates: two identical rows are an obvious duplicate; two rows “Marie Dupont” and “M. Dupont” with the same email are the same customer, which is harder to detect.

Some teams add integrity (references point to existing objects), precision (a sufficient level of detail), or accessibility.

From requirement to measurable rule

A vague requirement (“customer data must be clean”) cannot be measured. You translate it into rules:

  • Completeness: “email is filled in for at least 98% of active customers”;
  • Validity: “the French postal code has exactly 5 digits”;
  • Plausibility (a verifiable form of accuracy): “the date of birth is in the past and after 1900”;
  • Uniqueness: “no email is shared by two active customers”;
  • Freshness: “the previous day’s sales are available every day at 7 AM”.

Each rule has a threshold (100% is not always realistic or necessary) and an owner.

Computing an indicator

A quality indicator is a simple ratio: compliant rows / checked rows. Mind the denominator: the validity of an email is computed on the emails that are filled in, otherwise you mix completeness and validity.

SELECT
  100.0 * COUNT(email) / COUNT(*)                                AS pct_email_renseigne,
  100.0 * SUM(CASE WHEN email LIKE '%_@_%' THEN 1 ELSE 0 END)
        / COUNT(email)                                          AS pct_email_valide
FROM clients;

In SQL, COUNT(column) ignores NULLs: this is handy for completeness. Watch out for empty strings, which are not NULL: depending on the source, a missing email can be '', NULL, or 'n/a'.

Tracking quality over time

A one-off measurement is not enough: you compute the indicators at every load and record them. A chart of the trend shows whether quality is degrading (new version of a form, new source) and whether corrective actions work. Transformation tools (dbt tests) and specialized tools (Great Expectations, Soda, or the quality features of cloud platforms) automate these checks. But the choice of rules and thresholds remains a business decision, made with the users of the data.

Exercises

Solutions and tests stay on the server. In the student space, your code is run and checked, and the tutor gives hints before the solution.

  1. TaskDifficulty 22 hints

    Measuring the quality of a customer file

    Data (clients.csv):

    client_id,email,code_postal,date_naissance,pays
    1,[email protected],69001,1990-04-12,FR
    2,,75011,1985-11-30,FR
    3,[email protected],7501,1979-02-15,FR
    4,[email protected],33000,2031-01-01,FR
    5,[email protected],69001,1990-04-12,FR
    6,[email protected],13008,,France
    7,sam.ex.fr,44000,1992-07-08,FR
    8,[email protected],59000,1988-09-21,FR
    9,[email protected],31000,1995-03-03,FR
    10,,67000,2000-12-24,FR

    Compute, by hand, in a spreadsheet, or in SQL:

    1. the completeness of the email (as a % of the 10 rows);
    2. the number of emails that are filled in but invalid (without an @ preceded and followed by at least one character);
    3. the number of invalid postal codes (not exactly 5 digits);
    4. the number of impossible dates of birth (in the future);
    5. the number of rows whose pays is not in the expected FR format;
    6. the number of duplicate rows (same email, same date of birth as another row).
    • Email completeness computed over 10 rows
    • Email validity computed only on the emails that are filled in
    • Each indicator is linked to its dimension (completeness, validity, accuracy/plausibility, consistency, uniqueness)
    • The affected rows are identified by their client_id
  2. Multiple choiceDifficulty 11 hint

    Which dimension?

    At 9 AM, the sales dashboard does not yet show the previous day’s sales, which arrive around 11 AM. Which quality dimension is failing?

    • Uniqueness
    • Freshness (timeliness)
    • Validity
  3. TaskDifficulty 11 hint

    Writing measurable rules

    Customer service expresses three expectations: “we must be able to reach customers”, “no duplicate customers”, and “addresses must be up to date”. Translate each one into a measurable rule, with the dimension concerned, the indicator formula, and a threshold.

    • Each rule names a dimension (completeness, uniqueness, freshness, or accuracy)
    • Each rule has an explicit calculation formula (numerator and denominator)
    • Each rule has a numeric threshold
    • The address rule specifies how accuracy or freshness is verified (date of last confirmation, address reference database)

End-of-lesson quiz

  1. An email [email protected] actually belongs to another customer. Which dimension is failing?

    • Validity
    • Accuracy
    • Completeness
  2. “Country = France” in one table, “FR” in another: which dimension?

    • Consistency
    • Freshness
    • Uniqueness
  3. Why compute email validity only on the emails that are filled in?

    • So as not to mix completeness and validity
    • Because SQL requires it
    • To get a better score
  4. Who sets the acceptance threshold for a quality rule?

    • The data engineer alone
    • The business users of the data, with the data team
    • The quality tool, automatically

The rest of the course

  1. Data Contracts16 min
  2. Data Catalog and Lineage16 min
  3. GDPR and Personal Data in Data Platforms18 min
  4. Building a Quality Improvement Plan16 min

Data Quality and Governance · Intermediate · 1 h 22 min