Introduction to Data Analytics · Lesson 3 of 3

Data Quality, Privacy, and Responsible Analysis

Check whether records are complete, valid, consistent, and appropriate to use.

Lesson

1. Introduction

Analysis inherits the strengths and weaknesses of its source data. Missing identifiers, duplicate events, stale extracts, and inconsistent definitions can produce precise but misleading charts. Responsible analysis also asks whether data may be collected and shared for the proposed purpose.

2. Understand the concept

A data-quality rule should describe a testable property: an order ID is present, a quantity is positive, or a timestamp falls inside the reporting window. Track how many rows fail each rule and why. Privacy means using only the fields and level of detail required, with approved access and retention.

3. Syntax and interpretation

required = ['order_id', 'order_date', 'amount']
missing = orders[required].isna().sum()
duplicates = orders['order_id'].duplicated().sum()
print(missing.to_dict(), duplicates)

These checks quantify missing required values and duplicate keys; the analyst must decide whether duplicates are invalid events or valid repeated line items.

4. Apply it to an analysis

Check whether records are complete, valid, consistent, and appropriate to use. Start with a small, known example and state what each row or value represents. After the transformation, check row counts, types, keys, and units against the original data. If an assumption fails, stop and investigate instead of presenting the output as a valid result.

5. Common mistakes

Replacing every missing value with zero even when zero means a real measurement.
Exporting identifiable or confidential records into an unapproved tool.

6. Practice task

A customer table contains duplicate customer IDs. What should you inspect before removing them?

7. One possible solution

Check whether the ID is meant to be unique, compare duplicate records and timestamps, and establish whether they represent accidental duplicates, history versions, or legitimate one-to-many data.

8. Summary and next step

Check whether records are complete, valid, consistent, and appropriate to use. Continue to the next course section and apply this foundation to a different analytics tool or workflow.

Visual guide

Working through Data Quality, Privacy, and Responsible Analysis

Step 01 Define
State the question, data grain, and expected output.
Step 02 Apply
Use the method and check its assumptions on a small example.
Step 03 Verify
Reconcile the result and communicate limitations.

Code examples

Check allowed values

PYTHON

valid_statuses = {'new', 'paid', 'cancelled'}
invalid = [status for status in ['paid', 'unknown'] if status not in valid_statuses]
print(invalid)

Expected output

['unknown']

Report missingness before cleaning

SQL

SELECT COUNT(*) AS total_rows,\n       SUM(CASE WHEN email IS NULL THEN 1 ELSE 0 END) AS missing_email\nFROM customers;