Probability & Statistics for Data Analytics · Lesson 2 of 3

Sampling, Confidence, and A/B Test Interpretation

Understand sampling variability and interpret experiment results beyond a single p-value.

Lesson

1. Introduction

An observed difference can arise from a real effect, sampling variation, or bias in how data was collected. Statistical inference helps quantify uncertainty, but it cannot repair an experiment whose groups were selected unfairly.

2. Understand the concept

Random assignment supports causal comparison when other conditions are controlled. A confidence interval describes a method’s long-run coverage under assumptions. A p-value measures how surprising data would be under a specified null model; it is not the probability that the null hypothesis is true.

3. Syntax and interpretation

difference = treatment_rate - control_rate
relative_lift = difference / control_rate
# Report effect size with a confidence interval and sample sizes

Absolute percentage-point change and relative lift answer different questions; report both only when their definitions are clear.

4. Apply it to an analysis

Understand sampling variability and interpret experiment results beyond a single p-value. Start with a small, known example and state what each row or value represents. After the transformation, check row counts, types, keys, and units against the original data. If an assumption fails, stop and investigate instead of presenting the output as a valid result.

5. Common mistakes

Stopping an experiment as soon as a result becomes significant without a planned stopping rule.
Treating statistical significance as proof that an effect is large or commercially useful.

6. Practice task

Control conversion is 10% and treatment is 12%. State the absolute change and relative lift.

7. One possible solution

Absolute change is 2 percentage points. Relative lift is (12%-10%)/10% = 20%.

8. Summary and next step

Understand sampling variability and interpret experiment results beyond a single p-value. Continue to the next course section and apply this foundation to a different analytics tool or workflow.

Visual guide

Working through Sampling, Confidence, and A/B Test Interpretation

Step 01 Define
State the question, data grain, and expected output.
Step 02 Apply
Use the method and check its assumptions on a small example.
Step 03 Verify
Reconcile the result and communicate limitations.

Code examples

Compute percentage-point change

PYTHON

control = 0.10
treatment = 0.12
print((treatment - control) * 100)

Expected output

2.0 percentage points

Compute relative lift