A statistical deep dive into a real 588,000-user ad campaign, testing whether the ads worked, how much they mattered, who they worked on, and when.

This study set out to answer four connected questions:

• Did showing people ads actually cause more conversions, or could the gap be explained by chance?
• If ads did work, how big was that effect really, statistically significant doesn't always mean practically important.
• Did the amount of exposure matter, does seeing more ads change someone's odds of converting, and does that effect work the same way for people who only saw a PSA (or nothing)?
• And finally, does timing matter, are some days or hours simply better windows for reaching people who convert?

Each section below builds on the last. First confirming the ads actually work, then sizing that effect honestly, then digging into exposure and timing to understand not just whether but why and when.

A note on sample size

This dataset has ~588,000 rows, a considerably large dataset. With this many observations, even a tiny, practically meaningless difference can come back as "statistically significant." So alongside every p-value, I also report an effect size, that's what tells you whether a result actually matters, not just whether it's technically nonzero.

Loading and checking the data

The raw columns come with spaces in their names (test group, total ads), so the first step is renaming them into something I can reference easily in code.

df <- read_csv("marketing_AB.csv") %>%
  rename(
    row_index    = 1,  # the unlabeled index column
    user_id      = `user id`,
    test_group   = `test group`,
    converted    = converted,
    total_ads    = `total ads`,
    most_ads_day = `most ads day`,
    most_ads_hour = `most ads hour`
  )

print(colnames(df))
cat("\nRows:", nrow(df), "\n")
print(table(df$test_group)) 
#size of each group. ad=564577, psa=23524
print(colSums(is.na(df))) #no missing values found

No missing values. The size of both groups is imbalanced by design of the experiment, most people saw ads (564,577), only a small group saw the PSA (23,524). That's expected here, not a data problem, so I moved straight into measuring conversion

Conversion summary

A simple function to measure the conversion rate for each group.

Code

Ad converted at a higher rate than psa.

But a raw gap in two numbers isn't proof of anything on its own, it could be random noise. That's what the chi-square test checks next.