Statistical Significance in PPC and a Helpful Tip

“How much do I spend to test an ad?”

I’m guilty of asking this question at some point regarding PPC advertising and I see the question asked a lot on affiliate marketing forums and super affiliate blog post comments. I’ve also seen ridiculous answers like 30 clicks (wtf?) and $100 in spend (WTF!?)- are you serious?

If we were optimizing a campaign and in a single ad group we had two ads running, one at 1.74% CTR (43 clicks) and the other at 1.58% CTR (38 clicks), which is doing better? If we’re playing “winner stays on,” which should be replaced? Most people might say “the ad with 43 clicks and a 1.73% CTR is best and should stay up, DUH!” Well, unfortunately, the conclusion is not so simple.

The answer isn’t a magic number of clicks or how much you’ve spent on the ad, but whether the difference in clicks is significant or just statistical noise. I have a tip to help you but we’ll get to that later.

How do you tell if the difference in clicks is significant? You can think of this in terms of a binomial distribution (yes, I’m taking you back to first semester stats class) where you flip a coin ten times and count the number of times you land on heads. We should expect the coin to land on heads 5 times (the mean) because there are two sides of the coin and ten experiments- a 50% chance. It also may land on heads 4, 3, 2, 1, or 0 times and 6, 7, 8, 9, or 10 times with a decreasing probability in either direction. Landing on heads 3, 4, or 6 times would represent a large percentage change from the expected mean: -40%, -20%, and +20% away respectively. Now let’s say we flip the same coin 10,000 times. I’m not sure how long this experiment would take to complete but tell me how many times you would expect the coin to land on heads. You might say 5,000 times but the chances of that are pretty slim, again. The coin is more likely to land heads something like 4,695 times or 5,129 times, -6% and +2% respectively. Notice how as the number of experiments grow, the percentage away from the mean becomes smaller, despite the fact the number away from the mean is larger- this is called regression towards the mean.

What does regression towards the mean have to do with our ads? Well the trick for determining boundaries of significance is taking the square root (√)of clicks (n) and using that number above (n+√n) and below (n-√n) as boundaries. If the boundaries of competing ads cross over one another, the difference in clicks is probably noise. In the case of our earlier example, √43 is almost 7 and the √38 is about 6. Therefore, the the lower boundary of 43 is 36 and the upper boundary of 38 is 44, thus the two boundaries overlap and the difference is noise and not significant, therefore you should keep the ads running until your boundaries are not overlapping.

What if the number of clicks was 380 and 440? The √380 is about 19 and the √440 is about 21. The upper boundary of 380 would be 199 and the lower boundary of 440 is 419, the boundaries do not overlap, the difference is significant and the ad with 440 clicks is a clear winner.

* There is a caveat to this rule as PPC is much more complicated than CTRs and clicks. You should also be looking at conversions, among other factors! What if an ad has 5 times more conversions because the psychology behind that particular ad attracts a buyer that's more willing to buy or call? That's a clear winner in my book! Be careful and use this as one of your many tools in optimizing PPC campaigns.