At SMX Advanced 2016 in Seattle, Adria Kyne, SEO Manager (North America, Australia, and New Zealand) for Vistaprint, gave a presentation on doing search marketing testing — or any kind of marketing testing — that provides valid results.
Kyne made the point that by not understanding the importance of validity of sample size, we often end up unintentionally lying to ourselves, and we may actually “prove” the opposite of what we think our tests are showing.
Adria Kyne, Vistaprint
Common problems with marketing testing
We need to start with understanding the point of our testing:
We want to know what is really happening when people visit our site.
We want to be sure we can use the results to predict likely future behavior.
We also need to understand the basics of hypothesis testing:
We want to know if the variation we are testing is better, worse or the same as the original.
We don’t want to see a positive outcome that isn’t really there (a false positive, or Type I error).
We don’t want