5 Reasons Why I Stopped Following A/B Testing Case Studies

[bobjones]

5 reasons why you shouldn’t blindly follow A/B testing case studies:

1. What may work for one brand may not work for another.
2. The quality of the tests varies.
3. The impact is not necessarily sustainable over time.
4. False assumptions and misinterpretation of result.
5. Success bias: The experiments that do not work well usually do not get published.

Read in full here: https://www.dynamicyield.com/2014/08/ab-testing-case-studies/

 

[superbeaver]

@bobjones You can't take another company's successful efforts and apply it to your company. That's assumption and contradicts the whole purpose of split testing

 

[daniisa]

@bobjones they're only suggestions on what to try... not that they work unconditionally.

i will agree especially on #2 though. it drives me absolutely nuts when someone publishes an A/B test when the sample size was something like 100 tests. F that. After so many insignificant tests, I flat out told my devs and ops 2k at absolute minimum always. And then I still make them run their results through a significance calculator... occasionally they're short, but usually it's good enough.