When do you call something an experiment or not?

[diannemc1957]

I am a Digital Marketing Manager and introduced the growth hacking mindset in my team. Build, measure, learn (experiment driven WoW).

All our experiments we log in Excel, but I am struggling about whether it’s an experiment or not.

For example:
- is raising your budget on a campaign an experiment?
- is a change in targeting an experiment?

Of course are new channels or A/B-tests on copy or images experiments.

But what do you think about when is something an experiment and when not?

 

[lc4yah]

@diannemc1957 From my experience, I see experiments are tactics to reach particular growth objectives that you can later introduce them in your regular strategy.

You can call an experiment when:

1. Establish where you want to experiment (budgets, new images, new text)
2. create some hypothesis
3. establish a deadline
4. After the deadline, analyze what happened, or you can measure it against a benchmark

You can't call an experiment an experiment if you don't analyze its results and learn from them.

 

[irefinn]

@lc4yah The last line

 

[namhoang217gv]

@diannemc1957 I think the difference between “experiments” and “optimization” are often misunderstood in the marketing community and a lot of “experiments” that people run are actually just “optimizations.”

An experiment or test should be structured with a hypothesis based on data, account for seasonality & bias, have a control group, have proper sample size, and meet statistical levels of significance. Otherwise you’re just creating variations and more “noise” in your data.

There’s a reason why the larger companies who test a ton, also have whole teams dedicated to it. It’s laborious, time consuming, needs be defensible statistically, and most importantly repeatable.

 

[saved1994]

@diannemc1957 Anything that moves the needle (impacts the north star metric) can be considered an experiment. There are no hard and fast rules.