Running better A/B tests: How to stop wasting time and money on bad...

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Running Better
A/B Tests:
How to stop wasting time and
money on bad tests

About
Mammoth
Growth
2
Mammoth Growth are a team of 70 customer data and
marketing technology experts.
Since 2015, we’ve h...

85% of A/B tests fail

Why does it matter?
● Running an A/B test is expensive
○ Build variant(s)
○ Set up experiment
○ Find bugs
○ Analyse result...

Data is the answer

Case Study: Social Sign-in
Context
● E-commerce business
● Use email to drive repeat customers
● Have Facebook sign-in opt...

Case Study: Social Sign-in
7
Hypothesis
If we add more sign-up options
Then conversion will increase
Because registering w...

Case Study: Social Sign-in
8
Plan
A/B test adding more sign-in
options

Case Study: Social Sign-in
9
Pre-test analysis

Case Study: Social Sign-in
10
We found
Users who sign up with
Facebook place less orders

Case Study: Social Sign-in
11
Digging deeper
Email open rates are much
lower for Facebook users

Case Study: Social Sign-in
12
New Plan
A/B test removing Facebook
sign-in option

Case Study: Social Sign-in
13
Conversion rate
Repeat rate

What did
we learn?
14
● Our intuition isn’t always right
● Analysing data is much faster than
running an A/B test
● You mi...

How can we learn faster?

Hypothesis
● User research
● Session recordings
● Exploratory analysis
Simple learning process
16
Experimentation
● A/B te...

Hypothesis
● User research
● Session recordings
● Exploratory analysis
An integrated learning process
17
Post-test
● Post-...

Hypothesis
What are you testing?

Clear
Hypothesis
19
If we…
Then…
Because…

Validation & Design
What’s the most valuable test?

● How many users drop-off here?
How many can we save?
● Will enough people interact with
the change for it to be impactful...

Simple Funnel Example
22
Landing Page Visited CTA Clicked Signed Up

Simple Funnel Example
Top of Funnel Ideas
● Ease - Make CTA more prominent
● Trust - Add social proof
● Convince - Make va...

Simple Funnel Example
24

Behavioural Analytics
Tools provide rapid user-centric analysis
● Funnel drop-offs
● Conversion paths
● Heatmaps
● Session...

Post-test
What else can we learn?

Case Study: Inﬁnite Scroll
27
Hypothesis
If we add infinite scroll
Then conversion will increase
Because people will be ex...

Case Study: Inﬁnite Scroll
28
A/B Test Idea
Add infinite scroll to product
listing page

Case Study: Inﬁnite Scroll
29
A/B Test Result
People given infinite
scroll spend less

Case Study: Inﬁnite Scroll
30
Post-test analysis

Case Study: Inﬁnite Scroll
31
Why?
They view less product pages

Case Study: Inﬁnite Scroll
32
Why?
They scroll much further
before viewing a product page

Case Study: Inﬁnite Scroll
33
Alternative Hypotheses
● Too much selection leads
to indecision
● Users who view more
produc...

Ask why?
● How many users saw / interacted
with the experiment?
● How did behaviour change at each
stage of the funnel?
● ...

Final Lesson
Document everything

Documentation
36
● You’ve worked hard to learn
something
● All for nothing if learnings are
siloed or don’t stick
● Docume...

Conclusion
● Make your A/B tests count
● Use existing data ﬁrst
● Analyse losing tests too

Share Slideshare

LinkedIn
Facebook
Twitter

Embed

Size (px)

Show related Slideshows at end

WordPress Shortcode

Link

Share
Email

2024 Trend Updates: What Really Wor... by Search Engine Jou... 1061471 views
Storytelling For The Web: Integrate... by Chiara Aliotta 979505 views
Artificial Intelligence, Data and C... by OECD Directorate ... 919456 views
How to Leverage AI to Boost Employe... by SocialHRCamp 390372 views
2024 State of Marketing Report – by... by Marius Sescu 220814 views
Everything You Need To Know About C... by Expeed Software 237095 views

View on Slideshare

1 of 37 Ad

View on Slideshare

1 of 37 Ad