meta-ab-test-planner

A/B Test Planner

Overview

This skill helps set up statistically sound A/B tests in Meta Ads, including hypothesis formulation, test structure, sample size calculations, and result interpretation for reliable optimization decisions.

A/B Testing Fundamentals

Why A/B Test in Meta Ads?

┌─────────────────────────────────────────────────────────────────┐
│  A/B TESTING vs "JUST RUNNING"                                  │
│                                                                 │
│  Without A/B test:                                              │
│  ├── Algorithm picks "winner" based on early signals            │
│  ├── No statistical certainty                                   │
│  ├── Seasonal/timing effects confuse results                    │
│  └── Learnings not reproducible                                 │
│                                                                 │
│  With A/B test:                                                 │
│  ├── Equal conditions for both variants                         │
│  ├── A winner with a confidence % (Meta simulates outcomes)     │
│  ├── Isolated variable = clear learning                         │
│  └── Reproducible results                                       │
└─────────────────────────────────────────────────────────────────┘

Meta's Native A/B Test Tool

LOCATION: Ads Manager toolbar → A/B test
          or Experiments → Create test → A/B Testing

BENEFITS:
├── Automatic audience split (nobody sees both versions)
├── Winner on cost per result, with a confidence %
├── Controlled test environment
└── Winner declared when the data supports one

LIMITS (Meta):
├── Test schedule of 1 to 30 days; Meta recommends at least 7
└── Experiments compares up to 5 existing campaigns

WHEN TO USE:
├── Creative testing (image A vs B)
├── Audience testing (LAL vs Interest)
├── Placement testing (Auto vs Manual)
└── Optimization goal testing (the "Custom" variable)

Sources (Meta Business Help Center): "About A/B testing", "Best practices for A/B Testing", "How winning campaigns are determined in A/B tests without a holdout", "Create an A/B test in Experiments tool" and "Selecting a variable for your A/B Test".

Hypothesis Framework

Formulating a SMART Hypothesis

HYPOTHESIS STRUCTURE:
"If we change [VARIABLE] from [A] to [B],
then we expect [METRIC] to improve by [X%],
because [RATIONALE]."

EXAMPLE:
"If we change the video hook from 'product-first' to
'problem-first', then we expect View Rate (3 sec)
to improve by 15%, because people first recognize their
problem before they're interested in the solution."

Good vs Bad Hypotheses

Test Prioritization Framework

ICE Score Method

PRIORITIZATION FORMULA:
ICE Score = (Impact + Confidence + Ease) / 3

IMPACT (1-10):
├── How much influence on results?
├── 10 = Core element (hook, offer)
└── 1 = Minor detail (button color)

CONFIDENCE (1-10):
├── How sure are you of improvement?
├── 10 = Supported by data/research
└── 1 = Pure guess

EASE (1-10):
├── How easy to execute?
├── 10 = Copy change
└── 1 = Remake entire video

Test Priority Matrix

Rules of thumb, not Meta figures: check them against the account's own history.

PRIORITY    │ WHAT TO TEST                  │ ICE
────────────┼───────────────────────────────┼─────
HIGH        │ Video hook (first 3 sec)      │ 9.0
HIGH        │ Headline                       │ 8.5
HIGH        │ Offer/promotion               │ 8.5
MEDIUM      │ Primary text length           │ 7.0
MEDIUM      │ CTA button                    │ 6.5
MEDIUM      │ Image style                   │ 6.5
LOW         │ Description text              │ 5.0
LOW         │ Emoji usage                   │ 4.5
SKIP        │ Button color                  │ 2.0

Test Types & Setup

The 14-day durations below are rules of thumb, not Meta figures: check them against the account's own history. Meta's own bounds are a 1 to 30 day schedule, with at least 7 days recommended.

Type 1: Creative Test

GOAL: Which creative performs better?
VARIABLE: Image, video, or carousel

SETUP IN META:
├── Campaign: [Existing campaign]
├── Test Type: Creative
├── Variants: 2-5 creatives
├── Metric: cost per result (e.g., CPA or CPC)
├── Budget Split: Equal (50/50 for 2 variants)
└── Duration: Min. 7 days

EXAMPLE:
├── Variant A: Product lifestyle photo
├── Variant B: Product on white background
├── Variant C: UGC-style photo
└── Primary Metric: Purchase CPA (or CPC)

Meta picks the winner on cost per result ("How winning campaigns are determined in A/B tests without a holdout") and recommends the same budget for each version ("Create an A/B test in Experiments tool"); Meta's API runs creative tests across 2 to 5 cells ("Split testing", Marketing API).

Type 2: Audience Test

GOAL: Which audience has best CPA/ROAS?
VARIABLE: Targeting

SETUP IN META:
├── Campaign: New test campaign
├── Test Type: Audience
├── Variants: 2 audiences
├── Creative: SAME for both (!)
├── Budget: Equal per variant
└── Duration: Min. 14 days (more data needed)

EXAMPLE:
├── Variant A: LAL 1% Purchasers
├── Variant B: Interest Stack "Fitness"
├── Control: Same creative set
└── Primary Metric: CPA or ROAS

Type 3: Placement Test

GOAL: Auto placements vs Manual?
VARIABLE: Where ads are shown

SETUP IN META:
├── Campaign: Test campaign
├── Test Type: Placement
├── Variants:
│   ├── A: Advantage+ Placements (auto)
│   └── B: Manual (e.g., Feed + Stories only)
├── Creative: Same
└── Duration: 14 days

WHEN RELEVANT:
├── You suspect Meta Audience Network wastes budget
├── You want to isolate Stories performance
├── You want to test adding Threads (it runs only with Instagram feed)
└── New account without placement data

Meta's placement rules: Threads requires the Instagram feed placement, and Audience Network cannot run alone ("Placement targeting", Marketing API).

Sample Size & Duration Calculator

Minimum Test Duration

Rules of thumb, not Meta figures: check them against the account's own history. Only the 7-day minimum and the 30-day maximum come from Meta ("Best practices for A/B Testing", cited above).

RULE OF THUMB:
├── Minimum 7 days (Meta's recommendation; Meta's maximum is 30)
├── Minimum 100 conversions per variant
├── Or minimum 1,000 clicks per variant (for CTR tests)
└── Statistical significance >90% reached

CALCULATOR INPUTS:
├── Baseline conversion rate: [X%]
├── Minimum detectable effect: [Y%]
├── Statistical significance: 95%
├── Power: 80%
└── Daily traffic/conversions: [Z]

Quick Reference Table

When to End a Test?

END IF:
├── 95%+ statistical significance reached
├── Both variants have >100 conversions
├── Test has run for minimum 7 days
└── No major external factors (holidays, etc.)

DO NOT END IF:
├── <90% significance (unless budget exhausted)
├── <50 conversions per variant
├── <7 days run time
└── During abnormal period (Black Friday, etc.)

Test Structure Templates

Single Variable Test Template

The thresholds in the success criteria are rules of thumb, not Meta figures: check them against the account's own history.

A/B TEST PLAN
=============

TEST INFO:
├── Test Name: [Descriptive name]
├── Hypothesis: [SMART hypothesis]
├── Start Date: [Date]
└── Expected End Date: [Date]

TEST DETAILS:
├── Type: [Creative / Audience / Placement]
├── Variable: [What are we testing]
├── Primary Metric: [CPA / CTR / ROAS]
└── Secondary Metrics: [CTR, Frequency, etc.]

VARIANTS:
├── Variant A (Control): [Description]
└── Variant B (Test): [Description]

BUDGET:
├── Total Test Budget: €[X]
├── Per Variant: €[X/2]
└── Daily Budget per Variant: €[X]

DURATION & SAMPLE SIZE:
├── Minimum Duration: [X] days
├── Target Conversions per Variant: [X]
└── Stop Criterion: 95% significance OR [date]

SUCCESS CRITERIA:
├── Winner if: [Metric] difference >10%
├── With: >95% statistical significance
└── Decision: [What do we do with the winner?]

Result Analysis Framework

Interpretation Guide

Rules of thumb, not Meta figures: check them against the account's own history. Meta reports a confidence percentage for the winner, which it describes as the chance of similar results if the test were repeated; that is not the same as a classical significance level, so treat the bands below as rough guides.

RESULT SCENARIOS:

SCENARIO 1: Clear winner (>95% sig, >15% difference)
├── Action: Implement winner
├── Learning: Document why it worked
└── Next: Test the next variable

SCENARIO 2: Marginal winner (90-95% sig, 5-15% difference)
├── Action: Consider collecting more data
├── Learning: Difference may be real but small
└── Next: Implement or retest with larger budget

SCENARIO 3: No winner (<90% sig)
├── Action: Make no changes
├── Learning: No difference detected at this sample size
└── Next: Test a larger/different change

SCENARIO 4: Surprising loser (your favorite version loses)
├── Action: Implement the winner (data > gut feeling!)
├── Learning: Document the difference
└── Next: Investigate WHY your expectation was wrong

Post-Test Documentation

TEST RESULTS REPORT
====================

TEST: [Name]
Period: [Start] - [End]

RESULTS (example values):

Metric          │ Variant A │ Variant B │ Difference
────────────────┼───────────┼───────────┼──────────
Impressions     │  50,000   │  50,000   │    0%
Clicks          │   2,500   │   3,000   │  +20%
CTR             │   5.0%    │   6.0%    │  +20%
Conversions     │    125    │    150    │  +20%
CPA             │  €20.00   │  €16.67   │  -17%
ROAS            │   2.5x    │   3.0x    │  +20%

STATISTICAL SIGNIFICANCE: [X%]

WINNER: Variant [A/B]

KEY LEARNINGS:
├── [Learning 1]
├── [Learning 2]
└── [Learning 3]

NEXT STEPS:
├── [Action 1]
├── [Action 2]
└── [Next test]

Testing Roadmap Template

Quarterly Testing Calendar

Q[X] TESTING ROADMAP
====================

MONTH 1:
├── Week 1-2: Creative Test - Hook variations
├── Week 3-4: Creative Test - Format (video vs static)
└── Apply learnings

MONTH 2:
├── Week 1-2: Audience Test - LAL vs Interest
├── Week 3-4: Copy Test - Long vs short
└── Apply learnings

MONTH 3:
├── Week 1-2: Offer Test - % discount vs € discount
├── Week 3-4: Winning combo test
└── Quarter review & planning

PRIORITY ORDER:
1. Hook/first impression
2. Offer/value proposition
3. Audience targeting
4. Copy elements
5. Technical optimizations

Example Interaction

User: "I want to test whether UGC-style videos work better than my current studio content. What's the best way to set this up?"

Claude response:

"Great test! Here's your A/B test plan:

---

A/B TEST PLAN: UGC vs Studio Content

HYPOTHESIS: 'If we use UGC-style videos instead of studio content, we expect CPA to decrease by 20%, because UGC feels more authentic and better fits social media behavior.'

TEST SETUP:

IMPORTANT: Keep constant:

• Same message/script direction
• Same offer/CTA
• Same audience
• Same budget split

BUDGET & DURATION:

• Budget per variant: about 100 × your current CPA [from this account's history]
• Minimum 14 days (Meta allows at most 30)
• Goal: 100+ conversions per variant

METRICS:

• Primary: CPA (Cost per Acquisition)
• Secondary: CTR, Hook Rate (3 sec views), ThruPlay Rate (Ads Manager only; not available through meta_get_insights)

SUCCESS CRITERIA:

• Winner at >95% significance
• Minimum 15% CPA difference
• Both >100 conversions

The budget, duration and success figures above are rules of thumb, not Meta figures: check them against the account's own history.

---

Would you like me to also write a UGC script you can test?"

MCP Tool Usage

Pull current creative performance before setting up a test:

# Get performance data to establish your baseline metrics
meta_get_insights(
    account_id="act_XXXXXXXXX",
    level="ad",
    date_preset="last_30d",
    fields=["impressions", "clicks", "ctr", "cpc", "spend", "actions", "cost_per_action_type"]
)

Use this to identify which creatives to test against and what your current CTR/CPA baseline is before writing your hypothesis. meta_get_insights returns one page of rows: if the response has paging.next, fetch the rest with meta_query(account_id, "ads", pagination_url=<paging.next>).