To conduct A/B testing for a marketing campaign, you write down one hypothesis, change one variable, pick one primary metric, calculate the sample size before launch, split the audience at random, and wait until the planned sample is in before you declare a winner. Everything else is detail. Most tests that mislead people break one of those rules, and it is usually the sample size or the waiting.
This playbook follows the order you actually work in, with a sample size example you can rerun in Python, the current names of the ad platform testing features, and a plan template to fill in before your next launch.
Start with a hypothesis you could lose
A usable hypothesis names the change, the audience, the metric and the expected direction: “Replacing the stock hero photo with a short product demo on the pricing page will raise trial signups among paid search visitors.” If you cannot say which number should move, you are not ready to test.
Change one variable per test. If version B has a new headline, a new button color and a shorter form, a win tells you nothing about which change mattered. Multivariate testing can separate the effects, but it needs far more traffic than most campaigns get.
Write down one primary metric before launch, plus one or two guardrail metrics that must not get worse, such as refund rate or cost per acquisition. The primary metric decides the test. A guardrail can veto it.
Work out sample size and duration before launch
Sample size depends on three inputs: your baseline conversion rate, the smallest lift worth detecting (the minimum detectable effect), and how much risk of error you accept. The usual defaults are a 5% significance level (a 1 in 20 chance of a false positive) and 80% power (an 80% chance of catching a real effect of that size).
Here is a worked example. A landing page converts at 4.0%, and you care about a lift to 4.8%, which is 20% in relative terms. The standard formula for comparing two proportions gives 10,317 visitors per variant, or about 20,600 in total. At 1,500 visitors a day that takes just under 14 days, so you schedule exactly two full weeks. The significance test you will run at the end is the two proportion z test described in the NIST Engineering Statistics Handbook.
# pip install scipy
from math import sqrt, ceil
from scipy.stats import norm
def visitors_per_variant(p1, p2, alpha=0.05, power=0.80):
z_a, z_b = norm.ppf(1 - alpha / 2), norm.ppf(power)
p_bar = (p1 + p2) / 2
top = z_a * sqrt(2 * p_bar * (1 - p_bar)) + z_b * sqrt(p1 * (1 - p1) + p2 * (1 - p2))
return ceil(top ** 2 / (p1 - p2) ** 2)
n = visitors_per_variant(0.040, 0.048)
print(f"Visitors per variant: {n:,}") # 10,317
print(f"Days at 1,500 visitors a day: {2 * n / 1500:.1f}") # 13.8
print(f"Smaller lift (4.4%): {visitors_per_variant(0.040, 0.044):,}") # 39,475Two lessons fall out of the math. Halving the lift you want to detect roughly quadruples the sample, which is why the 4.4% target needs about 39,500 visitors per variant. And small accounts should test bold changes (a new offer, a new layout) instead of button colors, because only large effects are detectable on small traffic. Run in whole weeks even if you hit the number early, because weekday and weekend visitors behave differently, and set a hard stop date.
Randomize cleanly and resist peeking
Randomize by person, not by pageview, so a returning visitor keeps seeing the same version. The platform tools below handle assignment, but check the split anyway. If you planned 50/50 and a large sample came back 53/47, something is broken (redirects, bots, caching). That is a sample ratio mismatch, and a test with one should be discarded, not interpreted.
The most common way to manufacture a fake winner is peeking: checking the dashboard daily and stopping the moment the result crosses significance. With a fixed sample design, every extra look pushes the real false positive rate well above 5%. Read the result once, at the end. If you truly need to stop early, use a tool built for sequential testing, which adjusts the math for repeated looks.
Where to run tests, channel by channel
Email subject lines
Most email platforms include a split test that sends two versions to a sample and the winner to everyone else. The catch is the metric. Apple Mail Privacy Protection, introduced with iOS 15 in 2021, preloads tracking pixels and inflates opens, and Mailchimp itself warns that open based A/B results may not be accurate. Judge subject lines on clicks or, better, on conversions.
Landing pages
Google Optimize and Optimize 360 stopped working on September 30, 2023, and GA4 has no native replacement. Google named AB Tasty, Optimizely and VWO as integration partners and published a specification so any testing tool can send variant data into GA4 with this event:
gtag('event', 'experience_impression', {
// Format: TOOL_ID-EXPERIENCE_ID-VARIANT_ID
exp_variant_string: 'ABC-F2948574-3495F49'
});Register exp_variant_string as an event scoped custom dimension under Custom definitions in GA4 Admin, then break conversions down by variant in an exploration. Google documents the format in its experiment integration guide. If you are new to custom events, our guide to tracking a custom event in Google Analytics covers the basics, and our walkthrough on installing Google Tag Manager on WordPress helps if your tags run through GTM.
Google Ads experiments
In Google Ads, open the Campaigns menu and choose Experiments. Custom experiments work with Search, Display and several other campaign types and test bidding strategy, match types, landing pages or audiences against the original campaign. Ad variations handle ad text tests, and there are separate Performance Max, Demand Gen and video experiment types.
For custom experiments Google recommends a 50% split. The cookie based split, which shows each user only one arm, is the recommended method; the search based split reassigns on every search. Results show a confidence interval at a default 80% confidence level, with a blue asterisk when a difference is statistically significant, and Google suggests running 2 to 3 weeks. Its guide to monitoring experiments explains the scorecard.
Meta A/B tests
Meta’s A/B test is available in Ads Manager and in the Experiments tool. You can duplicate a published campaign, ad set or ad and change one variable (creative, audience or placement), or compare two existing campaigns or ad sets. Meta makes sure nobody sees both versions, lets you schedule 1 to 30 days, shows an estimated power figure (it recommends at least 80%) and picks the winner on cost per result. It explicitly advises against testing by switching ad sets on and off by hand. Our guide to setting up a Facebook ad campaign covers the structure you will be duplicating.
Read the result: significance versus real lift
When the planned sample is in, run the test once. Suppose version A converted 412 of 10,400 visitors (3.96%) and version B converted 498 of 10,380 (4.80%):
# pip install statsmodels
from statsmodels.stats.proportion import proportions_ztest
z, p = proportions_ztest([498, 412], [10380, 10400]) # B first, then A
print(f"z = {z:.2f}, p = {p:.4f}") # z = 2.95, p = 0.0032A p value of 0.0032 is far below 0.05, so the difference is unlikely to be chance. That is statistical significance, which is not the same as a result worth shipping. Ask three more questions: is the lift big enough to pay for the change, does it hold in revenue per visitor, and did any guardrail get worse?
Resist slicing the result into a dozen segments after the fact; slice enough ways and one will look like a winner by luck. If a segment matters, make it the next hypothesis. To tie the winner to spend and profit, use the approach in our guide to measuring and analyzing marketing ROI.
A/B test plan template
Fill this in before launch and store it next to the results. Writing it down prevents most of the mistakes above.
| Field | What to write | Example |
|---|---|---|
| Hypothesis | Change, audience, expected direction | A product demo on the pricing page raises trial signups from paid search visitors |
| Variable | The single thing that differs | Hero image only |
| Primary metric | The one metric that decides | Trial signup rate |
| Guardrails | Metrics that must not get worse | Cost per trial, page load time |
| Baseline and MDE | Current rate and smallest lift worth detecting | 4.0% baseline, detect 4.8% |
| Sample and duration | Per variant sample, rounded up to full weeks | 10,317 per variant, 14 days |
| Split and unit | Ratio and what gets randomized | 50/50 by user |
| Tool and tracking | Where the test runs and how variants are recorded | Testing tool plus the GA4 experience_impression event |
| Decision rule | What you do with each outcome | Ship B if significant and cost per trial is flat or lower |
| Result and learning | Filled in after the stop date | What won, by how much, what to test next |
Troubleshooting
The traffic split is lopsided. A sample ratio mismatch usually comes from redirect tests where one URL loads slower, from bots, or from caching that serves one version to everyone. Fix the cause and restart rather than correcting the data afterward.
No winner after the full run. That is a result: the true effect is smaller than the lift you designed for. Keep the control or test a bolder change. Extending a test until something turns significant is just peeking with extra steps.
The winner stopped winning after rollout. Novelty fades, seasons change and lucky streaks regress toward the average. Retest important winners, or keep a small holdout on the old version for a few weeks.
The ad platform and GA4 disagree. They attribute and count conversions differently. Pick one source of truth for the primary metric before launch and record it in the plan.
Frequently asked questions
How long should an A/B test run?
Long enough to reach the sample size you calculated, and never less than one full week. Two full weeks is a sensible default for most sites because it covers weekday and weekend behavior twice. Google suggests 2 to 3 weeks for Google Ads experiments, and Meta lets you schedule tests from 1 to 30 days.
What sample size do I need for an A/B test?
There is no universal number. It depends on your baseline conversion rate and the smallest lift you care about. At a 4% baseline, detecting a lift to 4.8% needs about 10,300 visitors per variant at a 5% significance level and 80% power. Detecting half that lift needs roughly four times as many visitors.
Can I test more than one change at once?
You can, but you lose the ability to say which change caused the result. Meta allows several variables in one A/B test and warns about exactly that tradeoff. Multivariate designs solve it statistically but need much more traffic, so for most teams a series of single variable tests teaches more per dollar spent.
What replaced Google Optimize?
Nothing from Google directly. Optimize shut down on September 30, 2023. Google pointed users to integrations with AB Tasty, Optimizely and VWO and published an event specification so other testing tools can send experiment data into GA4. For paid media, the testing features inside Google Ads and Meta Ads Manager cover most needs without extra software.
The bottom line
Good A/B testing is mostly discipline: one variable, one primary metric, a sample size calculated in advance, clean randomization and a single read at the end. The Google and Meta tools handle the mechanics well, and a short written plan keeps you honest.
Treat every test as a step in a series. Winners become the new control, losers still teach you something, and the same method works for the follow up campaigns in our guide to remarketing techniques for better conversions.
