How to test a landing page headline without a developer: one change, one goal, one rule for when to stop

By Ranjit Bhinge, founder of Blur Studio · Updated · 11 min read

Short answer: To test a headline, show half your visitors the current headline and half a new one, count how many in each group take the same action (a call, a booking, a form), and decide using a sample size you set before you start. You can do it without a developer with a testing tool or by hand. First check that the page gets enough visits, because a small site will not reach a reliable result. Ploy, an AI platform that builds and hosts websites, can run a test on one section of a page, and it is one option among several.

Ranjit Bhinge

You are paying for clicks and cannot tell whether the headline is why people leave. Here is how to test it properly, or decide it is not worth testing yet.

I'm Ranjit, and I run Blur Studio: 15+ years building software, 120 apps taken live, and clients who have gone on to raise $10M+. I build AI systems and processes that help businesses do a lot more with a lot less time. I also work differently from most of what else is out there, more on that further down this page. Say hello.

The short answer: change only the headline, pick one action that counts as a win, work out how many visits you need before you start, run both versions at the same time, and do not stop early because one looks ahead. If the page gets too few visits to reach that number in a few months, do not run a formal test. Use your judgment, improve the obvious problems, and compare before and after with the limits in mind.

This guide is for owners and agencies who can edit a page but do not have a developer. It covers writing a testable headline, the traffic math, the options for running the test with prices vendors show in November 2026, the guardrail that protects bookings, and rules for calling the result. It complements how to test two versions of a landing page, which covers the Netlify case and manual before-and-after tests, and why ad clicks do not convert.

This page may contain affiliate links from which Blur Studio earns commissions.

Before you test: is the headline the problem?

A headline test answers one question: does version B get more people to take the action than version A. It cannot fix a page where the offer is weak, the form is long, the phone number is hidden on mobile or the page loads slowly. Check these first, because they usually matter more:

  • Does the headline match the ad or link that brought the visitor? A visitor who clicked "same-day boiler repair" and lands on "Quality service you can trust" leaves. See ad landing pages that match the ad.
  • Does the page work on a phone, with a visible call or book button above the fold (the part you see without scrolling)?
  • Do you track the action? If you cannot count calls, bookings or form submissions by page, fix that before testing anything.

If those are in order and people still leave, the headline is a fair thing to test.

Write a headline that is worth testing

Change one thing. Rewriting the headline, image and button at once tells you the bundle won, not why. Use a hypothesis in this form: "We think that [change] will increase [action] because [reason], and we will know when [count] visitors have seen each version." Pick a version B that is truly different, not a word swap. Four types of difference worth testing, with made-up examples for a cleaning company:

TypeVersion AVersion B
Outcome firstProfessional home cleaningCome home to a clean house, booked in 2 minutes
Specific offerCleaning services in LeedsFixed-price deep clean from a stated price, same-week slots
QuestionWelcome to Bright HomesTired of doing the weekend clean?
Match to the adOur servicesThe exact words of the ad that sent the click

Do not state results you cannot support: a headline claiming "5-star rated" or "the cheapest" must be true and provable. Keep your phone number, form and booking link identical in both versions, so the only difference is the words.

Do you have enough visits? The part most guides skip

A/B tests need a sample size fixed in advance. Evan Miller, who wrote a widely used set of testing calculators, publishes a free sample size calculator. It asks for the baseline conversion rate (the share of visitors who take the action now), the smallest lift you care about, power (how often the test spots a real lift) and the significance level (how often it calls a win when there is none). As of November 2026, the page opens with a 20% baseline with a range of 15% to 25%, and shows 1,030 visitors per version at 80% power and a 5% significance level. Calculators use slightly different maths. The standard two-proportion formula gives about 1,094 per version for 20% to 25%. That second figure is derived, not Miller's.

Small lifts need far more visitors. These figures are derived with the standard two-proportion formula at 95% confidence (5% significance level) and 80% power. They are my arithmetic, not a vendor figure, and a calculator may give slightly different numbers:

Baseline to the lift you want to detectVisitors needed per version (derived)Both versions together
2% to 3%3,8257,650
3% to 5%1,5063,012
4% to 6%1,8633,726
5% to 6%8,15816,316
5% to 7.5%1,4702,940
10% to 12%3,8417,682

A made-up example: your landing page gets 700 visits a month and converts at 5%. To detect a jump to 7.5% you need about 2,940 visits in all, which is about four months. To detect a jump to 6% you need about 16,316 visits, which is about 23 months. That is too slow to be a useful test. Use the calculator with your own numbers before you build anything.

What to do with low traffic:

  • Test a bigger change. Big differences need fewer visitors to show. A small tweak to a word does not.
  • Test where the traffic is. Your homepage or main ad page, not a page with 40 visits a month.
  • Drive more traffic first, or accept a judgment call. Then watch the results over time and be honest that it is weaker evidence.

The rules of a fair test

  1. Fix the sample size first. Miller's "How Not To Run an A/B Test" explains that the maths assumes the sample size was fixed in advance. In his worst-case example (a 50% baseline, a change that does nothing, and a significance check after every visitor, up to 150), stopping as soon as the result looked significant produced a false "significant" result 26.1% of the time, against the 5% you think you are getting.
  2. Do not peek and stop. Looking is fine. Stopping because B is ahead is not.
  3. Run whole business cycles. Optimizely, a testing tool vendor, says a test should cover at least one full cycle, typically seven days. Convert, another vendor, says one to two full business cycles. Sources differ, so use your own cycle: a service business with weekend slumps needs whole weeks.
  4. Run both versions at the same time. Time-based changes (a holiday, a promotion, a competitor's sale) confuse a version-A-then-version-B comparison.
  5. One goal, plus a guardrail. The goal is the action you want more of. The guardrail is something that must not get worse, such as bookings, calls or revenue. If B raises clicks on a button but cuts bookings, B loses.
  6. Do not change the ads in the middle. A new audience mid-test ruins the comparison.
  7. Confidence is not certainty. A 95% confidence level means a 5% chance of a false positive, per Convert. Optimizely usually runs at 90%. Either way a "win" can still be wrong.

If you split traffic across different web addresses, Google's guidance on website testing says not to show Googlebot (its crawler) a different set of URLs from people (that is cloaking), to use rel="canonical" (a tag that points to the preferred version of a page) on alternate URLs, to use a 302 (temporary) redirect rather than a 301, and to run the test only as long as necessary and then remove the test elements. A test that swaps one section on the same address avoids most of that.

Your options, from least to most effort

Prices are as vendors show them in November 2026. Check each vendor's page, because plans and billing terms change.

RoutePrice shownFits whenWatch for
No tool: change one thing, wait two weeks, compare enquiriesFree (your time)Low traffic, a rough direction is enoughNot a true test. Seasonality and ad changes can fool you
Unbounce, a landing page builderPer month, USD: Starter $22, Build $74, Experiment $112, Optimize $187 when billed yearly; $29, $99, $149 and $249 when billed monthly. Agency starts at $349 a month, billed annuallyYou want dedicated A/B testing. Unbounce's Experiment plan lists unlimited A/B tests and confidence intervalsPlans have visitor limits (Build lists up to 20k, Experiment 30k, Optimize 50k). Pages live in Unbounce unless you publish to your own domain
Framer, a design-led website builderFree $0; Basic $10 a month and Pro $30 a month billed yearly ($15 and $45 billed monthly); extra editors $20 a month eachYour site is already on FramerFramer lists A/B tests under an add-on called Convert. Framer lists it at $50 per 500,000 analytics events
GrowthBook, an experimentation toolStarter free for up to 3 users; Pro $40 per seat a monthYou have a developer to set it upBuilt for developers, so it may not fit "without a developer"
Optimizely or VWO, two enterprise testing toolsNot publishedLarger teams with sales-led buyingYou pay by quote. Google Optimize, once the free option, ended on 30 September 2023
Ploy, one option of severalFree $0, Starter $50 a month, Pro $300 a month, Enterprise by quoteYour site is built on Ploy and you want a section testExperiments is on request, not on by default. See below

Where Ploy fits

Ploy's docs describe Experiments as an A/B test of one section of one page, such as a hero (the top block of a page) or a pricing table. You choose a goal event, describe the new version in a chat with Ploy's Agent (its built-in AI assistant), and send between 10% and 90% of visitors to it. The current section is called the champion and the new one the challenger. The form takes one guardrail and the Agent can set up to three. You then promote one version and publish the site.

What to know before you count on it:

  • It is available on request. Per the docs, you email Ploy to turn it on for your workspace. It is not listed as part of every plan.
  • Your plan caps how many experiments run at once. The number is not published.
  • Requirements: the site must be published at least once, Ploy's analytics must be on, and the goal and guardrail must be events your site already tracks.
  • Visitors who refuse cookies through your cookie banner always see the champion and are not counted.
  • The statistics method, minimum sample size and significance level are not published. Ploy shows a "Guardrail at risk" warning when the challenger is very likely doing worse on a guardrail, and a warning if you promote early. It does not decide for you. Use the sample size you set above, not the dashboard's mood.
  • It tests one section of one page, not multi-page journeys.

As of November 2026, Ploy's pricing page shows Free at $0 (500 AI credits a day up to 2,000), Starter at $50 a month (4,000 credits) and Pro at $300 a month (24,000 credits), with credits (Ploy's unit for AI work) that do not roll over. See Ploy pricing, Ploy vs Unbounce and Ploy vs Framer. Ploy's own pages say Unbounce has A/B testing inside its builder, and for a dedicated ad-page testing job Unbounce is a better-known pick.

A worked example with made-up numbers

The business and numbers are invented to show the method. "Bright Homes" runs ads to one landing page. It gets 3,000 visits a month, with a 4% form conversion rate (120 forms).

  1. Hypothesis. Replacing "Professional home cleaning" with "Come home to a clean house, booked in 2 minutes" will lift form submissions because it names the outcome and the effort.
  2. Goal and guardrail. Goal: form submitted. Guardrail: completed bookings, checked weekly.
  3. Sample size. The owner decides the smallest lift worth acting on is 4% to 6%. The standard formula (derived, 95% confidence, 80% power) gives about 1,863 per version, so about 3,726 visits in all, and she checks it in a calculator. She writes the number down. At 3,000 visits a month that is a little over a month, so she runs it for six whole weeks to cover full weekly cycles.
  4. Setup. A 50/50 split, both versions live together. She checks on a phone that both versions show the same phone number and that the form event fires for each.
  5. Run. She looks weekly only to check nothing is broken. She does not stop early.
  6. Decide. At 1,863 or more per version and six whole weeks, B has more forms and bookings did not fall. She promotes B, publishes, and logs the date, the numbers and what she learned. If the result were unclear, she would keep A and test a bolder change next.

Test plan with pass and fail rules

CheckPassFail
Tracking before launchGoal and guardrail events fire on both versions in a test submissionAny event missing: do not start
Sample size written downA number per version, set before launch"We will see how it goes"
Both versions on a phoneSame phone number, form and booking linkDifferent contact details or a broken button
Mid-test health checkTraffic split roughly as set, no outageSplit skewed or page down: fix and restart
DecisionSample reached, whole business cycles, guardrail heldStopped early, or guardrail fell
After shippingWinner published, result and date loggedOld test elements left running

Who should skip this, and who should skip Ploy

  • Your page gets fewer than a few hundred visits a month. Improve the obvious issues and compare before and after, with the limits in mind.
  • You cannot track the action. Set up tracking first. See your first month of ads: landing page, tracking and budget.
  • You need to test many pages or whole journeys. Use a dedicated tool such as Unbounce, or a developer.
  • Skip Ploy if your site is not on it. Moving a site just to test one headline costs more than it saves.

How I'd approach this

I would not start with the tool. I would first make sure the headline matches the ad, the phone number works on mobile and the action is tracked. Then I would write the hypothesis and the sample size on a page before touching anything. If the sample takes more than a few months, I would drop the formal test, change the headline to the best guess, and watch weekly numbers while being clear it is weaker evidence.

If the page is on a platform with built-in testing, use that. If you want to see a section test on a page built in Ploy, email Ploy to turn Experiments on and ask how many experiments your plan allows. If you need custom tracking, or a test tied to your own booking or order system, Blur Studio builds that with CRM and calendar connections and ongoing care. For a related method, see the best landing page builders for ads.

First-week plan. Day 1: check message match, mobile and tracking. Day 2: write the hypothesis, goal and guardrail. Day 3: run the sample size calculator and write the number. Day 4: write version B and set up the test. Day 5: test both versions on a phone and launch. Then wait. Check weekly for faults only, and decide only when the sample is reached.

For a closer look, see trying a new version of a weak ad landing page.

Related guide: running a website test without losing bookings.

Questions people ask

How long should a headline test run?

Until you reach the sample size you set in advance, and for whole business cycles. Sources differ: Optimizely says at least one full cycle, typically seven days, and Convert says one to two full business cycles. There is no single number of days that fits every page.

How many visitors do I need?

It depends on your baseline conversion rate and the smallest lift you care about. Miller's calculator opens with a 20% baseline and shows 1,030 per version for a range of 15% to 25%. Lower baselines and smaller lifts need more: by the standard formula (derived), 5% to 6% needs about 8,158 per version while 5% to 7.5% needs about 1,470. Run the calculator with your own numbers first.

Can I test a headline for free?

You can change one thing, wait, and compare, but that is not a true test because time-based changes can fool you. Google Optimize, once the free tool, ended on 30 September 2023. GrowthBook has a free Starter plan but is built for developers. See how to test two versions of a landing page.

Does Ploy run A/B tests?

Ploy's docs describe Experiments for one section of one page, with a 10% to 90% split, a goal and guardrails. It is available on request, not on by default, and its statistics method and per-plan experiment caps are not published. See what Ploy AI is.

Why not stop the test as soon as one version is ahead?

Because early leads are often noise. In Evan Miller's worst-case example, checking after every visitor and stopping at the first significant result gave a false win 26.1% of the time against the 5% people expect. Decide the sample size first and wait for it.

Will a headline test hurt my Google ranking?

Not if it is done as Google advises: do not show crawlers different URLs from people, use rel="canonical" on alternate URLs, use a 302 redirect rather than a 301, and remove the test when it ends. A same-address section test avoids most of this.

What if the test result is inconclusive?

Keep the current version and test a bolder change next, since small changes need very large samples to show. An inconclusive result is still information: this change did not matter much.

Do I need a guardrail?

Yes. A headline that gets more clicks but fewer bookings is a loss. Choose a guardrail such as bookings or calls and check it weekly. Ploy lets you set one in the form and up to three through its Agent.

More in this topic: Websites: testing and proving page changes (4 guides) →

Sources

Ready for the next step?

A 30 minute call. If you do not need me, I will tell you.

Ranjit Bhinge You talk to me, Ranjit. I reply within a few hours.

Not sure what you need? Tell me what is going on. I will point you the right way, even if it is not me.

Blur Studio · 30 minute call

Book a 30 minute call

You talk to me, Ranjit. If you do not need me, I will tell you.

  • We talk about your business and what is going on.
  • You leave with a clear next step, even if it is not me.
  • I reply to every message within a few hours.
Loading the calendar…