Decide what the test can answer
Write the question before assigning leads: does the proposed follow-up process change an agreed outcome compared with the current process? If follow-up is what you change, the test estimates the effect of that process on the leads included. It does not prove that a campaign produces inherently better people or that the result applies to every future campaign.
Define one primary outcome, such as an attended appointment that meets an agreed qualification definition. Keep bookings, attendance and sales distinct. Choose the definition before seeing results so it cannot be changed to favour the outcome you prefer.
Create comparable groups
- Define which new leads are eligible, and exclude existing customers or duplicates using rules agreed in advance.
- Assign eligible leads randomly to the proposed process or a holdout that continues the existing process. Keep the same person in the same group if they enquire again.
- Balance important sources of variation, such as campaign, location, language and arrival period, when planning assignment. Do not let staff pick the most promising leads for one group.
- Keep the offer, qualification definition and other treatment consistent. Record unavoidable differences rather than hiding them.
A holdout should still receive the agreed normal service. It is not a group to ignore. Record crossovers, such as a holdout lead receiving the new workflow, and keep the original assignment in the main comparison to avoid selecting only convenient cases.
Set an observation window
Choose a window that fits the outcome and the business’s actual decision cycle. Give each lead the same time from entry to observation. If enrolment ends today, recent leads may still need time before they can be compared fairly with earlier leads.
Agree the end date and any early stopping rules before launch. Checking repeatedly and stopping at the first favourable result can create a misleading conclusion. Record major campaign or staffing changes during the test.
What to record
- A lead identifier, entry time, original group assignment and eligibility decision.
- Campaign or source, relevant comparison characteristics and duplicate links.
- Contact attempts, responses, handoffs, tool failures and process crossovers.
- The primary outcome, its timestamp and the evidence used to verify it.
- Missing records, complaints and unresolved outcomes, with reasons when known.
Restrict access to the records and collect only what the test needs. Use the same measurement method in both groups. Report counts and denominators, including how you handled missing outcomes, so a result can be checked.
Why small samples prove little
A difference based on a few outcomes can reflect chance or an unusual mix of leads. No universal lead count makes a test conclusive: the required sample depends on baseline outcome frequency, the change you want to detect and the uncertainty you can accept. Plan those with someone qualified to analyse the test.
Report uncertainty alongside the observed difference. An inconclusive result is not proof of no effect, and a positive result is not a guaranteed future return. Use tentative findings to design the next test rather than advertise invented success.
Using AceOS
AceOS handles calls and WhatsApp and schedules follow-ups through enabled workflows. Its improvement suggestions are accepted by a person; use your agreed holdout method to evaluate changes rather than treating a suggestion as evidence of an improvement.