How do I run a 30-day AI pilot and know if it worked?
Short answer: Pick one repeat process, time 30 real items by hand, then run AI on the same kind of work for 30 days with a person approving every output. At the end, compare time per item, how often the AI output needed changes, and whether any serious mistake got through. Decide in advance what counts as pass, fix or stop, and write it on a one-page scorecard before you start, so the result is a number and not a feeling.
You want to try AI on one process before you commit, and you want to know on day 30 whether it earned its place.
I'm Ranjit, and I run Blur Studio: 15+ years building software, 120 apps taken live, and clients who have gone on to raise $10M+. I build AI systems and processes that help businesses do a lot more with a lot less time. I also work differently from most of what else is out there, more on that further down this page. Say hello.
If you want help choosing the best process to test, the AI Strategy Sprint finds it in 3 days.
What a pilot is, and is not
A pilot is a small, time-boxed test of one process, run with real work, with a stop date and a pass mark. It is not a rollout, and it is not a general play with chatbots. The aim is to find out cheaply whether the idea works before you pay for a proper build.
The US National Institute of Standards and Technology (NIST, a federal standards body) publishes an AI Risk Management Framework built around four functions: govern, map, measure and manage. You do not need the whole framework for a small pilot, but its order is a good one: decide who is responsible, describe what the system does and where it could go wrong, measure it, then act on what you find. The plan below follows that order in plain terms.
Pick the right process
A good pilot process has five traits.
- It repeats. At least 30 items in the 30 days. Fewer than that and you cannot tell luck from skill.
- It has a clear right answer a person can check in under a minute: a drafted reply, a sorted inbox, a summary of a call, a filled-in form.
- A mistake is cheap and catchable. A person approves the output before it reaches a customer.
- One person owns it and can say what good looks like.
- The data is safe to use. Before you paste customer or staff information into any tool, check the plan's data terms. My guide on putting company data into ChatGPT gives a simple rule sheet.
Bad first pilots: anything that spends money, sends legal or medical advice, or makes a decision about a person without a human check. Start where being wrong costs you five minutes. OWASP, a nonprofit that publishes security guidance, advises requiring a person to approve high-impact actions of AI systems, and the EU AI Act (Article 14) warns overseers about automation bias, the habit of trusting output too easily. That is why this pilot keeps a person approving and makes the approval a real read. For personal data in general, the US Federal Trade Commission's guide for business says to keep only what you need, so leave names out of the pilot where the task allows.
The 30-day plan, week by week
Days 1 to 3: set up
Write down the process in 5 to 10 steps, name the owner, and name the person who approves the output (they can be the same person). Pick the tool. A general chatbot such as ChatGPT (OpenAI's chat tool) or Claude (Anthropic's chat tool), or a tool you already pay for, is enough for most pilots. Do not build anything yet. If you are unsure what to try, see AI for small business.
Week 1: measure the baseline
Time how long the work takes today. Do this for 30 real items, or the last two weeks if you see fewer than 30. For each item record minutes taken, whether it had to be redone, and anything that went wrong. A phone stopwatch is fine. This is the step people skip, and without it you cannot prove anything later. You also learn which items are odd, which becomes your exception list.
Week 2: run it in the background
Have the AI produce an output for every item, but do not send or use it. Compare it with what the person did. Mark each as: good as is, good with small edits, wrong. Fix the instructions you give the AI and try again on the misses. This week is for learning where it fails, so do not judge the result yet. Think of it as a trial run with no customers involved.
Week 3: live, with a person approving every output
Now use the AI output for real work, but a named person reads and approves every item before it goes out. Time the approval. Log each edit and each rejection with a one-line reason. Keep every item in one sheet so the numbers are easy to total.
Week 4: keep going, then score it
Run the same way for another week without changing the instructions, so you measure a stable setup. On day 28 to 30, fill in the scorecard and decide. Resist tweaking in week 4, because every change resets what you are measuring.
How big a sample do you need
For a small business, 30 items for the baseline and at least 30 live items in weeks 3 and 4 is a workable minimum. That is enough to see big problems, such as one output in three needing a rewrite. It is not enough to prove a 2 percent error rate is really 2 percent: one error in 30 items is 3 percent, and a second one doubles it. So if a mistake would be serious, do not rely on a pilot of this size to prove the rate is low. Use review rules and a longer run, as in my guide to the human in the loop workflow.
The one-page scorecard
Write the pass mark before you begin. These are my own rules of thumb, not published standards. Change them to fit your business, but fix them in writing first.
| Measure | Baseline | Pilot | Pass | Fix | Stop |
|---|---|---|---|---|---|
| Minutes per item, including approval | Your week 1 median | Your week 3 and 4 median | 30 percent or more lower | 10 to 30 percent lower | Under 10 percent lower, or slower |
| Items approved with no edit | Not applicable | Share of live items | 70 percent or more | 40 to 70 percent | Under 40 percent |
| Serious mistakes caught in review | Count from week 1 | Count | No more than baseline | A few, with a clear cause | Repeating with no clear cause |
| Serious mistakes that reached a customer | Count | Count | Zero | One, with a fix in place | Two or more |
| Staff view | Not applicable | Would the approver keep using it | Yes | Yes, with changes | No |
Decide the rule for the overall result too: pass means every row is in the pass column or one is in fix. Stop means any row is in the stop column. Anything else is fix: change one thing, run another two weeks, score again.
Stop rules you set in advance
- Stop the pilot at once if an output with private or wrong information reaches a customer. Find out why before you restart.
- Stop if approving takes longer than doing the item by hand in two weeks in a row.
- Stop if the data terms of the tool turn out not to fit your client contracts.
- Stop if the owner cannot find 30 minutes a week for the log. A pilot nobody checks proves nothing.
Where a pilot ends and a build begins
A pilot usually runs on copy and paste, with a person moving the work between tools. That is fine for 30 days and does not scale. If the scorecard passes, the next step is to connect the tool to your email, your records or your phone system so the work flows without copying, and to add the checks and logging that a real workflow needs. That is a build, and it should be priced against what you measured. Turn your pilot numbers into a payback figure with my guide on calculating the ROI of AI automation. See also AI implementation, AI workflow automation for small business and business process automation.
If you are unsure which process deserves the pilot, the AI Strategy Sprint is a paid 3-day first step that looks across your business and writes down exactly where to start.
Common ways pilots fail
- No baseline. The AI "feels faster" and nobody can prove it.
- Too many processes. Three pilots at once means three half-measured ones.
- Changing instructions every day. You never learn what works.
- Judging the first week. The first outputs are the worst. That is what week 2 is for.
- Counting subscriptions you already had. Check your AI tool subscriptions first so you do not pay twice.
- Nobody owns it. Give the pilot one named owner and one named approver.
How I'd approach this
I would pick the most boring repeat task in the office, not the most exciting one. I would time 30 real items, write the pass, fix and stop numbers on one page, and give the whole thing one owner. I would keep a person approving every output for the whole 30 days, even if it looks perfect in week 2. Then I would be willing to stop. A pilot that ends with "no, not for this process" has done its job: it cost you a month of light effort instead of a build.
Questions people ask
How long should an AI pilot last?
Thirty days is enough for one repeat process with at least 30 items. If the process runs only a few times a week, extend it until you reach 30 live items, or choose a process with more volume.
What does an AI pilot cost?
Mostly time: the owner's hours for the baseline, the log and the scorecard, plus any tool subscription. Use a plan you already pay for where you can. Price the staff time as well, using the method in my guide on the ROI of AI automation.
Do I need a developer for a pilot?
Usually not. A chat tool and a spreadsheet cover most pilots. You need a developer when the pilot passes and you want the tool connected to your systems, or when the process needs automation from day one.
What should I do with customer data during a pilot?
Check the tool's data terms before pasting anything. Consumer and business plans differ on training, and some tools keep conversations for a time. See is it safe to put company data into ChatGPT for a simple rule sheet.
What if the pilot lands in the middle?
That is the fix result. Change one thing, such as the instructions, the type of item, or the checking step, and run two more weeks. If it still lands in the middle, narrow the scope to the items where it works well.
Can I run the pilot without a person approving each output?
Not in a first pilot. The approval step is how you measure quality. You can loosen it later, as described in my guide to the human in the loop workflow.
More in this topic: Problem solving: AI strategy and adoption (24 guides) →