A small, careful trial can reveal where a tool helps, what still needs checking, and whether the finished work is actually better.
By Arbab Naseebullah Kasi, Chief Executive Officer - CEO, Feel Worldwide Foundation Inc.
A painting of a vermilion mechanical bird and its natural bird-shaped shadow, a visual reflection on the difference between an impressive appearance and a tested result.
An AI demonstration can be wonderfully persuasive. A page of rough notes becomes an orderly summary. A difficult paragraph becomes easier to read. Several possible approaches appear before you have finished thinking of the first one.
It is reasonable to feel curious about that. For a small organisation, an educator, or someone managing a business alone, help with repetitive work can be appealing. There is no virtue in struggling through a task simply because it has always taken a long time.
The useful question comes a little later, when the demonstration meets the actual work. Did the summary preserve the exception? Did the clearer paragraph change the meaning? How much time did someone spend finding and correcting mistakes?
Before making a tool part of a routine, give it a modest practice test. Choose a task you understand, decide what a satisfactory result must contain, and examine the finished work closely. That first test can help you make a more informed decision about where the tool belongs.
Choose a task you can judge
“Help us use AI” is too broad to test. “Turn this public event description into a short draft announcement without adding any facts” gives you something concrete to assess.
Choose work with a clear boundary and a consequence you can safely manage. Keep the output private during the trial. Use material that is public or that you are authorised to use in the particular service, without entering confidential records or personal details for convenience.
A useful starting task should also be one you can evaluate without relying on the tool itself. If you cannot tell whether an answer is correct, its confident presentation may leave you with very little evidence of quality.
For an illustrative example, imagine a community organisation testing whether AI can shorten its public workshop announcements. The original descriptions already contain the approved dates, locations, costs, and attendance instructions. A staff member knows what must remain accurate. Nothing is sent automatically.
That is a manageable trial. Asking the same tool to decide who qualifies for a sensitive service would involve a very different level of responsibility. Success with announcements would tell the organisation almost nothing about whether that second use is appropriate.
Expect ability to vary across tasks
A well-known experiment makes this distinction tangible. Researchers working with Boston Consulting Group studied 758 consultants using a 2023 version of GPT-4. For a set of 18 consulting tasks chosen to be within the tool's capabilities, participants with AI completed more tasks and worked more quickly on average. On a separate managerial problem chosen to fall outside those capabilities, the combined AI groups were about 19 percentage points less likely to give the correct answer than the group without AI. The paper was published in Organization Science in March 2026. These were particular tasks, participants, and an older model, rather than a verdict on every current AI product. Read the original research.
The practical lesson I draw is to test each proposed use on its own terms. A tool that helps draft an invitation may still need close supervision when extracting eligibility conditions or interpreting an ambiguous document.
Start with the work you need done, rather than a general impression that a system is impressive.
Write down what a good answer must preserve
Before generating anything, make a short checklist. What would make the result usable? What would make it unacceptable even if the writing sounded polished?
In the illustrative workshop trial, the organisation might require the draft to preserve every date and time, retain the correct location, distinguish a free session from a paid one, and keep registration instructions intact. It might also require the tool to flag missing information rather than supply a plausible replacement.
These are suggested criteria for this example, not a universal testing standard. A different task needs different checks. A translation needs a reviewer who understands both languages and the subject. A spreadsheet formula needs to be checked against known inputs and expected results. A summary needs to preserve important qualifications as well as the headline message.
The US National Institute of Standards and Technology's July 2024 generative-AI profile recommends evaluating performance in conditions similar to the intended setting. It also cautions against generalising from narrow or anecdotal assessments and recommends checking sources and citations in generated outputs. The profile offers voluntary risk-management guidance, not a guarantee that a tool is suitable. See NIST's guidance.
A small organisation does not need to pretend that a short internal trial is a laboratory study. It does need to be clear about what it checked and what remains unknown.
Include the awkward examples
An easy demonstration may conceal the situations that create the most extra work.
For the illustrative announcement trial, include one description with two different dates, another with an exception to the usual opening time, and another where the venue has not yet been confirmed. Add a description that is already concise, so the tool has little reason to rewrite it extensively.
Prepare the expected facts before inspecting the outputs. Otherwise, it becomes tempting to accept the answer that happens to appear.
Pay special attention to missing information. A useful draft can say that a detail needs confirmation. It should not quietly turn an unknown venue into the organisation's usual address or transform an approximate duration into an exact finishing time.
Keep a few examples aside while improving the instructions. After making changes, try those untouched examples too. That can help reveal whether the instructions work beyond the particular cases you have been repeatedly adjusting.
None of this proves reliability across every future situation. A short trial can uncover a problem and help narrow a use case. It cannot establish that a rare but consequential error will never occur.
Count the work after the first answer
The first draft is only part of the job. Someone may still need to compare it with the source, check links, restore an omitted condition, correct the formatting, and obtain approval.
Time the whole task to the same standard of completion. Compare that with the ordinary way of doing the work, using examples of similar difficulty. Avoid always doing the AI version second, after the first attempt has already taught you what the answer should contain.
Consider an illustrative calculation. Preparing an announcement manually takes 20 minutes. Using AI takes four minutes of preparation, one minute to generate a draft, and eleven minutes to review and correct it. The total is 16 minutes, a saving of four minutes for that example. Calling it a one-minute job would hide most of the effort. Those numbers are invented solely to show the calculation, not to predict anyone's results.
Research also gives a reason to measure rather than rely entirely on a feeling of speed. In a small randomized study reported in July 2025, METR examined 16 experienced open-source developers completing 246 tasks in familiar repositories. Tasks with AI access took 19 percent longer on average, although participants believed AI had sped them up. The researchers explicitly limited the finding to that setting and period. Read METR's original study report.
The later evidence matters too. In February 2026, METR reported that a follow-on experiment had substantial selection and measurement problems. The researchers thought newer tools were probably helping these developers more, but said the new data did not reliably establish the current effect. It would be misleading to present the earlier slowdown as a permanent fact about AI. Read the follow-on update.
For your own trial, record both the time and the quality of the final result. A small time saving accompanied by a serious factual mistake is not a satisfactory improvement. A slower process might still be worthwhile if it produces a demonstrably better result, but that is a different benefit and should be described honestly.
Give the reviewer a real job
“A person will check it” sounds reassuring, but the arrangement needs substance. Who is that person? What must they check? Can they stop the output from being used? Do they have the relevant knowledge and enough time?
In the illustrative announcement trial, the reviewer could compare dates, fees, locations, and registration requirements directly with the approved description. They would then read the final draft for clarity. If the original description were contradictory, they would return to the person responsible for it rather than ask the AI to decide which version was true.
The reviewer should also be able to reject the draft and use the ordinary method. A trial becomes less informative when everyone feels obliged to demonstrate that the new tool was worth adopting.
Record the recurring errors in plain language. “Sometimes drops the registration deadline” is more useful than “needs improvement.” It tells the team what happened and helps them decide whether clearer instructions, a narrower task, or abandoning that use is the sensible next step.
Make a decision that fits the evidence
At the end of the trial, there are several reasonable outcomes.
You might keep using the tool for the tested task, with the same review. You might restrict it to producing alternative wording while retaining the original factual text. You might improve the instructions and run another trial. Or you might decide that the checking effort outweighs the benefit.
Keep a brief record of the tool and version where available, the date, the task, the examples used, and the problems found. If the model, instructions, source material, or intended use changes, revisit the test. A satisfactory result under one set of conditions should not become a permanent permission slip for unrelated work.
The value of a practice test is that it gives curiosity something solid to work with. You can welcome useful technology while asking it to demonstrate a useful role.
Begin with one task whose quality you can recognise. Follow it all the way to a checked, usable result. Then decide what deserves to become part of your working day.
