Plan an AI pilot: from use case to a defensible go/no-go
A good AI pilot is not simply a small AI project. It is a controlled experiment around a concrete business question.
The wrong starting question is:
"Which model or tool should we try?"
The better question is:
"Which assumption do we need to prove or disprove with the least effort before investing further?"
That turns innovation activity into a defensible decision process.
The short version: an AI pilot in 10 steps
- choose a concrete user and process,
- formulate the problem and value hypothesis,
- bound the scope tightly,
- identify real data sources,
- clarify privacy, permissions and security boundaries,
- sketch the minimum architecture,
- define real test cases,
- set measurable success criteria,
- test with real users,
- decide go/no-go from evidence.
Once these are clear, tool selection becomes much easier.
1. Start with a task, not "AI"
A use case is not:
- "company chatbot",
- "we need RAG",
- "Copilot for everyone",
- "introduce agents".
A useful use case describes a concrete task.
Too broad:
"Knowledge assistant for customer service."
Better:
"Service staff for product group X can find the current technical policy, receive a grounded answer and jump directly to the original source."
Now user, data, outcome and quality requirements are visible.
2. Make the value hypothesis explicit
A pilot needs a falsifiable assumption.
Example:
If service staff retrieve policies through a source-grounded RAG assistant, search effort drops while unsupported answers remain below a predefined threshold.
The hypothesis should clarify:
- What improves?
- For whom?
- In which process?
- What would count as measurable success?
- Which failures are unacceptable?
Without this, almost any impressive demo can be labelled "successful".
3. Bound scope aggressively
The fastest way to ruin a pilot is excessive scope.
Useful boundaries include:
- one department,
- one document corpus,
- one process,
- one language,
- one user group,
- read-only rather than write actions,
- no full automation,
- one target application.
Example:
Not "AI for HR", but:
"RAG over approved Austrian HR policies, read-only, with citations, for HR case workers."
That is testable.
4. Inspect the data first
Look at real data before building architecture.
Questions:
- Where does the required information live?
- Are there multiple versions?
- Who owns the content?
- How fresh is it?
- Which file formats exist?
- Which permissions apply?
- Is sensitive or personal data involved?
- How often do sources change?
Many pilots fail because the real data landscape only becomes visible after the technical build.
For RAG, see Connect RAG data sources.
5. Clarify permissions and risk early
A pilot should not rely only on artificial demo data if production will depend on real company information.
Clarify:
- which data is processed,
- who may access it,
- which systems participate,
- which vendors process data,
- whether content is replicated,
- how logs are handled,
- which approvals are required.
For RAG, permissions also need to survive retrieval. See RAG permissions & security trimming.
Technical guidance does not constitute legal advice.
6. Build the minimum architecture, not the final platform
A pilot does not need the full enterprise platform.
It needs the smallest architecture that tests the key hypothesis realistically.
Example RAG pilot:
source → ingestion → search/vector index → retrieval → LLM → small user interface
Not necessarily required yet:
- multi-region,
- full self-service platform,
- dozens of connectors,
- complex agent orchestration,
- large admin console.
But do not remove the critical risk from the test. If permissions will matter in production, include them in the pilot.
7. Define real test cases before the build
AI pilots become much stronger when evaluation starts before implementation.
Collect 20–50 real cases, depending on the domain.
For each case:
- input/question,
- expected result,
- expected source,
- critical failure,
- user role if relevant.
Example:
| Test question | Expected outcome | |---|---| | Which travel policy applies now? | current policy + citation | | Which approval is required above amount X? | correct process step | | Question with no supported answer | system abstains clearly | | User without HR access asks about HR data | no restricted retrieval |
This is more useful than "we tried a few prompts".
8. Set success criteria
Metrics depend on the task.
Business
- handling time,
- task completion,
- manual steps,
- adoption,
- follow-up questions.
Quality
- domain correctness,
- retrieval recall,
- groundedness,
- citation accuracy,
- failure classes.
Operations
- latency,
- cost per task,
- error rate,
- data freshness.
Security
- unauthorized retrieval,
- incorrect tool calls,
- permission violations.
You do not need perfect target values for every metric. But go/no-go needs explicit thresholds.
9. Test with real users
A pilot tested only by developers proves little about the actual workflow.
Real users reveal:
- different query phrasing,
- whether citations are understandable,
- whether the answer fits the process,
- where trust breaks,
- which failures actually matter.
A small representative cohort is more valuable than a broad rollout without measurement.
10. Decide go/no-go explicitly
The final question should not be:
"Did we like it?"
Use explicit outcomes.
Go
- value hypothesis supported,
- critical quality reachable,
- data and permission path viable,
- integration realistic,
- operating economics plausible.
Iterate
- value broadly supported,
- specific technical or organisational blockers are solvable,
- next hypothesis is clear.
No-go
- insufficient value,
- unsuitable data,
- critical quality cannot be reached,
- integration is disproportionate,
- risk or operating model is not viable.
A no-go can be a successful pilot when it prevents a larger bad investment.
Example: a four-week AI pilot
Not every pilot needs four weeks. As a planning frame:
Week 1 – scope & baseline
- use case,
- data,
- users,
- risk,
- evaluation set,
- baseline process.
Week 2 – minimum build
- core pipeline,
- data integration,
- UI/workflow,
- logging,
- first tests.
Week 3 – quality & integration
- analyse failure classes,
- improve retrieval/prompts/tools,
- test permissions,
- test with real users.
Week 4 – evidence & decision
- evaluate against baseline,
- cost/latency,
- failure map,
- production gaps,
- go/iterate/no-go.
Pilot vs PoC vs MVP
| Term | Primary question | |---|---| | PoC | Is it technically possible at all? | | Pilot | Does it work in a bounded real setting? | | MVP | Is the smallest usable product available for real use? | | Production readiness | Can it be operated reliably and responsibly? |
An AI pilot should be closer to the real process than a PoC while remaining deliberately bounded.
What does not belong in a good pilot
Common anti-patterns:
- choose the tool first and search for a problem later,
- test too many use cases,
- use only demo data,
- no real users,
- no baseline,
- no stop criteria,
- postpone governance entirely,
- automate everything immediately,
- call it successful because one answer looked impressive.
AI pilot canvas
For initial planning, twelve fields are enough:
- user
- task
- current process
- value hypothesis
- data sources
- permissions
- critical risks
- pilot scope
- architecture
- test cases
- success criteria
- go/no-go rule
A structured workshop can turn this into an actionable decision basis.
When a workshop is better than immediate build
If several use cases compete, or data, scope and success criteria are still unclear, a short structured workshop can save more time than a premature prototype.
My AI workshop from use case to pilot is designed for exactly that: prioritise, clarify data and risk, define the pilot.
If the target architecture itself is still open, use AI consulting & solution architecture.
When scope and architecture are clear and the pilot needs to be built, continue with RAG implementation & AI integration.
After the pilot: production readiness
A successful pilot is not a production release.
Before go-live, additional areas need attention:
- regression testing,
- monitoring,
- roles and ownership,
- security,
- permission boundaries,
- failure handling,
- load,
- cost,
- rollback,
- data freshness.
That is the role of AI Production Readiness.
You have several AI ideas but no defensible first pilot yet?
→ Prioritise use cases and define the pilot