Finding a first AI automation project
Choose a measurable AI pilot with a baseline, human review, evaluation criteria and a reversible rollout.
Published by DME with AI assistance. Examples are educational and identify their assumptions and limitations. Editorial approach and corrections.
A useful first AI project starts with a recurring task and a measurable decision. Consider a fictional service team that reads incoming requests, selects a queue, and drafts a response. The opportunity is specific: reduce preparation time while keeping routing and replies under human control. This guide shows how to choose that pilot, measure it, and decide whether to continue. The figures below are illustrative calculations, not DME client results.
Measure the existing workflow
Record a representative sample before building. For each request, capture input type, preparation time, final queue, whether information was missing, and corrections made by the reviewer. Include unusual cases and ambiguous wording; a sample of only tidy requests will underestimate operational risk. Remove customer identifiers from any development dataset and confirm that you are permitted to use the remaining material.
Suppose 200 requests take an average of eight minutes to prepare. That is 1,600 minutes of weekly work. A prototype that reduces preparation to three minutes saves 1,000 gross minutes. If review and correction add two minutes per request, the net saving is only 600 minutes, or ten hours. Measure the whole workflow, including verification, rather than counting model response speed as a business benefit.
Select the smallest reversible scope
| Question | Suitable first scope | Reason to pause |
|---|---|---|
| Can an error be corrected? | Draft a queue suggestion | Automatically issue refunds |
| Is there a baseline? | Logged handling time and reviewed labels | No agreed definition of success |
| Are inputs usable? | Approved documents with clear owners | Unresolved data permissions |
| Is there an accountable owner? | A reviewer can stop the pilot | No team owns incorrect outputs |
Avoid collapsing every task into a single score. A fast, high-volume workflow can still be unsuitable when mistakes are expensive or difficult to detect. Use rules for stable cases first. If a product identifier already determines a queue reliably, keep that deterministic path and reserve the model for ambiguous language.
Define an evaluation before prompting
Split examples into development and held-out evaluation sets. Adjust prompts against development cases, then evaluate the unchanged candidate against the held-out set. Report routing accuracy, unsafe suggestions, abstention rate, correction time, latency and cost per completed request. Compare against both the current workflow and a simple rules baseline. Treat a model's confidence statement as text, not a calibrated probability.
Write acceptance criteria before observing results. For this fictional pilot, the owner might require no unreviewed customer messages, no disclosure of restricted fields, a lower median handling time, and no increase in serious routing errors. Averages alone can hide poor performance for uncommon request types, so break results down by category and inspect the worst cases.
Run in shadow mode, then introduce review
- Generate suggestions without affecting live routing.
- Have reviewers compare suggestions with their independent decisions.
- Investigate disagreements and missing-context failures.
- Enable optional draft acceptance for a small group.
- Keep the original workflow available and document how to disable the prototype.
Untrusted request text must never grant permission to use a tool, change a destination, or reveal another customer's records. Keep authorization in application code. Log the prompt version, model configuration, retrieved document identifiers, and review outcome without retaining unnecessary personal data. Re-evaluate when the model, workflow or document collection changes.
Make the next decision from evidence
Continue when the pilot produces repeatable value and manageable errors. Narrow the scope when failures cluster around one input type. Stop when correction costs exceed savings or when reviewers cannot reliably detect bad outputs. The next step is usually better data, clearer boundaries or a better evaluation—not immediately a larger agent platform. For a practical implementation boundary, read Building a bounded Python agent; for document-grounded answers, read RAG architecture and evaluation.
Technical reference
NIST AI Risk Management Framework provides a broader framework for identifying and managing AI risks. The worked pilot and selection worksheet above are original illustrative material prepared for DME.