Investigate the gaps.
Map the data path, model calls, permissions and dependencies. Agree the users, operating conditions and criteria the service must meet.
Prototype to production
The demo works. Build the service around it.
Move an AI prototype into use, or review a system already running. Test the workflow, address the important control gaps and agree how the service will be owned, monitored and changed.
Talk to an engineer Start with a scoped reviewEngineering focus
01Assess the prototype
02Close the critical gaps
03Validate and deploy
Possible applicationUK public sector
Test a staff policy-search prototype against current documents, access rules and representative enquiries. Record the gaps and agree the work needed before release. Explore this sector.
Not everything needs rebuilding.
Examine what already works and where the evidence is missing. The next useful change may be an access check, a stronger interface or better tests. Keep the existing architecture where it meets the requirement.
Engineer the foundations.
Fix the critical failure modes, automate deployment and add appropriate controls. Make failures and changes in behaviour visible to the people operating the service.
Test the operating model.
Validate the workflow with representative users and test recovery. Agree ownership, documentation and how the service will be supported.
A prototype demonstrates an idea, not all the requirements of a production service. Scope the missing work before promising a release date.
AI evaluation & change assurance
Test the task. Not just the answer.
Measure completed tasks, answer quality, tool use, response time and cost. Repeat the checks when models, instructions, data or integrations change, then compare the evidence before release.
Build representative cases.
Create repeatable tests around the task and its risks. Agree success criteria and suitable test data, including missing information, denied access, ambiguity and adversarial inputs.
Measure the system.
Combine automated checks with human review where judgement is needed. Record model and runtime versions, instructions, data and tool configuration so results can be compared and investigated.
Govern changes.
Agree release criteria, review responsibilities and what to do when a check fails. Repeat the relevant tests after changes and investigate regressions in production.
AI operations & governance
Built to run. And to change.
Make ownership clear once AI is in use: who monitors it, who can change it and who responds when something goes wrong. Define the review, approval and support arrangements alongside the technical controls.
Watch the whole request.
Trace retrieval, model and tool activity. Monitor quality, latency and cost without collecting unnecessary sensitive content.
Prepare for interruption.
Define retry, rollback or recovery procedures around the actions the system can take. Account for changes to external systems, keep runbooks current and test the important paths.
Improve deliberately.
Review incidents, usage and access. Assign responsibility for approving changes, run the relevant evaluations and keep a record of what was deployed.
Agent identity & security
Give it an identity. Not your admin account.
An agent needs an owner, an identity and a limit to its authority. Control what it can reach, what it can change and how its access is removed.
Know what is running.
Record the agent, its owner, purpose and connected resources. Distinguish its own identity from actions performed on behalf of a person.
Restrict execution.
Apply least privilege, short-lived credentials, network controls and sandboxing where appropriate. Separate reading, drafting and committing changes.
Test and revoke.
Exercise prompt-injection and misuse scenarios. Log relevant actions, protect the logs and provide a tested way to stop work and revoke access.
A closer look
Good questions. Straight answers.
Can you review code another team wrote?
Yes. Access, documentation and a clear review scope help establish the starting point.
Do we have to start again?
Not by default. Preserve useful work and explain any proposed replacement in terms of the risks and requirements.
Can evaluations be automated?
Many checks can be. Human judgement remains important for ambiguous quality questions, consequential outcomes and review of the tests themselves.
The first step
Know what is ready, and what is missing.
Bring the prototype or live AI workflow, its intended users and the evidence from testing so far. We scope a review of readiness, change assurance or operations around the decision you need to make.
What you get. A prioritised assessment, agreed acceptance criteria and a scoped plan for testing, controls and operation.
Talk to an engineer