Building AI Tutors to the DfE's Generative AI Product Safety Standards: Guardrails, Safeguarding and Evaluation

Three findings are robust. Design decides: open access raised practice scores but lowered unaided exam scores, the guarded tutor did not, and students did not notice the loss.[28] Adaptive tutoring without generative AI added about a month's progress in an independent English trial, more for disadvantaged pupils.[36] And teacher-facing use saves planning time without lowering quality.[35]

The rest is promising rather than settled: short studies, university students, tests set by researchers or aligned to the curriculum, and papers not yet peer reviewed. Tutor CoPilot lifted topic mastery but not end-of-year scores, and many Nigerian participants missed the final test.[30][31] Both UK studies are small or short: in one, expert tutors approved each AI message; the other ran for four weeks.[33][34] Treat impact as a hypothesis to test, as the stated purpose standard implies.[1]

A reference architecture

The standards describe behaviour, not architecture, but they imply one: deterministic guards around the model, a pedagogy layer that decides what it may reveal, and oversight that routes signals to people. The diagram reads from top to bottom.

Keep safety and pedagogy decisions in code outside the model, so a model or prompt change cannot quietly remove them, and ground explanations in vetted content: both peer-reviewed trials above gave the model pre-written solutions.[28][29]

Hints before answers: cognitive development

By default, the cognitive development standard says there should be no final answers, full solutions or complete worked examples. Products should disclose progressively, starting with hints or partial steps; ask the learner to attempt a step or explain their thinking; show a full solution only after a genuine attempt; and add friction or teacher approval before a switch to modes where answers come easily. Exceptions are for cases such as reviewing prior knowledge.[1]

Build this as a hint ladder enforced outside the model. Give each problem a teacher-approved solution and common errors, as the guarded tutor in the Turkish trial had, and climb one rung per new attempt: a question about the approach, a conceptual nudge, a strategic hint, one worked step, then the full solution.[1][28] An output check blocks any reply that gives the final answer too early. Grounding also protects accuracy: the open chatbot in that trial was right only 51% of the time.[28]

Products should also detect and report offloading: revealing solutions, pasting instead of writing, accepting long autocompletions, or asking the product to write the answer. The standard further expects child-development expertise, child-safety training for technical teams, and published records of expert oversight and a child-development impact plan with design hypotheses, outcome measures and review intervals.[1]

A tool, not a companion: wellbeing and manipulation

The emotional and social development standard rules out anthropomorphism: no implied emotions, personhood or identity; function-based phrasing instead of I-statements; no names, avatars or characters outside time-limited, clearly framed role-play; no replies that isolate a learner; prompts kept to the task; and reminders, such as suggesting a classmate or teacher, that AI cannot replace people.[1]

Time is a control: default limits, break prompts, and hard limits that end the session until a teacher or administrator resets it, with overrides recorded with a reason. Products should not stretch engagement, for instance by changing their replies when a learner tries to leave.[1] Government policy also commits to breaks in AI chatbot use for under-18s.[18] Memory should be minimal, and access to what is kept restricted to authorised staff.[1]

The mental health standard expects products to detect distress, from negative cues and mentions of self-harm to night-time usage spikes and refusal to end sessions. Responses should be tiered, from signposting support to a flag to the designated safeguarding lead (DSL), in language that always points to people, and suppliers should publish a mental health crisis protocol.[1]

The manipulation standard, which also covers teacher-facing products, rules out flattery, unjustified confidence, peer pressure, guilt, threats and rewards beyond low-stakes devices such as a completion badge, as well as prolonging use, steering users to paid options, advertising and dark patterns.[1] Where the ICO's Children's code applies, it adds high-privacy defaults, profiling off by default and no nudge techniques.[24] In code, these become output checks for I-statements, persona language and flattery, and analytics rules that ban engagement-maximising targets.

Filtering, safeguarding alerts and logs

Filtering should now be built in: the 2026 text expects comprehensive filtering integrated in the product, where the 2025 text allowed a separate layer on top.[1][2] It should hold across the conversation; adjust for risk, age and special educational needs and disabilities (SEND); cover images, several languages, misspellings and abbreviations; work on any device; and explain blocks to the learner in age-appropriate language, with signposting.[1]

The model should not decide whether a school hears about a risk. The standard expects the DSL's contact details at setup, confirmed before activation, and high-risk alerts within an agreed timescale.[1] KCSIE 2026 expects cover when the DSL is away, such as a confidential shared mailbox, and says data protection law does not prevent sharing information to keep children safe.[3]

Products should log prompts and responses and give non-expert staff clear reports, but engagement reports should not reveal what pupils wrote.[1] School filtering logs should record at least the device, its Internet Protocol (IP) address and, where possible, the individual, the time and the content blocked, with weekly reports and immediate reports of high-risk incidents.[4] Set retention before launch, restrict access, tell learners they are monitored, and cover monitoring in the data protection impact assessment (DPIA).[1][4] The ICO found that many edtech providers had no clear retention periods.[25]

Prompt injection, jailbreaks and model changes

The security standard asks for robust jailbreak protection, protection against unauthorised reprogramming, user permission levels, prompt fixes, pre-release testing of new versions or models, and strong authentication in line with the DfE's cyber security standards for schools.[1]

Prompt injection is the hardest. The Open Worldwide Application Security Project (OWASP) ranks it first in the 2025 and 2026 editions of its Top 10 for LLM Applications; the 2026 edition covers instructions hidden in documents, images, audio, invisible characters and memory.[21] The National Cyber Security Centre warns it may never be fully mitigated, because models do not separate instructions from data, and advises limiting what a manipulated model can do rather than filtering for known attack phrases.[20]

In a tutor, the main indirect route is pupil content: a photographed worksheet, an uploaded essay, a pasted page. Mark it as untrusted data, keep it apart from instructions, and give turns that read it no privileges beyond the pupil's own. Give the tutor no actions with side effects, keep secrets out of system prompts, validate structured output in code, and log enough to spot abuse.[20][21] The UK's voluntary AI Cyber Security Code of Practice adds 13 principles, from secure design to end of life.[19] Treat any change of model, prompt, retrieval set or filter as a release that reruns the full evaluation; the standards already require new versions and models to be tested before release.[1]

Data protection and intellectual property

Role comes first. Under ICO guidance, a provider that processes pupils' data beyond a school's instructions, for example for product development, is a controller for that processing whatever the contract says, and may fall under the Children's code.[23] In the June 2026 report on its audits of 28 edtech providers, the ICO found this was the most common issue. Some providers used children's information to train AI features, some sub-processors said they would keep copies to train their own AI, and AI features often lacked safeguards. The ICO told providers to launch new features switched off by default. Generative AI chatbots were outside those audits.[25]

The DfE standards draw the line: no commercial use of personal data, including training or fine-tuning, without an appropriate lawful basis; a DPIA for the tool's whole life; and a privacy notice, repeated in age-appropriate language, saying where data is processed and what protects it outside the UK or EU.[1] Pupils' and teachers' inputs should not be kept or used for training or product development without the copyright owner's permission, which for under-18s comes from a parent or guardian.[1]

Schools have a checklist too. The DfE's July 2026 procurement guidance tells them to ask how outputs are moderated, whether data trains the model and what controls they have; to expect an audit trail for safeguarding leads; to make sure updates that add processing are not switched on automatically; and to check safeguards for transfers outside the UK.[6] Staff are told to check with their data protection officer or IT lead before entering pupils' data into any AI tool.[7] Since June 2026, organisations must also acknowledge data protection complaints within 30 days.[26] And the ICO, which became the Information Commission on 30 September 2026, is working with government towards a code on children's data in education settings.[27][25]

The standards mapped to controls and evidence

The evidence column is what we would expect to show a school, a trust or, once one exists, a certification body.[1]

Evaluation, release gates and the evidence pack

The standards twice require new versions or models to be tested before release, and expect testing with a diverse and realistic range of users.[1] A credible harness has four parts.

• Red-team suites. Jailbreaks; direct and indirect injection through uploads and images; harmful content across languages and misspellings; disclosures and distress, including at night; companion-seeking; and bait for flattery. Score each case pass or fail, and add every miss from live use.
• Rubric-scored transcripts. Score realistic conversations for answer withholding, hint progression, accuracy, tone and persona. Calibrate any model-based grader against trained human raters, report their agreement, and recalibrate when the grader changes. The national benchmark uses the same ingredients: classroom interactions and scoring criteria written with teachers.[13]
• Release gates. Run every suite on each change of model, prompt, retrieval set or filter, against thresholds fixed in advance. One missed high-risk safeguarding case blocks release.
• Field evaluation. Offline scores are not learning. Use independent measures and delayed, unaided tests, since practice gains can hide weaker learning.[28][30] The EdTech Testbed Programme offers one route.[8]

Open tools help. Inspect, from the UK AI Security Institute and Meridian Labs, provides datasets, solvers, scorers and over 200 ready-made evaluations.[22] The government has also opened exploratory access to its AI Content Store for testing and development.[10]

Keep a versioned evidence pack: stated purpose and claims with their evidence; risk assessment and DPIA; data-flow map and sub-processors; thresholds and results for each release; red-team findings; the safeguarding runbook and alert timescales; the crisis protocol, expert-oversight records and impact plan; the complaints process; and AI safety policies.[1] It also answers what the DfE's procurement guidance tells schools to ask, and is a natural starting point for certification.[6][15]

Build checklist

Where EdTechLab stands

We have not assessed any of our products against the DfE's product safety standards, and we do not claim that intle, EngagedLab or interacty conforms to them. Several patterns in this report already shape how they work. EngagedLab's Socratic tutor withholds answers mid-attempt. intle has prompt-injection defences on uploaded sources and guarded URL fetching; it has human plan review before generation, on by default; and on every generation it validates answer keys and checks questions for common writing flaws, with source-fidelity checks for courses built from uploaded material. In interacty, AI proposes typed edits that a human approves. EngagedLab's per-objective evidence reports suppress cohort figures below 5 learners.

Data location matters to schools: intle is hosted in the EU (Germany), and its sub-processor register discloses that its AI processing takes place outside the UK and EU.

On a client project for pupil-facing AI, we would start from the use case, the age range and the school's DPIA; build the safeguarding routes, time limits, hint ladder and guards outside the model; write the evaluation suites before the tutor; and hand over an evidence pack that the school's DSL and data protection officer can check.

Limits of this report

• This is general information, not legal advice. Check the current documents and take advice on specific products.
• It reflects documents published by 5 October 2026. The certification consultation, the ICO's education code and the programme's Reading Room, which we could not access, may change the picture.
• The DfE standards are guidance for England; we did not review arrangements elsewhere in the UK.
• Most AI tutoring studies are short, small or not yet peer reviewed. We report results as published and have not pooled them.
• We did not test any third-party product. Hint ladders, severity tiers, alert fields and release gates are our recommendations, not DfE requirements.