AI inside the care workflow: govern the decision, not the product label

What the official position establishes

CQC says it regulates the quality and safety of AI-enabled care in England but does not approve individual technologies. Its stated expectations include human oversight, appropriate information for people receiving care and maintaining access through non-digital routes where needed. That is a boundary around service delivery, not a regulator endorsement of a particular product.

The September DHSC guidance describes care planning, sensors, analytics and administrative uses. It says evidence for truly personalised AI-generated care plans remains limited and requires staff review for accuracy. It also warns against putting identifiable confidential information into free or online tools without appropriate arrangements. These are reasons to examine a workflow carefully, not proof that every described application produces better outcomes.

A fluent record can conceal an altered meaning

An illustrative review meeting contains two statements: a person usually prefers to walk independently, and today they felt dizzy. A summary that keeps only the first statement changes the practical meaning of the record. A polished paragraph can be easier to read and less useful for a worker who needs to know what changed. This is a constructed example, not an incident investigated by Care Circle.

The approval step must consequently involve someone able to compare the draft with the underlying information. Giving a manager a button labelled approve does not establish that the review is meaningful. The service needs to know whether uncertainty, chronology, disagreement and the person’s own language survive the transformation. Reviewing grammar alone cannot answer that question.

The unseen cost is checking the output

A supplier demonstration may measure the time taken to produce a draft. A provider needs the time required to create an accurate, usable record, including corrections and any additional conversation. If checking becomes slower for complicated cases, a headline average may hide the cases where assistance is least reliable. Efficiency should be measured at the completed task.

A useful evaluation would compare the same kinds of task before and during a limited trial, record the reasons for substantive corrections and examine who performs the checking. It should include awkward or incomplete source material rather than only clean examples. Such an evaluation can inform a decision locally; it does not become a general claim about the entire care sector. The trial should retain a route for reporting problems.

Supplier assurances need an operational counterpart

A contract can describe security controls while leaving a practical question unanswered: which function is enabled by default, and can a worker turn it on independently? Providers should ask how model changes are communicated, where submitted information travels and whether it is retained or reused. An assurance about the main application should not silently stand in for an assurance about every added service.

The useful record is a description of the approved use, its data, its reviewer and its stop condition. A low-risk drafting function need not have the same review as a recommendation affecting care. Equally, familiar branding should not make a new high-consequence use seem routine. The assessment must follow the actual decision and the people affected.

Choice must remain workable after implementation

A person declining a digital interaction should not discover that the alternative route has become slower or inaccessible. Staff also need a realistic way to challenge a generated recommendation without being treated as obstructing efficiency. A system can formally allow overrides while its working culture discourages them.

The service should examine what happens after someone exercises that choice. Is the alternative documented and resourced? Can the reviewer explain the final decision? Are corrected drafts distinguishable from approved records? These questions focus on what people experience rather than whether the software can display a consent field. They also make an eventual reported case more informative than a testimonial about time saved.

The supplier discussion should distinguish an advertised capability from a demonstrated local workflow. Care Circle would ask who reviewed the generated record, what was changed and whether a later reader could identify the original information. Those questions make the account testable without treating a product demonstration as evidence of improved care.