Human Oversight in AI: How to Design It and Prove It

What human oversight really means

Human oversight means that one or more people can exercise effective control over the use of an AI system.

It is not simply a matter of being present.

The person must be able to:

• understand what the system does
• interpret its outputs
• detect unexpected behaviour
• challenge its recommendations
• intervene
• escalate
• stop the process when necessary.

Human oversight therefore has two components.

Capability

The person understands what they are doing.

Authority

The person is able to act.

Where knowledge exists without authority, oversight is limited.

The same is true where authority exists without sufficient knowledge.

Oversight becomes effective when both are present.

What Article 14 of the AI Act requires

Article 14 requires high-risk AI systems to be designed and developed so that they can be effectively overseen by people while in use.

The aim is to prevent or minimise risks to:

• health
• safety
• fundamental rights.

Measures must be proportionate to:

• the risks
• the level of autonomy
• the context of use.

Article 14 also requires the people assigned to oversight to be able, where appropriate, to:

• understand the system's capabilities and limitations
• monitor its operation
• detect anomalies
• remain aware of automation bias
• interpret outputs correctly
• disregard, invalidate or reverse outputs
• intervene in the system's operation
• stop the system safely.

Human oversight is therefore not only an organisational function.

It also has a technical dimension.

The deployer’s role

The AI Act does not place every responsibility on the provider.

The deployer also has obligations.

It must assign human oversight to people who have:

• competence
• training
• authority
• the necessary support.

An organisation using a high-risk system therefore needs to answer practical questions.

Who provides oversight?

Do they have sufficient knowledge?

Can they stop the process?

Can they challenge the output?

Do they need further approval?

Where do they escalate?

Which evidence is retained?

Oversight must form part of real operations and the allocation of AI governance roles and responsibilities.

Human-in-the-loop, human-on-the-loop and human-in-command

Different oversight models are used in practice.

Human-in-the-loop

A person intervenes directly within the decision process.

For example, a system recommends candidates and a person reviews the recommendation before making the decision.

Human-on-the-loop

A person monitors the system's general operation and can intervene when anomalies arise.

For example, an operator monitors an automated system and acts when deviations are detected.

Human-in-command

A person or the organisation retains overall authority over the system.

They can decide:

• when it should be used
• when it should be restricted
• when it should be suspended
• when it should be withdrawn.

These models are not closed legal categories under the AI Act.

They are useful ways of designing different levels of control.

Start with risk

Oversight should not be designed before risk is understood.

Start by asking:

• what could go wrong?
• who could be affected?
• what impact could occur?
• how likely is it?
• which other controls already exist?
• which residual risk remains?

For example, light oversight may be enough where a system recommends low-impact internal content.

But where it influences:

• employment
• access to services
• credit
• health
• safety

human intervention may need to be much more robust.

Oversight must be proportionate and connected to AI risk management and, where appropriate, an AI impact assessment.

Define who provides oversight

One of the most common mistakes is to write:

“the user will review the result”.

That does not define accountability.

The organisation should specify:

• role
• department
• authority
• competence
• deputy arrangements
• escalation.

For example, an HR manager may review recommendations made by a recruitment system.

But that person may not have authority to correct a technical problem.

They may therefore need:

• escalation to IT
• escalation to Compliance
• escalation to the provider.

Oversight may require more than one level.

Competence, training and authority

A person can only oversee a system if they sufficiently understand what they need to observe.

They may need training on:

• the system's intended purpose
• capabilities
• limitations
• risks
• bias
• data
• indicators
• controls
• escalation procedures.

This competence can be developed through role-based AI literacy measures.

Training alone is not enough.

The person must also have authority.

They must be able to say:

“I do not accept this output.”

Or:

“We are stopping this process.”

Without organisational consequences that make the function impossible to exercise.

Understand capabilities and limitations

Effective oversight requires an understanding of what the system can do.

And what it cannot do.

For example:

• which data it uses
• which population it represents
• its level of accuracy
• the scenarios in which it fails
• the conditions that degrade its performance
• the changes that may affect its output.

The person providing oversight does not need to become a developer.

But they do need enough information to interpret the output in context.

A result showing 90% confidence may appear convincing.

But that figure is meaningless without understanding how it is calculated and what it represents.

Automation bias: when humans stop questioning

Automation bias is one of the specific risks identified by the AI Act.

It occurs when someone places excessive trust in the recommendation of an automated system.

They may think:

“if the machine says so, it must be right”.

This is particularly likely when:

• the system appears highly accurate
• its interface looks convincing
• the person has limited time
• the volume of decisions is high
• challenging the system creates more work.

Effective oversight should be designed to reduce this effect.

Measures may include:

• displaying uncertainty
• explaining relevant factors
• requiring decision rationales
• making review easy
• avoiding interfaces that automatically push users towards one option.

Approve, disregard, override, escalate or stop

Oversight requires clear actions.

A person may need the ability to:

Approve

Accept the system's recommendation.

Disregard

Choose not to use the output.

Override

Alter a decision previously suggested or executed.

Escalate

Send the case to someone with greater competence or authority.

Stop

Interrupt the system or process.

Not every system requires all of these options.

But the necessary ones must be defined.

The human-machine interface and the procedure should make them genuinely executable.

How to design human oversight step by step

A practical approach can follow these stages.

Step 1. Identify the system

Document its purpose, users, affected persons and context. The AI inventory, supported by a practical AI register and inventory, can keep that information connected.

Step 2. Analyse risk

Determine which risks remain.

Step 3. Decide which decisions require human intervention

Not every decision needs the same level.

Step 4. Define the model

Human-in-the-loop, human-on-the-loop or a combination.

Step 5. Assign accountable roles

Define who provides oversight.

Step 6. Define authority

Establish what they can:

• approve
• reject
• reverse
• escalate
• stop.

Step 7. Provide information

Ensure that the person understands capabilities and limitations.

Step 8. Train

Adapt training to the role.

Step 9. Design evidence

Record significant decisions.

Step 10. Monitor

Check whether the model works.

Step 11. Review

Update it when there are changes to:

• the system
• data
• context
• risk.

Human oversight also has a lifecycle.

What evidence demonstrates effective oversight

An organisation should be able to demonstrate how oversight works.

Relevant evidence may include:

• the procedure
• a responsibility matrix
• appointment of oversight personnel
• training
• instructions for use
• review records
• human decisions
• overrides
• escalations
• interruptions
• incidents
• audits
• periodic reviews.

For example, where a person changes a system recommendation, the record may include:

• the original output
• the human decision
• the reason
• the date
• the accountable person.

This creates traceability and an audit trail.

How to monitor whether it works

Designing oversight is not enough.

The organisation must also check whether it is being used properly.

Relevant indicators may include:

• percentage of outputs reviewed
• overrides
• escalations
• incidents
• review times
• decisions reversed
• errors detected by reviewers
• decisions challenged
• audit results.

A zero override rate does not necessarily mean that the system works perfectly.

It may also mean that nobody is challenging it.

Metrics therefore need context.

Common mistakes

Placing a person at the end of the process

Presence does not mean effective oversight.

Failing to give them authority

Someone who cannot stop or correct the process has limited control.

Trusting the output blindly

Automation bias.

Failing to train

Oversight requires competence.

Reviewing too late

In some processes, intervention must occur before the result takes effect.

Failing to record decisions

Without evidence, control is difficult to demonstrate.

Failing to adapt oversight to risk

One model for every system is usually ineffective.

Failing to review the model

Changes to the system may require changes to oversight.

What an organisation should do now

A reasonable starting point would be:

1. Review the inventory

Identify which systems require oversight.

2. Classify systems

Determine which are high-risk through an AI Act classification process.

3. Review the provider's instructions

Identify the intended oversight measures.

4. Analyse risk

Determine which human intervention is necessary.

5. Assign accountable roles

Provide competence, training, authority and support.

6. Design the workflow

Define when to:

• review
• approve
• reject
• escalate
• stop.

7. Record evidence

Create traceability.

8. Monitor

Check how it works.

9. Review periodically

Update oversight when the system or context changes.

The aim is not to be able to say:

“a person provides oversight”.

The right question is:

“can that person genuinely change the outcome when necessary?”

Oversight is not watching: it is the ability to intervene

Human oversight only adds value when the person has a genuine ability to act.

That requires:

knowledge, context, authority and evidence.

An organisation may design sophisticated interfaces and detailed procedures.

But if the person:

• does not understand the system
• does not challenge its output
• cannot reverse it
• does not know when to escalate

oversight can become a formality.

The aim is different.

It is to ensure that artificial intelligence supports decisions without removing people's ability to control them. This design can be integrated into an AI governance framework.

References

• Regulation (EU) 2024/1689 — Artificial Intelligence Act.
• Article 14 — Human oversight.
• Article 26 — Obligations of deployers of high-risk AI systems.
• Article 13 — Transparency and provision of information to deployers.
• European Commission — Navigating the AI Act.