I recently read a really good post by Aya Jaff about autonomous weapons systems and the need for meaningful human control. And following on from my latest piece about the new AI centre announced by the Danish Armed Forces, I simply couldn’t leave the topic alone.
Aya had been at the UN in Geneva, where she followed diplomats working on what she described as something that could potentially become some of the first international rules governing AI and autonomous weapons.
This is no small debate.
She had observed the work of the UN’s Group of Governmental Experts on Lethal Autonomous Weapons Systems. The group is currently working on elements for a possible international instrument on autonomous weapons systems, including some fairly fundamental questions about human control, accountability and the use of lethal force.
Aya’s core message was one I have a great deal of sympathy for.
When we use weapons that can potentially identify, select and engage targets on their own, we need to preserve meaningful human control. There must be a human being who can be held accountable, and we need to be extremely careful about handing decisions over life and death to an algorithm.
It is a noble, humanistic and, at first glance, very sensible position.
I just don’t agree.
Not because I believe we should remove human accountability.
Quite the opposite.
I’m just not convinced that human involvement in each individual decision is the same thing as meaningful human control.
Because if the human does not realistically have the time or the information needed to independently assess the system’s judgement, how meaningful was that control in the first place?
So I very much agree with the intention.
And yet I think I disagree with the conclusion.
Because there is a risk that we begin to equate human involvement with human control.
They are not necessarily the same thing.
Imagine an autonomous system that has only a few seconds to identify and respond to a number of incoming threats. If we insert a human into the process who has to approve every individual action, have we ensured meaningful human control?
Or have we simply inserted a human who, under severe time pressure, ends up pressing “OK” on the system’s recommendations?
In the worst case, we have not added better control. We have simply added delay.
That does not mean I believe we should remove the human being or human accountability. Quite the opposite.
But I think we need to distinguish between decision and responsibility.
Military responsibility does not begin only when a soldier pulls the trigger.
We already have principles of accountability within military organisations and chains of command. A commander can be responsible for decisions and actions even if they do not personally carry them out.
That is also why I think it is too simplistic to reduce the question of autonomous weapons to whether a human being is sitting there approving each individual engagement decision.
Perhaps the more interesting question is:
Who decided to deploy the system, under what conditions, with what limitations, and based on what knowledge of what the system can and cannot do?
Those are still human decisions.
And this is where I think the debate becomes particularly important.
Because of course the idea of having a human being in the decision-making process appeals to me. It is intuitive, humanistic and, in many ways, sensible.
But we do not design military capabilities in a vacuum.
We design them for a reality in which a potential adversary is also developing autonomous systems, and may not necessarily impose the same restrictions on themselves.
If an autonomous system can identify, prioritise and respond to a threat faster than a human can understand the situation and approve the action, then a requirement for human approval may ultimately become an operational limitation.
That is an uncomfortable realisation.
But conflict between states is unfortunately not a competition over who can formulate the most sympathetic principles. It is, among other things, a competition over the ability to act faster and more effectively than an adversary.
That does not mean we should abandon law, ethics or human accountability.
But it does mean that we need to be very precise about what it is we actually want to preserve.
For me, human accountability is absolutely central.
But I am not convinced that human accountability necessarily requires a human being to make every single decision.
Perhaps, in some situations, meaningful human control lies before the individual action: in the decision to activate the system, the rules governing what it is allowed to do, the area and timeframe in which it is allowed to operate, the requirements for the system’s reliability, and the ability to stop it again.
That is a less comfortable discussion than simply requiring a human in the loop.
But I think it is the discussion we are going to have to have.
References
Link to Aya Jaff’s post
Link to UN/ The 2026 Meeting of the High Contracting Parties to the Convention on Certain Conventional Weapons