- Posted by: anyak
- Category: Cybersecurity
Reflections on autonomous weapons systems and their impacts on Defense.
Roberto Gallo CEO KRYPTUS, President ABIMDE
Thiago Carneiro, Chief of DIPROD/MRE
As each day goes by, military capacities are more dependent on platforms of sophisticated and automatic weapons. Such undeniable technical advance may bring overwhelming advantages to the battlefield, dangerously increasing the gap between countries that develop technologies, and those that merely utilize them. However, this phenomenon also implies big strategic risks, typically misunderstood by the makers of military doctrines and by field operators, particularly in countries that only buy those technologies and equipment.
More and more, these changes present themselves as challenges to every armed force, and, equally so, to the defense industry. In order to better comprehend those risks, it is necessary to understand how the automation of the combat platforms and its systems have exponentially increased over the past 40 years.
Automation in Platforms
If we observe the platforms from the 80s, we notice its embarked components and subsystems operated in a majorly isolated way. Airplanes, for instance, had subsystems such as communications link, mission computer, countermeasures, weapons systems, navigation and motor control, and some had some sort of shared reading in the cabins, if that.
If that type of organization – despite assisting with the control of critical areas – still imposed a major workload over the crew, it also kept the platform in hand. One failure of an individual component many times could be mitigated by the crews themselves, or even during ground maintenance.
1 The opinions presented are strictly personal and do not reflect the Ministry of Foreign Affairs’ position.
Still using airplanes to exemplify the matter, four decades later, we may observe the contrast with the case of Boeing 737 MAX 8. The level of automation, integration and independence of the aircraft is such that the architecture of systems is conceived in a way that the airplane may fly safely despite the crew. The human operator, who used to be crucial for the aircraft to function, is increasingly growing as a supporting character in the (limited) platform control [Figure 1].
In the military world, that level of automation has led to the autonomy of weapons systems (LAWS) which, supported by artificial intelligence, communications and advanced sensors (C4ISR), have the ability to plan missions, identify and acquire targets, perform lethal actions and withdraw, with no need for human intervention.
This increasing level of complexity of systems and platforms presents a series of challenges to the defense scholar, in multiple orders:
- Strategic (mastering technical knowledge and geopolitical implications),
- Tactical (reformulation of military doctrines)
2. Lethal autonomous weapons systems.
3. Command, Control, Communications, Computers, Intelligence, Surveillance and Reconnaissance.
- Operational (use of technology in combat and deny its use to enemies)
- Moral (how far does the operator’s responsibility goes? And who is the operator to be held responsible?).
In this article we will approach, in a very brief manner, each of these aspects, with particular emphasis on the operational matter – in which the concepts of “kill switch” and “switch to kill” stand out, and its implications to Brazil.
Eminence in a war, and therefore a victory at an armed conflict have always been associated with ability, quantity and mastery of combat means. It’s long been known that it is not enough to have the best military strategy if you don’t have the support of an important technical and logistic chain. Steel forges, powder stock, observation balloons, submarines, long-haul aircrafts and long-range missiles have all been, each at a time, key elements for supremacy.
If, during the whole history of conflicts, supremacy has marched alongside technical superiority, only now do we witness a paradigm shift in the men-weapon relationship. Even with all the evolution of war during the last millennia, systems and platforms have always somehow been under the operator’s command. We now witness a revolution as quiet as it is scary. The weapons systems no longer rely on a person in order to be effective during combat.
There is a “depersonalization” of the soldier, who went from operator to system manager in the current days, and in the future, they’ll practically be a simple spectator. That goes, of course, for countries that possess such critical technologies. As for the others, there will always be a place for field operators, or even for “cannon fodders”. Such changes profoundly alter the strategic prospect. The world is divided between countries that can make war as offensive as it can be at minimum cost of their citizens’ lives, and those that will suffer the consequences of a clash against an adversary who never goes hungry, thirsty or tired, and can function 24/7 in the most hostile environments.
Similarly, the military doctrines present themselves according to the systems and platforms of its time. From the Romans’ “turtle” formation, able to use their shields and spears for sustained attack of the legions, to Nazi Germany’s “Blitzkrieg”, to the expedited advance of “panzer” divisions and air support from “stukas”, up to USA’s “Shock and Awe” doctrine, war has always been fought with troop risk mitigation and greatest possible damage imposition to the adversary in mind.
Autonomous platforms emerged for a new generational leap to these doctrines. 21st Century war centers around nets, intensive use of technology and lower human involvement to those who can afford to do so. “Drone swarms” and almost automated armies are a new reality that will significantly alter how States will start win and lose wars. This is the context in which three concepts must be well understood by the makers of public policy and military doctrines. The essentiality of the national Industrial Defense Base (BID) principle as a final means of effective protection of the country’s technical sovereignty, which is connected to the strategic and tactical aspects, and concepts like “kill switch” and “switch to kill” are related to the operational aspect.
The use of autonomous weapons systems in combat reveals a great deal of challenges to the modern soldier. Decision making in combat – a fundamental element to the success of any maneuver – becomes increasingly less dependent on men, significantly changing the way armies will define their strategies in combat. Autonomous weapons systems may “talk” among themselves, implementing the true concept of “netcentric warfare”, by means of encrypted communications. The ability to process and fuse data, together with the dissemination of information to the attack unities is the difference between victory and defeat. More precise and refined algorithms, modeled by supercomputers, will not only better aid the generals of the future, but, in some cases, replace them altogether.
Whoever has the best combination of software and hardware will have the best ability. However, these changes on how to “make war” lead to two conceptual matters that are as fundamental as they are complex. How to re-establish domination of men over machine in extreme situations and how to make (or avoid) the machine to turn on its user. That’s when we have the concepts of “kill switch” and “switch to kill”, which we’ll begin discussing now.
In English, “kill switch” is the term that refers to command, key, button or any resource that shuts down or disables a system when so decided. Kill switches normally work for the benefit of the system’s owner, as a fast stop resource, for protection. Some examples of it are panic buttons in lathes and the remote wipe of an iPhone in case it’s lost or stolen.
Any given system may have many “kill switches”, some activated in person, and others remotely, including through the narrowest of communication bands, after all, one single bit of information is what needs to be transmitted. “Kill switches” may be clearly identified as a “big red button” or insidiously buried in a single electronic component among thousands that may form a complex system – sometimes without the knowledge of the system’s manufacturers themselves!
The situation is so serious and the challenge so big that General Keith B. Alexander, former director of the feared National Security Agency – the NSA, reputed the profound “kill switches”, inserted with no knowledge of a technology’s owners, as one of the biggest challenges of computer platforms in the future.
4. Live declaration during RSA Conference 2015.
Alexander fully comprehended what he was talking about: during the 2010 decade (after revelations by Edward Snowden, Figure 2) until recently in February 2020 (the case of the company Crypto SG, run by the CIA and BND), the United States were caught many times practicing what is referred to as “supply chain intervention”.
That category of attacks serves as a prep. action for purposes of intelligence, creation of “distractions”, sabotage and use denials. While for intelligence activities the examples of “supply chain intervention” are recurrent, take the case of Huawei with their 5G technology banned from many countries throughout 2019 and 2020, other finalistic are way less reported, but no less real or less impactful.
As for the military scope, any nation minimally equipped has formal or informal concerns about technical denial and end-user agreements that limit acquisitions, information, and also possible employment of certain technologies. The final form of use control, however, is the operational impossibility,
5. ‘The intelligence coup of the century’, Greg Miller Feb. 11, 2020, https://www.washingtonpost.com/graphics/2020/world/national-security/cia-crypto-encryption-machines-espionage/
classically imposed over missiles and combat systems that respectively require launch codes or “software licenses” to operate.
As a classic denial example in our proximities we could (and should) mention that inflicted by the French over the Argentinians during the Falklands War against the British in the 1980s. At the time, even without the launch codes, the Argentinian Airforce was able to operate the EXOCET missiles and surprise the British Navy.
However, that kind of scenario would never repeat itself with today’s levels of platform automation – we may suppose, far from fiction, that if the Falklands War took place today, “kill switches” on French fighter-bombers wouldn’t even allow the aircrafts to take off. Let alone that missiles be launched.
That type of ability in current days, to remotely deny use through hidden “kill switches” spread across platforms, has massive strategies implied, some of them mentioned ahead.
However, there are still some even more serious outcomes when you combine high levels of automation and supply chain intervention attacks.
Switch to Kill
Not meaning to sound repetitive, but what is obvious should be mentioned: the same control technology that is used to disable a system against the owner’s volition may also be used to enable self-destruct routines.
For example, a Trojan Horse in hardware, inserted in the logistics chain of a missile, activated by satellite, may be used not only to disable the weapon, but to detonate it in the hands of their owners, whenever it’s the most damaging. Whether at the bunker or embarked on the platform, the potential damage is enormous.
However, the worst is yet to come: with the advent of the Lethal Autonomous Weapons Systems – LAWS – the subject becomes even more complex. Named “killer robots” by human rights activists, the employment of such systems has raised many discussions among the international community as the technologies have been employed on real operations by the leading nations of the war industry, Figure 3.
In a military context, however, the broad discussion that is lacking is around the fact that a “kill switch” may be a “switch to kill”, a plausible situation in which target designation of a LAWS is subverted, and, instead of attacking foes, it could turn against friends.
As the LAWS, by definition, possess minimally autonomous capacities to designate targets and shooting, they also become perfect targets (or means) to accomplish actions that go against the weapons systems owner’s interests: (i) supplier/developer nations have great geopolitical and strategic incentives to demand their war industries to include “kill switches” and “switches to kill” in weapons systems supplied to other States, (ii) at the same time, different supplier nations may want to compromise the supply chain of their adversaries.
In fact, the matter is so grave that private statistics demonstrate that about half the suppliers of American DoD had already faced problems of supply chain intervention in 2015 [CHASE Workshop on Secure/Trustworthy Systems and Supply Chain Assurance, University of Connecticut]. No wonder the USA has SHIELD, the great program led by DARPA to mitigate supply chain interventions.
The strategic developments of the above exposed reality are wide and deserve a long reflection, particularly when it comes to those of second and third order. That being said, the following list should only have purposes of summoning or starting material for studies in Schools of War.
Noting that reservation, we call brief attention to the following:
- The essentiality of the national Industrial Defense Base principle as a vector of technology development and fundamental component in keeping real sovereignty and the ability to have effective military deterrence. Without the “know how” and “know why” of these technologies, the complex weapons systems are just expensive toys in the taxpayers’ eyes and, to a limit, are threatens to the country’s sovereignty.
- Higher instability in conventional conflicts. Because they require less trained military personnel and may be controlled remotely, LAWS can be supplied to a larger number of countries, including in buying options, such as “leasing” or rental (real “mercenary robots”), posing particular risk to civilians;
- Catastrophic reduction of strategic employment hypothesis. No one should expect that a highly automated weapons system (LAWS or not) purchased from a given country could correctly operate against said country’s interests, right? However, it is even worse than that: with foreign LAWS, said platform could even fight against those who purchased them (how about that?);
- Acquisitions of defense materials should be widely revised. While central powers have great programs of supply chain assurance to mitigate and control the risks of “kill switches” (for example the American programs Trusted Foundry and DARPA SHIELD), this subject can’t even be comprehended in many other countries. Besides the elementary and necessary solution of strengthening the national industry, it is important to make plans so that in the strategic aspect the choices of technology partners favor control and visibility over the subcomponents of military platforms and interoperability among singular forces. We should also pose the question: at the end of the day, who benefits from occasional purchases?
In Brazil, the subject of supply chain protection has gained some traction, even if it’s not in a structured manner. On one hand, the bill 12.598/2012 collaterally covers the subject. On the other hand, recent decree 10.222/2020, which establishes the National Strategy of Cybernetics offers more advance – and it’s already reflected on the limitation of Huawei’s 5G critic infrastructure in Brazil.
However, facing the problem directly still hasn’t resulted in strategic action, unfortunately, even if the subject has been under discussion for a few years at the technical scope of the Armed Forces and the Brazilian Intelligence Agency. We must establish a policy and a national system of supply chain assurance for national defense, because strategic actions of intervention have already been implemented by the great powers for years.
As for the matter of reorganizing the purchase process, we must initiate an open and clear debate about the current Brazilian template, particularly when we consider the great changes the autonomous weapons systems impose on the future. The purchasing of defense material, thought and made only by singular Forces, or taking minimal intervention from other areas, reveals a short-sightedness of an action and a culture of operations that is still too compartmentalized, which is unfit to the new setting of conflicts in which technology mastery should be a priority.
A new multidisciplinary, inter-ministerial and inter agencies structure is not only desirable, but it also seems the only possible way to handle these challenges. The eventual creation of a Special Inter-Ministerial Secretary (SEIPRODE), which encompasses all the state actors involved with the defense product areas (MD, MRE, ME, MCTIC and APEX), to deal with the subject of defense products from its conception, industrial development, exportation, control and financing, in constant and fluid contact to the private sector presents itself as an urgent demand, even if it offers a mid-term resolution.
In the moral aspect, there are more questions to be asked about the combat operations. Who is in charge of the actions performed by a machine? Who is the operator in charge? How about humanitarian issues, under international law, such as the Geneva Convention, if now the machines decide what, when and
6. O Hardware Comprometido: Uma Importante Ameaça a Ser Considerada pela Atividade de Inteligência, Gustavo Andrade Bruzzeguez, Clóvis Neumann, João Carlos Félix Souza / Compromised Hardware: an important threat to be considered by the Intelligence Activity, Gustavo Andrade Bruzzeguez, Clóvis Neumann, João Carlos Félix Souza.
how to destroy? Given the goal of this article, we will not delve on the subject, but we already observe the necessity to reflect on the moral implications – and international law – that the use of autonomous weapons brings.
There is a revolution in place regarding weapons systems. Based on technology, such changes imply disruptive effects for national defense and security. In order not to let our National States appear in strategically unfavorable positions, it is important to fight technology captivity and recognize the essentiality of their respective BIDs.
In light of LAWS emergency, it is crucial that in Brazil, as well as in other countries of similar stature, we advance in studies of strategic, tactical, operational and moral implications, which should serve as subsidies for the necessary realignment of national policies and State agencies.
In comparison to other countries, particularly to Turkey, to us it seems that at the Age of LAWS, Brazil can’t do without a new State structure, one that’s multidisciplinary, inter-ministerial and inter agencies, so that we can keep up with the revolution in place, considering the prominence the country deserves.