Decision Making in the Age of the Machine

When there is more data and analysis behind what is presented to the battlefield commander than he can know, what does that mean for the impact of his decisions? Can he ever even know it? And if he can't, is he really in control?

The modern commander must answer these questions now. With the emergence of AI in his equipment and workflows, the velocity of his own systems continues to become faster than he can keep pace, and those systems continue growing ever more rapidly in scale and scope.

In part 1 of this article I identify this phenomena as automation momentum, where the sheer speed of machine processing can outpace the commander's ability to exercise meaningful control over their own forces, for the advantage of lightning fast speed of computation and battlefield execution. To enable commanders to leverage this velocity, we must adopt a new principle of war, the principle of cognitive integration, which is the deliberate balancing of human decision making with the power of AI.

The Problem of Automation Momentum

A conservative estimate places 9,000 drone-based sensors flying per day on the battlefield in Ukraine. The amount of data created by those sensors is tremendous; roughly 6 terabytes a day. You could stream the entire series of Game of Thrones in 4K 12 times over with that amount of data. Modern command and control systems attempt to present that data to the commander in the form of reports and visualizations such as dashboards, PowerPoint, or word document reports, and this largely governs the commander's situational awareness in the operation.

Cyberneticists in the 1950s like W. Ross Ashby and Stafford Beer foresaw this situation at the earliest days of the digital age, theorizing that for a regulator (the commander) to maintain control, their own internal "variety," or possible responses, must match the immense variety of the system he seeks to control (the battlefield). Because a human mind cannot process thousands of variables at once, the machine is forced to filter reality down to a manageable size, effectively surrendering the regulator's agency to the algorithm's choice of what to show and what to hide. Cyberneticists of the era focused their efforts on designing tools that amplified a regulator's reach while using smart filters to prevent them from being buried by the noise.

Today, our solution has emerged in layering on more systems, sensors, and methods of presentation in a continual race to bring more awareness and control to the commander. Paradoxically, this creates an explosion of variety because of the exponential increase in potential data states that the addition of more systems brings us. The recent requirement to add AI into our systems accelerates this process by generating non-deterministic layers of complexity. The aggregate of these systems creates a loop that demands ever more sensors and processing layers to be ever more effective.

This is Automation Momentum: a self-reinforcing feedback loop that occurs when process-oriented systems like AI drive organizations to produce, utilize, and act upon ever-growing volumes of data, resulting in the production of more models and insights that require more data to fine-tune. As the requirement for AI is increased, so does the need for more sensors, automation, and integration, creating a momentum that increases the velocity of decisions beyond human ability to keep pace.

The IDF's reported use of AI targeting systems illustrates automation momentum's dangers. According to Israeli intelligence officers, the Lavender system identified up to 37,000 potential targets with human personnel "often serving only as a rubber stamp," sometimes approving targets in 20 seconds by merely confirming gender. As one source stated: "Because of the system, the targets never end. You have another 36,000 waiting." The head of an Israeli intelligence unit describing human personnel as a "bottleneck" and lamenting: "We cannot process so much information." This is a larger issue than ensuring a human is in a loop on a kill chain. When the question is could a human meaningfully assess what they were approving? it is an issue of decision makers having authority, but not comprehension.

The situation is increasing in scope as militaries increase the speed of their race to integrate AI into their workflows. The confluence of rapid increases in data, speed, and integration manifest the phenomena of automation momentum.

The Cognitive Dynamics of AI in Warfare

Modern command and control systems act as filters by design, that discard the vast majority of environmental variety (data) to fit the human requirement for a dashboard, situation map, or some other method of presentation. This reliance on a filtered digital reality over physical ground truth is the primary driver of automation momentum. Within this automation momentum, commanders and analysts become increasingly boxed-in to algorithmically generated understandings of reality, mistaking the speed and precision of automated systems for depth and insight.

Let's return to the Ukraine example. If 9,000 drones generating 6 terabytes of data is funneled down to a presentation of 100 4MB targets to the commander, the commander is only seeing 0.006% of the reality the drones see. Quantitatively, if the machine is discarding 99.9% of the variety of the battlefield, the commander is essentially looking at the battle through a pinhole.

A paper on Ukraine last March by CSIS noted the number one challenge for ISR was that drones were generating large volumes of data that "exceed human processing capabilities." Real world studies confirm this phenomena of narrowing perception, such as NASA's findings that over 70 percent of information transfer deficiencies occur during high-workload conditions, degrading situational understanding. In this environment, the commander's capacity to make decisions has essentially been eroded by a sea of data necessary for our AIs to take action at scale.

This perceptual funnel creates a situation where the commander remains "in the loop" by pressing buttons or giving orders, yet is cybernetically decoupled from reality. Automation momentum creates a form of perceptual inertia where human judgment becomes subordinated to the logic of automation itself. At the decisive point, this inertia can cause the commander to mistake the most calculated option for the correct one.

Even the U.S. Air Force's experimental DASH program, designed to enhance human-machine teaming, reveals the tensions. While AI generated recommendations 90% faster than humans with 97% validity compared to humans' 48%, commanders remain uncertain. As one general admitted: "Even though there may be significant automation, there will have to be, at some point, a human to decide this is the right thing to do." But which decisions? And how can humans maintain meaningful judgment at machine speed?

The transformative impact automation momentum has on human judgment is that on the cybernetic battlefield, as the power of any one decision made within it is exponentially increased, the decision maker's connection to the reality those decisions occur in is tremendously decreased. This has the greatest impact on uniformed decision makers at the highest echelons, and the policy makers above them for the same reasons.

Automation momentum is the imperceptible terrain of the modern battlefield, and affects the nature of warfare itself, at the strategic, operational, and tactical levels. It is a fundamental rule that should guide how organizations should approach and think about the conduct of operations.