On the late evening of September 26, 1983, I was probably at home getting ready for bed. Maybe I was cramming for a test the next day, things that could feel terribly important. Meanwhile, about 2400km towards the east, Stanislav Petrov was deciding whether to kill me or not. Not me personally, my family, or my schoolmates. But since military Airbase Soesterberg was only 22km from my parental home in Leersum, the Netherlands, it is a valid assumption that we were in the strike zone of incoming nuclear missiles.
The problem with AI
Petrov was deputy chief of combat algorithms at the town Serpukhov-14. This town south of Moskow was home to the Soviet Union’s early warning system for missile detection. Just after midnight, the radar screen showed that a single missile had been launched in the US and was heading towards the USSR. As Petrov tried to calm his 200 subordinates, the radar system showed that four more missiles had been launched. Giant blood-red letters appeared on the screen saying START.
“All I had to do was reach for the phone and raise the direct line to our top commanders. But I couldn’t move. I felt like I was sitting on a hot frying pan.”
Petrov needed to act as he had been trained: pick up the phone and talk to the aging and ailing Yuri Andropov. The man with his finger on the trigger.
“I had a funny feeling in my gut,” he later told The Washington Post. “I didn’t want to make a mistake. I made a decision, and that was it.”
In the two calls that followed, Petrov bluntly told the duty officer that these were false alarms though he did not know this for sure. He went against training and protocol. The algorithms of the Soviet early warning system indicated high reliability.
A multitude of simulations, war games, etc. has shown that it would not have stopped at a few missiles back and forth. The next steps are firing at military goals and cities alike. Most likely the end of mankind.
It turned out that the false alarm had been created by a rare alignment of sunlight on high-altitude clouds above North Dakota.
Petrov later indicated that the influences on his decision included that he had been told a US strike would be all-out, so five missiles seemed an illogical start. He was also a software coder, and knew that the launch detection system was new and, in his view, not yet wholly trustworthy.
The problem with humans
That day AI failed. But humans are also lousy decision makers. In one of the best books I have ever had the pleasure to read, Thinking fast and slow, Nobel prize winner Daniel Kahneman explains a model of thinking in which the brain has two modus operandi:
- System 1 thinking is effortless, automatic, and near-instantaneous. This is the brain function that drives you to work, recognizes the mood of your vis-a-vis, and forms an opinion about a job applicant in a matter of seconds.
- System 2 thinking is slower and requires more effort. It drives on logic and reasoning. Making deliberate choices, looking for something or solving complex problems are examples of system 2 thinking.
Through our thinking run biases, heuristics, and rules of thumb (See lion! Run!).
Biases and heuristics make our thinking efficient and fast, but the problem is that they produce systematic errors. Systematic error is not a word used lightly in economics. Psychology has fewer problems with it. The systematic part of the error means that given a situation, humans will take an illogical, not optimal decision. We’re wrong!
A few examples[1]:
Confirmation bias: our tendency to notice, focus on, and give greater importance to evidence that fits our existing beliefs. Judging the book by its cover is very harmful in recruitment processes.
Overconfidence bias: roughly 80% of drivers indicate that they are better drivers than average. This number is higher for men than for women. Maybe this could be considered an argument towards prohibiting men from running countries from avoiding the mess we currently face.
[1] Here is a list of sources of our systematic errors: https://thedecisionlab.com/biases
Loss aversion: the pain of losing is psychologically two to three times more potent than the pleasure of gaining. It would be logical to take a flip of a coin bet when the pay-out is more than the loss. But the fear of a loss makes people hesitant to make such a logical decision.
Fundamental attribution error: is the tendency to relate your own mistakes to circumstances and other people’s mistakes to their character flaws.
Consider also that humans make thousands of decisions every day. They are yet not good at handling 21st-century overwhelming information challenges. And they are hugely impacted by their physical condition. Our brains don’t give each decision equal attention to save time and mental energy. Instead, we take mental shortcuts.
The jury is still out
AI is flawed, and humans are flawed (yes, humans make the AI. For now…)
However, digitalization is happening and still expanding at a massive scale, including in the safety domain. This has the implication that more and more critical decisions are made through or by computers. Should we be worried?
AI is getting better and better. Since May 11, 1997, we have known that the world’s best chess player is a computer. And also, the game of Go, which has a near infinitive amount of possibilities, is now in AI’s hands thanks to the new learning of AI focused on pattern recognition. But still… Don’t you think that self-driving car takes an awfully long time? A couple of years ago, I wondered if my kids, who are now 8 and 10, would ever have a driver’s license, but now I’m pretty sure they will.
And humans can become aware of our fundamental decision errors, learn and adapt. In the HSE universe, we focus our attention on shaping the environment in such a way that people are less likely to make preventable mistakes.
Nobel prize winner Richard Thaler call that choice architecture. His book Nudge is great read if you want to learn ways to prevent these predictable mistakes.
So humans or technology?
Maybe for more optimal decisions, we need better collaboration between humans and AI. Here is an example of Udacity, a company offering online courses. It used AI to analyze transcripts of sales calls and created an AI agent that helps sales agents to make suitable proposals to customers. The combination of a human supported by an AI agent has them much more effective. We also introduce this approach at Unite-X: first engage people and then embed technology.
As AI evolves into something much better than the version that nearly wiped out humanity in 1983, the fundamental question of who will make the ultimate decision arises. Who decides, and who is the agent? Judges rule much harsher on identical cases just before lunch when they are getting hungry. Should AI intervene in such a case?
AI can monitor people’s movements on a plant, and there are developments where control measurements like PPE are monitored. And how? Can AI decide to revoke the permit to work and site clearance immediately? Or signal to the shift supervisor what is going on? Big brother in the plant!
I learned early in school that I could pass exams by learning a bit and by gambling on all the unknowns with the longer answer that was usually the correct one. No short answers here. It will depend. How accurate can we get the data to feed AI? Can we clearly define what needs to be optimized? And how do we decide who is the agent and the decider? Big decisions await us.
Digitalization is a fact, but managing the opportunities and risks is up to humankind in the decades.
***
We were saved that day because of a brave man’s gut decision, or pattern recognition, going against procedures. 12.000 soviet nuclear warheads stayed in their silos. I cycled to Revius Lyceum in Doorn the next day, worrying about school tests and other teenage problems.
P.S. Are you wondered what happened to Petrov after saving the world? He received some praise for his decision but also got reprimanded for lousy paperwork. Soon after the event, he was moved sideways and put into retirement. He lived poorly until his story became public and then received international recognition and around $33k in award money. A quick estimate of global assets in 1983 stands at about 1200 trillion USD (1.2*10^15). Since all of this would have been destroyed if not him, I guess his bonus scheme did not match his performance.
Written by Sjoerd Nanninga, Unite-X
