The Great Brake: Essential Shift in Perspective for Technologies with Unanticipated Risks
The Great Brake
I believe that we need to seriously consider implementing the concept of the “Great Brake” for technologies that clearly involve risks that researchers cannot foresee. In other words, the research and development of such technologies should be halted, or nearly so, with strong regulations imposed.
This comes from the conviction that such unknown-risk technologies have asymmetric impacts on the level of risk and the measures against it. This conviction comes from my experience of working in internet security research for about eight years in my youth, and as a researcher and system engineer, thinking about countermeasure technology while understanding the thinking of attackers.
Generally speaking, technologies that contain such unpredictable risks reduce the minimum human-induced catastrophic cost, and at the same time, they also reduce the cost of the countermeasure technology to prevent it. However, unfortunately, there is a law at work here that reduces the former even more. If there is any counterargument to this view, it would only be in cases where it is known that the technology hardly contains any unknown risks. To deny this asymmetry while recognizing the existence of unknown risks is a clear logical contradiction.
However, the “Great Brake” is indeed a concept fraught with great difficulties.
One is the existence of resistance. Researchers, developers, and corporations engaged in technology development will likely react skeptically to the Great Brake. Similarly, I would expect government officials and beneficiaries in countries that are supported by economic growth through technology development, or in countries that are eyeing such opportunities, to react in the same way. This is not only due to their personal motivations such as curiosity, economic desires, and ambition, but also to a sense of mission and a sense of obligation to protect their comrades and compatriots. And in some cases, there may even be movements to suppress the opinions of those who express the idea of the Great Brake. However, if we sympathize with their opinions or succumb to pressure, it directly links to the risk of losing our future. For the sake of our own future, our children, our grandchildren, and the future of our future children, we need to deeply consider the risks posed by these technologies and the “Great Brake”.
Another major obstacle when thinking about the Great Brake becomes apparent when we start to think about when to apply the brake. We tend to wonder when is the best timing to hit the brake. However, we cannot think about this. Because essentially, the best timing is something we cannot know.
Normally, the best timing becomes clear when we have a prospect of when a problem will occur, or when signs of a problem appear. However, with this human-made catastrophe, we cannot know when the problem will occur. Because, when and how much technology advancement will lower the cost of a human-made catastrophe is a matter of the future, and even experts cannot know. In addition, considering that it is planned intentionally rather than occurring naturally, there is the possibility that it could start suddenly without any warning, foresight, or omen. In this way, we need to make that decision without foresight or omen to think about when we should hit the brake. This is very difficult, and it means that no one can depict a rational and certain strategy, being swayed by the subjective opinions of individuals and the opinions of those who speak loudly.
This difficult situation makes us pessimistic. It will be difficult to make a decision that satisfies many people in the face of this unprecedented challenge. And among some opinions, society and the world are likely to be divided. This situation will waste time without deciding on any measures and will lower the minimum human-induced catastrophe cost. This almost certain tragic future prediction says that it is impossible for humanity to avoid the catastrophe we create.
<Fable of the Golden Island>
A large cruise ship wrecks, and nearly 100 people wash ashore on an island in the middle of the sea. Gold is strewn everywhere, which astonishes them, but survival is the immediate priority. They unite their strength. They procure drinking water and food from the forests and rivers, and build simple houses to shelter from the rain and wind. They learn how to make fire, collect useful materials, and create necessary tools.
At one point, they notice countless inscriptions in an unknown language deep within the island’s many caves. Apparently, people used to live here. Moreover, the characters are unlike anything they’ve seen before, giving off an aura of ancient civilization. They recall seeing remnants of columns and walls that seemed man-made while exploring the island.
As their island life and procurement of water and food becomes somewhat stable, everyone starts talking about the necessity of preparing for winter. They experiment with methods of food preservation and urgently collect surplus food. But it becomes increasingly clear that it’s going to be extremely tough for nearly 100 people to secure enough food to get through the winter.
These crisis situations highlight differences in thinking, and two opposing groups are formed. The groups grow larger, and almost everyone must join one group or another. To strengthen their unity, each group creates a flag, and they start calling each other the Reds and the Blues from the colors of the flags.
In the midst of this, the Reds manage to create crops to get through the winter, prepare cold protection, and build homes suitable for warmth. The Blues are surprised to see that techniques and abilities to solve many winter problems have emerged somehow. The Reds’ leaders tell the Blues’ leaders that there is an important issue related to the survival of everyone on the island that holds the secret to this. On the condition of not prying into the matter, the two groups reconcile, and they manage to survive the winter.
At some point, the Blues’ leaders discover the Reds’ secret. A few of the Reds’ leaders had used a mysterious artifact left by the island’s ancient civilization, gaining useful tools and knowledge. The Blues’ leaders grew suspicious that they were monopolizing this technology to live a better life than other members. They gathered members of the Blues and those within the Reds who were dissatisfied with the current situation, and succeeded in restraining the Reds’ leaders and seizing their technology.
The Blues’ leaders independently learned to use the artifact of the ancient civilization and shared it with all the islanders. The artifact was a soft, bluish stone. They were lying around everywhere in the caves, and they would appear if you dug a little. By engraving the ancient characters into the stone, its power was activated. With the power of this artifact, everyone could use that technology and knowledge to live a rich life.
However, a while later, the Reds’ leaders, who were allowed to meet with the Blues’ leaders, tell them this:
The characters written on the walls of the countless caves dotted around this island could gradually be deciphered using the artifact. The Reds’ leaders were gradually able to draw power from the artifact through this deciphering. And among the countless characters on the cave walls and the potential power of the artifact, there might be a technology that could turn everyone living on this island into a lump of gold in an instant.
The Blues’ leaders were surprised to think that the lumps of gold that existed on the island might have originally been the inhabitants of this island.
The Reds’ leaders say, now everyone can freely pick up the artifact from the caves, and the deciphering of the cave characters has been advanced. If such dangerous technology really exists, sooner or later someone will know how to turn everyone into a lump of gold.
It cannot be denied that there might be people who become fed up with life on the island and become desperate, or become weak-hearted, and think about using that technology.
In fact, fights and troubles often occur due to stress from unfamiliar environments, and some people are neurotic. Some people frequently ignore rules and behave crudely. The Reds’ leaders, who are still restrained, strongly criticized the Blues’ leaders for teaching everyone the existence and use of the artifact in this situation.
One of the Blues’ leaders raises his voice. “Weren’t you guys keeping the technology a secret to benefit yourselves?” he says.
An elder from the Reds says, “Then what should we have done for you to believe us? You doubted whatever we said.”
And he slowly continues.
“We had said that it was a matter of life and death for all the islanders. Did you think it was okay to put everyone’s lives in danger, as long as you could strip us of our vested interests?”
“Even with problems, I thought we could manage. We’ve always overcome crises that way.” Another member of the Blue Team spat out these words in a dismissive manner.
“You knew it was an unknown technology. Do you not understand that there can be things that are simply beyond control within the unknown, unless you experience it once?” said the elder from the Red Team, dropping his gaze in resignation.
The Blues’ leaders, unable to say anything, could only exchange glances.
Reality of the Situation
Regrettably, the state of the current real world is far from my considerations about the risks of major disasters.
Many people often say that the advancement of these technologies is important for the humanities. They argue that there are many possibilities and benefits for us, and regulations on these technologies should be minimal. In other words, there are still many people on the blue team in the real world, despite many red team members expressing concern and caution about the unknown risks of these technologies.
Even if all nations and most people recognized that we are in the situation I am concerned about, there are many challenges in this field that seem difficult or impossible. This is already the worst-case scenario for us.
However, the reality is even far from that. We are still at a point where many people believe that the blue team represents justice and the red team represents evil or fear.
If there is no risk, there is no need to evolve. If there is a risk, we need to solve and manage it. This is the most important point when thinking about serious risks like catastrophes.
The essence of the red team’s thinking is risk control. In the fable, it was the red team that first used technology. They used this technology because they were in a dangerous situation by winter. If there were no risks to food, water, and climate, they might not have used the technology.
On the other hand, the blue team fundamentally believes that technological development is a good thing. They prefer to eliminate vested interests and solve problems. They do not prioritize risks highly.
The difference in opinion with people who belong to the blue team is whether there is a need to evolve or must grow unless there is a significant need. The blue team believes that it is important to always evolve and grow. However, my and the red team’s view is that there is no need to evolve or grow unless there is a serious need to resolve it.
From this perspective, there is something very important to consider. That is global collaboration.
If we were in the 19th or 20th-century situation of warring nations, there would clearly be risks requiring social, economic, and technological evolution and growth.
But now, in the 21st century, we can easily talk to other countries. And for various reasons, the major powers are essentially unable to go to full-scale war. Given these circumstances, it may also be possible to discuss the great brake globally.
In today’s world, the main concerns for many country’s elites are terrorism and criminals. The evolution and growth of technology make them richer and more powerful because they also use technology for their activities and business. Consider this, most of the country’s elites and governments should want to cooperate with other people to control the level of technology.
From this perspective, many people should be able to agree with the concept of the great brake. I believe that borders are not a major concern and are not a major resistance to the concept of the great brake.
On the other hand, the biggest barrier is the bias of competitive thinking that cannot change its thinking, like the blue team, which wants to develop technology, avoid vested interests, and overcome problems.
They do not prioritize risks highly. These people often say there is geopolitical risk, but I do not agree with that opinion.
When faced with geopolitical risks and the risks of terrorism and criminals, which can be resolved through dialogue and negotiations? Who can we cooperate with to resolve other risks? If I could choose, I would bet on dialogue and negotiation with geopolitical stakeholders, that is, other governments and elites.
