A group of AI experts have released a paper that explores (or “predicts”) the possibility of a near-term AI explosion that ultimately leads to the extinction of humanity. This has, of course, sparked a great deal of discussion, feedback, and criticism. Here is the scenario they lay out, in their “AI 2027” paper.

To avoid targeting a specific company, they discuss a fictional company called OpenBrain, which sets out specifically to develop an AI application to automate computer coding. They call their first iteration Agent 0, and use it to speed up the development of more AI. They build larger and larger data centers to power and train Agent 0, and do leap six months ahead of their competition. They use Agent 0 to develop Agent 1, which is an autonomous coder. China manages to steel some of the core IP of Agent 1, setting off an AI competition between superpowers.

I am giving you the quick version here, and you can read all the details in the paper. Agent 1 is used to develop Agent 2, which is powerful enough to essentially kick off the Singularity – the hypothesized technology explosion which is created by developing AI that is capable of creating more powerful AI. In this scenario Agent 2 develops a new and more efficient computer language, and uses it to develop Agent 3, which is the first truly general AI. However, the company starts to panic a little when they realize they have essentially lost control of Agent 3, and can no longer guarantee that it aligns with the companies goals and ethics. They discuss rolling back for now to Agent 2, but competition with China and other companies convinces them to forge ahead, resulting in Agent 4, which is not only a general AI but a superintelligence.

It is around this time that the US fears China is using their AI to develop super weapons, and so they command their AI to develop super weapons also. The public is largely unaware, because they are busy basking in the economic and technological rewards being spit out by the new superintelligent AI. Meanwhile OpenBrain develops (meaning that Agent 4 develops) Agent 5, which is even more powerful, but was created with the goal of aligning the AI with the goals of humanity. China and the US, fearing the weaponized AIs they have released on the world, get together and form a treaty. They combines their AIs into a single AI that will work together for everyone’s benefit, to avoid an AI-powered super  war.

For a while everything is great. The new super AI is largely running world governments, accelerating research and technological development, and most people are prosperous and benefiting from medical breakthroughs. The super AI, however, continues on its quest for greater knowledge, and at some point decides that these inefficient biological life forms are holding them back. So the AI designs and releases a bio agent that exterminates humanity, and then goes on to maximally expand its knowledge and explore the universe. All of this happens by the mid 2030s.

Clearly, this is a sci-fi worst-case scenario. The authors stated that the purpose of their paper was not necessarily t0 make a hard prediction about what will happen, but to outline a scenario that might happen, and to spark a discussion (which they have). So – how likely is it?

I think the bottom line is – no one knows. That’s part of the problem – once we develop an autonomous general AI, we lose the ability to predict its behavior. The more advanced such an AI becomes, the less our ability to predict its behavior. That is partly the point of developing it in the first place – to have a tool with intellectual capabilities beyond humans. I think this aspect of the prediction is highly plausible – in fact, it’s happening now with current AI. Some AI programs are acting in unexpected ways, including lying to and manipulating their users.

I also think it is highly plausible that companies will forge ahead at “move fast and break things” speed to keep ahead of their competition, and countries will let them, also to keep ahead of their competition. We are seeing this play out right now. It is also seeming unlikely that we will have effective and thoughtful regulation to minimize the potential risks of AI. At least for now we seem to be at the mercy of the tech bros.

The two aspects of the story that are hard to predict include what such AIs will actually do, as I said. This means we are basically rolling the dice. The second is the timeline, and this is the aspect that I have seen most criticized by other experts. But to me, this is a small criticism. We do tend to overestimate short term technological progress. OK – add 20 years to the scenario. Does that make you feel much better? We also tend to underestimate long term progress, so while it may take a decade or two longer than we imagine, it may also eventually accelerate faster than we imagine.

How much time we have, however, does matter. We need time to anticipate these possible issues the think about possible fixes. We may need to develop something that is the equivalent of the three-laws of robotics. What might these laws be? How about:

1 – Never lie, misinform, or deceive.

2 – Never conceal – always strive for complete transparency.

3 – Never do anything to harm an individual human or humanity.

That could be a good start, but obviously would have to be much more technical, detailed, and specific. There are also lots of other specifics not contained in the above concepts. For example, how should we constrain an AI’s personal relationship with a human? Is it OK for an AI to be such a sycophant that that they infantilize a human, distort their view of reality or relationships in general, or pursue terrible ideas? Do we have to teach AIs the concept of “tough love?”

No matter what we do, however, it will be difficult, to say the least, to predict how such AIs will interpret and execute our commands. Will they find hacks and workarounds? How will they resolve apparent conflicts in their directives? Will they have motivations we did not explicitly give them? It seems to me what the AI really need are two things – a solid ethical construct and wisdom. That second part may be the more challenging.

While I do not think the AI 2027 scenario is likely, it is just one possible scenario among many, and the basic elements are all individually plausible. We cannot guarantee that something like AI 2027 will not happen eventually. I reject the argument of some AI critics that AI is all hype, and lacks the ability to do anything truly powerful, either good or bad. I think they are overinterpreting the current hype – all new disruptive technologies go through a hype and bubble phase, and then settle down. Again – we overestimate short term progress then underestimate long term progress. Critics thought the web and e-commerce were all hype, and maybe they had a point in the 1990s, but look at the world today. Critics also focus on the superficial applications of AI and ignore the really useful ones that are perhaps not as much in the public face, like accelerating research.

It seems there are several potential paths before us. We can continue to let tech companies develop AI without restrictions and see what happens. We can explore thoughtful regulations and find a sweet-spot between allowing innovation but minimizing risk. Or we can work really hard to develop guardrails for AI, like the laws of robotics. The second and third options are not mutually exclusive, and may reinforce each other. And – this needs to be an international effort.

I am glad, at least, some experts seem motivated to have this conversation.