Generative AI applications seem to be on the steep part of the development curve – not only is the technology getting better, but people are finding more and more uses for it. It’s a new powerful tool with broad applicability, and so there are countless startups and researchers exploring its potential. The last time, I think, a new technology had this type of explosion was the smartphone and the rapid introduction of millions of apps.

Generative AI applications have been created to generate text, pictures, video, songs, and imitate specific voices. I have been using most of these apps extensively, and they are continually improving. Now we can add another application to the list – generating virtual environments. This is not a public use app, but was developed by engineers for a specific purpose – to train robots.

The application is called holodeck, after the Star Trek holodeck. You can use natural language to direct the application to build a specific type of virtual 3D space, such as “build me a three bedroom single floor apartment” or “build me a music studio”. The application uses generative AI technology to then build the space, with walls, floor, and ceiling, and then pull from a database of objects to fill the space with appropriate things. It also has a set of rules for where things go, so it doesn’t put a couch on the ceiling.

The purpose of the app is to be able to generate lots of realistic and complex environments in which to train robot navigation AI. Such robotic AIs need to be trained on virtual spaces so they can learn how to navigate out there is the real world. Like any AI training, the more data the better. This means the trainers need millions of virtual environments, and they just don’t exist. In an initial test, Holodeck was compared to an earlier application called ProcTHOR and performed significantly better. For example, when asked to find a piano in a music studio a ProcTHOR trained robot succeeded 6% of the time while a Holodeck trained robot succeeded 30% of the time.

That’s great, but let’s get to the fun stuff – how can we use this technology for entertainment? The ability to generate a 3D virtual space is a nice addition to the list above, all of which is contributing to a specific application that I have in mind – generative video games. Of course there are companies already working on this. It’s a no-brainer. But let’s talk about what this can mean.

In the short run generative AI can be used to improve the currently chumpy AI behind most video games. For avid gamers, it is a cliche that video game AI not very good, although some are better than others. Responses from NPCs are canned and often nonsensical, missing a lot of context about the evolution of the plot in the game. The reaction of NPCs and creatures in the world is also ultimately simplistic and predictable. This makes it possible for gamers to quickly learn how to hack the limitations of the game’s AI in order to exploit it.

Now let’s imagine our favorite video games powered by generative AI. We could have a more natural conversation with a major NPC in the game. The world can remember the previous actions of the player and adapt accordingly. AI combat can be more adaptive and therefore unpredictable and challenging.

But there is another layer here – generative AI can be used to generate the video game itself, or at least parts of it. This was referenced in the Black Mirror episode, the USS Callister. The world of the game was an infinite generated space. In many ways this is an easier task than real-world applications, at least potentially. Think of a major title, like Fallout. The number of objects in the game, including every item, weapon, monster, and character, is finite. It’s much less than a real-world environment. The same is true for the elements of the environment itself. A generative AI could therefore use the database of objects that already exists for the game an generate new locations. The game could become literally infinite.

Of course, generative AI could be used to create the game in the first place, decreasing the development time, which is years for major titles. Such games famously use a limited set of recorded voices for the characters, which means you hear the same canned phrases over and over again. Now you don’t have to get actors into studios to record script (although you still might want to do this for major characters), you can just generate voices as needed.

This means that video game production can focus on creating the objects, the artistic feel, the backbone plot, the rules and physics for the world, and then let generative AI create infinite iterations of it. This can be done as part of game development. Or it can be done on a server that is hosting one instance of the game (which is how massive multiplayer games work), or eventually it can be done for one player’s individual instance of the game, just like using ChatGPT on your personal computer.

This could further mean that each player’s experience of a game can be unique, and will depend greatly on the actions of the player. In fact, players may be able to generate their own gaming environments. What I mean is, for example (sticking with Fallout), you could sign into a Bethesda Fallout website, choose the game you want, enter in the variables you want, and generate some additional content to add to your game. There could be lots of variables – how developed the area is, how densely populated, how dangerous are the people, how dangerous are the monsters, how challenging is the environment itself, what is the resource availability, etc. This already exists for the game Minecraft, which generates new unique environments as you go and allows players to tweak lots of variables, but the game is extremely graphically limited.

Also, I am just thinking of using AI to recreate the current style of video games but faster, better, and with unlimited content. Game developers, however, may think of ways to leverage generative AI to create new genres of video games – doing new things that are not possible without generative AI.

It seems inevitable that this is where we are headed. I am just curious how long it will take. I think the first crop of generative video games will come in the form of new content for existing games. Then we will see entirely new games developed with and for generative AI. This may also give a boost to VR gaming, with the ability to generate 3D virtual spaces.

And of course gaming is only one of many entertainment possibilities for generative AI. How long will it be before we have fully generated video, with music, voices, and a storyline? All the elements are there, now it’s just a matter of putting them all together with sufficient quality.

I am focusing on the entertainment applications, because it’s fun, but there are many practical applications as well, such as the original purpose of Holodeck, to train navigation AI for robots. But often technology is driven by entertainment applications, because that is where the money is. More serious applications then benefit.