Microsoft Flight Simulator (MSFS) 2020 doesn’t just feel real; it almost is real. You see the same objects and relationships when you look out the window that you would see out of a physical plane. Down there are your own cities and streets, and you can even fly into a live hurricane [1]. Because of its fidelity to the real world, MSFS represents an inflection point in virtual worlds. Almost all video games are based on fictional worlds, but MSFS is played within our own. By representing our planet so well, MSFS shows how having games set in our world creates value beyond entertainment.
Entertainment is weird. It hijacks our drive for information and puts us in a stupor, but entertainment’s beneficial side effect is that it gives us life experience on the cheap. The Shawshank Redemption taught us what prison in the 1940s might have been like. Of course, movies lie, but they provide initial models of new experiences that we can later refine with thoughts and further evidence. The experience gained through video games is even more potent because we take actions, and MSFS brings that experience with actions to our world.
I’m not an aviation guy (I don’t like the noise pollution private planes and helicopters make [2]), but I now have some sense of what it’s like to fly, and I can take my kids to see the Grand Canyon in the middle of the pandemic. I’ve always dreamed of video games that took place in my neighborhood. Imagine shooting zombies in your local grocery store or driving to work on your morning commute and thinking, “I always get shot down over this intersection.”
Video games set in a simulation of the real world will do more than enrich our experience: they will also teach our future robot helpers. To become intelligent, robots need real-world experience for two reasons. First, we could never explain the world in enough detail for them to understand it. You have to be there. Think of how hard it would be to describe a gas station to someone who had never been to one. No matter how hard you tried, they wouldn’t get the correct big picture, and you would almost certainly leave out small details, like all the questions you have to answer before they let you pump the gas: zip code, car wash, rewards program, and favorite food. It’s the same with a grocery store or even flying as a passenger in an airplane (did you remember to mention the little vent above your seat?)
The second reason robots need real-world-like environments is that they must create knowledge on their own by generating and testing hypotheses. Robots need causal models of the world to be able to handle novel situations, and the easiest way to build causal models is to run little experiments, “What happens if I flip this switch?” Experimentation, of course, is how children learn [3], which is why we keep them away from dangerous or fragile equipment until they are older. In a simulated world, the robots can break things all they want, and simulated experience is good for robots for the same reason it is good for us—it’s cheap. Cheap experience is even more critical for robots because they require a lot of maintenance, especially if we plan on letting them go around flipping random switches.
Besides having all of the familiar landmarks in place, another way to characterize a virtual world’s realness is a measure we can call “conceptual depth.” Conceptual depth is how well the dynamics of the game match the dynamics of the real world. For flying, MSFS is obviously good, that’s the stated point of the game, but we need a lot more if we want to get out of the airplane. Does paper tear? Does water spill if you pour a big cup of water into a small one? Does the spill make the counter slippery?
In most games, the ground is a 2D surface with some assets on top of it (apparently even in MSFS [4]). In the real world, when you walk outside on the grass, the ground is a maze of plants and dirt and organisms. It’s limitless. For humans or AIs to learn in virtual worlds, we need the freedom of action and consequences that match those of the real world. We’re getting there. In racing games from the 1980s, you were just a square that only moved left and right as objects came toward you, but now real-world details are slowly and consistently being added. To have common sense, a robot needs to “grow up” in a rich environment, and as the virtual worlds become more real and detailed, our AIs will get smarter.
I currently can’t kill zombies behind my local Arby’s, and my robot assistant can’t yet run amuck at the neighborhood grocery store. MSFS shows the correct roads, but it only renders generic buildings in areas outside of major cities or well-known attractions. We can do better. For each neighborhood, we could find the standard floorplans used when that neighborhood was built and generate from that. Using such realism brings up intellectual property issues. Should a company be able to scrape the real world to populate a virtual one? I hope so, because it serves a public good. Otherwise, our virtual worlds won’t be as fun. Trying to contact and pay every rights holder could be endless. One positive example is that Activision won a court case to depict Humvees without a license [5]. By contrast, according to [6], Sony removed the One World Trade Center (Freedom Tower) from its depiction of New York in its Spider-Man game, possibly because buildings can be trademarked. Another issue is privacy. If I own a Honda Accord, can a virtual world show that in my driveway? What about my poorly maintained lawn [7]? Although privacy could get messy as well, we can see that Google already shows my front yard with Street View.
I’ve always wanted a Google Street View with a better interface so I could explore new places, and MSFS moves us toward a Street View where you can fly. MSFS is an inflection point for virtual worlds because these aren’t just video games anymore; they are mirror worlds [8] where we can enrich our life experience, and robots can approximate what it is like to be human. There’s the old idea that the entirety of our world is merely a flight simulator built by a more advanced civilization. Maybe that idea subconsciously contributes to our drive to understand the universe, so we can outgrow our sandbox.
References and Comments
[1] https://www.theverge.com/2020/8/27/21403769/hurricane-laura-microsoft-flight-simulator
[2] They should have to pay me $20 every time they fly over while I’m trying to enjoy nature.
[3] Alison Gopnik, The Philosophical Baby: What Children’s Minds Tell Us About Truth, Love and the Meaning of Life, 2009.
[4] Video flying over Austin (with a better computer than my son’s) ending with sitting on the surface of water https://www.youtube.com/watch?time_continue=367&v=-XP5RymsqK0&feature=emb_logo&ab_channel=Squirrel
[5] https://www.theverge.com/2020/4/1/21203414/activision-wins-humvee-call-of-duty-trademark-infringement-lawsuit
[6] https://www.stevenbuccini.com/why-1wtc-isnt-in-spiderman with Hacker News discussion https://news.ycombinator.com/item?id=24502706
[7] When my parents came to visit once, my mom commented, “It looks like nobody lives here.” I tried to explain that my preferred aesthetic is “apocalypse chic.”
[8] David Gelernter, Mirror Worlds: Or: The Day Software Puts the Universe in a Shoebox…How It Will Happen and What It Will Mean, 1991.