The most realistic simulations today can be found in the latest computer games on PC and game consoles such as Playstation 5 and Xbox Series X. Triple A games like Cyberpunk 2077 by CD Project, Red Dead Redemption by Rockstar Games or Star Citizen by Cloud Imperium Games typically require thousands of years of work by a large project team of artists, software developers and game designers and cost well over 100M euro to bring from concept to release. Wikipedia list of most expensive video games to develop.

If we aim to design a true simulation that satisfies the requirements we set in the previous chapter, we need a high-level conceptual design and the most logical and obvious place to start is to understand how next-gen games are developed and in particular the game engines that power these games.

The term game engine came into use in the mid 1990s with the popular Doom game. Doom made a clean division between the game mechanics, the physics, collision, animation, rendering and all the game level design and artwork. Game engines enable game designers and developers to concentrate on the actual game itself and lower the actual cost of development through reuse of common compontents.

Modern game engines such as Unreal, Unity, and CryEngine being used by thousands of games on all kinds of PC, console and mobile platforms. In our high-level conceptual design in Figure A we therefore make a distinction between the multiverse simulation (game) and the multiverse simulation engine (game engine) that together are the run-time simulated reality (the game running on your PC or game console at home). We will discuss the simulated reality system in section x.1 in more detail.

The simulated reality system itself relies on a primary support system that provides all the functionality that is directly needed by the simulated reality to operate. This includes the software, hardware and infrastructure organizations that develop and improve the simulated reality but also the organizations that ensure the multiverse remains operational both from a technical point of view (e.g. provide computing power, telecommunications, electricity etc.) and economic, political, cultural point of view (e.g. open marketplaces, democratic governement to set laws, law enforcement etc.). We zoom into the primary support system in the second half of this chapter.

The simulated reality and primary support system together form the simulation platform (for lack of a better word) that is part of a broader secundary support system. This secundary support system contains the part of the global economy that is not directly related to providing commodities, goods, services and experiences to the simulated reality. This includes agriculture, transportation, manufacturing, construction, financial services, retail and more. The secundary support system also includes non-profit, public government services such as healthcare, education, utilities, law enforcement and protection and defense against external threats such as earthquakes, floods and meteor collisions. We will not describe the secundary support system further in this chapter but touch upon its implications and changes in next chapter as we discuss the benefits, opportunities and threats of the simulated reality.

Simulated Reality

A computer game can be defined as a soft real-time interactive agent-based computer simulation. If we look closer at the individual terms in the sentence above we get a better understanding of what a game actually is. The term computer simulation refers to the imitation of (a part of) our reality over time by a computer program. The simulation is agent-based because the simulation mimics the behavior of both real and virtual agents; real people represented by their avatars and virtual characters, people, animals, robots that operate.

according to their internal logic encoded in the simulation. A game is interactive because we can affect the way the agents in the computer simulation behave according to the actions we take. The term real-time means that the computer simulation runs at the same clock speed as the clock on our wall. Time in the simulation happens in small fixed discrete steps, the more steps in a second, the higher the frame rate of the real-time simulation and the more smooth and realistic the game will feel. The time criticality of games has severe implications for the design of the game (engine) as we will see shortly. The term soft real-time applies to the fact that games can occassionally skip a frame if they cannot complete the computation of a scene in the time interval.

In Game Engine Architecture, Jason Gregory lead software developer at Naughty Dog Inc. the game studios that developed game titles like The Last of Us and Uncharted on Playstation, Xbox and PC platforms, presents a system architecture view of the components of a state-of-the art game engine. It is beyond the scope of this post to provide all the necessary background to develop a game engine. For readers who are interested to learn more about how a modern game engine works in detail, I can really recommend this excellent book. The focus of this chapter is to summarize each component to give you a basic understanding of the function of each component and explain the additional improvement we need in order to satisfy our requirements.

Hardware, networking, and operating system

The hardware layer provides the physical computing platform that is required to run the simulation. Today’s hardware platforms such as the PC, Xbox, Playstation and mobile System-on-a-Chip (SoC) semiconductor products that are used in Android and iOS smartphones all implement the so-called Von Neumann architecture. In this Von Neumann architecture the processor (CPU) takes a central position to perform calculations. The CPU communicates with memory to read and write instructions and data. The CPU also sends external input and output to devices such as mouse, keyboards, speakers, monitor. Today hardware platforms use multiple CPU cores to perform more calculations in parallel and use special Graphics Processing Units (GPUs) to render 3D graphics which are comprised of billions of polygons efficiently. Offloading graphics processing to high-powered GPUs make modern gaming possible. For a good introduction to modern CPU architecture and more in-depth information see Architecture All Access: Modern CPU Architecture Part 1 – Key Concepts – YouTube.

A true simulation will demand much more from the hardware layer. The hardware layer will develop further by adding more transistors on a chip, special purpose chipsets to optimize computational expensive operations and new forms of computing.

Since the 1960s when Gordon Moore made the prediction that the number of transistors on a chip would double every year (in 1975 he adjusted this rate to every two years), Moore’s Law has held up until today. Although scientists have claimed the end of Moore’s Law, the semiconductor industry and chip makers such as ASML have always been able to push Moore’s Law further. Intel considers 3 ways to increase the amount of transistors further: First, using the traditional approach of shrinking transistors and placing them closer together which should triple the density. Second, new transistor designs that continue the current transformation of transistors from flat circuitry elements into 3D structures. These steps, called nanowires and stacked nanowires, should quadruple density. The third step are packaging innovations which chips stacked into a layer cake of processor elements (quadruple density). How Intel will keep Moore’s Law cranking for years to come – CNET

Besides advances in GPU chipsets following general CPUs, we can see other specialized chipsets to run computational expensive operations in a cost-effective fast manner. AI chips are being developed to enable cost-effective fast deep learning artificial neural network applications for Internet of Things devices that have limited access to the Internet AI chips in 2021: Guide to cost-efficient AI training & inference (aimultiple.com). For a true simulation, the cost of updating the hypergraph containing trillions of objects and their relationships may similarly lead to the development of optimized chipsets.

The third development that will push the hardware layer further are new computing paradigms such as quantum computing. Quantum computers exploit properties of quantum states such as superposition and entanglement to perform computations. Quantum computers will break widely used public key encryption, solve certain optimization problems faster like optimization in data analytics and by simulating molecules and materials more efficiently which aids in the understanding of diseases and the design of better medicines. Quantum computing (tudelft.nl).

The network layer builds upon the hardware layer to exchange of data between computers across telecommunication networks. The wired or wireless network layer implements the TCP/IP protocol to communicate with other PC, game consoles, mobiles and Internet of Things devices and has enabled the emergence of massively multiplayer online games like Doom, World of Warcraft and the emergence of games as a service. Today various business models exist to turn games into services; monthly subscription services to access a large library of games (e.g. Xbox Game Pass, Playstation Now), micro-transactions (e.g. Roblox, Grand Theft Auto Online) and more recently Cloud Gaming (e.g. Google Stadia).

The true simulation will push development of the network layer further. The trend of cloud gaming will continue because of the high demands placed by the simulation engine components on the physical computing infrastructure. From our homes, offices and the public space we will interact through the human input/output device system. Advances in wireless networking technology and mobile telecommunications will continue to enable end-users to roam freely without cables, easily install new I/O devices and provide the necessary high bandwidth, low-latency connections required. Satellite internet providers such as Starlink will provide end-users fast internet access to remote rural areas anywhere on the planet. Optical fiber networks are widely rolled out by telecommunications providers, but increasingly also electronic routers and other networking equipment that is now still electronics will be based on photonics, i.e. photons instead of electrons to carry information at the speed of light. Quantum internet Quantum Internet – QuTech will enable sending and receiving information using quantum bits (qubits) that follow the rules of quantum mechanics. One application of the quantum internet will be a fundamentally secure way of communication in which privacy is guaranteed by the laws of physics. We return to security and privacy in 2.2.8.

The driver layer enables the operating system layer to interface with the hardware and networking layer. Whenever you install a new peripheral device such as headset, mouse or printer, the operating system looks for so-called software drivers to control the hardware device. The operating system manages the computer hardware, software resources and provides common services for computer programs such as memory allocation, disk access, interrupts, security, networking and user interface to start and stop programs.

In a true simulation, the operating system layer will become less visible than it is today. Instead of consciously switching between applications as end-users we will see the game engine as the way we accomplish our work and activities in the simulated universe and forget about the underlying OS. The simulated universe will provide a context-dependent, task-dependent private or shared user interface wherever and whenever we need it. Also, when we switch between universes, we likely switch between cloud streams instead of applications running on our PC or smartphone in the future.

Low-level engine support system

The low-level engine support system provides a range of software support services, middleware and APIs that are used by the game engine components.

In game engine architecture this can be further broken down into a 3rd party SDK/platform dependent layer, platform independent layer, core system layer and resource manager layer.

The low-level engine support system will ensure an eco-system of software, hardware and networking system and service providers can be plugged into the game engine central system to allow specialization and avoid lock-in in a specific hardware or software operating system.

The low-level engine support system is also the place to manage security, privacy, integrity and other quality attributes of the game engine because it provides a single point of control.

3rd party SDK/platform dependent systems

On top of the software operating system layer 3rd party Software Development Kits (SDKs) offer Application Programming Interfaces (APIs) and software libraries that are optimized for the hardware and operating system layer these include:

• Data structures and algorithms optimized for different hardware and operating systems. In a true simulation we may see quantum algorithms optimized for specific hardware platforms or compilers that automatically translate a classical program into a quantum program.
• SDKs for graphics (DirectX, OpenGL) and audio rendering support game engine developers already. In the future touch, scent and taste rendering SDKs may complement graphics and audio so that game engine developers do not need to worry about optimization for a specific touch suit or scent system.
• Collision detection and rigid body dynamics (physics). Havok Havok and Nvidea Physx NVIDIA PhysX System Software are examples of companies that offer software products that help reduce the effort to implement a physics engine. A true simulation physics engine will have to approximate physical, chemical and biological processes more precisely ranging from the quantum, nano level to the world we experience with our senses all the way to general relativity to describe the laws of physics on a large scale.
• Animation packages such as Granny (Granny 3D (radgametools.com) provide an animation studio for artists to build characters with complex animation behavior and make it easy for programmers to export, process and animate what artists have produced. In the future animation will be much more procedural generated to speed up the work of artists and developers using AI techniques.

Platform independent layer

The platform independent layer makes use of 3rd SDK middleware layer and OS services and provides an abstraction layer so that the game engine can run on different hardware / software platforms. In a widespread simulation it can be expected that billions of users have millions of configuration of hardware, networking and operating systems. The platform independent layer is necessary to support plug and play of new human input / output devices and enable previous generation platforms to continue to work.

The platform independent layer (and core system layer) may be open source to enable inspection and transparency by the primary support system (government) of what the game engine reads as input and writes as output in a way that is independent of the way it is implemented on different hardware/software platforms. Examples are file systems, networking, graphics and physics wrappers that use the 3rd party SDK/platform dependent and OS layer.

Core systems

The game engine core system provides enabling services for the game engine. It builds on the platform independent layer to provide a range of basic functions like module start-up/shutdown to initiate game engine components in the correct order, math libraries for e.g. collision detecting/physics, animation or rendering, memory management to load and unload parts of game world, profiling to test performance and debugging to test animation sequences or the behavior of agents and much more.

The core systems layer of a true simulation will include the functionality describe above but likely also include a host of other services of which we want to mention two. First, the simulation must provide the physical and mental welbeing and safety of the observer at all times. The core system needs to interrupt the game engine when the safety of the observe is in danger by disconnecting from the simulation in a graceful way. Second, the simulation will have to protect the privacy of users and ensure connections are trusted and secure for exchange of information. The core system layer may use blockchain technology to ensure the authenticity of virtual objects and quantum computers to make private connections unbreakable by hackers, governments or companies that want to exploit this personal data without the consent of the user. In general the rule of thumb will be that critical functions to protect observers from physical or mental damage are handled by the core system layer to interrupt the engine.

Resource manager and resources

The resource manager layer in a modern game provides access to all the in-game objects and their properties. This includes 3D models, sound clips, skeletons, textures, materials, behaviors, collision shapes and the game objects that use these properties. The resource layer also contains the world maps and world properties.

The resource manager provides a unified interface to all resources in the simulation. Most of the resources in our simulation will not exist as traditional game objects with associated 3D models, sound effects, textures and materials etc. but as procedures that when computed will give rise to game objects that exist in the simulation. Resources also include the world map and settings that describe the environment in which the agents both human and artificial interact. These environment resources will also exist more as algorithm than as an object. The set of resources loaded into the simulation at any specific point in time determines the next state of the universe. The resources may perform operations on themselves and other resources so that they load or unload themselves into the simulation based on their behavior and the input of human observers. The resources may be cached and stored electronically close to the user for temporary storage or written in DNA to be stored for hundreds of years in permanent storage, archived until one needs to read them.

The resource management layer will rely on the privacy, security and integrity provided by the core system layer to prevent that users can experience (parts of) universes or objects they have no priviledges to. The government system which we describe in next section must be able to inspect resources to monitor for unsafe or illegal actions under strict rules and rely on the resource management and core system layers. Making modifications in these two layers is conceptually similar to making adjustments in the constitution and needs to be highly controlled.

With the basic infrastructure in place, we can now turn our attention to the most important process in the game engine, the game loop that brings reality into existence.

Computation clock system – The game loop

Games are real-time simulations as we saw earlier in this post. Time in the simulation happens in small fixed discrete steps, each time interval is called a frame and generally, the more frames per second, the more smooth and realistic the game will feel. In the digital physics theory we discussed how time itself may seem continous at larger scales but when we zoom in we find that spacetime is granular.

The core process in a game engine is the game loop that runs in sync with the clock speed set by the game engine. To have a smooth game experience, modern game engines need to operate at a frame rate of 40-60 frames per second (FPS). Within this time interval the following steps are typically executed:

1. Reading the human input device system to collect the actions performed by the player through their input device(s)
2. Updating the game state to determine the location and other properties of each game object at the end of the time interval.
3. Calculate the motion to determine how game objects will reach the target state according to the laws of the phyiscs engine
4. Detecting collisions and resolving the collision to determine the final location and properties of each game object.
5. Rendering the scene from the point of view of the game player in a frame buffer and audio buffer that can be displayed on a digital screen and played back over speakers.

The game loop is the most time critical aspect of the game engine and therefore game software developers take great care to ensure the game loop is highly optimized. Modern game engines exploit both the parallelism provided by the underlying hardware and networking layer and the multithreading support offered by the operating system layer. By running these calculations in parallel at the end of every time interval a frame buffer is completed in time. In massive online multiplayer games, game state updates have to be performed multiple times in a game loop so the central server often runs at a lower clock speed so the clients can achieve consistent high frame rate.

If we assume an advanced simulation will use a similar game engine architecture, it will also have to implement a game loop. It may be hard for us to imagine, but imagine each of us implements its own game engine in our brain that performs this game loop while we are conscious and creates the universe we observe. Through a central server the game state is shared so we all participate in one giant simulation. Our brains are wired to be conservative with energy so our mind will only calculate what is needed, what is seen. While the number of calculations required for a frame would exponentially grow larger if we moved from macroscopic to molecular chemical reactions and even smaller scales, the real question is for whom this is needed and why. If we would want to experience the quantum level, we could reduce the clock speed of our game loop and scope so we can observe the simulation at quantum level, if we want to experience the simulation on cosmic scale we could increase the clock speed and scope. Nobody of us has ever directly experienced the fast chemical reactions, or effects in a particle accelerator that smashes sub-atomic particles, we built machines that slow down the clock so we can observe these interactions. It does not matter whether this is real or not, the question is whether we would engineer reality this way if we build a true simulation. Let’s assume we design our simulation this way, depending on the clock speed we choose we would have to change the sub-system configuration to ensure we use the quantum collision detection module and rendering engine to render reality at quantum level.

The control of time in the game loop would give us superpowers, as we could also use it to slow down time in a car race so we can observe obstacles or speed up time so we can travel faster to the stars. We could pause time and see the scene from different points of view like Neo in the movie the Matrix. This would be similar to profiling and debugging the game running in a game engine – we come back to this later. We could rewind time and move backward in the simulation or restore a saved game in memory.

Human input device system

One of the first steps in the game loop in games involves reading out the input device values. Today games use a combination of discrete and continuous input received from devices. Keyboard, mouse, game controllers give discrete values like a specific key or button pressed, continuous input is typically given by touch sensors, motion recognition sensors (like Xbox Kinect) and joysticks. The human input device system in a game engine captures and processes these events. For some devices only the changes in the state need to be communicated to systems in the game engine that want to receive notifications (event listeners), examples are for example mouse events. For other types of devices like keyboards all events must be captured. In case of devices with high update frequencies like there may be some many events per seconds that the human input device system must poll the device at a certain frequency to avoid the event listener is overloaded with calls.The human input device system is typically in close proximity to the user as a wearable, handheld device, or sensor in the same room and builds and maintains an accurate and up-to-date representation of the state of the observer in the real world that can be use in the virtual simulation.

Human input system

The more advanced the human input device system, the more control observers of the simulation have over their avatar in the simulation. Compared to the keyboard and mouse I am using to type this sentence, a true simulation will have put almost no constraints on the way we can interact with the simulation. If we move our body, arms or legs, it will detect the gestures we are making and transform this into walking, swimming, jumping or other activities. For this we need technologies like omni-directional threadmills that keep us fixed in a place while we are walking, running, or siting down. With our fingers we can type on a virtual keyboard, feel an object by applying pressure (and feel the response to determine if it is a soft or hard object for example) and play a virtual violin maybe. To implement this we need high-resolution touch gloves that can track complex finegrained behavior. High-resolution touch input will also be provided by sensors we wear on our feet or sensors for sexual pleasure. If we use our voice, not only the words and sentences are detected, the emotions in our voice may be sampled so the simulation can understand not only what is said but also how it was said. This will be enabled by improvements in microphone and sound processing technology. Not only explicit input derived from the things we say and do, but also implicit body functions such as our blood pressure, temperature, feelings of hunger or pain are captured and processed by specific sensors. This implicit input will increase the empathic powers of a true simulation; imagine a virtual character recognizing you just had a work-out or don’t feel so well today and adjusting its dialogue and expression accordingly.

A different way to implement an advanced human input device system would be through a brain computer user interface so that you only need to think about e.g. moving your arm to actually move your virtual arm in the simulation. The main advantage of a brain computer user interface is that you would only need one input device and not be surrounded by a cloud of input devices to track your motion and interactions with the real word. The disadvantage is that freedom of thought could be more in danger as the input device may also record your thoughts and emotions you do not want to share Mind-Reading AI Optimizes Images Reconstructed from Your Brain Waves – The New Stack. Brain computer interface technology is still in its infancy and nowhere near as precise as state of the art sensing and motion detection technology for example, but brain computer interface technology may also help solve to taste and smell rendering problem.

The human input device will recreate the existing way we interact with the world in virtual but also enable new ways to interact as we saw in the previous chapter. To be someone else, another living being or a collective consciousness, at our plane of existence or elsewhere, the human input device system will be the same but the routines to control our avatar will be different. We will have to learn how to interact like we once learned to walk or ride a bicycle. While this may seem strange and out of this world, gamers are very familiar with this when they need to control the avantar in a first person shooter or control an army in a real time strategy game.

Game state system

After the human input device system has collected the user input, we have to determine the game state at the end of the frame but in order to do this the game engine needs to retrieve the as-is game state so it knows the start state. The game state system therefore maintains a representation of the input received from all observers and the state of all game objects and its relationships with other game objects.

In a true simulation the game state system would maintain the dynamically evolving hypergraph that represents the entities and their relationships in the observers’ universe. Most game engines assume the dynamic and static entities are fixed, but in a true simulation engine the static entities of one level of existence are dynamic entities at a different level. From a distance, a water surface can be a static game object, but when we come close the static game object expands into dynamic game objects; we see the waves, the fishes and so on. To implement this, the game state system may updates the section of the hypergraph that is observable and use a probability functions on non-observable sections. Next time when we zoom and observe the waves in detail, the section of the hypergraph expands into the existence of fish game objects with a specific position, orientation and speed- very much like what the Copenhagen interpretation of quantum mechanics we discussed in the digital physics chapter earlier would predict. This expansion could run several layers deep so we could experience the fish biology, chemistry down to the atoms and sub-atomic particles.

The game state system will need to determine which parts of the animation, physics and rendering systems will be invoked. In today’s game engines there is only one system for each step in the game loop but in a true simulation we need a different physics engine to simulate the laws of physics on very small scales than compared to our normal daily experience in the real world. The animation of chemical reactions will be very different from the mechnical classical movements we see in the world we live in. How we would render the quantum or cosmic scale so our senses can experience is something we may even have to imagine first, but if we do it may increase our knowledge and understanding of these strange parts of universe by experience.

When the game state has been determined and the properties of the static and dynamic entities are known at the start of the frame, the location, speed, momentum, mass and other properties need to be determined at the end of the frame.

Motion and animation systems

Modern game engines implement a motion system to calculate the trajectory of a cannon ball, the movement of a car or particles that are blown by the wind. The motion system is closely connected to the animation system that determines how moving or static objects must be animated. The animation system is for example used to determine how virtual characters can move their arms and legs or how a car door opens based on the the game state.

Most game engines use skinned / skeletal animation. Characters, animals and other objects are modelled as skeletons which are a hierarchical set of interconnected parts (also bones and joints). The joints can move around their axis. By combining bones and joints (edges and vertices – again graphs!) complex human skeletons can be created from simpler skeletons such as arms and legs. A smooth triangle mesh is finally bounded to the joints to provide the skin of the character. Each vertex of the mesh can be bound to multiple bones so the skin moves naturally.

Every animation takes place over time, the animation comes to life by letting a character go through different poses that together in sequence let the character walk, run, swim, sit or jump in a natura way. A skeleton is posed by rotating, translating or scaling the joints. The pose of a skeleton is simply the set of all the joints poses.

Animation clips define the sequence of local or global poses a character or objects goes through over a period of time in a certain situation like falling off a cliff or behavior like moving the arm to pickup an bag. Characters can have thousands of animation clips because real time games are highly interactive and the right sequence is dependent on the users actions. Animation clips can be blended to combine several local clips into a complex animation. Before the skeleton as posed and blended by animation clips and with skinned mesh completely mapped is ready for the rendering engine, post processing steps takes place to e.g. adjust the final pose of the hand to the exact position of the target object (inverse kinetics).

It is beyond our scope to provide an in-depth discussion, for more detailed descriptions of animation systems see e.g. Learning Blender: A Hands-On Guide to Creating 3D Animated Characters: Villar, Oliver: Amazon.nl or The Art of 3D Computer Animation and Effects: Kerlow, Isaac V.: Amazon.nl

In an advanced simulation the animation engine system is not limited to characters, animals, the trees in a forest on a windy day or the waves of a ocean. When we zoom in to the molecular level and slow down our clock the animation system may define how the bonds between hydrogen and oxygen atoms change when temperature increases. If we zoom out and speed up our clock the animation system determines the poses of distant galaxies, stars and gas nebulae.

The animation system plays an important role in reducing the computational complexity. Even if humanity would build computers that are several orders of magnitude faster, it would be a waste of computing power to animate things in too much detail if we cannot observe at a particular level of abstraction. The animation engine system effectively prepares the dynamic entities in the universe for rendering.

Physics and collision system

The primary purpose of the physics and collision system in modern game engine is to determine whether any two objects have contact. In case objects have contact, the collision system must determine the change in direction and speed on the object, and update the animation so that a soft ball correctly bounces of the surface it hits or a ray of white light is scattered in its colors when it hits prism glass. The collision system keeps track of all the collidables, the game objects that can interact with other game objects.

In essence, each object in a game engine is represented by a geometric shape to describe its contact surface, which can be a simple bounding box, sphere or a complex convex or concave geometry. The geometric shape is rotated, translated and scaled together with the game object by the motion and animation system and can also dynamically change as the game object is modified.

Besides determining if two object make contact, the collision system must also support so called collision queries. Collision queries are asked by the game engine to the collision system to e.g. determine the first game object that is hit by a bullet from the point of view of the player or all the enemies in a certain radius around the player. One popular way to perform these collision queries is ray casting which draws a line segment between a starting point (e.g. the game object that fired the gun) and an end point (maximum distance the bullet can travel) and determines if a game object is on the line segment. Shape casting is used to cast e.g. a sphere around the object.

Modern game engines are concerned with the movement of objects over time and therefore the physics engine usually supports classical rigid body dynamics. Classical means that the motions are expected to obey to Newtonian physics to simplify computations. Rigid body dynamics assumes that objects are solid and cannot be deformed and this further helps to reduce computational complexity.

The collision and physics engine are often tightly integrated and even refered to as the physics/collision system. Typically, the following steps are taken:

1. The forces and torques applied on objects are calculated so their position in the next frame, time interval is determined,
2. The collision system checks if any new objects have collided.
3. The collision response is calculated taking into account the energy and angle of impact.
4. Constraints are resolved if objects are constrained in their degree of freedom of motion. This happens for example when a door is closed while there is a character in the door opening and the door has a hinge.

In a true simulation we need a digital physics engine like we outlined in chapter Reality and bits. The purpose of such a digital physics engine will be to compute the hypergraph that is the universe. The game state system describes the hypergraph at the start of the frame, the physics engine computes the hypergraph at the end of the frame and updates the game state system. Space time and gravity in the simulation emerges from the evolving hypergraph at the most detailed level. Forces, matter, light, energy, emerge on a higher quantum and atomic level, and through successive layers of abstraction in the hypergraph we eventually reach at the Newtonian classical physics engine for game objects that we are familiar with. On larger and faster scales still, we see Einstein general relativity physics becoming prominent. At MIT, research built general relativity in a game to explain and visualize the strange effects that happen if we would approach light speed http://gamelab.mit.edu/research/openrelativity/

Like modern game engines make assumptions like classical rigid body dynamics, a true digital physics engine may choose to approximate the hypergraph by making the physics / collision detection a function of the observer’s viewpoint as discussed for keeping track of the game state for the animation system. If we want to zoom in on the level of chemical reactions we need to speed up the game engine clock to ensure that as observers we can see these reactions as animations unfold. At the same however, the dynamic game objects at higher levels of abstraction will appear to be static and can be removed from the list of collidables to track. The digital physics engine may also skip computation of sections of the hypergraph completely if there is no observer and with approximations or probabilities instead of a fully computed graph. If there is an observer, the probabilities eventually collapse and the hypergraph is computed to a more detailed level. This principle may not only hold at the quantum level but any level, at cosmic level the position of the planets and stars in the simulation is determined when there is an observer. One of our requirements is that we can tweak the parameters in the physics system and see what happens in the universe if we change nature’s constants. The physics system therefore needs to be configurable at least at design time and these parameters affect the motion and physics engine.

It may well turn out that quantum computers as we discussed before are uniquely equiped to implement the physics system of a true simulation game engine in the hardware layer as superposition is a key feature of quantum computing. Hardware acceleration could also be achieved with special Hypergraph Processing Units (HPC) that excel in fast hypergraph computations as the game state system may need to scan quintillions (a billion billion) objects per frame, outperforming today’s supercomputers.

For more details on physics in game engines see e.g.

Game Physics Engine Development (The Morgan Kaufmann Series in Interactive 3D Technology): Millington, Ian: 9780123694713: Amazon.com: Books .

Game Physics: 9780123749031: Computer Science Books @ Amazon.com

We leave the topic of collision detection for now and turn to the final step in the game loop, the rendering step.

Rendering engine

When all the responses are calculated and constraints have been resolved, the game objects are in their updated position in the game world and the rendering engine must be called to render the scene so it can be experienced by the player.

The purpose of the true simulation rendering engine is to construct a holographic projection of the hypergraph that represents the state of the calculating universe at the current time frame so observers can see, hear, feel, touch and taste the simulated reality. We are going to look at each of the senses separately, starting with graphics and audio first.

The rendering engine in state of the art game engines is a complex software system that consists of a graphics and audio component. The graphics rendering engine system in essence boils down to the following steps:

1. The virtual scene is described in terms of 3D surfaces in some mathematical representation, typically through triangulation, a process of subdividing surfaces into triangle shapes.
2. A virtual camera is positioned and oriented in the virtual scene to define a viewpoint. In first person view the camera is positioned on the observer and oriented towards the direction the observer is looking, but the camera can also be slightly above the observer like in racing games or far above as in real time strategy games.
3. Virtual lighting sources are positioned and oriented in the virtual scene. Lighting sources are typically individual point sources, spot lights, directional light and ambient light levels.
4. Visual properties of surfaces in the virtual scene are described. These properties define how light sources will interact. Surfaces can absorb, reflect, transmit or diffuse lighting as light interacts.
5. An imaginary screen is placed in front of the camera to render the frame image, the pixels on your computer screen.
6. For each pixel in the imaginary screen the rendering engine calculates the color and intensity of the light rays that converge on the focal point of that pixel on the screen.
7. The pixels in the frame buffer are sent to the video card driver.

Getting lighting and shading to behave realistically similarly often required a lot of manual work to create light maps. The CryEngine 3 was one of the first game engines to introduce a real-time diffuse global illumination system so that developers could change the path of the sun or move around spot lights in a room to create the right ambiance. https://advances.realtimerendering.com/s2009/Light_Propagation_Volumes.pdf.

Similarly, game developers traditionally modelled surfaces with triangle meshes because triangles have specific properties that work well with real-time rendering. Triangles are the simplest type of polygons, triangles are planar (flat surface) and triangulation is supported by software drivers and specialized GPU hardware. Fixed triangulation can lead to blocky surfaces especially if the camera is close by so game engines use dynamic triangulation to increase the number of triangles as the camera gets closer. Unreal Engine 5 (https://www.unrealengine.com/en-US/blog/a-first-look-at-unreal-engine-5) Nanite virtualized micropolygon geometry frees artists to create as much geometric detail as the eye can see and supports dynamic triangulation without having to worry about importing geometry data, manage polygon counts and memory allocation or manually creating level of details (LOD).

For more details on graphics rendering see: Real-Time Rendering, Fourth Edition: 9781138627000: Computer Science Books @ Amazon.com

The graphics rendering engine in a true simulation must be capable of rendering frames with 16K resolution per eye and at 240 Hz refresh rate as we saw in chapter 2. To achieve these resolution and refresh rates future GPU may be in the cloud to do the hardware acceleration before sending the frame to the homes of people as we saw in 2.1.1.

The audio rendering engine system basically receives as input all the 3D sounds that can occur in the game world, either manually created or procedurally generated, and as output returns the sound over a number of sound channels that the observer would hear at her specific position in the game world. The 3D audio rendering system typically needs to achieve the following tasks:

1. Generate sounds that occur from events in the game world (sound synthesis)
2. Spatialize the sound in the game world to create the illusion that sound is coming from the proper location to the point of view or should we say hearing of the observer.
3. Accoustic modelling; like lighting, sound waves are absorbed and reflected by surfaces and substances and to create a realistic soundscape the audio rendering engine needs to determine the vibrations and cancellations.
4. Doppler shifting may be applied if the observer and sound source are moving relative to each other.
5. Determine sound volume over each sound channel (Mixing).

Readers who are interested to learn more on 3D audio engines we refer to Game Audio Implementation: A Practical Guide Using the Unreal Engine – Kindle edition by Stevens, Richard, Raybould, Dave. Arts & Photography Kindle eBooks @ Amazon.com.

A true audio immersion rendering engine will have to support the audio capabilities described in Matter of degree – immersion – Reengineering Reality.

For the true touch rendering engine the simple touch feedback through the PS or XBox game controllers will not be a good starting point and novel architectures, software and algorithms will have to be developed. Because our body shapes and how we experience warm and cold differ, the touch pattern must be customized to the observer of the simulation. To achieve true touch immersion we need to render pressure and temperature of high resolution in places where we have many receptors and lower resolution where we have few. Optimizations and hardware acceleration may be less critical as for true visual immersion but as there are more touch pixels than visual pixels to render. A true touch rendering engine will need to be efficient and choose to render a part each frame or render a full touch image at a lower frequency if we can’t feel the subtle fast touches and temperatures. Typical steps to render touch could be:

1. Use the collision/physics engine to determine the body surface of the avatar of the observer that collides with game objects.
2. Use the collision/physics engine to determine temperature, wind, rain and other weather effects that collide with the body surface of the avatar of the observer.
3. Calculate the combined touch pattern on the avatar body, optimizing for the differences in touch resolution.
4. Map the avatar body touch pattern to the observer’s body, scaling, adjusting the pattern to the body shape.

A true scent rendering engine is difficult to imagine but we give it a try. Scent is gas molecules that interact with our nose and create signal to our brains. The scent rendering engine may do steps like:

1. Generate scents that are emitted from objects in the game world.
2. Environmental modelling: calculate wind, temperature, humidity to determine the scent dispersion. Flow and temperature are Partial Differential Equations that are hard to solve. Deep learning techniques may help to solve PDEs and enable real-time scent dispersion modelling.
3. Determine the composed scent (composition and intensity of each scent) at the observer’s location (mixing the right scent).
4. Send the composed scent (mix) to the scent hardware unit in your home.

A true taste rendering engine would have to simulate the liquids and solids on your tongue including the touch sensations when you chew on a virtual banana or ice cream. It seems almost impossible to render taste virtually without a direct brain computer interface or creating an illusion of taste by using the other senses so the mind ‘creates’ the taste in our brain. The taste rendering engine would need to be tightly coupled to the touch engine and scent engine to create the sensation of drinking or eating something:

1. Calculate the substance touch response on tongue and teeth.
2. Determine the temperature sensation.
3. Compute the taste stimulus.
4. Calculate the combined (person independent) response
5. Map combined response to brain signal.

The true simulation rendering engine needs to synchronize the visual, audio, touch, scent and taste rendering to achieve true immersion through the devices that render the multisensorial experience for the observer at home. The five rendering engines need to work in parallel to achieve this synchronization of frame level.

For true scent, taste and even touch, brain computer interfaces may turn out to be a better solution to render scent, taste and touch than through our senses because of the engineering challenges for the human input device system. Especially the combination brain computer interfaces with deep learning algorithms could turn out to be promising to see how brain signals could be turned into imagery. Deep learning algorithms trained to recognize scents by comparing our brain signal with the stimulus and then recreate these brain signals to create the illusion of scent. This would enable people who lost their sense of scent to smell the real fragrances in front of their nose and have people experience fragrances that do not exist in real. However with this mind control comes significant privacy and security concerns.

Before we discuss the local rendering devices as part of the primary support system, we need to discuss the actual simulation content, the game objects and their behaviors. All the systems we described sofar, no matter how sophisticated they are, only become meaningful through the simulation content they process.

A game is defined eventually not by the human input device, game state, animation, physics and rendering systems, but by the game rules, the goals players can achieve, the abilities players have, the non-player entities and other human entities and the overall flow of the game.

Gameplay foundation system

Most modern game engines have a set of run-time components called the gameplay foundation system to construct the game rules, objectives and flow. If a line can be drawn between the game engine and the game itself, the gameplay foundation system is just below this line.

The gameplay foundation system in modern game engines provides typically:

• Runtime game object model: The runtime game object model takes care of e.g. creating and destroying game objects, realtime simulation of object behaviors, link to low-level engine systems, finding game objects, network replication to ensure game state is shared, saving and loading state to pause and resume at a later stage.
• Level management and streaming: the game world is divided into game chunks that are automatically loaded (and freed from memory) when the player is in a particular section of the game world. A game level can consist of several game chunks is the size of the level is large.
• Realtime object model updating: game objects must be able to behave autonomously and periodically updated to bring the game to life.
• Messaging and event handling: game objects must be able to communicate with each other by sending messages and responding to events to trigger behaviors.
• Scripting: to reduce the threshold to program the high-level game behavior, scripting languages such as Python or Lua instead of general purpose programming languages such as C or Java are often used so non-programmers can write the game logic themselves and speed up development time. The scripting engine is integrated in the game engine.
• Objective and game flow management: tracks the player’s mission progress and controls the flow of the game based on the player’s progress.

For a true simulation, we can expect a number of changes:

• Runtime game object model/updating: With billions of simulation objects that can emerge and disappear, the runtime game object model will be implemented by low-level system that is optimized for this purpose and the physics engine system. The gameplay foundation system will provide an interface to the functionality that is needed for the world editor system to create and manage procedural generated content / artificial intelligence.
• Level management and streaming: The ability to divide the universe in observable sections based on the location of the observer will remain important to avoid loading unnecessary objects. State of the art branch prediction algorithms that are also used in CPU architecture will be deployed to predict which parts of the universe must be loaded into memory to reduce computational effort and save bandwidth.
• Scripting: Designing behavioral scripts for all game objects will be virtually impossible, especially considering how game objects are generated from algorithms. In Matter of degree – world size – Reengineering Reality we discussed how games like No Man Sky use procedural generation to create galaxies, planets, sky, land, water, plants and animals from algorithms. The gameplay foundation system will be much more procedural like NMS and force simulation designers like we saw in chapter 3 to design algorithms that generate game objects and their behavior. This will also require new world editor tools, which we will discuss in the next section.
• Objective and game flow management: Specific universes may have objectives that must be completed by observers like a global climate emergency or the emergence of superintelligent AI, but most universes will be much more open ended and do not have strong flow management system. Observers find meaning in the universe as it calculates itself and the interaction with other humans and AI into existence.

In sum, we can conclude the gameplay foundation system will be more dynamically generated at run time and this will profoundly change the way we design games as we will see next.

World editor and tooling support

All commercial game engines have a world editor that enables artists, game designers and developers to test and debug the gameplay logic at design time and often even in real-time. In some engines (e.g. Unreal) the world editor builds on top of the gameplay foundation system and calls the same systems and functions, other engines like Unity have a separate stand-alone world editor tool. Regardless of this architectural decision, the world editor system provides the following services:

• World creation and management: create terrain, clouds, structures and buildings, and other static entities and save them in world chunks.
• Import content from the content asset pipeline – a collection of tools to turn artwork by designers into content suitable for the game engine – and assign them to static and dynamic entities.
• Game world visualization: provide a three dimensional perspective view on the world so authors can see how the game objects look like.
• Game world navigation: change the camera perspective so the author can see the world chunks from all possible angles and distances and move around in the visualized world,
• Game object selection and layering: the primary purpose of a world editor is to select static and dynamic objects and world editors typically enable authors to assign objects to layers to quickly select or deselect the light objects for example and turn the effects on/off to see the difference.
• Game object property grid: each game object has many properties such as it is coordinates in the world, its class or type, the collidable, visual characteristics, audio clips and so on that can be inspected and changed,
• Game object placement and alignment: the world editor enables authors to drag and drop objects, scale, rotate and align objects.
• Special game objects – lighting can be added and moved dynamically to create the right ambiance in a room or path of the sun, particle emitters to make flames or water flowing.

Most modern game engines make use of a virtual world model that consists of two main categories of game objects: static and dynamic game objects. Static game objects include the terrain, water, air, infrastructure like roads, bridges and buildings, everything that does not move. Dynamic game object include avatars, non player characters, animals, cars, planes, destructable objects or objects that can be picked up like health packs, guns, keys, anything in the foreground. The ratio dynamic vs static entities determines the degree and depth of interaction with the world. Early racing games only had a race car and the road and other competing race cars would float under the players car. Modern games get their realism from many game objects that can be interacted with. Still, this often feels very restricted: You cannot go to the supermarket to buy food and prepare it in the kitchen of the players house in Grand Theft Auto V because it was never part of the objectives of the game to cook food. Therefore increasing the ratio of dynamic game objects is necessary to achieve a high degree of realism.

In the early days, much of the game design work would have to be done by programmers because the tools available were not suitable artists. Defining game objects and scripting behavior required special programming skills which meant game software developers had to do part of the work that could be done by artists themselves. Data driven game engines today empower artists to define complex game objects and script behavior themselves which speeds up the game production process. With Unreal Engine 5 for example, developers can log in and instantly gain access to thousands of assets to use in their game, with Quixel Bridge developers can simply drag and drop an asset into the workspace and design large open worlds faster and with much less effort [6 Ways Unreal Engine 5 Will Improve Video Games (makeuseof.com) ].

In a true simulation we see at least three ways how designing games will change fueled by advances in deep learning / artificial intelligence and improvements in world universe editor system

• Game asset generation for game artists
• Game object generation for game designers
• Game interaction as writing for gamers

Art creation tools like Photoshop already employ AI techniques to speed up the work of artists by context-aware fill, smart object selection and sky replacement Photoshop: Now the world’s most advanced AI application for creatives (adobe.com). Game asset generation will further help artists to work more efficiently by use deep learning tools to create materials, textures, 3D models and other assets needed to design game objects. An early example is Dall-E DALL·E: Creating Images from Text (openai.com) which can create images from text using a state of the art machine language model so game artists can generate game art even faster. All blog images of Reengineering Reality are generated by AI. Human game artists will be complemented with artificial game artists. Nvidia Omniverse is a multi-GPU, real-time simulation and collaboration platform for 3D production pipelines that aims to create interoperability between different applications NVIDIA Omniverse™ Platform | NVIDIA Developer to further accelerate asset generation for game artists.

Game object and game creation can also be made more efficient and effective through deep learning. AI Dungeon AI Dungeon: Dragon Model Upgrade. You can now play AI Dungeon with one of… | by Latitude Team | Medium is a dungeons and dragons text adventure game that is generated of a deep learning language model which works surprisingly well. Try it out yourself. Next generation language models will enable game designers to enter “apocalyptic desert landscape with a desolated fuel station and abandoned cars” and the game objects and their behavior immediately appear in the 3D workspace of the game designer to start tweaking. Human game designers could create these queries for an AI, but with procedural generation techniques the algorithm itself could generate these queries to an AI and start to generate dynamically evolving game objects – parts of the hypergraph that make up the simulated reality. Human game designers will be complemented with artificial game designers.

Human game artists and game designers in turn will be more like producers, putting multiple AIs to work to help them build, review and test game assets and game objects. Although the work of artists and designers may be speed up by automation, the size and depth of the game world will extend to a vast universe so the job of game artist/designer won’t disappear, on the contrary, there will be much higher demand. The work will also become more creative as the AI will take care of the technical implementation.

The third, and most profound level of change is the boundary between authoring and running the game that will blur further and eventually disappear. In our everyday reality we don’t think about reading or writing, consuming or producing, we just interact and experience reality. When we observe our reality (read) we interact with the simulation (write) at the same time, millions of actors human and artificial simultaneously. In a true simulation we will all be game artists and designers!

Some of us will want to design experience sandboxes for others to explore like our world designer did in chapter 3 and some will choose to explore universes made by others, but even if we only explore the worlds made by others we would still write the simulation. We can pause the simulation and make large changes in the simulation as if we would download a world chunk and modify it before reimporting it back or we could make subtle changes in the simulation as it is running, but in both cases we would still be writing the simulation. The universe editor system has effectively become the interface to the simulation run-time.