From player to maker

I just turned 30. As a kid in the 90’s and 00’s, I grew up alongside the most iconic gaming consoles created. Think Gameboy Original, Color, and Advance. SNES, N64, GameCube, and Wii. Playstation One, Two, Three. The list goes on.

That just totally dated me for all younger folks in the program, with their PS5s and Oculus Rifts and whatnot. Honestly, my last major foray into console gaming was the Nintendo Switch a few years ago, which is my dedicated Zelda and Mario Party device.

All that long while, these consoles were magic. Real, tangible portals into other worlds, where a kid could be or do pretty much anything. So many hours of my youth were spent immersed in these worlds. But, ultimately, it was always beyond my horizon at the time to comprehend that some real human like me was out there somewhere building and creating these things. I knew they were people making games, but these weren’t ordinary mortals in my kid eyes. They must be superhuman. Geniuses, even.

Our team is building a game in Unity3D for our capstone project. Despite that whole spiel, this is actually the first time ever touching game development in a serious, legit kinda way. Not even in the two years I’ve been pursuing my computer science degree. In a lot of ways, computer graphics and game development are the last foggy mystery that really feels clouded in my mental model of the computer science universe.

ELI5: Graphics and Games

Part One – How to go from Steam to pixels

Let me try to fill in an “Explain Like I’m 5” rundown of how graphics and games fit into the bigger picture of computer science. We’ll take a top-down dive starting from a newly released game.

The game you just launched on Steam is just a program like any other running on your computer. The same goes for the game you launched on your shiny new Playstation or whatever. It’s just running on a purpose-built OS and hardware specifically designed for that product. Either way, the program is just a process running on the OS and competing for time slices on the CPU alongside all your Chrome tabs and everything else (we’ll get to the GPU in a minute). Since the game is just a program like any other, it’s really a binary executable (read: machine code) that was compiled from source code. Again, just like any other program. That source code was typed out by one or more humans, possibly in a variety of different programming languages, possibly even more than one. The most common languages used for typing in source code for games are C++ and C#, but venture out a little farther and you’ll certainly see a fair amount of Javascript, Python, and other exotic choices, particularly out on the web – anyone from the late 2000s remember Flash games?

Here’s where graphics comes into play. The what in what the source code is programming is pretty different from your run of the mill square root calculator or binary search program for school. The whole point of a video game, after all, is that we can represent an entire 2D or 3D world virtually by drawing things on a screen. What’s really happening, in fact, is that we’re drawing a whole lot of things over and over again, really really fast, like 60 or 144 times per second (that’s frames per second or FPS for game nerds). By drawing, what we really mean is that for every pixel on your screen (of which there are tens of thousands or millions) we are assigning a particular array of numbers that corresponds to a color value we want that pixel to be. Again, we’re doing that for millions of pixels, over and over again hundreds of times per second. All the while, we’re changing what colors (really, what numbers) to send to the pixels on the screen based on a dizzying amount of math, all of which really boils down to figuring out the locations of a bunch of triangles representing the things in our game in 3D and translating all that to how it should look on a flat 2D screen. This is a lot like how the Rennaissance artists figured out linear perspective and managed to make real, three-dimensional representations of their world on the flat canvas. Except the computer needs math, not intuition. On top of that, we’re changing all those calculations on the fly based on all kinds of game world logic – think of input from the mouse and keyboard, or the numbers representing the reality of our game world – hearts, attack points, coins, so on and so forth.

The sheer amount of data being passed along to the screen all at once and so constantly is rightfully overwhelming. It’s possible for the CPU to do all that work, and in fact a lot of CPUs are built with graphics in mind, but really beyond basic environments like your Word doc and Excel sheets they tend to feel overwhelmed themselves and overheat. It takes a whole heck of a lot of math to calculate precise locations of all these triangles in order to make Hyrule look and feel real in Breath of the Wild.

That’s because CPUs are really great for sequential processing, but what we really want and need for graphics is for a lot of things to happen in parallel. The best way to do that is really through the GPU, which is a kind of processing unit (the PU bit) that is purpose-built for performing massive amounts of the same kinds of computations over and over again in bulk. To do that, the whole architecture, the circuitry of the hardware needs to be different. So what we end up with is a system where the CPU off loads all of the heavy, expensive graphics processing to the GPU over a big fat data bus and let’s the GPU do all of that work to make the calculations happen and push the data over your HDMI cable to your screen.

If your head hurts, don’t worry, there really is a lot of complexity here and honestly, it’s a marvel of human innovation that graphical computing exists at all. It really is that amazing. But like everything else in computer science, we’ve devised layers of abstraction to make tackling building graphical programs easier than pushing physical bits over the wire to the pixels on the screen.

Next time, we’ll take a deeper look at how computer scientists and programmers have devised layers of abstraction that help to make sense of and manage the complexity in the gargantuan task of placing three-dimensional, visual worlds onto your screen. We’ll see how frameworks, followed by game engines, bring us back up from pushing pixels to the realm of high-level graphic rendering and game development.