I've posted a new video showing off my SSAO implementation. I think it looks pretty much awesome. Check the video further down. First though, I just want to post some of my thoughts about AO.

As I understand it, AO is often applied as a sort of post-process, combining a final render of the screen with the results of an AO pass by simply multiplying one by the other. This means the AO will darken the lighting contributions from all lights in the scene. The trouble, though, is that there's already a couple of things that do that: the standard lighting equation, which lessens the illumination from a light hitting a surface at a glancing angle; and shadows, which lessen/block the illumination from a light when there's something in the way. There's really no need to add an extra occlusion factor to the direct illumination.

Instead, ambient occlusion should be exactly that: occlusion of ambient light. Ambient light is a rough approximation of scattering of light, to give the illusion that even though a light is blocked by some object, some of the light has been scattered in the air and a small amount of it is still able to reach the shadowed area. Since this, in principle, comes from all directions, there is no easy way to cast a proper shadow for it, or adjust the illumination based on surface-light angle. Instead, the amount of light that reaches a point is proportional to the "accessibility" of that point, which is found by firing rays in many directions to see if other surfaces are nearby. This is what AO does.

It makes sense to apply AO to the ambient lighting contributions, rather than applying it to the ambient light and direct illumination from light sources. The end result is that it has much less effect where lights are shining directly on a surface, since the direct illumination far outweighs the ambient light. But, it still takes full effect where there is little to no direct illumination and only ambient light, and this is really where it helps. Especially with my game, where every light casts shadows, the direct illumination already has pretty much all the occlusion it could need (granted, the shadows could be a little more accurate, but whatever). Major crevices are darkened simply because (particularly with higher shadow quality) shadows are sampled at a fairly large range, so a point can easily be occluded. With some adjustments I could have even small creases get shadowed (way back in some of the early videos, I even showed this, the shadows were creating a pseudo-AO-looking effect). When there is little or no direct illumination, then the scene starts to look flat, and that's where AO really shines, showing definition where there is otherwise nothing.

So, that's my little rant about AO. It should occlude ambient light, not everything. Because only the ambient light is otherwise missing any sort of occlusion.

Now, watch the video! As always, you can see all of my videos on my Youtube channel.

So, I am using the FreeType library to generate fonts at runtime! Had to write a small wrapper around it so I can use it from my C# code, but it's pretty sweet. Each time I request a character for a given font in a given size, a cache is checked to see if that character has already been used. If it has, then no further work needs to be done. If it's not in the cache, then FreeType will render the character into a memory buffer, and then the character is packed into a giant texture containing all cached characters. While it could probably be better, the packing is pretty tight (it's not just based on a fixed size grid) and works plenty well for right now.

I don't have text rendering happening yet, but I did run a test on the cache to see how the packed textures are working. To the left you can see the results. This is every typeable character on my keyboard, in 4 font faces (2 fonts, with one of them having regular/bold/italic styles), sizes 12/15/18/21/24/27/30/33. The order the characters were added is as follows:

• Each font was processed in a fixed order;
• For each font, the various sizes were processed in order;
• For each size, the characters were added in a shuffled order.

I haven't done much testing with different cache orders, but I suspect it may be improved by doing larger fonts first. However, my intention is to just add characters as they are needed, rather than taking a big chunk of time up front populating with characters that may or may not be needed. When building this texture, the rectangles of each character is recorded so that drawing a string of text is as simple as drawing a series of quads, making direct lookups in the font texture. If needed, I may extend the system to allow multiple textures if it fills up. Or, I may allow it to remove characters that have not been used for a while. But, unless I find myself running out of space, it should be fine as-is.

I am planning to allow custom effects to be used when drawing text, with hooks for using more textures and such. This should make for a pretty powerful text rendering system.

If the texture packing system works out well, I am also considering using it (or something like it) to make my sprites a little more manageable. Instead of loading every frame of every animation for every type of sprite into video memory all at once (which is a lot, even using DXT compression), I would keep all that in system memory and then pack just frames I need to draw the scene into one or more (most likely more) dynamic textures. Basically, it would work similar to Virtual Texturing/Mega Textures, except likely simpler. Of course, that depends on how quickly I can copy data into a texture, since it will be happening quite frequently.

But, I decided to give it a go! In Blender, I can create "Empties" which are basically a non-object with a transform. I can display them as boxes, sets of axes, cones, spheres, images, whatever. By displaying as a box, I can use an Empty to mark out a rectangular area of space (which I've used for setting initial spawn locations for enemies). This works nicely for setting up the zones for a portal system; it limits each zone to a rectangular shape, but that's not so bad. In addition, I can use Empties to mark portals -- just make them thin. Again, it limits the portals to a rectangular shape, but again not a problem. With a couple custom properties set on each portal, I can have my level all set up for portal culling!

Then, in the game code, it was pretty easy to take those Empty markers, set up the relationships between portals and zones, and do the culling. Drastically lowers the amount of stuff I'm drawing in a frame, without having to go back to the occlusion culling stuff I had before (which didn't work out so hot). So now I can cull objects so less geometry is drawn, as well as lights, so fewer light passes need to be done. Got a pretty nice framerate boost!

I've put up a new video on Youtube showing it off. Check it out below, or on my Youtube channel.

1. Handy-dandy installer for the game. Run it and it should install everything you need (DirectX, .NET 4, XNA 4, etc).
2. New enemy. This is the subject of my most recent video.
3. Improved performance. I've made several large steps towards better performance in the game. Still not where I want it, and it uses a ridiculous amount of video memory, but it's getting better!
4. Yeah, that's about it... for now!

So, head over to the Guess What, You're Dead project page, and download the game!

Introduction

First off, I would like to welcome you to my first article type thing! I hope to write more, documenting some of the techniques I use in my various games, and this is a start. I apologize in advance if it is less-than-coherent! Also, I'm deeply sorry that there aren't any pretty pictures.

Okay, now that that's over with, a little background. I've been working on Guess What, You're Dead! for some time now. Until recently, I used a fairly simple forward renderer. For those who aren't aware, the basic process is this:

1. Make a list of everything that I wanted to draw.
2. Do an initial depth/ambient pass where I draw everything with a small amount of ambient lighting (constant, directionless illumination that helps prevent absolute blackness in poorly-lit areas)
3. For each visible light, do the following:
1. If it is a shadow-casting light, generate a shadow map containing all shadow casters within the light's range
2. Draw all geometry, applying lighting and shadows.

This worked pretty well but without careful culling of objects, it resulted in a lot more geometry getting sent through the graphics pipeline than was necessary. In addition, since I make extensive use of shadow-casting point lights all over my level, each light that gets drawn must have up to six shadow maps generated. A naive approach, using only frustum culling to eliminate objects and lights that were outside the camera's field of view, would easily result in drawing upwards of 800k triangles each frame for my small ~60k triangle level. My framerates were low and something obviously needed to be done.

Enter Occlusion Culling

Because of my level's setup, in any average scene a lot of the objects within the camera's frustum is hidden. Obviously, trying to draw something behind a wall is a waste of effort. So, I decided to add in occlusion culling to eliminate objects and lights that could not be seen. The game uses DirectX 9, so using occlusion queries with predicated rendering was out of the question. I do not like the thought of having to manually place portals or otherwise pre-processing visibility, so that too was out. So, I was left with two options:

1. Use a hardware occlusion query for each object, and grab the results during the next frame when they're ready, or
2. Get a copy of the current screen's depth information and compare an object against this to determine if it is visible.

Delaying visibility determination can have awful consequences, especially with lights. I actually did use this for lights at one point, and turning around quickly or turning around a corner would present you with a lovely dark patch before it realized a light was now visible. This was not acceptable to me, so I was left with one option.

With XNA, I do not read access to the actual depth buffer used for rendering, so I added an extra render pass where I would write out the distance from the camera for each pixel drawn. This would go into a small render target that I could then read into system memory, do some extra work, and then do occlusion tests with it. One immediate issue with this is that reading the render target data immediately after drawing the scene is going to stall the pipeline while the GPU catches up. I hoped that with a simple enough bunch of shaders, and limiting the drawing to specially-flagged objects that were most likely to occlude large parts of the scene, this wouldn't be a problem. Reading the depth information averaged about 7-10ms, which is not so good. I could do my occlusion tests very fast, but not fast enough to make up for the time taken to get the depth data. Even if I delayed the read until the start of the next frame, it still took a very long time, which leads me to believe that just reading texture data in general should not be done.

But all was not lost yet. I could write a simple triangle rasterizer to generate my depth information in software, avoiding the expensive texture read. And so I gave this a try. After a little searching around on the internet, I found what appeared to be a very nice bit of code for it. A short time later, it was implemented, worked mostly perfectly, but was still much too slow. I'm sure I could have sped it up, but it would have taken so long!

So then, what could I do instead of occlusion culling (as I've said, I do not want to put in a portal system or some complex visibility pre-process step)?

Exit Occlusion Culling

I deleted all of my occlusion-related code and decided to start fresh. The big problem seems to be that I'm drawing way too much geometry each frame. My level doesn't have very much itself, but because of how many passes were required (between the lighting passes and shadow map generation), it became prohibitively large. If occlusion culling is out, what else could I use to reduce the amount of drawing, as well as filter out lights that will have no effect on the scene?

Well, for the first problem, I have in the past written a deferred shading-based renderer. This has a different process compared to the forward renderer described above:

1. Make list of everything to be drawn.
2. Draw everything into several render targets, writing information like color, depth, normal direction, specular reflectivity, etc. This collection of render targets is known as a G-Buffer.
3. Render ambient into main render target, by reading the color data from the G-Buffer.
4. For each visible light:
1. If it's a shadow-caster, generate the shadow map as before
2. Light the scene by drawing a rectangle covering the area that the light affects, using the data in the G-Buffer to calculate the light contribution.

With this, and by drawing to multiple render targets in a single pass, the geometry is only drawn a single time. I could split it into two passes, where the first does not write to any render targets but just fills the depth buffer (which could speed the G-Buffer generation up), but in either case it's quite a bit less stuff being drawn.

Unfortunately, this was still not enough. It helped a little, but my framerate was still pretty bad. I was still generating a lot of shadow maps every frame.

Trading Space for Speed

It seems that my shadow maps may be causing my performance issues. So, let's look at how that works. I won't go over the actual shadow mapping algorithm here, since that is not relevant. I'll limit the discussion to generation of the shadow maps. For a single point light, it goes something like this:

1. For each face of a cube texture:
1. Create a view/projection matrix to view the screen from the light's position in the direction of that cube map face (±X, ±Y, ±Z).
2. If the frustum created by this matrix is not visible (does not intersect the main camera's frustum), skip this face.
3. Make a list of all shadow casters within the light's frustum.
4. Draw those objects into a shadow map cube texture, writing the distance from the light.

So the scene gets drawn up to six times for every light within the camera's field of view. Not good!

If you look at the level in GWYD, you'll notice that not much moves. The level is static, the plants don't move, the lights don't move, it's pretty much just the sprites and a few objects. So why should I have to re-draw all this static geometry when it was in the same place last time? What if I rendered anything that doesn't move into a separate cache cube texture, and instead of re-drawing those things every frame, just copy the contents of the cache into the main shadow map and re-draw only objects that could move? Worth a try!

Normally, I just have a single cube texture that gets reused for all lights (actually, I have several of various sizes, but that's not important). Now, I'll need each light to have its own dedicated texture in addition to the one I already have. Now my shadow map creation looks like this:

1. For each face of a cube texture:
1. Create a view/projection matrix to view the screen from the light's position in the direction of that cube map face (±X, ±Y, ±Z).
2. If the frustum created by this matrix is not visible (does not intersect the main camera's frustum), skip this face.
3. Copy this face from the light's cached shadow map into the current shadow map texture.
4. Make a list of all non-static shadow casters within the light's frustum.
5. Draw those objects into a shadow map cube texture, writing the distance from the light. Use a MIN blend mode.

Note at the last step there, the use of MIN blending. This is important since when we draw the non-static objects, the depth buffer does not contain any of the static geometry. The use of blending will slow it down a little, but that's a small trade-off compared to not having to draw most of the scene.

So this again helps out quite a bit, but I want to do better. I'm still processing lights that were on the other side of a wall and so have no impact on the rendered scene. How can I stop that?

Re-enter Occlusion Culling

Yes, that's right. Occlusion culling! That very thing that I claimed did nothing but harm up above a little ways! But, things have progressed since then, so maybe I can figure something out?

Okay, so the big problem with occlusion queries was that it would result in lights not showing up for a frame or two while the result of the query was made available. Well, what if we don't stop the light from drawing entirely? With the deferred shader, drawing a single light is not very time-consuming; it's just a matter of drawing a single quad, applying standard lighting calculations. Pretty much the only cost is from reading from 4 32-bit textures, but I'd be doing almost the same with the forward renderer to get color, do normal mapping, and possibly a specular/gloss map. So that's no big deal.

The big expense with the lights is generating this stupid shadow map every frame, copying the cached map and then drawing non-static objects. Maybe if I can avoid doing that for non-visible lights (or lights that have just become visible) the short delay won't be noticeable. As it turns out, this seems to be the case! So, when rendering my lights, it now looks something like this:

1. If this light's occlusion query is ready, get the results. The light is not visible if the query reports 0 pixels.
2. If it's a shadow-caster and is visible according to the latest occlusion query results, generate the shadow map as before
3. If the light's occlusion query was ready, start a new query.
4. Light the scene by drawing a rectangle covering the area that the light affects, using the data in the G-Buffer to calculate the light contribution.
• If the light is visible, use the generated shadow map
• If it is not visible, use the cached shadow map filled with static objects
5. End the occlusion query, if we started it.

We use the cached shadow map, so that the light will still cast mostly correct shadows for the moment between it showing onscreen and the game registering it as visible.

After making this change, my framerate is finally up closer to where I want it. I would like to still make it better, but I might have to look elsewhere (getting the list of objects to draw into the shadow maps is actually showing a couple milliseconds per frame, so I may be able to improve things with a CPU optimization).

Unfortunately, even with still applying shadows using the cached shadow map, there were some noticeable issues. For example, when the center room becomes visible, there was a short period where the walls would be red, since the falling platforms are not static and so do not block the red lights below from affecting the rest of the room.

Repurposing the Static Shadow Maps

The solution to this last issue was pretty simple. While still not perfect, it is a much better approximation with less obvious side effects.

When a light's visibility changes, I trigger a refresh of its cached shadow map. If it is no longer visible, I draw the entire scene into the cached map. If it is visible, I draw only static objects. So now, for that ever-so-brief moment between the light coming on-screen and the light's visibility changing, shadows are cast according to where everything was at the moment the light become hidden. This means the falling platforms will generally cast appropriate shadows, blocking the lights below from shining through.

As I said, it is still not entirely perfect. For example, if a large dynamic object is near the light when it becomes hidden, and is no longer there when the light is shown again, there will be a short time where that large shadow will show up. But, that's a trade-off I am okay with.

But Wait, Something's Missing!

The astute reader may have noticed that a not-so-small detail is missing here. With the deferred renderer, I'm drawing a quad for each light within the camera's view. I already have the scene drawn, so there's no depth buffering to allow things to block a light from view (and with XNA, I can't reuse the same depth buffer with a different render target without some trickery). Which means that the occlusion query should always report pixels drawn, just that they'll be black and not contribute any light. So then every light will be visible as long as it's within the camera's frustum. And in that case, we're no better off from the original version.

The easy solution is to discard, in the lighting pixel shader, any pixels that are too far away from the light to be affected. And it turns out this works quite nicely! This also has the effect of making hidden lights faster to draw, since they will have most/all of their pixels discarded before the shader gets to any expensive shadow/lighting calculations.

The Future

Although I have made vast improvements compared to where I started, there is still more that I can do.

I just got the book GPU Pro, which has one article of particular interest to me (well, a lot of the articles are interesting, but whatever), discussing a shadow mapping technique for omni-directional lights using a tetrahedron mapping instead of a cube. This would allow me to draw the scene only 4 times for each light instead of 6 to generate the shadow maps. The article also provides a way to pack all faces of the tetrahedron into a single texture (I realize a cube can be mapped onto one 2D texture too, but I did not do that). So that's something I want to look into using.

In addition, I may experiment with using a Light Pre-Pass renderer instead of my current Deferred approach. This adds an extra geometry pass, but cuts the number of intermediate render targets in half (only normals and depth, instead of normals, depth, color, misc material stuff).

In terms of this article (or others like it), I'd like to to put together small code samples demonstrating techniques. Unfortunately, I just don't have the time for that right now.

There's a new GWYD video up on Youtube, showing off the walk cycle for a new enemy I'm working on. The footage for the enemy is already recorded, but I still have the tedious task of splitting out each different animation, making them loop, removing backgrounds, etc.

Also featured in the video, is a quick look at the occlusion culling system I am using for determining potentially visible objects and reducing the amount of geometry I send for rendering. If I can find some, I'd like to write up some articles here about some of the techniques I'm using for various things.

Here's the video. You can also check it out, along with the others, on Youtube.

At this time I have no plans to renew and get back into iPhone development, so those projects are probably dead. But, if people really want to see them again, I might be persuaded to re-make them for PC/Mac. Otherwise, I think I'm working on better games now, so I'll stick with those.