Another minor hiring setback

It’s been a bad week ending with another bad hire.

This time I got a contractor to implement the C++ and stored procedures side of the database. We agree on $1500 for his estimate of 30-40 hours of work done over two weeks. One week later he does his first check-in. It’s about 100 lines of C++, two stored procedures, and some project changes, which breaks the build for 4 days. Repeated requests to fix it did not get addressed promptly, so I had to stay up to 4:30 AM on the 4th night to fix it myself. Generally speaking, when 3/4 of a contract duration passes by with no significant work done, the contract isn’t going to get done. So I send him an email canceling the contract and offer him $100 for the effort. He replies that he was actually 80% done, and wanted to finish, but if I cancel now I owe him 80% of the payment ($1200).

We finally agree that I will make the full-payment if he works over the weekend to get everything done. To his credit he put in effort, working about 20 hours over the weekend, and some more hours the following night. However, I effectively had to write all the C++ myself. I didn’t have time to check when he claimed to be finished. Grateful that he did work over the weekend, I sent him the payment. This turned out to be a mistake, since even a cursory check reveals bugs with every query, and his work was incomplete as well. There is still about 10 more hours of work to do. My email to him asking him to finish never got a response.

End result is that he did about 50% of the total work, I did about 30%, and I still have to do about 20%.

Lessons learned:
1. Stick with your gut feelings
2. A contract should have clear payment milestones, and only on acceptance of a milestone is any payment for that milestone due.
3. Short-term contracts are a good way to judge a potential hire because they limit your loss on a bad hire. Test potential hires through contracts before offering a regular position.


Kidney infection (?)

Last Saturday I woke up at 6 AM in the morning with the worst pain of my life in my left kidney, and this terrible pressure sensation in my bladder. I couldn’t even stand with the waves of pain, and was shaking and moaning. I thought I would need to go to the hospital and woke up my wife, and sent an email to my workers in case I wouldn’t be back for a while.

My wife gave me a bunch of medicine and acupuncture and about 5 hours later I was feeling good enough to walk again, although tired. I ultimately didn’t go to the hospital because the costs would basically close down development of my game.

My guess from reading around on Google is that it was a kidney and bladder infection caused by not drinking enough water. In truth I had been barely eating or sleeping the last 3 months since all my time was spent focused on working. I had some antibiotics so am taking those along with some Chinese medicine.

My kidney is still sensitive to the touch but doesn’t hurt continually anymore. I get too tired to keep working after about 8 hours so take it easy the rest of the day. Each day I feel better, especially in the mornings.

I don’t think this will delay the game too much, since most of the delay is waiting for the artists anyway. It just means a little less polish.

Game Development

Manifest hell

Whoever thought of manifest files for VS2005 should be shot. Right now my artist’s can’t run the game due to this crap. Already wasted 4 hours trying to figure this out, and am getting nowhere.

No longer can you include the necessary DLLs. Now they have dependencies embedded into the assembly, and it checks in weird system paths using the registry. Unless the person trying to run the game has these dependencies already (or Visual Studio 2005 installed) they can’t run it. I can’t actually test since I do have VS2005, so I have to grab people to help me try installing the game.

Your choices: Either (maybe??) use the HORRIBLE installer included with Visual Studio 2005, or … um I don’t know. The reason their installer is horrible is that it’s hard as hell to use, and I have to go through every single file in my game, one by one, and add it to the installer. No way am I going to do that. It also can’t find dependencies by 3rd party DLLs. It also doesn’t generate a single file for the installer, but 2 files (???). So there’s not one download but 2. Forget about it.

I tried copying a .manifest file from another program, along the DLLs. that should work I think, except that I’m using a whole bunch of libraries. Do I need to somehow find the source for those other DLLs and turn off manifest generation there too, and rebuild everything? I’m not sure that’s even possible, and if it were it would take me days to hunt down the source. Anyway, it doesn’t work.

I tried having my testers install the VS redistributable runtime but it also didn’t work.

I’m sort of running out of options here aside from installing VC6, getting things to build there, and using that instead.

Goddamn it. I just want to get this freaking game done and it’s always something to waste my time.

* EDIT *

I finally figured it out. I didn’t know there was a VS2005 redistributable for versions both with and without SP1. I didn’t see the SP1 version on my prior google search, and had my artists install the one without. I had one user install it with the SP1 version and it worked.

I’m also going to distribute the .manifest and dlls alongside the application. This was actually one of the first things I tried, and it didn’t work, so I think it’s more for versions of Windows that don’t have side by side assemblies at all. I saw that FMOD designer and most other installed applications did this, so in all probability it will work.

Game Development

Two disasters narrowly averted today

I lost like a year off my life today in stress.

The first looming disaster of the day was the ongoing problems getting setup support with our billing service provider. It was getting to the point where unless it were resolved immediately, and it didn’t look that way, the project would have to be canceled for lack of ability to take payment and lack of time to write a solution. Ouch. Just by fortunate chance, the CEO of the company had posted on my blog before, back when I required registration to stop spammers. As a last resort I grabbed his email address, shot him an email asking for help, and got a phone call an hour later. I was really impressed by the positive attitude of the new person I was working with at that point. I’m pretty sure things will work out now and this problem is behind me.

The second disaster was, as I previously posted, CEGUI is SOOOO slow at rendering text. It does a full draw call for every letter, recalculating the screen every update. I vacillated between writing my own system, fixing CEGUI, switching libraries, or just bypassing the problem in some other way. There was no easy solution in sight, and they all came with drawbacks. But my programmer in Portugal found a design flaw with CEGUI – we were hiding the root window, rendering the compositor, and then showing it again. Doing this caused a full-rebuild of every element on the screen. When combined with how CEGUI treats every individual letter as a widget, it dropped the FPS between 50%-75%.

I added a hack to System::renderGUI(void) block the variable “d_gui_redraw” from changing between the compositor start and end and that fixed that part. Another problem is that calling setAlpha, setFont, or setText with the same as the current value signals an unnecessary rebuild. So I changed CEGUI to check for that. The last thing is that CEGUI redraws the screen anytime any element changes. So I had to disable the clock, coordinates, and energy text since they update almost every tick. I switched over to displaying them with Ogre overlays instead.

I still have a lot of stress as Alpha is several days late now, but with these problems out of the way, and a few annoying bugs, I can at least start looking at making an Alpha build again.


How I got to 30 FPS to 10 FPS

Just when I was about to start alpha another problem strikes. I added help text messages to the chat window, and this is the first time we tested a full chat window full of messages.

In theory (and usually in practice) rendering static text is a trivially fast operation. Once you calculate the dimensions you save to a billboard and save the billboard. If you need to resize the text you just calculate when necessary.

I have no idea what the hell CEGUI is doing but in the same scenario in debug I go from 30 FPS to less than 10 FPS. Even if I just do


My FPS drops by 10. So I’m profiling CEGUI with Devpartner Studio to see if I can find what the problem is. It’s not too hopeful because I don’t know their code. But it’s worth a shot.

They recommend in the forums to use multi-line edit boxes, but I think rather than do that I will abandon CEGUI for the chat Window and render to texture the font, then just store that the correct way.

Profile attached. This is with 6 static text lines on the screen. Click to see the full image.

CEGUI Profile


How I got from 24 FPS to 264 FPS

I brought the FPS from 24 to 264 today. My optimizations started this morning when I bought DevPartner Studio and ran the performance analysis tool. Honestly, I’m not sure why people waste time writing their own versions in games because the cost is like a weeks’ salary and is 100X better than what you could write yourself. It’s not cheap, but I actually got something useful for my money whereas with 90% of the people I hire I get nothing for the same price.

Anyway, the performance analysis tool pointed out some serious problems.

1. The keymapper, as per my prior post, was taking 28% of the total CPU
2. Per-tick Script calls were wasting 10%. I refactored them to support single calls on the first tick, which is usually what I use them for.
3. CEGUI does not optimize out duplicate calls to setAlpha or setting the image property. I was doing that per-tick assuming it would ignore unneeded updates. That got me another 5% or so.
4. A few other minor tweaks here and there.

Then on my own, I turned off exception handling. Doing this generated about 6000 warnings because Ogre and CEGUI both use exceptions. Bad design decision on their part, but whatever.

Also, all my shaders are customized for the specific situations they used in. You don’t get this at most game companies but since I’m doing both the programming and half the art too I have the opportunity for that. They are optimized down to individual lines of code present or not present for the situation. The reason I’m doing this is that my models and textures are super detailed, yet the ships are far away on the screen. So in order to really see what is going on you have to be able to run at very high resolution. This means the pixel shader has to be super fast, and if the vertex shader is also fast then so much the better. Also, as I’m setting up all the materials for the level I can pick intelligently so there’s no waste.

I have another 25% or so boost coming in the next few days. Once the server and client are split that’s one thread and a duplicate copy of the game code out.


Botched Keymapper = wasted half day

I did some performance analysis today and found that the freaking keymapper was taking 28% of the total CPU time! It even took longer than the render cycle.

The keymapper was one of the first tasks I gave out, about 6 months ago, to a junior programmer. It’s supposed to load an XML of strings of actions, and bindings. It would map the strings to numbers, and create an association between the two. So that given a list of pressed keys I can get the corresponding actions, etc.

The main problems are dealing with multiple keys to trigger a single action, actions that take multiple keys to trigger, and how to handle the situation where more than one action is possible given multiple keys.

The original programmer loaded the XML files and just kept them in memory. He then did string lookups only, not using the numerical equivalents like I told him to. Worse, these were XML based string lookups, so it would have to parse the whole XML file to look up a particular binding, then a particular string. Worse, in order to handle the case with duplicate mappings, he would scan the whole list again. So it’s slow * slow * slow.

On top of that he didn’t handle the situation with multiple bindings for a given action, so you couldn’t do left control OR right control to fire. Only one or the other. And he didn’t correctly handle the situation with multiple keys to multiple actions, so if I pressed a key combo to trigger an action, and that combo included other actions, those other actions would trigger too.

Basically he made an O(1) operation (if you’re smart about it), or an O(n) operation (if you’re not), into an O(n^3) operation.

I rewrote it. I got each call from 160.8 nanoseconds to 4.5 nanoseconds, has the two features his didn’t, and I did it in half a day, where he took 3 days.

Incidentally, it was another bad programmer that wrote the XML lookup to begin with. He also didn’t follow my instructions on how to do it efficiently. Sort of bad programmer squared.


A few images from my game

I’m almost ready to start closed alpha. Here’s a few screenshots. I didn’t turn on anti-aliasing although it is supported.

GreenEye Level

This is a final level the artists did. There’s a few rough patches they have to fix but this will be the background for the first level. This is used for players to play in tournament style matches, 1 on 1.

Ship Customization Screen

You can pick/buy your weapons and items and ship. No weapon or item is straight out better than any other, they all have trade-offs in mass/cost/ammo usage, and weapon slot.

Heavy weapons ship

This is the most massive fighter and can carry extra ammo for ammo based weapons. It is also good to use for mining since you can hold extra Neutronium. I don’t have LOD in yet and it’s harder to zoom the camera than it should be. Stuff to work on. But it has pretty good lighting and the weapons you pick are on your ship.


Built-in memory leak detector

I didn’t know this until today but Visual Studio has a built-in memory leak detector. One day out of the blue my game starts to take a long time to shut down and in the output window I see

Detected memory leaks!

Followed by a bunch of numbers.

I looked it up today and after some research, I found out that the allocator counts memory allocations, and prints out the memory allocation number that caused the leak. I don’t know why in all the time I’ve been programming I’ve never heard about this before but it’s really useful!

This only works if your program state is repeatable (such as on startup) but it works really well from what I’ve seen.

All you have to do is put

#include <stdlib .h >
#include < crtdbg .h >

Somewhere at the top of your program.

Then either put at the start


or put


at the end.

Lastly, to break at the specified allocation number, call


where X is the allocation number.

Alternatively, put this in your watch window:


It should show -1. Change that -1 to the number of the allocation.

For example:

Detected memory leaks!
Dumping objects ->
{614841} normal block at 0x123A7D30, 8 bytes long.
Data: <> 7F 00 00 01 08 E6 CD CD

I put F10 to step one line into my program.

I put {,,msvcr80d.dll}_crtBreakAlloc into the watch window.

I change the -1 to 614841.

I press F5 to run.

It should break when it hits this allocation.

I just fixed 2 leaks in about 5 minutes that I would have never found otherwise and only have 614,841 to go 🙂


Upgraded Ogre + oFusion problems again…

I upgraded Ogre today to 1.4.1. It looks like a much better release than I was using. I especially like that they assert now rather than throw exceptions. Things already feel more snappy.

Once I did that oFusion caused problems again. It turns out there was a bug in their mesh exporter, the only thing left I use it for, such that bounding boxes are calculated incorrectly. Judging by the forum this has been a known bug for a while yet the author didn’t fix it. So I wasted an hour figuring out how to run the mesh upgrader recursively, and I had to mod it to get rid of the exceptions so if a mesh didn’t have a material it wouldn’t crash generating tangent vectors. Now artists can’t export models anymore, although I’m not sure if they were to begin with.

The silver lining is that oFusion has another bug where it wasn’t generating tangent vectors anyway, so in hindsight it wasn’t really wasted work as I eventually should have done this regardless. Pre-generated tangent vectors let the game load faster.

Definitely, the first thing I’m going to do post-ship is to hire a programmer to write a usable exporter for the artists.