Month: May 2007

It’s been a bad week ending with another bad hire.

This time I got a contractor to implement the C++ and stored procedures side of the database. We agree on $1500 for his estimate of 30-40 hours of work done over two weeks. One week later he does his first check-in. It’s about 100 lines of C++, two stored procedures, and some project changes, which breaks the build for 4 days. Repeated requests to fix it did not get addressed promptly, so I had to stay up to 4:30 AM on the 4th night to fix it myself. Generally speaking, when 3/4 of a contract duration passes by with no significant work done, the contract isn’t going to get done. So I send him an email canceling the contract and offer him $100 for the effort. He replies that he was actually 80% done, and wanted to finish, but if I cancel now I owe him 80% of the payment ($1200).

We finally agree that I will make the full-payment if he works over the weekend to get everything done. To his credit he put in effort, working about 20 hours over the weekend, and some more hours the following night. However, I effectively had to write all the C++ myself. I didn’t have time to check when he claimed to be finished. Grateful that he did work over the weekend, I sent him the payment. This turned out to be a mistake, since even a cursory check reveals bugs with every query, and his work was incomplete as well. There is still about 10 more hours of work to do. My email to him asking him to finish never got a response.

End result is that he did about 50% of the total work, I did about 30%, and I still have to do about 20%.

Lessons learned:
1. Stick with your gut feelings
2. A contract should have clear payment milestones, and only on acceptance of a milestone is any payment for that milestone due.
3. Short-term contracts are a good way to judge a potential hire because they limit your loss on a bad hire. Test potential hires through contracts before offering a regular position.

Last Saturday I woke up at 6 AM in the morning with the worst pain of my life in my left kidney, and this terrible pressure sensation in my bladder. I couldn’t even stand with the waves of pain, and was shaking and moaning. I thought I would need to go to the hospital and woke up my wife, and sent an email to my workers in case I wouldn’t be back for a while.

My wife gave me a bunch of medicine and acupuncture and about 5 hours later I was feeling good enough to walk again, although tired. I ultimately didn’t go to the hospital because the costs would basically close down development of my game.

My guess from reading around on Google is that it was a kidney and bladder infection caused by not drinking enough water. In truth I had been barely eating or sleeping the last 3 months since all my time was spent focused on working. I had some antibiotics so am taking those along with some Chinese medicine.

My kidney is still sensitive to the touch but doesn’t hurt continually anymore. I get too tired to keep working after about 8 hours so take it easy the rest of the day. Each day I feel better, especially in the mornings.

I don’t think this will delay the game too much, since most of the delay is waiting for the artists anyway. It just means a little less polish.

Just when I was about to start alpha another problem strikes. I added help text messages to the chat window, and this is the first time we tested a full chat window full of messages.

In theory (and usually in practice) rendering static text is a trivially fast operation. Once you calculate the dimensions you save to a billboard and save the billboard. If you need to resize the text you just calculate when necessary.

I have no idea what the hell CEGUI is doing but in the same scenario in debug I go from 30 FPS to less than 10 FPS. Even if I just do

a
a
a
a
a

My FPS drops by 10. So I’m profiling CEGUI with Devpartner Studio to see if I can find what the problem is. It’s not too hopeful because I don’t know their code. But it’s worth a shot.

They recommend in the forums to use multi-line edit boxes, but I think rather than do that I will abandon CEGUI for the chat Window and render to texture the font, then just store that the correct way.

Profile attached. This is with 6 static text lines on the screen. Click to see the full image.

I brought the FPS from 24 to 264 today. My optimizations started this morning when I bought DevPartner Studio and ran the performance analysis tool. Honestly, I’m not sure why people waste time writing their own versions in games because the cost is like a weeks’ salary and is 100X better than what you could write yourself. It’s not cheap, but I actually got something useful for my money whereas with 90% of the people I hire I get nothing for the same price.

Anyway, the performance analysis tool pointed out some serious problems.

1. The keymapper, as per my prior post, was taking 28% of the total CPU
2. Per-tick Script calls were wasting 10%. I refactored them to support single calls on the first tick, which is usually what I use them for.
3. CEGUI does not optimize out duplicate calls to setAlpha or setting the image property. I was doing that per-tick assuming it would ignore unneeded updates. That got me another 5% or so.
4. A few other minor tweaks here and there.

Then on my own, I turned off exception handling. Doing this generated about 6000 warnings because Ogre and CEGUI both use exceptions. Bad design decision on their part, but whatever.

Also, all my shaders are customized for the specific situations they used in. You don’t get this at most game companies but since I’m doing both the programming and half the art too I have the opportunity for that. They are optimized down to individual lines of code present or not present for the situation. The reason I’m doing this is that my models and textures are super detailed, yet the ships are far away on the screen. So in order to really see what is going on you have to be able to run at very high resolution. This means the pixel shader has to be super fast, and if the vertex shader is also fast then so much the better. Also, as I’m setting up all the materials for the level I can pick intelligently so there’s no waste.

I have another 25% or so boost coming in the next few days. Once the server and client are split that’s one thread and a duplicate copy of the game code out.

I did some performance analysis today and found that the freaking keymapper was taking 28% of the total CPU time! It even took longer than the render cycle.

The keymapper was one of the first tasks I gave out, about 6 months ago, to a junior programmer. It’s supposed to load an XML of strings of actions, and bindings. It would map the strings to numbers, and create an association between the two. So that given a list of pressed keys I can get the corresponding actions, etc.

The main problems are dealing with multiple keys to trigger a single action, actions that take multiple keys to trigger, and how to handle the situation where more than one action is possible given multiple keys.

The original programmer loaded the XML files and just kept them in memory. He then did string lookups only, not using the numerical equivalents like I told him to. Worse, these were XML based string lookups, so it would have to parse the whole XML file to look up a particular binding, then a particular string. Worse, in order to handle the case with duplicate mappings, he would scan the whole list again. So it’s slow * slow * slow.

On top of that he didn’t handle the situation with multiple bindings for a given action, so you couldn’t do left control OR right control to fire. Only one or the other. And he didn’t correctly handle the situation with multiple keys to multiple actions, so if I pressed a key combo to trigger an action, and that combo included other actions, those other actions would trigger too.

Basically he made an O(1) operation (if you’re smart about it), or an O(n) operation (if you’re not), into an O(n^3) operation.

I rewrote it. I got each call from 160.8 nanoseconds to 4.5 nanoseconds, has the two features his didn’t, and I did it in half a day, where he took 3 days.

Incidentally, it was another bad programmer that wrote the XML lookup to begin with. He also didn’t follow my instructions on how to do it efficiently. Sort of bad programmer squared.

I’m almost ready to start closed alpha. Here’s a few screenshots. I didn’t turn on anti-aliasing although it is supported.

GreenEye Level
This is a final level the artists did. There’s a few rough patches they have to fix but this will be the background for the first level. This is used for players to play in tournament style matches, 1 on 1.

—

Ship Customization Screen
You can pick/buy your weapons and items and ship. No weapon or item is straight out better than any other, they all have trade-offs in mass/cost/ammo usage, and weapon slot.

—

Heavy weapons ship
This is the most massive fighter and can carry extra ammo for ammo based weapons. It is also good to use for mining since you can hold extra Neutronium. I don’t have LOD in yet and it’s harder to zoom the camera than it should be. Stuff to work on. But it has pretty good lighting and the weapons you pick are on your ship.

I didn’t know this until today but Visual Studio has a built-in memory leak detector. One day out of the blue my game starts to take a long time to shut down and in the output window I see

Detected memory leaks!

Followed by a bunch of numbers.

I looked it up today and after some research, I found out that the allocator counts memory allocations, and prints out the memory allocation number that caused the leak. I don’t know why in all the time I’ve been programming I’ve never heard about this before but it’s really useful!

This only works if your program state is repeatable (such as on startup) but it works really well from what I’ve seen.

All you have to do is put

#define _CRTDBG_MAP_ALLOC
#include <stdlib .h >
#include < crtdbg .h >


Somewhere at the top of your program.

Then either put at the start

_CrtSetDbgFlag ( _CRTDBG_ALLOC_MEM_DF | _CRTDBG_LEAK_CHECK_DF );

or put

_CrtDumpMemoryLeaks();


at the end.

Lastly, to break at the specified allocation number, call


_CrtSetBreakAlloc(X);


where X is the allocation number.

Alternatively, put this in your watch window:


{,,msvcr80d.dll}_crtBreakAlloc


It should show -1. Change that -1 to the number of the allocation.

For example:


Detected memory leaks!
Dumping objects ->
{614841} normal block at 0x123A7D30, 8 bytes long.
Data: <> 7F 00 00 01 08 E6 CD CD


I put F10 to step one line into my program.

I put {,,msvcr80d.dll}_crtBreakAlloc into the watch window.

I change the -1 to 614841.

I press F5 to run.

It should break when it hits this allocation.

I just fixed 2 leaks in about 5 minutes that I would have never found otherwise and only have 614,841 to go 🙂

I upgraded Ogre today to 1.4.1. It looks like a much better release than I was using. I especially like that they assert now rather than throw exceptions. Things already feel more snappy.

Once I did that oFusion caused problems again. It turns out there was a bug in their mesh exporter, the only thing left I use it for, such that bounding boxes are calculated incorrectly. Judging by the forum this has been a known bug for a while yet the author didn’t fix it. So I wasted an hour figuring out how to run the mesh upgrader recursively, and I had to mod it to get rid of the exceptions so if a mesh didn’t have a material it wouldn’t crash generating tangent vectors. Now artists can’t export models anymore, although I’m not sure if they were to begin with.

The silver lining is that oFusion has another bug where it wasn’t generating tangent vectors anyway, so in hindsight it wasn’t really wasted work as I eventually should have done this regardless. Pre-generated tangent vectors let the game load faster.

Definitely, the first thing I’m going to do post-ship is to hire a programmer to write a usable exporter for the artists.