Category: Game Development

I posted an job ad on Gamasutra about 4 days ago. It’s $350 for a single posting. I’ve only gotten 3 responses so far and my posting is the most appealing one there – work from home, high pay, small company. That’s pretty disappointing, I’m inclined to ask for a refund.

Of the 3 that applied, I’m now sending a pre-interview test. It’s based on how smart you are, rather than how much you can research stuff, and is a good indicator of programming ability.

1. Suppose you want to find the median of a list of a billion numbers. You have a cluster of networked computers at your disposal. You need to find the median as fast as possible. Describe a technique to do so.

2. I have a large set of data where each datum is a key/data pair. I need the operations insertion, deletion, lookup, and linear access and all these operations must be as fast as possible. How should I represent this data?

3. You are tasked with writing a logger class in a 3D game that does not already have a logger. It will be used for a wide range of features, writing everything from networked chat logs to graphics errors events to database lookup errors. You only have one day to write it. How would you architect this class? What features would you support?

4. You are planning to write a client/server game with an unattended server. Because the server is unattended, any crashes will mean the players cannot play for days. You have one week to implement whatever features / security you want to deal with this problem. What would you do?

In the past, I’d hire based on how badly I needed somebody, and how much I wanted to save money. So if the position really needed to get filled, and cheaply, I’d end up hiring a bad programmer for little money. This actually made me worse off – the bad programmer would waste my time (costing money) and accomplish nothing (costing money) and the job wouldnt’ get done (so now I needed somebody even more).

Now I’m hiring solely based on “Is this someone I would like to work with, and who would provide value for the money?” I’d rather pay 4X more for someone 10X more productive, and as we know there is a 100X difference between the best and worst programmers, so this isn’t hard to make happen.

Ever since I started in the games industry, I, and most games professionals I know, have a pet game design, and a dream to start our own companies. Most of us never have a chance to do that, but with a promise to my wife to ‘only’ spend 3 months and 20K (so about 50K counting my own lost salary), I started down the path of potential riches. That was 5 months ago.

The initial schedule called for 4 months of game programming work for myself. In order to cut that down to the required 2 months, I decided to hire 3 juniors, plus one senior graphics programmer so my game would have all the cool new graphics effects. Of course I can’t afford that for 20K, and I heard that India was the best outsourcing location and that the programmers there were cheap and hardworking. So I decided to give outsourcing a try.

To get my programmers, I posted on a general job site, naukri.com. I thought I would get 20-30 resumes, interview maybe half of them, and pick the best of the bunch. What actually happened was I got got flooded with about 800 resumes, about 200 of which came in the first 24 hours. Because of the way their site is designed, anyone can apply to any job, regardless of qualification, and without reading the job description, and with a single click. You can imagine the resulting average quality applicant I got. I also had no idea what Indian programmers were worth at the time. I got prices ranging from $250 a month to $5000 a month, all of whom claimed a college education, and most of whom could not even program “Hello World” in a real-life application.

I ultimately hired 4 guys at, on average, $1500 a month. One disappeared on the day he was supposed to start work. One was fired after wasting 3 days just trying to compile the game and had to get help even opening and copying files. Another supposedly worked for two days, then disappeared, and told me he took another job else a week later. The last was the best qualified of the bunch: my former graphics programmer – a diamond in the rough comparatively speaking. He claimed to be a former lead programmer at nVidia, claimed 6 years of experience, could speak well, and could answer most of my general interview questions.

Yet at the same time, his graphics knowledge wasn’t what I was expecting. I’m a networking guy, not a graphics guy, so was prepared to interview a graphics programmer who knew so much more than I did it was like Chuck Norris beating up a schoolyard bully. What actually happened was closer to an untrained sumo wrestling match in a mud ring. I couldn’t really get a grip on what he knew, I knew some things he didn’t, and overall we were pretty evenly matched as graphics programmers. But I attributed this lack due to in part the language barrier, in part because he was awesome compared to the 849 buffoons before him, and lastly because I was so desperate to save money. My own inexperience in hiring didn’t help either.

I thought hiring an ‘expert’ with 6 claimed years of experience would enable me to be very hands-off about the graphics, and focus just on the gameplay and networking. Don’t meddle in affairs you know nothing about, right? Unfortunately, this approach messed things up even more. He put a lot of hours in – easily 7 days a week, over 10 hours a day. He got work done, from the standpoint of writing code and checking it in. Yet the game never made much graphical progress. The problem he solved were the wrong problems, solved the wrong way, and even that inefficiently and poorly written. Many tasks ended up taking 4X longer than they should have – once to explain every minor nuance, once for him to program it, once for him to redo it after I point out obvious errors by glancing at the code for 30 seconds, and the final time for me to give up in exasperation and do it myself.

As time progressed, and this kept happening, I had to learn more and more about graphics programming in Ogre 3D just to keep the game from degrading. Eventually it reached the critical threshold where it was easier and faster to do everything myself. I forced him to resign, and only regret not doing so sooner.

What conclusions can one draw from this? I have a good sample set of applicants and interviewees – the average applicant on a general job site is worthless. To be fair, this is probably also true on general job sites in the US as well. There is a huge variation among asking salaries – a twenty-fold difference between the cheapest and most expensive. I’ve found that you get what you pay for – ignoring the outrageous endpoints, the more expensive guys are clearly better.

I hired 4 guys – not a statistically meaningful sample set, but for what it’s worth, all 4 of the guys that did work worked hard and put a lot of time in. The comments about Indian programmers being cheap and hardworking is actually true. It’s only their competence that is lacking.

If I have to pick one common problem among all the applicants and workers I’ve had from India it’s literalism. If you say “Write a ship explosion effect” they will – but it will look stupid, be inefficient, and ignore the game design you just told them about two days before. And they honestly won’t know what’s wrong. So if you be more specific, and say “Write a ship explosion effect and that looks like this image, and do so efficiently” they will bombard you with a dozen stupid questions. It’s like programming a computer – you have to give ALL the information. Whatever information you don’t give will produce arbitrary and meaningless results, such as only supporting 1 lights in shaders (I didn’t say we needed more than one, I just said we needed light support), or doing 3D effects in 2D (Our game is in 3D, but I didn’t say the effect had to be in 3D), or making a cloaking effect where the enemy player’s 3D name still shows up (I didn’t say the name had to be invisible, only the ship).

Not all outsourcing is bad – I have some artists and a programmer in Europe that are doing an excellent job. But outsourcing to India has been a definite bust for me.

Fortunately, the project is not at risk, since graphics is just marketing, 90% of graphic quality is your artists anyway, and I have fantastic artists. But you definitely get what you pay for. Next time I’ll just hire someone in the US, at full salary.

I have debris from shooting reactors now. When the reactor mesh loads, I analyze the mesh and generate a face list, which I store in a cache map with the filename of the mesh as the key. When you shoot the reactor, I take the bullet position and find the nearest face. I then find the nearest point on that face to the bullet. Lastly, I randomly rotate the face normal within a certain tolerance and use it the vector to shoot out the debris from. I had to account for some transformation problems, such as the face list is in local space, while the bullet is in world space, but I got it working in the end. What this means is that when you shoot a reactor, a piece of debris comes out from the surface of the reactor where you shot it, out away from the surface. Most games just fake it, by spawning debris from the bullet position, but I have damage from a proximity radius, so the bullet doesn’t always directly hit the reactor. This way, the debris still comes out of the reactor.

This technique can be extended more usefully for walls which explode outwards.

I’m not sure if anyone will actually appreciate this, given how far out the game is zoomed, but it’s good practice for me. Just 6 months ago I would have really struggled with this. But with my 3D programmer training at The Collective I not only did it, but in passing reinvented the function to get the nearest point on a plane just from thinking about it for a few minutes.

Next up is when reactors explode, the lights go out and you switch spawn points. This should be no work at all, but it seems like every time I start some minor thing I get sidetracked with a dozen little bugs and changes.

I spent several hours looking over PhysX and NxOgre, a wrapper for PhysX to bridge the gap between the two systems. I also experimented with the modelling program plugin for PhysX.

Conclusions so far:

PhysX: Powerful, pretty easy to use, professional quality level. Documentation is wrong in some places but nothing to really kill my productivity. Code documentation is generally sufficient – again while there are holes, it’s not enough to kill my productivity. Overall it’s pretty much pleasant to work with and I’m making good progress. The only thing I don’t like is that it requires its own installer.

NxOgre: Very easy to use if you already know what you’re doing. Otherwise about 70% of the documentation is missing and it’s definitely not clear what to do in these cases. Coding quality is above average relative to other freeware stuff, but still not up to the level I like to work with. In the end, I decided not to go with it. This is for three reasons:
1. Unclear memory management. While it’s clear that NxOgre handles most memory management for you, I don’t feel very comfortable putting new calls all over the place and trusting they were handled correctly. The architecture pretty much requires you do this.
2. Unique name convention for objects. This is similar to what Ogre does, and is one of the biggest annoyances about Ogre. I don’t want to name every damn class I want with a text string, especially not if I have to constantly work to make sure it’s unique.
3. Overhead. The documentation and samples for PhysX are just much more clear and straightforward, with better documentation, and it seems relatively easy to use. I don’t see the point of using NxOgre except to save me time, and without clear documentation it won’t save me much of anything. Plus, if I’m going to use a library it needs to be written at a higher quality level than what I saw or I’m not going to trust that it won’t crash or leak memory somewhere.

Modelling plugin: A buggy terrible mess, with outdated and wrong documentation. It looks like it was written halfway, then abandoned. I’m going to just integrate with the exporter I already have and parse out objects.

Right now I have PhysX integrated with my engine and creating a physics scene. I look forward to seeing how fast I can get the ship flying around.

In the end, after another few hours, I got Newton sort of working. Sometimes thrust would stop working for no reason, until I died and respawned, usually. Sometimes it would stay not working. And I wanted to fix the location of the fighter’s shield to the fighter’s body, and that never worked. Also, doing the same simulation on the client and the server was less accurate than with my own code.

It took me 2 days to get Newton sort-of-working. It took me 15 minutes to rewrite my old code back in, as all I wanted was a rotating ship in space with thrust.

My biggest problem with Newton is that if something doesn’t work, you’re basically fucked. The forum is unhelpful, the manual is incomplete, the tutorials don’t help much, and there’s no feedback from Newton as to why. So even if I were to attempt to fix the problem of “Why does my thrust suddenly stop working, but rotation still works?” I have absolutely no idea where to even look, with no source code to rely upon. These kind of things take hours, where just writing my own system velocity = thrust / mass * time takes about 10 seconds and consistently works.

It also lacks in easy-to-use features. For example, people have asked for efficient trigger bounds in the forum over a year ago, and it’s not there. I tried using my own system with AABB collision boxes and Newton was so slow I gave up on that and wrote my own system, again in not much time, which worked perfectly and was O(1). I have no way to update a single entity given a past time and state, which I need for networking. I don’t have access to the physics octtree which I also needed for network culling. There’s no easy way to do explosions or springs, and it goes on and on. I could spend another day guessing and probably get everything working, until I run into the next problem, and then lose another day, and so on.

So PhysX is free for PC games now and I got a developer account with them. I’m going to read over what they have and see if I can make the switch in a day or less. The nice thing about them is
1. A lot more people use that than use Newton, so I can get better support
2. There is a plugin for 3D modelling programs
3. It’s from a professional company so I have a feeling the quality and speed will be much better.

After doing the schedule, if I work 10-15 hours a day with no breaks until the end of Feb. I can get the game out at the start of March. Hopefully I can get it done faster than that but right now things take a lot longer because it’s my first pass, so I spend a lot of time refactoring code.

The big question mark right now on the release date is the people I hired. It’s possible they can help me with gameplay code, to get things done faster. It’s also possible that things might get screwed up and draw away my time. I’m going to have to be much less kind-hearted and drop workers that aren’t producing. Because I was too forgiving, I lost the first month of the project. I won’t make that mistake again.

Right now the total cost of the game looks to be 110K US, which is pretty cheap for an MMOG. It’s about average for a $20 on the shelf-game though. In order to make my money back in 1 year, after counting ongoing costs, I need about 2500 paying subscribers. Of course that is assuming I don’t pay myself. If I do, I need another 1000 subscribers or so.

I think this is still reasonable. Even the worst MMOG gets that many subscribers and although mine is a lot smaller and simpler, it’s still going to be a fun and nice looking game.

I’m getting better at interviewing people now. What I’ve found is that Joel really hit the nail on the head with The Guerrilla Guide To Interviewing. Only two things really matter in an interview: smart, and gets things done.

Resumes only tell part of the story. Some people who come to me with excellent resumes and a lot of experience are terrible. Some people who come to me with no experience at all turn out to be great. This is because of two additional factors: communication and enthusiasm.

People with bad communication don’t get things done because they do the wrong things. By communication I don’t mean reading and writing skills, but the willingness to have one’s ideas discussed and reviewed by others. Enthusiasm means a desire to do a good job. That seems obvious, but many people in the game industry only do what they are assigned, and no more, especially those with many years of experience, although this is more the fault of the management than of the programmer. A good portion of my day-to-day management is spent trying to build enthusiasm and facilitate communication.

The reason for smart is not just that it is a good quality in itself, but that it leads to better working independence (needing less communication) because of the ability to make good decisions and see past a specification. The problem with specifications, especially over the internet, is any specification less detailed than the code itself is open to interpretation. The more open, the more ways to do it wrong. The less open, the more possible it is to get fixated on the details, and less on the result. This is part of the reason why government contracts always turn out mediocre – the specifications are exact and rigid, meaning the great programmers never have a chance to improve on it.

A day to day decision I face is how much detail to put in a specification. For example, a loose specification is “Write a class to display text on the screen.” The implementer could do anything from writing a class that only writes a single line of text at a specific location to writing a class that supports multi-language true type fonts, with scrolling, fading, and animation. They may spend a day or a week on the task, working the whole time or slacking the whole time, and I no real way to know until I see the results. On the other hand, if I give a strict specification, such as “you must use true type fonts” the implementer will likely not even think about other, better, ways to do the job, such as using fixed-width fonts that look worse in some contexts we can avoid but double the framerate.

You can’t blame someone who consistently does something according to specification because if something is missing it’s your own fault for not specifying it. At the same that person isn’t going to be very productive in the sense of making a good game.

Enthusiasm is related to independence and is important in any job, but especially over the internet. At my very first job my boss was extremely hands-off. I don’t think he checked up on me more than once a month and never did code reviews. At the time he only had 2 programmers – myself and another guy. I just got out of school, it was my first game job, and I was smart, it was a game type I liked in a field I liked, and so I had a lot of enthusiasm. So I worked 12 hour days and was happy to do it. didn’t need specifications. I just thought of what was good, and I did it. The other programmer had neither positive quality. He did nothing the entire project up until the last month it was due, and then only did the bare minimum, and a bad job at that. He also never communicated, except in anger, when I would touch his code. That is the problem with a lack of enthusiasm.

So what do resumes tell you? One thing is if a person does a lot of side projects, they are probably enthusiastic, and they probably get things done. At least when they are working for themselves. But that is better than the alternative, which is a person with no side projects at all. A new graduate who has worked on freebie games is probably the most enthusiastic. This is why EA hires new graduates – they are the most likely to take crap and the most likely to work insane hours on the worst kinds of tasks.

As Joel said, there is a great difference in the productivity of the best and worst programmers. It’s better to hire someone twice as expensive if they can do twice the work. The trick of interviewing is finding out who these people are. Interviewing is probably the most difficult and important task that faces a company.

One problem with my current business plan is that if I want things done faster, I have to hire more programmers, which costs more per month. In theory, (aside from diminishing returns), this would pay for itself anyway. Twice as expensive for half as long still costs the same amount. In practice, I have a feeling this isn’t going to be the case. One problem is that, before hiring someone, it’s hard to get a feel for how fast they are. If I have 6 months of work for myself, and hire 1 programmer, is it going to now take 3 months? Probably not. Maybe it will only take 5 months instead. Or 4. Or 7 because they take so much of my own time that overall productivity is lowered.

This is hard to budget and hard to schedule.

What is much easier to budget is if I attach a fixed maximum and minimum cost to each item on the schedule. I now know the absolute minimum and maximum the game will cost, as well as the longest it could take to get done. It also lets me hire more people with safety, because it just spreads out the same amount of money between more people, rather than costing more with no guarantee the job will get done faster anyway.

I know from personal experience I actually like this better, from both sides. When I was an hourly contractor, I didn’t feel compelled to work when I don’t want to. As a manager, I don’t feel I need to check up on the schedule anymore. Either the work gets done in time or not. As a matter of fact, I liked putting down my hours, because I was fast and I could show it. And I worked hard, and I liked the extra money I made.

To try to help this along I’m also attaching minimum hours to each assigment. If a worker can get something done faster than I could myself, and still be up to spec, they still get paid for those hours. It’s a bonus for being especially smart. The maximum is to prevent cost overruns, and wasted time on dead-ends.

This might cause a rush of half-way finished code to try to beat the requirement. Hopefully not, but if it does, no big deal. It just costs me a bit more time to day to review code.

I’ll see how it works over the next couple of weeks.

Funny how things work out. I contacted the contractor MIA and he just quit voluntarily. Then I had a mid-level programmer apply suddenly and get hired pretty much the same day.

Looks like about 6 months of full-time work left if I were to do the whole game by myself. I think the mid-level programmer can cut that down by 1-2 months, and the junior programmer possibly by 1 month. The graphics programmer might save some time if he finishes the effects early and helps implement the effects in gameplay. It would probably average out to 3-4 more months of programming, less the week already spent.

One thing I’m worried about is that simply finishing the game in 4 months is only a part of the whole process. Ramping up from 1-2 people during development to supporting 10,000 players is definitely non-trivial. Another thing is that good gameplay never comes out of a closed cycle. Only through beta testing, iteration, and feedback from the community. In the optimal situation, I would sit on the game in closed beta for 6 months. I have a suspicion that is going to happen anyway. There’s no point to releasing before the game is saleable. But that’s a bad thing, since it will essentially double the development cost. It’s going to be tough to figure that one out.

My original plan for programming was to do most of the game work myself and have some junior programmers help me out to make things go quicker and take care of the small details.

This actually wasn’t a very good plan, because as I fill out the schedule I find that tasks can be categorized by the skill level of the programmer needed to do them correctly. Some tasks can only be done by me, some by any senior programmer, some by any programmer with gameplay experience, and some by junior programmers. My original schedule was to take the raw total hours, divide by 4 (the number of programmers) and figure out how many months that is. But that won’t work because most of the tasks are senior to mid-level, meaning with the people I have right now only I can do it correctly, meaning that I’ll never finish in the 3 months allotted.

This wasn’t total naivety so much as that I had never worked with any truly junior programmers before until now. When I thought junior I thought of my own skill out of college, which actually was mid-level. So in essence, the programmers I hired up to this point, except the lead programmer, aren’t really going to speed up the schedule.

The smart thing to do at this point would be to dump one of the junior programmers and hire 1 senior programmer and possibly 1 mid-level programmer. I’m not so cold-hearted as to just fire a junior programmer for no reason and will just go over-budget if I have to. However, the problem may resolve itself naturally.