Kevin Jenkins' Blog

6 months ago, when I started my Indy game company Jenkins Software, I had virtually no interviewing experience and made some hiring mistakes. Here’s what I’ve learned, now that I know a bit more about the process. I can’t go back in time to send this to myself, but hopefully others starting companies can learn from my mistakes.

It takes time and costs money to find the good people.

Originally, I thought posting a job ad on Gamasutra would net me hundreds of applications, of which some small percentage (5%) would be great guys. And I’d hire those guys.

As it turns out everyone wants the great guys, not just me, and the other companies have a lot more money and perks to offer than I do. So great guys are rarely on the market, and when they are they tend to be snatched up very quickly. The only reason I’m competitive at all is that I offer telecommuting, which is such a perk that it outweighs everything else for many people. On my own resume I offer to work for $10K less a year if I am allowed to telecommute. I also allow everyone in the company to have a say in the game design, another huge perk most other companies do not offer.

My job ad on Gamasutra only got me 8 resumes in the end, only 1 of which I would say is a professional level candidate (where I was expecting hundreds of professional level candidates) and even then I don’t know if he was one of those top 5% great guys I was looking for.

I’m starting to understand the recruiter market. Because the pool of great candidates is so small, recruiters are there to pull people away from their existing companies and not so much to search for people who are already available. They charge a ridiculous amount for that small service (15%-50% of the first year’s salary) and I’d never use one for that reason but I can at least see the theoretical point now.

In 6 months of looking, I only ever found one good guy and just hired another, who will hopefully turn out to be good – we’ll have to see.

If I could do things over again, I’d start the hiring process about 1 month before I even started the company. Get some names down, some people to sign up for you for a start date two weeks later. And I’d continue hiring and looking, because some of those people won’t work out anyway.

A bad hire can ruin the company

Before I started the company I read a lot of management books and spent a lot of time trying to motivate the people I did hire. So when people didn’t perform, I tried to be a better manager, and blamed myself for their failings, so would respond with motivation and help, and kept giving more time, and more chances. As it turned out, the problem wasn’t bad management but a lack of raw talent – the bad management part was in hiring the wrong people to begin with. I confused inexperience with incompetence. The first is easily fixed, and in a way is positive because you can get someone cheaper and train them. The second is impossible to fix and will only drag your company down. If someone says “I don’t know how to …” this is OK – spend a day training them and you have a more valuable worker. If someone says “I can’t …” and they actually can, this is incompetence and you might have made a hiring mistake. For new hires, this is very important to watch for.

Hiring the wrong people is a triple whammy. You waste your time training, they waste your money learning, and you end up even later and more behind schedule.

This may sound harsh, but it’s true. There is a 100X difference between the best and the worst programmers and the curve is towards the bad end. So there’s a lot more bad programmers looking for jobs than good ones, and it makes it very important to not keep the bad ones.

You won’t know who is a bad hire until after you hire them

Maybe I’m not good enough at interviewing, but some people are good at interviewing but bad at programming. It takes me between a week and a month to sort out the difference. So when you hire, you should overhire, because some of the people you hire aren’t going to work out anyway.

Quality is more important than cost

With 100X the difference between the best and worst programmers, it’s worth paying double for someone 10X better. Of course it’s always a risk – say you hire a guy at double the cost, he doesn’t work out after a month, and now you are out 2 months of salary. But generally speaking, if you find a great guy just take him right away, despite the extra cost. This will save you money in the long run. Plus, the good programmers know they are good and tend to ask for more money. A very low asking price is a warning sign.

One of the mistakes I made early on was to pass up a very experienced guy in the US because he was slightly too expensive. As it turned out, I ended up paying far more in the long run because of this, and had to hire people nearly as expensive anyway.

Get the money straightened away first

Once I interviewed with a company that spent probably $1000 on the interview. They flew me in, put me up at a nice hotel, spent a lot of time on interviews, had a producer drive me around showing me the local area, and spent time doing paperwork. Their final offer was 20% below my best offer, and was also below the minimum I put on my resume. The offer was fair for the area, but was too low for me to accept given the number of other companies that also had offers for me.

To a lesser degree I had the same problem, spending a lot of time interviewing guys that wanted double what I could afford to pay. I may have liked them in the end, but it was a waste of time nonetheless.

Setup interviews in stages, and don’t be afraid to cut an interview short

Similar to above, a lot of guys just can’t cut it and it’s not worth the time to do a 2 hour interview with a weak candidate. It’s not rude, it’s business, if you have to just cut an interview off after a short time.

Go with your gut instincts

Lastly, if someone gives you a bad feeling but you can’t say exactly why, no hire. Every time I’ve hired someone because I had no other reason not to, I regretted it a few weeks later.

I have temp art for the energy bar in. It uses an animated texture to flip lights on and off of the base of the bar. The center part interpolates between the ’empty’ texture or the ‘full’ texture, picking along the UV channel based on the percentile of my total remaining energy.

Photoshop:

In the game

I posted an job ad on Gamasutra about 4 days ago. It’s $350 for a single posting. I’ve only gotten 3 responses so far and my posting is the most appealing one there – work from home, high pay, small company. That’s pretty disappointing, I’m inclined to ask for a refund.

Of the 3 that applied, I’m now sending a pre-interview test. It’s based on how smart you are, rather than how much you can research stuff, and is a good indicator of programming ability.

1. Suppose you want to find the median of a list of a billion numbers. You have a cluster of networked computers at your disposal. You need to find the median as fast as possible. Describe a technique to do so.

2. I have a large set of data where each datum is a key/data pair. I need the operations insertion, deletion, lookup, and linear access and all these operations must be as fast as possible. How should I represent this data?

3. You are tasked with writing a logger class in a 3D game that does not already have a logger. It will be used for a wide range of features, writing everything from networked chat logs to graphics errors events to database lookup errors. You only have one day to write it. How would you architect this class? What features would you support?

4. You are planning to write a client/server game with an unattended server. Because the server is unattended, any crashes will mean the players cannot play for days. You have one week to implement whatever features / security you want to deal with this problem. What would you do?

In the past, I’d hire based on how badly I needed somebody, and how much I wanted to save money. So if the position really needed to get filled, and cheaply, I’d end up hiring a bad programmer for little money. This actually made me worse off – the bad programmer would waste my time (costing money) and accomplish nothing (costing money) and the job wouldnt’ get done (so now I needed somebody even more).

Now I’m hiring solely based on “Is this someone I would like to work with, and who would provide value for the money?” I’d rather pay 4X more for someone 10X more productive, and as we know there is a 100X difference between the best and worst programmers, so this isn’t hard to make happen.

Ever since I started in the games industry, I, and most games professionals I know, have a pet game design, and a dream to start our own companies. Most of us never have a chance to do that, but with a promise to my wife to ‘only’ spend 3 months and 20K (so about 50K counting my own lost salary), I started down the path of potential riches. That was 5 months ago.

The initial schedule called for 4 months of game programming work for myself. In order to cut that down to the required 2 months, I decided to hire 3 juniors, plus one senior graphics programmer so my game would have all the cool new graphics effects. Of course I can’t afford that for 20K, and I heard that India was the best outsourcing location and that the programmers there were cheap and hardworking. So I decided to give outsourcing a try.

To get my programmers, I posted on a general job site, naukri.com. I thought I would get 20-30 resumes, interview maybe half of them, and pick the best of the bunch. What actually happened was I got got flooded with about 800 resumes, about 200 of which came in the first 24 hours. Because of the way their site is designed, anyone can apply to any job, regardless of qualification, and without reading the job description, and with a single click. You can imagine the resulting average quality applicant I got. I also had no idea what Indian programmers were worth at the time. I got prices ranging from $250 a month to $5000 a month, all of whom claimed a college education, and most of whom could not even program “Hello World” in a real-life application.

I ultimately hired 4 guys at, on average, $1500 a month. One disappeared on the day he was supposed to start work. One was fired after wasting 3 days just trying to compile the game and had to get help even opening and copying files. Another supposedly worked for two days, then disappeared, and told me he took another job else a week later. The last was the best qualified of the bunch: my former graphics programmer – a diamond in the rough comparatively speaking. He claimed to be a former lead programmer at nVidia, claimed 6 years of experience, could speak well, and could answer most of my general interview questions.

Yet at the same time, his graphics knowledge wasn’t what I was expecting. I’m a networking guy, not a graphics guy, so was prepared to interview a graphics programmer who knew so much more than I did it was like Chuck Norris beating up a schoolyard bully. What actually happened was closer to an untrained sumo wrestling match in a mud ring. I couldn’t really get a grip on what he knew, I knew some things he didn’t, and overall we were pretty evenly matched as graphics programmers. But I attributed this lack due to in part the language barrier, in part because he was awesome compared to the 849 buffoons before him, and lastly because I was so desperate to save money. My own inexperience in hiring didn’t help either.

I thought hiring an ‘expert’ with 6 claimed years of experience would enable me to be very hands-off about the graphics, and focus just on the gameplay and networking. Don’t meddle in affairs you know nothing about, right? Unfortunately, this approach messed things up even more. He put a lot of hours in – easily 7 days a week, over 10 hours a day. He got work done, from the standpoint of writing code and checking it in. Yet the game never made much graphical progress. The problem he solved were the wrong problems, solved the wrong way, and even that inefficiently and poorly written. Many tasks ended up taking 4X longer than they should have – once to explain every minor nuance, once for him to program it, once for him to redo it after I point out obvious errors by glancing at the code for 30 seconds, and the final time for me to give up in exasperation and do it myself.

As time progressed, and this kept happening, I had to learn more and more about graphics programming in Ogre 3D just to keep the game from degrading. Eventually it reached the critical threshold where it was easier and faster to do everything myself. I forced him to resign, and only regret not doing so sooner.

What conclusions can one draw from this? I have a good sample set of applicants and interviewees – the average applicant on a general job site is worthless. To be fair, this is probably also true on general job sites in the US as well. There is a huge variation among asking salaries – a twenty-fold difference between the cheapest and most expensive. I’ve found that you get what you pay for – ignoring the outrageous endpoints, the more expensive guys are clearly better.

I hired 4 guys – not a statistically meaningful sample set, but for what it’s worth, all 4 of the guys that did work worked hard and put a lot of time in. The comments about Indian programmers being cheap and hardworking is actually true. It’s only their competence that is lacking.

If I have to pick one common problem among all the applicants and workers I’ve had from India it’s literalism. If you say “Write a ship explosion effect” they will – but it will look stupid, be inefficient, and ignore the game design you just told them about two days before. And they honestly won’t know what’s wrong. So if you be more specific, and say “Write a ship explosion effect and that looks like this image, and do so efficiently” they will bombard you with a dozen stupid questions. It’s like programming a computer – you have to give ALL the information. Whatever information you don’t give will produce arbitrary and meaningless results, such as only supporting 1 lights in shaders (I didn’t say we needed more than one, I just said we needed light support), or doing 3D effects in 2D (Our game is in 3D, but I didn’t say the effect had to be in 3D), or making a cloaking effect where the enemy player’s 3D name still shows up (I didn’t say the name had to be invisible, only the ship).

Not all outsourcing is bad – I have some artists and a programmer in Europe that are doing an excellent job. But outsourcing to India has been a definite bust for me.

Fortunately, the project is not at risk, since graphics is just marketing, 90% of graphic quality is your artists anyway, and I have fantastic artists. But you definitely get what you pay for. Next time I’ll just hire someone in the US, at full salary.

I have debris from shooting reactors now. When the reactor mesh loads, I analyze the mesh and generate a face list, which I store in a cache map with the filename of the mesh as the key. When you shoot the reactor, I take the bullet position and find the nearest face. I then find the nearest point on that face to the bullet. Lastly, I randomly rotate the face normal within a certain tolerance and use it the vector to shoot out the debris from. I had to account for some transformation problems, such as the face list is in local space, while the bullet is in world space, but I got it working in the end. What this means is that when you shoot a reactor, a piece of debris comes out from the surface of the reactor where you shot it, out away from the surface. Most games just fake it, by spawning debris from the bullet position, but I have damage from a proximity radius, so the bullet doesn’t always directly hit the reactor. This way, the debris still comes out of the reactor.

This technique can be extended more usefully for walls which explode outwards.

I’m not sure if anyone will actually appreciate this, given how far out the game is zoomed, but it’s good practice for me. Just 6 months ago I would have really struggled with this. But with my 3D programmer training at The Collective I not only did it, but in passing reinvented the function to get the nearest point on a plane just from thinking about it for a few minutes.

Next up is when reactors explode, the lights go out and you switch spawn points. This should be no work at all, but it seems like every time I start some minor thing I get sidetracked with a dozen little bugs and changes.