Category: Uncategorized

When I first set out to start my company, I basically did the time calculation as follows:

w = Number of hours of work
m = Number of months I want to finish in
h = Number of people I have to hire.

The calculation is then simply
h = w / (m*hoursPerMonth)

Before doing that calculation, I actually doubled w, to account for time in beta. I also increased h a bit, to account for lost time to to communication, since programmers do not scale linearly, but inverse exponentially.

The final total was about 4 months for 3 people. This was about 50% on me, and 25% for each of the other two programmers. I didn’t account at all for things I considered to be trivial / intangible – billing, hosting, finding artists, setting up a server, finding tools for communication and development, time off (just assumed a 4 month crunch), delays due to sickness or hardware failure. It wasn’t that I didn’t know about these things, but that I considered them to be too trivial to schedule. My concept was “How long could it possibly take to find, do, or manage [x]? Just post an ad (or do a web search), interview a few guys (or read a few websites), and go with it.”

In the optimal scenario this is actually true. The problem is that the world is filled with people who have nothing better to do than waste your time. This ranges from hosting services that take your server down without warning, to Linux programmers that don’t even know how to open files in windows, to people that spend weeks discussing a project only to ask for 4X more money than is reasonable.

So each of these steps of running a business become very time consuming, and there isn’t always an easy way to tell the difference between the guys who are wasting your time, and those who are not. For example, if I want a quote on my art assets, I really need to spend a lot of time explaining the contract every time, because no matter how detailed my asset list is, people will not understand or appreciate certain aspects. This is especially true if you are trying to save money (as I am), because at the low end of the totem pole there’s a lot higher noise to signal ratio.

As a result, I’ve learned to appreciate the value of businesses beyond the amount of intellectual property they own. Each programmer represents not just any dude off the street, but a long, careful, and expensive search both to find and retain a worker. Making a new programmer productive costs about that much again, when you account for the months of initial downtime required to train someone to really be productive. Getting a working setup and source control reflects many hours of work, from setting up the server, to finding a reasonable host, to purchasing the license, to training everyone how to use it. Having a team handle your art outsourcing takes many hours to find a qualified team, one that can do the job cheaply, setting up communication channels, explaining the requirements (including documentation), and sweat and worry that you might get ripped off anyway.

My initial schedule of 4 months with 3 programmers is now 6 months with 4 programmers (double the man hours) and even with that I don’t think beta will be long enough, and time is of the essence. Half my time currently goes towards intangibles. Tasks originally assumed to be trivial (a few hours) sometimes end up taking weeks. And other problems come up, such as sick time, people quitting, and bad hires. For example, nearly a month of productivity was lost in the first two months just from dealing with bad hires.

My advice to would-be game companies is to add 3-4 man months on to the total schedule to account for these kinds of things.

Yesterday a friend and I spent some time talking to Aria Systems, a company which specializes in billing systems for games. Basically, they provide the webpage by which users can pay, they handle the payments, and they pass the result of the transactions back to the customer (me). The good thing about this is I then do not have to get a secure webpage and handle these things myself. Although admittedly it is not that hard – I’ve set up this billing system before and it only took me a few days, even with recurring billing. I just like the security of knowing that a 3rd party is handling this, so I am protected from credit card fraud, webpage issues, etc.

The bad thing about this is how much they charge – the setup fee is $1500, even for the most trivial case (such as mine – just a flat monthly fee). They also take 2%, in addition to what the credit card providers themselves take. This will amount to over $1,000 a month and would only get larger as the game gets more successful. This 2% does not go down until you take in a million dollars a month, at which point it goes to 1.8%. That seems pretty outrageous to me. 2% of a million a month is $20K a month, just to have your billing processed by a 3rd party. For 20K a month I could hire a team of 3 guys full-time and host my own servers, and I doubt it will take that once the initial systems are written.

Anyway, I’m willing to go with that, except for one part in the contract I’m seeing if I can negotiate out. They want a 3 year exclusive deal to provide this service. What I said to this was:

I’d like to add “This agreement may be terminated by either party at any time, with or without cause, with 3 months written notice.”

With this change I can agree to the exclusivity agreement. Without this change, if Aria were not able to compete on merits in the marketplace, I would be locked in to providing my customers with less than the best possible service. If you’re confident that Aria is and will remain the best provider of billing services, then there shouldn’t be any problems and I will have no reason to switch.

It seems ridiculous to me that they even ask for this. The way to keep my business for 3 years is to provide the best service at the best price for 3 years, not to force me into the service without my prior knowledge of how good the quality is. I have a feeling the reason for this is what I was saying above – at my initial levels of sales, it’s cheaper to pay them than it is to hire my own guy and do it myself. At higher levels of sales, only someone with no other choice would pay their prices, which is what I think they are counting on.

Their responses to my initial emails only took minutes to get returned and its half a day now with no response to my last one. While I could be wrong, there’s a good chance I’ll never hear from them again. And that’s fine with me. If a company themselves do not feel they can compete on the merits of price and service, then I shouldn’t feel that way either.

*** EDIT ***

They got back to me a day later with an OK.

In a surprise twist, the company in Brazil turns out not to have any people with game art experience. I asked a friend who has his own company how important this is, and he told me “Very” so I dropped them out of the running. Between the company in the US, headed by the former WoW guy, and the one in Slovakia, I just decided to go with the latter. This is because I just am not confident that the game can get done with what is basically 1 1/2 guys working at half salaries for 4 months as a favor to a friend. I’m guessing it’s a 50/50 chance in that scenario the guy would just drop out or not put enough effort in, and the game either wouldn’t get done or wouldn’t get done right.

The more I think about this, the more confident I am it is the right decision.

First, with the 1 1/2 man company working on the cheap, if there’s a problem and something takes longer than expected, there’s only two possible outcomes – it’s late or features are dropped. There’s no profit margin to cut into to hire extra guys, and with so few people they are already working at 110%. With an established company, guys can be moved from other projects and the art can still get done on time.

Second, with the 1 1/2 man company, the people who will be working are unknowns – sure there’s a certain level of reliability when hiring people you know and trust, but at the same time it’s not like they are established in the job. There are growing pains, and they may only be taking the project conditionally, until they get an offer they can’t refuse. People are much more likely to leave on a new, low paid, temp project than from a company they have already been working at and living off of.

Third, 1 1/2 people is so few people that if one guy goes down, so does the whole team. With a company, there are extra workers, computers, backup power generators, etc. – basically factors to deal with one person or computer going down.

Lastly, it’s the choice between two bad alternatives – having the game done at a worse quality, or not at all. The worse art quality will at most cost me some percentage of subscribers, which I doubt will be significant. All I need is the art to be good enough to get people to notice the gameplay. The other alternative, not having the art done at all, is guaranteed to cost me 100% of subscribers. So in that context the choice is obvious.

My art bidding process has narrowed down to 3 companies. I’m really not sure who to go with.

The first one is in the Slovak Republic. They want $30,000 for 4 artists for 4 months. This sounds like a great deal and I was going to move forward with it. But they sent me some sketches that show they totally ignored what my asset list specified, which is that I’m going for a believable military look. The reason this is such a major problem is that if they are making such fundamental mistakes that even I can see it, what kind of mistakes will they make for things I can’t see, and will just get a bad looking game? For example, maybe a poor choice of a color scheme, or bad contrast between the background and the ships. If I was already an artist it would be OK but I have to really entrust that the company I go with knows what they are doing and I didn’t see that here.

The second is the guy who was on World of Warcraft. This is $30,000 for 1 1/2 artists for 4 months, plus himself directing. From talking to him I can tell he understands the requirements and isn’t going to screw up direction, the way the first company might. Plus he wants to work for me after my game ships, and it would be great to have the same guy doing both pre and post release art direction. My major concern is that I only get 1 1/2 artists to do the work that other art companies have been estimating between 2 to 4 artists. This kind of estimate cuts no slack at all – if it turned out halfway through he thought it was an undoable amount of work, I’d be stuck high and dry, where with a bigger company, where I had slack to begin with, they could assign more people. Even if he refunded my money, which I doubt would happen, I’d be behind schedule and it would kill me in other ways.

The third is a company in Brazil. They’ve quoted $30,000 for 8 man months of work, done over a period of 4 months. That’s actually not a very good deal. Even a top-notch artists in Brazil makes about $2000 US a month, and I’m being charged double that. On the other hand, they are very professional and they’ve clearly done high quality work for major companies. If their estimate is accurate (2 guys for 4 months) then I could just as well go with the company in the US and avoid the time zone and communication problems. I’m also concerned that they are taking a literal interpretation of my asset list. While it covers most of what my game needs, it’s obviously impossible to think in advance of every possible effect my game might have, especially since I don’t plan to do the art direction myself.

I wrote back to the third company clarifying what I need. If they are taking a literal interpretation of my asset list, or they misunderstood the amount of content I wanted, I will go with the second company. If everything was clear I’ll probably just go with them.

I have 5 companies bidding to do the artwork for my game now.

Two are in preliminary stages and didn’t get cost estimates back to me yet. It’s possible they will go nowhere. One in China and one somewhere else (I think Europe).

One, in Brazil, quoted $60,000 which is absurd, because workers in Brazil make 1/4 of what we do. So it’s the equivalent of a US company asking for $240,000. This is 4 months of work on an Indy game. What the guy told me is he charges by meters, so if a spaceship is $100 a star destroyer 100x larger is $10,000. That’s a stupid metric. If this was a first person shooter and you were walking on the surface of the Star Destroyer, and the SD was the entire game, then maybe I’d use that much detail. But it’s a space game zoomed way out. Symmetrical with windows as filler is good enough. Either they are trying to rip me off or the communication is beyond horrible. Either way, I don’t want them, so I told them to forget it.

One, in the US, will be between $20,000 and $40,000, not sure yet. I think they could do a good job but they are very poor at responding when they say they will. So I might not go with them because if it’s this bad now I think it will probably be worse after I pay them.

Another, in the US, will be $30,000 and will get back to me in 2 days. It will be an art director, who formerly did WoW, and a couple of juniors. The communication is really good and I’m moderately confident that it will get done correctly.

I’ve interviewed 6 lead or art directors in the last 2 days.

The first guy seems honest and keeps his word. But my super art friend Jeff said 50% when I passed along the website, which I assume is bad. Pass.

The second guy sounds very professional on the phone which I like. But he said he’d get a quote back to me on art assets, and then didn’t. Plus he wants between 95 and 100K a year which is just too high for an artist at a small company. Pass.

The 3rd guy I think can do the work, but lacks assertiveness and wouldn’t be able to manage people effectively. Jeff said 50% when I asked him about the work quality. It doesn’t look good to me either, but I never know if that’s because of the graphic engine or what. Since Jeff said 50%. Pass.

The 4th guy sounds professional and is more honest than the other guys I’ve spoken to. His art looks good as well. But he also seems slow, saying it would take between 1 and 2 weeks to model a star destroyer, depending on the amount of detail. He’s also weak at 2D art, which is something I need. By slow I mean giving longer estimates than the other guys. I don’t know if 1-2 weeks is actually slow. He sounds truthful and if that’s how long it takes then that is how long it takes. For the 2D art I’ll just have to find workarounds, if I do indeed go with him.

The 5th guy spent the first 15 minutes of the interview complaining about his current managers and also only has 2 years of experience. I don’t think I’ll hire him, except as a junior artist and then only if a more senior artist recommends him. His only redeeming factor is that he’s cheap, and going for cheap has wasted more money than it’s saved with programmers. Pass.

The 6th guy said on his resume that he had extensive experience with C/C++, but on the phone said he wasn’t a programmer at all. The claimed experience came from looking at code or something. I didn’t get a clear answer on that. Everything he said I liked, including his time estimate (1 week for star destroyer), but since he lied on his resume for all I know he lied during the interview too. Pass.

Got another guy tomorrow, best looking art so far. Hopefully that goes well.

I’ve been spending a great deal of time looking for and interviewing artists lately.

A few points I’ve learned:

• Posting an ad on Gamasutra is expensive and not effective. It’s much more effective to browse resumes and contact people directly.
• People tend to come as a whole package. You don’t get too many idiot savants who can’t spell but are great artists, or who can’t keep a schedule but are great programmers. If an application email or portfolio looks bad, or has bad spelling, capitalization, or punctuation, it’s not worth responding.
• Older people are more reliable.
• You get what you pay for. Someone asking for a ridiculously low salary based on their experience and location is a sure sign of someone nobody else wants to hire.
• If someone doesn’t exhibit desired qualities in an interview, they aren’t going to exhibit them after you hire them, whether you try to coach them to do so or not.
• If you hired the right person, they will impress you within the first hour. If they don’t impress you in the first hour, you hired the wrong person.
• Positive management skills can polish a lump of gold, but can’t turn coal into a diamond.
• The world is filled with bad workers and the vast majority of people that apply to you are such. The good ones are rare and hard to find. Hold onto them if you find them.

I found a really neat site 3drt.com that may be the answer to most of my artist problems. They have a bunch of premade packs of spaceships for sci-fi games. While it’s not the most novel way to go, I otherwise see it as an excellent way to get a lot of art into the game with minimal effort. So I ordered 4 packs of art for about $350. While not free, it’s practically nothing compared to the equivalent cost of having an artist spend 2 weeks on a model ($4000 or so). Plus I save those two weeks meaning we can ship earlier.

Theoretically we have every fighter we need in the game right now, and also the starbase, and also the weapons. All that is left is a couple of carriers, the graphics effects, and the UI.

Speaking of which, I’ve been interviewing artists today and will continue to do so over the next few days. The problem, as I expected, is that it’s hard for me to tell who is good and who is not. So I just go by years of experience and my general feel for their enthusiasm as a person. Because my standards are so high, the people I’ve been interviewing are really expensive, making nearly as much as I used to make as a programmer (and I made a small fortune, which is why I’m able to afford this game now). However, you get what you pay for and when it comes to top quality professionals for artists I won’t compromise.

Tomorrow I have an interview with a guy that lives only 5 minutes away. That would be really helpful since I can physically see what they are working on, and really save a lot of communication time.

A difficult problem with action-based online games is data inaccuracy caused by the inability to trust clients.

On one extreme, a game can trust the client completely. So whatever the client tells you for position, orientation, velocity, etc, is correct. This way is both easy and results in very smooth gameplay for the clients. You never have the character jerk around or other funny stuff.

However, the problem is obvious: A hacked client can then run faster, teleport, move through walls, and do other things.

Sometimes this is tolerable. For very small games, nobody would bother. For heavily moderated games, this will be obvious to an administrator, who can ban the offender. However, even that solution isn’t that great because with human greed and stupidity as it is, you’ll end up banning a large portion of your player base. This is worse in a paid game, because not only do you lose that money, but it will result in angry chargebacks and complaints. “You banned my account!!! I didn’t do anything wrong!!! WAAAHHHH!!! I don’t know who put that program on my machine!!!! WAAAHHHH!!!”

Lets avoid that hassle. Lets not give the client more power than it usually has, namely pressing keys at certain time and that’s it. There are two problems with this approach: Past inputs and inaccuracy due to entropy.

By past inputs I mean that the commands you get from the client are in reference to a gamestate some number of milliseconds ago, rather than the present. Worse, you can’t even be sure of the times that you get due to variable pings. Worst, the client may be cheating and sending you the wrong times anyway.

Accuracy is a problem because even with zero latency, if all you were to do is process remote inputs, due to calculation, framerate, and timer differences, given enough time the positions would be off anyway.

Here is one way to semi-accurately fix the problem and still look good.

1. For every game object which has untrusted client input, we need to track the state history for the server. This is basically a list of states every 50 milliseconds for the last second. State history includes everything affected by non-discrete untrusted input. Actions, position, orientation, velocity (both angular and positional), etc.

2. Each client datagram should contain the the state of the client object (same info as the history in step 1) plus a timestamp.

3. Server: On receipt of a state update datagram.
A. Validate the timestamp by clamping between the lowest and highest of the last 5 pings. If outside these ranges, add 1 point to a “Cheat metric” (explained later) for this client.
B. Given the timestamp in A. find the nearest corresponding state in the history. Compare the state in the history against what is in the datagram for reasonableness. Reasonable means the position and orientation are not off by more than what is possible given that client’s ping, and that the client does not go through walls or warp. For each value that is unreasonable, add 1 or more points to the cheat metric, depending on how unreasonable the value is. If any values are unreasonable, use the state from the history. If no values are unreasonable, use what the client has given us.
D. For the state selected in C. extrapolate that state to the present given the set of actions specified by that state (such as running forward and turning right by 1 radian per second). This doesn’t have to be perfectly accurate. We just want to make sure that the client cannot run through walls. Fill out the history with the extrapolated values, and set the current state to what we have extrapolated.
E. If any values in B. were unreasonable, flag that we need to send a timestamped state update to the sender for the sender’s own avatar (basically force them to the right spot). Otherwise, do not send state for the sender’s avatar to the sender. Signal this object as updated, so that state updates are relayed to all viewers.

4. Client: On receipt of a state update datagram.
A. Extrapolate the state in the datagram, given the timestamp in the datagram.
B. If there is a wall or other blocking objects between our current position and the position in the datagram, just warp.
C. If the position in A is really far away, just warp.
D Otherwise, subtract the state in A from our current state. Record this as a state delta, and the current time.
E. Add the state delta over a short period of time (say 250 milliseconds) to the current state.

5. Server cheat metric.
When players cheat, they usually do not just try to cheat once, but will attempt to cheat repeatedly through the game over time. The cheat metric is simply a score that goes up on suspicious input, and down over time. When the cheat metric past a threshold, it should be logged and an admin notified to watch the player. The values can then be adjusted if they are too sensitive, or action taken against the cheater.

Most games, including all the shooters I know of, use repeated unreliable sends for player updates.

Repeating unreliable sends:
1. Send position, orientation, health
2. [50 ms later]
3. Go to 1

The advantage of this technique is it is tolerant of packetloss and ping spikes. If a packet is lost, it doesn’t matter, because another is coming in 50 ms later. So you never see the player falling to move, then all of the sudden warping somewhere.

Sequenced Repeating unreliable sends:
1. Send position, orientation, health, timestamp
2. [50 ms later]
3. Go to 1

On the recipient:
Message arrives.
if (timestamp in message > last timestamp we got) then process the message and set the last timestamp to the timestamp in the message
else ignore the message.

This way you don’t get out of order or old messages messing up the stream.

So this technique is good for removing the effect of latency spikes and packet loss, but is also inefficient. If the player isn’t moving you still send these messages. If the player is just running in a straight line, you still send these messages, although it should be possible for the recipient to get a “run forward” message and just keep doing that unless it hears otherwise. You can fix this with event based sends

Reliable event based sends:
If [player presses a key] then send this key, and the current player status.

That’s very efficient, but the problem is if the message is lost then the player may have run away a long time ago, yet you still see him standing there.

You can send this message reliably but then you are subject to ping spikes. By the time the system can assume that message was lost and retransmit, it may be half a second later, and you see a pop. This won’t happen very often (maybe 2% of the time) but with 1 command a second and 30 players on the screen, you would see 1 pop every second. It would look awful.

For an MMOG, I need to use the absolute minimum bandwidth, but also not have pops.

Event based networking with staggered unreliable datagrams

1. (Constructor) Set sendStaggerTime = [large value]; Set nextSendTime=0;
2. (On user action update)
Set sendStaggerTime=50.
If (nextSendTime+sendStaggerTime < currentTime) then
Set nextSendTime = currentTime;
3. (Update loop) If (sendStaggerTime <= 200 && nextSendTime <= currentTime) then
Send update.
Set sendStaggerTime*=2
Set nextSendTime=currentTime+sendStaggerTime;


What this means is that:
1. Updates will never be sent faster than every 50 ms.
2. Actions will be sent 0 ms later, 100 ms later, 300 ms later. It will then stop sending.

I send using RakNet’s unreliable sequenced capability, which means that I can be assured that I don’t get messages out of order.

The advantages of this technique:
1. If a ping spike affects one message so that it arrives late, in all likelyhood the next message will arrive within 100 or 200 ms, thus minimizing the effect of the spike.
2. For a message to completely not arrive, all 3 messages over a total period of 300 or 350 ms would have to be lost. This is unlikely enough that that if it does happen it won’t ruin the game.
3. Simply performing one action, and continuing to perform that action (or no actions at all) result in 3 sends, and then no further sends.
4. Messages are sent unreliably, but arrive reliability without incurring the memory overhead of tracking messages for resending.
5. The client doesn’t send needless duplicate updates while the server relays updates immediately.
6. If I get 3 updates for every action, I can use the update with the lowest timestamp value, thus resulting in the most correctness for interpolation.
7. Errors are minimized. If on the server I’m stuck on a corner, but I barely went around it on the client, enough time elapsed by the 3rd message that I can fix this and know the client actually went around.

Disadvantages:
The most common case is that a message will arrive on-time and without loss (97% of the time or so). Yet I’m sending each message 3X, which means I waste two sends most of the time.

The reason for the numbers I picked:
50 ms – players won’t notice granularity less than 50 ms. Also, hitting 2 keys at once will probably occur within 50 ms, so only one send occurs, rather than 2.
100 ms – If the first message is lost to due ping spikes or packetloss, the second will arrive soon enough that it won’t harm the game.
200 ms – If the first two messages are lost, this is likely due to a prolonged period of spike or packetloss. 200 ms is probably long enough to get past that temporary spike, with modern internet routers.