Month: December 2006

I’ve interviewed 6 lead or art directors in the last 2 days.

The first guy seems honest and keeps his word. But my super art friend Jeff said 50% when I passed along the website, which I assume is bad. Pass.

The second guy sounds very professional on the phone which I like. But he said he’d get a quote back to me on art assets, and then didn’t. Plus he wants between 95 and 100K a year which is just too high for an artist at a small company. Pass.

The 3rd guy I think can do the work, but lacks assertiveness and wouldn’t be able to manage people effectively. Jeff said 50% when I asked him about the work quality. It doesn’t look good to me either, but I never know if that’s because of the graphic engine or what. Since Jeff said 50%. Pass.

The 4th guy sounds professional and is more honest than the other guys I’ve spoken to. His art looks good as well. But he also seems slow, saying it would take between 1 and 2 weeks to model a star destroyer, depending on the amount of detail. He’s also weak at 2D art, which is something I need. By slow I mean giving longer estimates than the other guys. I don’t know if 1-2 weeks is actually slow. He sounds truthful and if that’s how long it takes then that is how long it takes. For the 2D art I’ll just have to find workarounds, if I do indeed go with him.

The 5th guy spent the first 15 minutes of the interview complaining about his current managers and also only has 2 years of experience. I don’t think I’ll hire him, except as a junior artist and then only if a more senior artist recommends him. His only redeeming factor is that he’s cheap, and going for cheap has wasted more money than it’s saved with programmers. Pass.

The 6th guy said on his resume that he had extensive experience with C/C++, but on the phone said he wasn’t a programmer at all. The claimed experience came from looking at code or something. I didn’t get a clear answer on that. Everything he said I liked, including his time estimate (1 week for star destroyer), but since he lied on his resume for all I know he lied during the interview too. Pass.

Got another guy tomorrow, best looking art so far. Hopefully that goes well.

I’ve been spending a great deal of time looking for and interviewing artists lately.

A few points I’ve learned:

• Posting an ad on Gamasutra is expensive and not effective. It’s much more effective to browse resumes and contact people directly.
• People tend to come as a whole package. You don’t get too many idiot savants who can’t spell but are great artists, or who can’t keep a schedule but are great programmers. If an application email or portfolio looks bad, or has bad spelling, capitalization, or punctuation, it’s not worth responding.
• Older people are more reliable.
• You get what you pay for. Someone asking for a ridiculously low salary based on their experience and location is a sure sign of someone nobody else wants to hire.
• If someone doesn’t exhibit desired qualities in an interview, they aren’t going to exhibit them after you hire them, whether you try to coach them to do so or not.
• If you hired the right person, they will impress you within the first hour. If they don’t impress you in the first hour, you hired the wrong person.
• Positive management skills can polish a lump of gold, but can’t turn coal into a diamond.
• The world is filled with bad workers and the vast majority of people that apply to you are such. The good ones are rare and hard to find. Hold onto them if you find them.

I found a really neat site 3drt.com that may be the answer to most of my artist problems. They have a bunch of premade packs of spaceships for sci-fi games. While it’s not the most novel way to go, I otherwise see it as an excellent way to get a lot of art into the game with minimal effort. So I ordered 4 packs of art for about $350. While not free, it’s practically nothing compared to the equivalent cost of having an artist spend 2 weeks on a model ($4000 or so). Plus I save those two weeks meaning we can ship earlier.

Theoretically we have every fighter we need in the game right now, and also the starbase, and also the weapons. All that is left is a couple of carriers, the graphics effects, and the UI.

Speaking of which, I’ve been interviewing artists today and will continue to do so over the next few days. The problem, as I expected, is that it’s hard for me to tell who is good and who is not. So I just go by years of experience and my general feel for their enthusiasm as a person. Because my standards are so high, the people I’ve been interviewing are really expensive, making nearly as much as I used to make as a programmer (and I made a small fortune, which is why I’m able to afford this game now). However, you get what you pay for and when it comes to top quality professionals for artists I won’t compromise.

Tomorrow I have an interview with a guy that lives only 5 minutes away. That would be really helpful since I can physically see what they are working on, and really save a lot of communication time.

A difficult problem with action-based online games is data inaccuracy caused by the inability to trust clients.

On one extreme, a game can trust the client completely. So whatever the client tells you for position, orientation, velocity, etc, is correct. This way is both easy and results in very smooth gameplay for the clients. You never have the character jerk around or other funny stuff.

However, the problem is obvious: A hacked client can then run faster, teleport, move through walls, and do other things.

Sometimes this is tolerable. For very small games, nobody would bother. For heavily moderated games, this will be obvious to an administrator, who can ban the offender. However, even that solution isn’t that great because with human greed and stupidity as it is, you’ll end up banning a large portion of your player base. This is worse in a paid game, because not only do you lose that money, but it will result in angry chargebacks and complaints. “You banned my account!!! I didn’t do anything wrong!!! WAAAHHHH!!! I don’t know who put that program on my machine!!!! WAAAHHHH!!!”

Lets avoid that hassle. Lets not give the client more power than it usually has, namely pressing keys at certain time and that’s it. There are two problems with this approach: Past inputs and inaccuracy due to entropy.

By past inputs I mean that the commands you get from the client are in reference to a gamestate some number of milliseconds ago, rather than the present. Worse, you can’t even be sure of the times that you get due to variable pings. Worst, the client may be cheating and sending you the wrong times anyway.

Accuracy is a problem because even with zero latency, if all you were to do is process remote inputs, due to calculation, framerate, and timer differences, given enough time the positions would be off anyway.

Here is one way to semi-accurately fix the problem and still look good.

1. For every game object which has untrusted client input, we need to track the state history for the server. This is basically a list of states every 50 milliseconds for the last second. State history includes everything affected by non-discrete untrusted input. Actions, position, orientation, velocity (both angular and positional), etc.

2. Each client datagram should contain the the state of the client object (same info as the history in step 1) plus a timestamp.

3. Server: On receipt of a state update datagram.
A. Validate the timestamp by clamping between the lowest and highest of the last 5 pings. If outside these ranges, add 1 point to a “Cheat metric” (explained later) for this client.
B. Given the timestamp in A. find the nearest corresponding state in the history. Compare the state in the history against what is in the datagram for reasonableness. Reasonable means the position and orientation are not off by more than what is possible given that client’s ping, and that the client does not go through walls or warp. For each value that is unreasonable, add 1 or more points to the cheat metric, depending on how unreasonable the value is. If any values are unreasonable, use the state from the history. If no values are unreasonable, use what the client has given us.
D. For the state selected in C. extrapolate that state to the present given the set of actions specified by that state (such as running forward and turning right by 1 radian per second). This doesn’t have to be perfectly accurate. We just want to make sure that the client cannot run through walls. Fill out the history with the extrapolated values, and set the current state to what we have extrapolated.
E. If any values in B. were unreasonable, flag that we need to send a timestamped state update to the sender for the sender’s own avatar (basically force them to the right spot). Otherwise, do not send state for the sender’s avatar to the sender. Signal this object as updated, so that state updates are relayed to all viewers.

4. Client: On receipt of a state update datagram.
A. Extrapolate the state in the datagram, given the timestamp in the datagram.
B. If there is a wall or other blocking objects between our current position and the position in the datagram, just warp.
C. If the position in A is really far away, just warp.
D Otherwise, subtract the state in A from our current state. Record this as a state delta, and the current time.
E. Add the state delta over a short period of time (say 250 milliseconds) to the current state.

5. Server cheat metric.
When players cheat, they usually do not just try to cheat once, but will attempt to cheat repeatedly through the game over time. The cheat metric is simply a score that goes up on suspicious input, and down over time. When the cheat metric past a threshold, it should be logged and an admin notified to watch the player. The values can then be adjusted if they are too sensitive, or action taken against the cheater.

Most games, including all the shooters I know of, use repeated unreliable sends for player updates.

Repeating unreliable sends:
1. Send position, orientation, health
2. [50 ms later]
3. Go to 1

The advantage of this technique is it is tolerant of packetloss and ping spikes. If a packet is lost, it doesn’t matter, because another is coming in 50 ms later. So you never see the player falling to move, then all of the sudden warping somewhere.

Sequenced Repeating unreliable sends:
1. Send position, orientation, health, timestamp
2. [50 ms later]
3. Go to 1

On the recipient:
Message arrives.
if (timestamp in message > last timestamp we got) then process the message and set the last timestamp to the timestamp in the message
else ignore the message.

This way you don’t get out of order or old messages messing up the stream.

So this technique is good for removing the effect of latency spikes and packet loss, but is also inefficient. If the player isn’t moving you still send these messages. If the player is just running in a straight line, you still send these messages, although it should be possible for the recipient to get a “run forward” message and just keep doing that unless it hears otherwise. You can fix this with event based sends

Reliable event based sends:
If [player presses a key] then send this key, and the current player status.

That’s very efficient, but the problem is if the message is lost then the player may have run away a long time ago, yet you still see him standing there.

You can send this message reliably but then you are subject to ping spikes. By the time the system can assume that message was lost and retransmit, it may be half a second later, and you see a pop. This won’t happen very often (maybe 2% of the time) but with 1 command a second and 30 players on the screen, you would see 1 pop every second. It would look awful.

For an MMOG, I need to use the absolute minimum bandwidth, but also not have pops.

Event based networking with staggered unreliable datagrams

1. (Constructor) Set sendStaggerTime = [large value]; Set nextSendTime=0;
2. (On user action update)
Set sendStaggerTime=50.
If (nextSendTime+sendStaggerTime < currentTime) then
Set nextSendTime = currentTime;
3. (Update loop) If (sendStaggerTime <= 200 && nextSendTime <= currentTime) then
Send update.
Set sendStaggerTime*=2
Set nextSendTime=currentTime+sendStaggerTime;


What this means is that:
1. Updates will never be sent faster than every 50 ms.
2. Actions will be sent 0 ms later, 100 ms later, 300 ms later. It will then stop sending.

I send using RakNet’s unreliable sequenced capability, which means that I can be assured that I don’t get messages out of order.

The advantages of this technique:
1. If a ping spike affects one message so that it arrives late, in all likelyhood the next message will arrive within 100 or 200 ms, thus minimizing the effect of the spike.
2. For a message to completely not arrive, all 3 messages over a total period of 300 or 350 ms would have to be lost. This is unlikely enough that that if it does happen it won’t ruin the game.
3. Simply performing one action, and continuing to perform that action (or no actions at all) result in 3 sends, and then no further sends.
4. Messages are sent unreliably, but arrive reliability without incurring the memory overhead of tracking messages for resending.
5. The client doesn’t send needless duplicate updates while the server relays updates immediately.
6. If I get 3 updates for every action, I can use the update with the lowest timestamp value, thus resulting in the most correctness for interpolation.
7. Errors are minimized. If on the server I’m stuck on a corner, but I barely went around it on the client, enough time elapsed by the 3rd message that I can fix this and know the client actually went around.

Disadvantages:
The most common case is that a message will arrive on-time and without loss (97% of the time or so). Yet I’m sending each message 3X, which means I waste two sends most of the time.

The reason for the numbers I picked:
50 ms – players won’t notice granularity less than 50 ms. Also, hitting 2 keys at once will probably occur within 50 ms, so only one send occurs, rather than 2.
100 ms – If the first message is lost to due ping spikes or packetloss, the second will arrive soon enough that it won’t harm the game.
200 ms – If the first two messages are lost, this is likely due to a prolonged period of spike or packetloss. 200 ms is probably long enough to get past that temporary spike, with modern internet routers.

I contacted a few more companies for art outsourcing and got some replies. From talking to them, art outsourcing is usually done when you have concepts and know exactly what you want. It’s not usually done for what I need, which is art direction and iterative level design. So I probably won’t go with outside companies for art.

I’m trying to find some artist websites now where I can post free job offers. My real offer is going to be posting an expensive ($350) ad on Gamasutra. I’m going to post it tomorrow, as most people will not work that day so my ad can stay on top for a while.

It’s going to be a problem hiring someone good enough. I’m just going to go by the strength of their resumes and see how smart they sound on the phone. I’ll also go with the hourly pay thing, so I don’t have to worry about wasting money paying someone who isn’t working.

I contacted about 5 companies that posted ads on Gamasutra advertising art outsourcing. All 5 ignored me. I also contacted about 5 developers regarding licensing their models. 2 replied, one of which ignored me after I sent the NDA over to be signed.

I don’t understand why this is so difficult. Admittedly I’m not EA, but I am offering tens of thousands of dollars for what amounts to a very small amount of work. Why bother posting an ad only to ignore customer inquiries?

This is a good thing when I think about it. If a company treats a potential customer this way, imagine how they treat customers who have already paid, or have some investment at stake? Whoever I ultimately pay to do the artwork needs to have my game as the first priority and being ignored doesn’t indicate to me that is the case.

It might work out for the best if I just hire someone after Christmas. At least that way I control the ultimate quality. With a 3rd party developer, they don’t have my best interests at heart.

My game has reached the point now where I need to hire an artist, and two new programmers are starting full-time, for a total of 4 programmers, not including myself. Additionally, I need to get a full-time artist on board. So I’m finally at the point of serious risk, where I should either fully commit to the project or drop it.

Any point before now and all I would lose is a little bit of money. After this point, I stand to lose a lot of money since my monthly expenditures are so high. So when evaluating risk factors, there’s really three points of failure right now.

The first is that the game doesn’t scale, which is similar the risk that I release too soon. If too many people see the game before it is fully tested and just really polished, those are permanently lost customers. And as I said in a previous post, you never have a better time than release to take off. If you botch the release, you will either never recover or it will take a very long time. The way to avoid this risk is through extended alpha and beta testing, but that is expensive, so there’s definitely a balance to be kept.

The second is that I make a bad hire for the artist. I’m only going to hire one artist, and whoever I hire needs to be basically one of the best in the world. Fast, intelligent, able to do any kind of work, AAA quality, self-motivated, experienced, knowledgeable, and communicates. I’m talking director-level quality – someone who likely has their own company already. The reason for this is that I’m not an artist myself, so there is no recourse if the person I hire does not know how to do something, or does it incorrectly. In all probability I won’t even know what is done wrong, or at a poor level of quality, until it’s too late. The risk here is that since I’m hiring one of the best in the world, I am planning to pay an according amount of money, and that means if I mess up and pay the wrong guy even for one month it’s going to be a huge risk to the entire project. I don’t have a good solution to this right now. All I can really think of is to hire someone with a ton of real experience, and to check references.

The third is billing. If customers try to pay and cannot, or credit card numbers are exposed, or anything else goes wrong related to money it will likely take down the project with it, especially if it results in chargebacks. This is probably the least risky, because I know someone good who I think can do it, but it’s one of those things you never can plan for.