Month: June 2007

Galactic Melee is turning out pretty well.

From the game side

Graphics: All positive compliments My goal wasn’t to make the best looking game in the world, but to make something marketable, and in that I succeeded.
Performance: Every one I’ve tested with has gotten over 60 FPS even with lots of bullets on the screen.
Sound: DSonic did a good job for a fair price. The game still needs tweaks and polish but I have no complaints.
Music: I bought a few songs, they are all pretty good. My original goal was to get something very memorable and distinct (like the SMB music, or Zelda music, or Star Wars music). While I don’t have that what I have is good enough. By buying premade songs I saved a lot of time and money and got something higher quality than I likely would have through direct contracting.
Gameplay: Most of the features I wanted have made it into the game. 5 game modes, about 20 items + weapons, a leveling system, and a database to store stats. The only major missing feature is the command screen, where you could give orders to your fellow pilots. However, I don’t see this missing feature as losing sales. It’s still to-be-seen how people like the game design but since I control this it’s not a major risk factor. Peer to peer voice chat is also missing but it’s also not an issue that will lose sales.

From the business side:

Website: It looks good, but lacks content. I think with content it’s good enough to drive interest, which is the whole point.
Cost: Massively overbudget and over time. On the other hand, if the game does well it can make it back. This is the risky part. I’m definitely out of time and money so it’s time to ship or die.
Art development: Late, overbudget, but things got done in the end. This was in large part because I learned how to do everything myself, so was able to and did fix everything.
Programming: Late, overbudget, plus had incredible difficulty finding good people. With the exception of my excellent programmer in Portugal, I was consistently disappointed. As with art, I eventually had to learn how to do everything myself, and was able to and did fix almost everything.
Marketing: Big risk point, we’ll see how well word of mouth does. I believe that marketing is what businesses do to deceive people into buying products that are not good enough to sell themselves. Plus, I don’t want a million customers at launch because I’m not able to support that yet. I’m looking for a few thousand sales to pay for further development before really branching out.
Server hosting: Hypernia‘s service was mixed at first, but once they moved their data centers things got much better. Now that they know I’m going to be a huge customer the service is really good. I have a game server and a database server and it’s easy to replicate game servers.
Billing: Aria is helping me here. It’s a good service because developing a billing system is more than just taking credit cards. You have to provide a way for users to manage their accounts, it has to be secure, and you have to be verified by various third parties. It would cost so much to do all that it wouldn’t be worth it for small games. Word of advice: I got very bad service from my original representative, so asked for another. I then got very good service. Don’t assume the service you get from one guy is representative of the whole organization.

All that’s left to do is get the game running bug-free, polish it so it’s fun, and then hope people like it and draw in sales.

My “slippery tail” interpolation worked well for rotation. Rotation has no inertia (start and stop immediately). The magic number to lag behind is half your ping to the server. This definitely wasn’t obvious why and it took me an hour of thinking to really see it.

It doesn’t work for position. Theoretically it should work, but in practice small errors in timing, ping variances, and high latency all cause failure. On the front side, when I am finishing a prior interpolation, if my new extrapolated position doesn’t match exactly, I get a position warp. This happens easily from ping variance. On the back side, with very high pings the extrapolated position differences are so large it just ends up looking like a warp.

For position I’m trying what I call trajectory intercept interpolation. When the datagram arrives I store the current visual position and physics velocity. The start interpolation position is where you would normally be (visual position + original velocity * elapsed interp time). The end interpolation is the intercept when we stop interpolating (real position + real velocity * remaining interp time).

This method doesn’t jerk (jerk is a change in acceleration) which I do get if I were to not count original velocity. It also gives me an accurate visual position. Hopefully it will work.

Other problems fixed:
PhysX jitters using fixed timesteps. Fixed by changing to variable.
Stopped using per-datagram timestamps. Just use half the average ping. This also removes a potential cheat.
I was previously allowing you to accelerate past your maximum velocity, then clamping the next tick. This caused inaccuracy. Now I have to duplicate the physics calculations and determine the force to apply to make you reach your maximum velocity exactly.
Previously network packets were processed after the tick update, so clearing acceleration didn’t take until one tick too late, resulting in slightly too high velocity.

These bugs were massively difficult to find. This has been a very painful day.

I’m having a problem with high latency players jumping around a lot.

Originally, I would extrapolate your position when a message came in. I would then do a lerp over some fixed amount of time between your current visual position and your new extrapolated position.

This is accurate in that you see where the object currently is. The problem is that when you suddenly change directions there is a hop.

Suppose I am moving y+10 per second and 100 milliseconds later I move y-10 per second. If the sender’s plus my own latency is 1 second, I would see that object 10 units up, then 100 milliseconds 10 units down. A huge hop.

I thought about this a long time and I came up with what I call slippery tail interpolation, although likely this isn’t novel and there’s a real name for it. But it just gives me the impression of holding onto a snake with oil on its tail and the tail keeps moving around.

I keep a transformation history of your position, orientation, and velocity for the last second. I also locally update your transformations based on your current velocity, acceleration, and rotational velocity. However, when a network message arrives, this negates part of the history (the slippery tail). So I recompute where you should have been over that amount of time, and rewrite history. My movement is complex, with a maximum velocity and a constant rotation which changes your thrust angle. I can’t think of any mathematical way to calculate your final position, so I just integrate over a fixed timestep.

Visually, I lag behind by half the sender’s average ping plus half your own average ping. This is the edge of the tail, just before the slippery part. So as history is rewritten, it is not likely to change what you see, only what you will see, and hence there is no popping around. However, if due to ping variances a message is so far in the past that it changes what we see, then I do linear interpolation.

Hopefully this will work. The hard part is going to be testing it. I’m also a little concerned that if you see everything slightly in the past it’s going to make gameplay less accurate. I might have a tweak where I move up the tail a bit so there’s some hopping but then visual positions are more accurate. Another possibility is to hit detection based on your past state.

PhysX doesn’t have a nudge feature, by which I can slightly adjust the position of an object, while still accounting for collisions. What I had to end up doing was adding a velocity to move me by an estimated amount for the next tick, then undo that velocity. Naturally this caused a lot of problems, such as breaking my maximum velocity limit, causing you to potentially hit walls at great velocity, and it’s inaccurate.

I’m surprised this feature doesn’t exist, since every networked game needs it.

I wonder what other physics libraries might have this feature?

Incidentally, I am not happy that PhysX requires you to use their system installer thing. Users are complaining about having to install so many dependencies, and that leftovers are left after they uninstall the game. I totally agree. This might even cost me sales. With a store bought game you have no choice, but with my game since there’s a free trial the user has a low investment and might just cancel the install.

Speaking of which, I am curious about why other programs I install don’t all prompt you to install the Visual Studio 2005 end user runtime. If I could get rid of that it would knock down my dependencies by 1/3.

I changed the movement algorithm today.

Formerly:

Client sends inputs (position, rotation, velocity, acceleration, angular rotation).
Check each field independently using some pretty loose and generous metrics. Do a raytrace to make sure the client didn’t go through a wall.
If any field is off by too much, just use the current values on the server, and force update the sender.

The problems with that are:
1. Requires a raytrace
2. Easy to cheat
3. Accuracy and visibility of lag depend on the client – slow clients give bad performance

I changed it to:

Client sends inputs (position, rotation, velocity, acceleration, angular rotation).
Clamp the timestamp to a maximum value
Look in the transformation state history to find out where the client was back when they created the packet.
Verify input values with a maximum error of 100 milliseconds. If a value is wrong, set the value to the transformation history value

This is better because:

1. Transformation history lookup is perfectly accurate
2. I can use a tight clamp because of this accuracy, so I don’t need to check for going through walls, since the timestamp is too low to ever go through one anyway.
3. Faster with no raytrace.
4. Performance is always good to other clients, no matter how bad the sender is.

However, it does give worse feel to the sender if they are very slow or lagged. They will warp around occasionally as the server corrects them. The warp is interpolated, but still noticeable.

I couldn’t figure out how to get the quaternion representing the rotation from A to B. Laugh if you want but I don’t use Quaternions enough to remember and have to reinvent the wheel every time.

20 minutes later of looking at Real-Time rendering

// start * delta = final
// start^-1 * start * delta = start^-1 * final
// delta = start^-1 * final

The inverse of a quaternion q is conjugate(q) / length(q)^2

For a unit quaternion, the length is 1.

For the conjugate, flip the sign of x,y, and z.

The first 3 years of my programming career I was a telecommuter. At that time I was entirely in favor of telecommuting for all information based jobs. In my opinion, the only reason companies didn’t allow telecommuters was because of anal managers grasping for control. Managers that didn’t trust people to do their work and foolishly thought that having you physically there made any difference. It was an outdated and pointless business model, and smarter more progressive people would excel in the marketplace over these dinosaurs.

The few times I applied to jobs and asked for telecommuting, I was told they didn’t allow it because of a lack of communication. I considered this to be a lame cop-out. Simply an excuse to justify their own anal behavior and lack of experience.

When I finally did start a non-telecommuting job, my opinion was only reinforced. I was twice as productive per unit time at home than in the office. Not only that, but at the office the computers were slower, there were a lot of distractions, I had to waste time driving to work, and I was more likely to work fixed hours than at all hours of the day as I do now. Overall, the company probably only got 1/3rd of their value for the money by forcing me to come into an office.

When I started this company my intention and initial design was to be 100% telecommuting. It was far cheaper for me to run the company, everyone would get those massive productivity games, I could hire anywhere in the world, and since people love to telecommute I could steal away the best talent from other companies.

That was 8 months ago. Now that my project is winding up I have some perspective on both sides of the issue. The truth is that telecommuting works, and it works and delivers the massive gains I always claimed it did – but only for programmers and then only for superstars. Everyone else fails miserably. A superstar has to excel in every relevant area: self-motivation, intelligence, attention to detail, communication skills, belief in the project, and organization. Since 98% of programmers are not superstars, and its very hard to tell who is and who isn’t just from an interview, you’re going to end up wasting a lot of money with bad hires, and firing a lot of people, as I did.

Self-motivation: One guy I hired worked hard the first three days, sort-of-worked the next week, then slacked off the remaining month. Emails queries just got excuses, until even under the most generous conditions I couldn’t believe it was taking him a month to write one UI screen. It’s very hard to keep motivated when nobody is forcing you to go into an office each day.

Attention to detail: The Indians failed here because of literalism, I believe due to cultural attitude of subservience. They do exactly what they think they are told, no more, no less, and don’t expect to think for themselves on any issue. Even if what they thought they were told is obviously stupid, or doesn’t make sense, they won’t think about it or bring up issues. So you just pay for a week to get some obviously wrong and useless check-in.

Communication: One guy I hired from the Ukraine would never communicate. He would take a task, guess at parts he wasn’t sure of, and work for a week only to check in something fundamentally wrong and unusable. Unlike the Indians, he would go against the written specification when it didn’t make sense, but based on usually wrong guesses as to the correct solution. Email checkups were always answered with short sentences. Over a month and a half I tried to get him to communicate – first nicely, then direct orders, finally threats, until I had no choice but to fire him. This wouldn’t have happened in an office setting, since I would have just went by his desk.

Lack-of-care: The database programmer I hired was competent technically but just didn’t give a crap. So he didn’t check for user string to server buffer overruns even when its only one line of code, implemented things the easiest, most inefficient way possible, and ultimately abandoned the work and ran with the money. People act this way for the same reason they may be loud-mouthed jerks on the internet but nice guys in person. There’s no face to face interaction, so you have much less investment.

With one exception, none of the guys I hired were truly bad such that they would fail under any circumstances. It was just that 90% of them were not superstars and were not able to cut it working over the internet.

The net result is I’m actually worse off because of telecommuting, because I got nothing at all from the fired individuals, where in an office setting I would have at least had some code to show for my money. However, my failure wasn’t the principle of telecommuting, but that I didn’t appreciate just how hard it was to find good programmers, or how only superstars would be able to successfully telecommutes. If I could do things over again, I would still telecommute, but I would be much more careful about hiring. Start with small, milestone based contracts, and don’t offer a job until an individual has proven himself.

With only two weeks left of funding on this project it’s now clear that my discovery of and subsequent decision to hire an overseas art-team was a mixed blessing.

On the positive side they are extremely cheap for the size of the team I hired. They try to do a good job. By hiring through a company I don’t have to manage individuals on a day to day basis. On certain tasks, such as drawing textures, or modeling objects when there are no technical requirements things worked well.

On the negative side, all games have technical requirements for art assets, and mine more than most. Trying to communicate and enforce these requirements was a constant major time loss on both sides. About 50% of the total time on art development with this company was either lost completely on unusable assets, or spent redoing work. I don’t think they did things wrong out of a lack of care, or of intentional sloppiness. It’s more like artists are very visual, and it’s unrealistic to expect them to follow written instructions without messing up a few times first.

I had to go in and do probably 1/4 of the total art development myself. For example, I knew it was hopeless to try to get the artists to setup physics when after 6 months they still don’t know how to center models correctly, or to reset transforms.. I probably spent a month in total doing work inside Photoshop or the 3D modeling program. This is a good part of the reason why the programming is late. Things look OK, but not half as good as I originally envisioned.

In the end I think I got what I paid for. 1/4 of the value I would have gotten from artists on-site, but 1/4 of the cost too. If I could do things over again, I would have hired on-site artists, and used off-site artists only for gross work with no technical requirements, or requirements that are already visually represented (such as concepts).

I’m not unhappy about how things turned out, I just thought it could have been better.

I added asteroids to my game levels now.

Asteroids are huge objects, about 3000 meters in diameter, that you can fly into. The fighter itself is 18 meters in diameter. The asteroid is about 40,000 polys.

My first approach was to render the physics geometry in the radar, and the regular mesh on the screen. However, my performance wasn’t very good – 50 FPS or so, about half of what I usually get on complex levels. There are about 500 boxes and 50 physics spheres for the large asteroid. So I stopped rendering the physics objects, figured out how to cut off the top of the asteroid, and render that instead. This brought me up to 60 FPS.

The next thing was that those 550 or so physics objects, even though they never render (visibility set to false), might still be processed through Ogre. While Ogre has many good qualities, efficiency isn’t one of its strongest strengths. So I deleted those objects after processing the physics bounds. This freed up some memory and got me from 60 to 62 FPS.

My next guess was that if any part of the asteroid renders, the whole asteroid renders, when in fact only the part you are looking at needs to be drawn. You only see a diameter of about 500 meters on the screen, about 1/(6^2) of the total asteroid, so there’s a lot of pointless vertex culling.

I decided to break the asteroid up into a grid, each cell of which is about half of what you can see on the screen. This means that at any given time it will render 4 cells. However, I went a little bigger and made each cell 800/2 meters. This is because I am anticipating a lot of players may complain the max camera zoom distance is still too low. If I made it higher later I wouldn’t want to resplit all the objects. At 400 meters per cell, and 3000 meters in diameter, this is slightly less than 8 cells on an edge, or 64 cells total. If I can render only 4 / 64 cells, this is a great improvement.

It took me a while to figure out how to do it in the modeling program. I eventually made an 8×8 grid of helper planes and split along each of these planes. After this, and rerunning the game, I now got 75 FPS. So someone with a machine half as fast as mine should still get at least 30 FPS, which is acceptable. (Actually it will be higher, as I was running the server too).

The problem as usual is this technique is too technically complicated for my artists to do. I’ll tell them to do it, and maybe get lucky, but I’ll give 10:1 odds they will mess it up somehow: the splits won’t be exactly 400 meters, won’t be evenly spaced, won’t be centered at the origin, splits will be missing, and/or it will take days and days. So I’m going to have to do it myself for all the models, just as I spent the last 3 days painfully creating physics geometry for every level and every asteroid.