Month: February 2008

I spent an entire day on Sunday thinking about the architecture on how to write a generic system to create, destroy, serialize, and control scope over networked objects in C++. It’s a very hard problem to solve because you know nothing about the game objects themselves, or what complexities the end-users might introduce with their specific architecture.

In the end I decided on 3 classes. One class, ReplicaManager2, is the base plugin for the system. All it does is essentially serialization, deserialization, transmission, and bookkeeping. Stuff you wouldn’t normally care about. I have the user pass a class factory that generates connection instances, one per connection. This controls high-level and non-object specific functionality such as how to create object instances. Lastly, there is the base class Replica2, which game objects should derive from.

The main reason I did it this way, rather than putting everything in the ReplicaManager2 plugin (as I did in its predecessor) was so that the functionality can be easily overridden. This should solve a problem I had with the old system, where I ended up passing a ton of data to the callbacks in order to force it to get it to do what I wanted. By assuming users will override functions, this also gives me the ability to make assumptions and thus much more elaborate functions.

Improvements it has over the old system are:

1. Relay across systems actually works correctly. So you could have a server cluster, with n clients on each server, and everything would just get sent out properly. It automatically deals with cyclical connection graphs up to a depth of 1 away from the source node.

2. Defaults to polling for changes. Each tick, if an object is now constructable, or in scope, and it was not in the last tick (or vice-versa) the object is created or destroyed appropriately. The object is also serialized every n milliseconds, and if it serializes differently under the same context, the newly serialized value is transmitted. This is easier to use than event-based updates (which are also supported) so the user has to write less code.

3. Downloads construct all objects first, then deserializes them, and in the order they were originally created. This is likely what a user would do were they to hand-write the download system, and maximizes the chance that pointers and other dependencies will be valid on deserialization. This also allows you to have objects that cross-reference each other when deserializing, each both have already been created anyway.

4. It’s easier to modify events and more information is given about what triggered those events. For example, it’s now trivial to send using different reliability types based on the context of what is being sent, something that was hard to do before.

5. Changes are easier to initiate on the client, especially construction, which before would have taken custom game code.

Here’s a video that goes over the system. I start out too fast because I was tired of starting over when I made mistakes, but later on it gets better.
http://www.rakkarsoft.com/raknet/manual/ReplicaManager2.html

I’m working on the second iteration of the Replica Manager class. For those that don’t know it, this class is supposed to handle the details of sending and synchronizing objects across the network as new systems connect.

My first approach was data based. All the code was in the Replica Manager itself, and it would query an interface to get the data it needed to perform operations, such as serializing objects. In most cases, this worked OK. However, the problem with games is there is always complexity to work around, especially with existing systems, and if cases came up that I didn’t anticipate it was hard to go around the system. One example in particular was the initial object download. The first system would gather up all objects, serialize, and send them to you. However, this didn’t work if objects already existed globally (such as created when the map loaded). It also didn’t work if you cared about what order objects were downloaded in, which you would if they had dependencies on each other (such as cross referencing pointers).

My new approach is more code based, with data queries exposed only for very common or unlikely to change operations. So now the Replica Manager class itself mostly just encodes and decodes queries, and holds the data structures, with the actual functionality in the Replica class, the class you derive your own game objects from. This gave me a degree of flexibility I didn’t have before, because now I can implement complex operations since I know the user can override the code if necessary. One example is having a client locally create an object which is verified by the server, and deleted or synchronized on the client accordingly. Another example is automatically scoping objects with a simple callback, where before you had to do this by writing your own system.

The test case I’m solving is:

// Preexists on all systems, has objectID, still want to register it so I can call serialize automatically
class World {
int totalKills;
}

class Player : public Participant {
Soldier *soldier;
char name[256];
int teamNumber;
char killMessage[256];
int gameInstanceId; // Players may be spread out between multiple game instances
};

class Soldier {
Player *owningPlayer, *lastDamagingPlayer;
Gun gun;
float health; // Sent reliable ordered
float positionX, positionY; // Sent unreliable sequenced
bool isCloaked; // If cloaked, do not send position to non-teammates
};

class Gun {
Soldier *owningSoldier;
Player *owningPlayer;
int ammo;
};

class Bullet {
Gun *firingGun;
Player *owningPlayer;
RakNetTime timeFired;
};

So you have:

1. Classes that have pointers to classes that have pointers to themselves in return
2. The world object is assumed to preexist, so should not be dynamically downloaded, yet still needs to be updated
3. Cloaking, where an object either won’t be serialized at all, or only partly serialized (if a teammate)
4. Pointers which may be NULL
5. Object composition with serialized objects (so their parents should be dynamically downloaded, but not the objects themselves).
6. Serializing the object using a different send type, depending on what part is serialized.

These could all be solved with the old system too, but it was much harder and you had to really understand the code to know what to disable. If I do my job this test case should be solvable trivially.

Quick note: I’m really glad a service like Sourceforge exists. After all the trouble recently with source control providers it is really awesome that I can update RakNet without having to think about if the server is up or down, or security for my code, or paying for it. Plus you get a mailing list. It’s hard to set up but I’m glad I did.

I heard from 3 unrelated sources today that Blizzard is on a mad hiring spree. I’ve never worked there but from what I’ve heard

A. They pay hourly, 9 hours a day max, with 1 hour of that overtime
B. They pay up to 40% of your yearly pay as a bonus (not sure if this is true, or how true it is, or who it is true for)
C. The work is easy (at least on WoW) because you are tweaking a game that’s already done.

They are only a couple of miles from where I live too. The problem is I don’t see anything on their job board I am well qualified for. I have too much experience for their network programmer position. Tools I could do, but I don’t know C# (though I could learn quickly enough). Also, I’m sort of ambivalent about working on tools, since that is most of what I’ve done in my career.

Looking over their listings makes me think of what kind of job I would like to do. In my opinion here’s how things rank:

Most to least enjoyable

Graphics – You get all the attention from the big wigs
API – You get to design systems
AI – See above
Gameplay – Fun to do a good job and make a good game, but can be high pressure
Audio – Easy to do, low pressure
Physics – Moderately hard, low credit and visibility
Networking – Very hard, doing a good job means people don’t complain.
Maintaining systems – Who likes guessing at bugs caused by other people?

Best pay to worst pay:

Graphics – Highly specialized, high demand
Networking – Highly specialized, critical in games that do MP
AI – Everyone thinks they can do it, but if you have experience you can get a higher price
Physics – Many middleware libraries drives down price
Tools – Slightly specialized
Gameplay – Not specialized
Maintaining systems – Grunt work

Difficulty

Networking – Takes less knowledge than graphics, but much more thought and bugs are harder to fix. Everything is asynchronous.
Graphics – Requires lots of up-to-date knowledge, great math
API – Doing a good job is hard, which is why so many do it badly
Maintaining systems – It’s all about experience at reading other’s code, which is hard
Gameplay – Average
AI – It’s not hard, it’s just a matter of designing what you want before you write it
Physics – Lots of APIs out there, just integration
Audio – Lots of APIs out there, just integration

Probably I would like to do graphics, but my skill isn’t great enough to get a job doing that on its own at the pay I usually get. It would be good to do this in a training capacity again. After that gameplay or AI. There’s too little demand for API authoring, and usually lead programmers take that upon themselves. Speaking of which, I’ve gotten 3 interview offers for lead programmer over the last few months. Galactic Melee really helps on this point. Maybe one of these days soon I’ll be moving up on the industry.

I am having a very difficult time developing on the PS3.

If the PS3 is not hooked up from the day before, I spend on average between 15 minutes and 2 hours getting it running again. Today I spent an entire day, and it still does not work correctly. I probably spent 20% of my time dealing with hardware or setup issues preventing me from doing my work.

There’s 3 things I think they need to work on

1. The causes of failures are often seemingly unrelated to the results of failures. Either A has to be done before B, or if you do A, B will not work, and one would not normally suspect that A and B are related. I’ve been thinking a lot about the analogy of how a chain is only as strong as its weakest link, and in this analogy the PS3 has a very long chain, with many potential failure points. It is actually similar to the difference between gaming on consoles and the PC. On the consoles you plug in the game and it just works. On the PC there are many reasons why a game would not work. Your video card is too old. You did not reboot after uploading drivers. You have some particular combination of drivers the game does not like. If any of these things go wrong, usually you just get a crash with some obscure message that does not necessarily indicate why things went wrong. The PS3 is like that, with some factors variable or semi-random.

2. Errors due to your failure to take some precondition often do not show up until much later, and the manner in which they show up often do not reflect the precondition. It’s like bad pointers when running in release mode. Or put more simply, you try to start it, and it says “Fubar because of 0x8327283” and it turns out it was because you didn’t hold down the reset button long enough when you turned off the machine the day before (that’s a made up example, but if it happened I would not be surprised).

3. On the API, I think a lot more attention should have been given to documentation and generally making things clear. Especially given the name and resources behind it. Or maybe I’m spoiled given the relative high quality of the documentation for the Windows API and DirectX.

I’ve never heard of such a job, but someone ought to make up the job “API Usability Engineer.” If they did I would apply. I would even help Sony if they asked me. I think I have relatively high standards of documentation and what construes acceptable quality for public APIs. A few things come to mind immediately:

* If B relies on A, and there is no reason not to do A, then B should do A implicitly. I can think of some cases immediately where this ought to be but is not true on the PS3
* If 95% of the users would do A the same way, and if that assumption is made B can be written much more simply, provide both A and B.
* If B will fail because A is not done, B should indicate this immediately, and not crash or cause problems later in apparently unrelated code.
* Documentation should cross reference itself, so that if A is related to B…n then B…n is linked to on the same page.
* Failures should not be silent
* All non-trivial functionality should be documented.
* Documentation should not just repeat the name of a function, except in the most trivial cases.
* Samples should be as simple and modular as possible, with an eye towards clarity over repetition or cleverness. By modular I don’t mean just focusing on that one feature, but writing the feature in such a way that the user could copy/paste it and use it with minimal problems
* Error messages should accurately reflect why something went wrong at the API level, not why something went wrong because of some internal reason. Error messages should always indicate how to fix the problem, not just state what the problem is.
* It’s worth spending a day of your time to safe a few minutes of an end-user’s time, because you only have to spend that day once but each end-user will have to spend that time over and over again.

I could think of more but I’m tired and frustrated right now. RakNet is not perfect, so if one wanted to bash me back they could. But I think it reflects these standards more than your typical library and my users sometimes write to tell me how much they appreciate this (And I appreciate the compliments!)