Categories
Game Development

Replica Manager 2

I’m working on the second iteration of the Replica Manager class. For those that don’t know it, this class is supposed to handle the details of sending and synchronizing objects across the network as new systems connect.

My first approach was data based. All the code was in the Replica Manager itself, and it would query an interface to get the data it needed to perform operations, such as serializing objects. In most cases, this worked OK. However, the problem with games is there is always complexity to work around, especially with existing systems, and if cases came up that I didn’t anticipate it was hard to go around the system. One example in particular was the initial object download. The first system would gather up all objects, serialize, and send them to you. However, this didn’t work if objects already existed globally (such as created when the map loaded). It also didn’t work if you cared about what order objects were downloaded in, which you would if they had dependencies on each other (such as cross referencing pointers).

My new approach is more code based, with data queries exposed only for very common or unlikely to change operations. So now the Replica Manager class itself mostly just encodes and decodes queries, and holds the data structures, with the actual functionality in the Replica class, the class you derive your own game objects from. This gave me a degree of flexibility I didn’t have before, because now I can implement complex operations since I know the user can override the code if necessary. One example is having a client locally create an object which is verified by the server, and deleted or synchronized on the client accordingly. Another example is automatically scoping objects with a simple callback, where before you had to do this by writing your own system.

The test case I’m solving is:


// Preexists on all systems, has objectID, still want to register it so I can call serialize automatically
class World {
int totalKills;
}

class Player : public Participant {
Soldier *soldier;
char name[256];
int teamNumber;
char killMessage[256];
int gameInstanceId; // Players may be spread out between multiple game instances
};

class Soldier {
Player *owningPlayer, *lastDamagingPlayer;
Gun gun;
float health; // Sent reliable ordered
float positionX, positionY; // Sent unreliable sequenced
bool isCloaked; // If cloaked, do not send position to non-teammates
};

class Gun {
Soldier *owningSoldier;
Player *owningPlayer;
int ammo;
};

class Bullet {
Gun *firingGun;
Player *owningPlayer;
RakNetTime timeFired;
};

So you have:

1. Classes that have pointers to classes that have pointers to themselves in return
2. The world object is assumed to preexist, so should not be dynamically downloaded, yet still needs to be updated
3. Cloaking, where an object either won’t be serialized at all, or only partly serialized (if a teammate)
4. Pointers which may be NULL
5. Object composition with serialized objects (so their parents should be dynamically downloaded, but not the objects themselves).
6. Serializing the object using a different send type, depending on what part is serialized.

These could all be solved with the old system too, but it was much harder and you had to really understand the code to know what to disable. If I do my job this test case should be solvable trivially.

Categories
Game Development

Sourceforge is a great service

Quick note: I’m really glad a service like Sourceforge exists. After all the trouble recently with source control providers it is really awesome that I can update RakNet without having to think about if the server is up or down, or security for my code, or paying for it. Plus you get a mailing list. It’s hard to set up but I’m glad I did.

Categories
Game Development

Blizzard hiring

I heard from 3 unrelated sources today that Blizzard is on a mad hiring spree. I’ve never worked there but from what I’ve heard

A. They pay hourly, 9 hours a day max, with 1 hour of that overtime
B. They pay up to 40% of your yearly pay as a bonus (not sure if this is true, or how true it is, or who it is true for)
C. The work is easy (at least on WoW) because you are tweaking a game that’s already done.

They are only a couple of miles from where I live too. The problem is I don’t see anything on their job board I am well qualified for. I have too much experience for their network programmer position. Tools I could do, but I don’t know C# (though I could learn quickly enough). Also, I’m sort of ambivalent about working on tools, since that is most of what I’ve done in my career.

Looking over their listings makes me think of what kind of job I would like to do. In my opinion here’s how things rank:

Most to least enjoyable

Graphics – You get all the attention from the big wigs
API – You get to design systems
AI – See above
Gameplay – Fun to do a good job and make a good game, but can be high pressure
Audio – Easy to do, low pressure
Physics – Moderately hard, low credit and visibility
Networking – Very hard, doing a good job means people don’t complain.
Maintaining systems – Who likes guessing at bugs caused by other people?

Best pay to worst pay:

Graphics – Highly specialized, high demand
Networking – Highly specialized, critical in games that do MP
AI – Everyone thinks they can do it, but if you have experience you can get a higher price
Physics – Many middleware libraries drives down price
Tools – Slightly specialized
Gameplay – Not specialized
Maintaining systems – Grunt work

Difficulty

Networking – Takes less knowledge than graphics, but much more thought and bugs are harder to fix. Everything is asynchronous.
Graphics – Requires lots of up-to-date knowledge, great math
API – Doing a good job is hard, which is why so many do it badly
Maintaining systems – It’s all about experience at reading other’s code, which is hard
Gameplay – Average
AI – It’s not hard, it’s just a matter of designing what you want before you write it
Physics – Lots of APIs out there, just integration
Audio – Lots of APIs out there, just integration

Probably I would like to do graphics, but my skill isn’t great enough to get a job doing that on its own at the pay I usually get. It would be good to do this in a training capacity again. After that gameplay or AI. There’s too little demand for API authoring, and usually lead programmers take that upon themselves. Speaking of which, I’ve gotten 3 interview offers for lead programmer over the last few months. Galactic Melee really helps on this point. Maybe one of these days soon I’ll be moving up on the industry.

Categories
Game Development

PS3 giving me a hard time / API usability engineer?

I am having a very difficult time developing on the PS3.

If the PS3 is not hooked up from the day before, I spend on average between 15 minutes and 2 hours getting it running again. Today I spent an entire day, and it still does not work correctly. I probably spent 20% of my time dealing with hardware or setup issues preventing me from doing my work.

There’s 3 things I think they need to work on

1. The causes of failures are often seemingly unrelated to the results of failures. Either A has to be done before B, or if you do A, B will not work, and one would not normally suspect that A and B are related. I’ve been thinking a lot about the analogy of how a chain is only as strong as its weakest link, and in this analogy the PS3 has a very long chain, with many potential failure points. It is actually similar to the difference between gaming on consoles and the PC. On the consoles you plug in the game and it just works. On the PC there are many reasons why a game would not work. Your video card is too old. You did not reboot after uploading drivers. You have some particular combination of drivers the game does not like. If any of these things go wrong, usually you just get a crash with some obscure message that does not necessarily indicate why things went wrong. The PS3 is like that, with some factors variable or semi-random.

2. Errors due to your failure to take some precondition often do not show up until much later, and the manner in which they show up often do not reflect the precondition. It’s like bad pointers when running in release mode. Or put more simply, you try to start it, and it says “Fubar because of 0x8327283” and it turns out it was because you didn’t hold down the reset button long enough when you turned off the machine the day before (that’s a made up example, but if it happened I would not be surprised).

3. On the API, I think a lot more attention should have been given to documentation and generally making things clear. Especially given the name and resources behind it. Or maybe I’m spoiled given the relative high quality of the documentation for the Windows API and DirectX.

I’ve never heard of such a job, but someone ought to make up the job “API Usability Engineer.” If they did I would apply. I would even help Sony if they asked me. I think I have relatively high standards of documentation and what construes acceptable quality for public APIs. A few things come to mind immediately:

* If B relies on A, and there is no reason not to do A, then B should do A implicitly. I can think of some cases immediately where this ought to be but is not true on the PS3
* If 95% of the users would do A the same way, and if that assumption is made B can be written much more simply, provide both A and B.
* If B will fail because A is not done, B should indicate this immediately, and not crash or cause problems later in apparently unrelated code.
* Documentation should cross reference itself, so that if A is related to B…n then B…n is linked to on the same page.
* Failures should not be silent
* All non-trivial functionality should be documented.
* Documentation should not just repeat the name of a function, except in the most trivial cases.
* Samples should be as simple and modular as possible, with an eye towards clarity over repetition or cleverness. By modular I don’t mean just focusing on that one feature, but writing the feature in such a way that the user could copy/paste it and use it with minimal problems
* Error messages should accurately reflect why something went wrong at the API level, not why something went wrong because of some internal reason. Error messages should always indicate how to fix the problem, not just state what the problem is.
* It’s worth spending a day of your time to safe a few minutes of an end-user’s time, because you only have to spend that day once but each end-user will have to spend that time over and over again.

I could think of more but I’m tired and frustrated right now. RakNet is not perfect, so if one wanted to bash me back they could. But I think it reflects these standards more than your typical library and my users sometimes write to tell me how much they appreciate this (And I appreciate the compliments!)

Categories
Game Development

Lobby server nearly done

Man what a crunch.

Here’s the capabilities of what it can do on the PC:

Commands:
(~) Quit
(A) Login
(B) Set TitleId
(C) Logoff
(D) Register an account
(E) Retrieve password recovery question from handle
(F) Retrieve password from password recovery question
(G) Update account
(H) Change handle
(I) Download friends
(J) Send add friend invite
(K) Accept add friend invite
(L) Decline add friend invite
(M) Get friends list (print them)
(O) Download ignore list
(P) Add to or update ignore list
(Q) Remove from ignore list
(R) Get ignore list (print it)
(S) Download emails
(T) Send email
(U) Delete email
(V) Update email status
(W) Read emails (print it)
(X) Send IM
(Y) Create room
(Z) Download rooms
(0) Get All rooms (print them)
(1) Leave room
(2) Join room
(3) Room chat
(4) Room invite
(5) Set ready to play status
(6) Kick room member
(7) Set room invite only
(8) Set room allow spectators
(9) Change room slots
(!) Grant moderator
(@) Start game
(#) Quick match
($) Cancel Quick match

It’s all about 15,000 lines of code (probably a quarter of that copy/paste hehe) written mostly over 6 or 7 days. It’s late so I’m making a lot of mistakes.

Tomorrow I will document the headers, test it, and move some files around. Then I can release it as the first beta.

Categories
Game Development

3000 lines of code in 2 days

This lobby server is turning out to be much more involved than I originally thought. I have to write every query between 6 to 11 times.

For example, to add a friend to the friends list for a user I have to:

1. Query the user for the friend
2. Validate the inputs on the client (we are connected, etc)
3. Serialize the inputs and send them to the server
4. Deserialize the inputs on the server
5. Check the inputs for sanity (don’t add yourself as a friend, you are connected, …). If bad, reply back.
6. Serialize the inputs again in database format, and send to the DB thread.
7. Deserialize the output from the db thread
8. Parse the output from the db thread (doing the actual work of maintaining the internal friends list).
9. Serialize the reply back to the client.
10. Deserialize the reply from the server on the client.
11. Parse the reply (modifying our friends list), and call the output callback for the user.

That’s not even counting that this relies on another previous query to ask/authorize a friend invitation.

This would be even harder and more involved if I had not yet already wrote and tested the DB queries in 3 other projects!

So I did 9 client commands in 2 days for about 3000 lines of code. I have like 50-60 more commands 🙁

The good part is the only painful commands left are the ignore list (which is easier than friends) and sending emails. Friends was probably the hardest to do.

I’ve sort of skipped per-user permissions for now, and moderator commands. Those will also be very involved and painful.

Categories
Game Development

Reflection failure :(

I feel pretty bad. For two weeks I have been trying to get reflective floors working in Gamebryo. At first I tried just rendering from a different camera angle. It never really worked. So I next looked into portals. But after a couple of days where I actually figured them out, I emailed support to clarify some questions and they said that portals were just for culling, and were not true portals. So then I found a sample where they have reflection. But the code was split among 3 files. So I extracted the part I wanted out but it didn’t work. It took me a couple of days to get a pink texture floor. Then a week to get a texture that moved with the camera, but was distorted and totally wrong. Then I updated the code after not looking at reflection for a couple of days and I couldn’t even get the texture to update anymore.

So I threw in the towel and told the boss to give it to someone else with more graphics experience.

* EDIT * Gamebryo support was helpful through this process. This shouldn’t be taken as a knock against Gamebryo, just my own lack of graphics knowledge. * END EDIT *

While the world doesn’t need another graphics engine, these experiences motivate me to continue work on Rak3D. There’s two features I want to have that I haven’t seen before:

1. Rather than letting users program most features themselves via low-level constructs, Rak3D would include every feature I can imagine, in modular classes. So reflection would be one line of code, the shader would be a multi-part super-shader that does normal, parallax, light mapping, etc. This would be organized into a two-level system, low and high. The low level system is typical of what you see in 3D engines right now. The high level system is like RakNet’s plugins, where they expose direct features. What I like about this is stuff just works, there is no debugging needed.

2. A big problem I have is stuff just doesn’t show up and I have absolutely no idea why (as above). I want to include an interactive on-screen debugger where I can look at my scene heirachy as a tree, which will indicate with colors what is visible. And I can select any object and get full details on it, esp. why it is not visible on screen, or why it is using a default or missing texture.

Categories
Game Development

RakNet features video

I think the RakNet website is too programmer-centric. If you look at other sites like FMOD they don’t have a single word about their technical superiority, but instead list their features. To remedy this I’ve made a RakNet Major Features Video. It took me about 4 hours to record, since I restarted many times. The first successful recording was 40 minutes long, I cut it by half twice. The current video blows through the features very fast, but I think is more exciting for it.

Categories
Game Development

Ranking server done

I’ve finished writing a ranking server for RakNet. It is a C++ interface to a PostgreSQL database that stores a table that contains a list of matches between two participants. It’s very general – participants, games, scores, ratings, are just numbers. You can store a text string and arbitrary binary data for each match. It uses my functor system so calls are asynchronous.

There are currently 4 functors:
SubmitMatch – Submits the match results between a pair of participants.
GetRatingForParticipant – Get the most recent rating for a single participants.
GetRatingForParticipants – Get the most recent rating for a list of participants.
GetHistoryForParticipant – Get all the matches and the data therein for a particular participant.

I store IDs as a pair of unsigned integers. The intent is that the primary integer can represent a foreign key. The secondary integer can represent a sub-type. In Galactic Melee for example I store rankings for both players and squads, so the secondary integer would differentiate between these.

The class is designed in such a way as to be easily extensible while still easy to use. Generally speaking the more extensible something is, the harder it is to use because there are more pieces involved, so you have a lot more to learn. This is true of Gamebryo for example, which has good division of code but then requires a lot of steps to do anything. I wanted to avoid this, so went to a lot of extra work to make as much automatic as possible.

One nice thing is you can add more queries without changing the original source code. This is because each query is just a derived class.

The current table format is:

CREATE TABLE rankingServer (
rowId serial UNIQUE,
participantADbId_primaryKey integer NOT NULL,
participantADbId_secondaryKey integer NOT NULL,
participantBDbId_primaryKey integer NOT NULL,
participantBDbId_secondaryKey integer NOT NULL,
participantAScore real,
participantBScore integer,
participantAOldRating real,
participantANewRating real NOT NULL,
participantBOldRating real,
participantBNewRating real NOT NULL,
gameDbId_primaryKey integer NOT NULL,
gameDbId_secondaryKey integer NOT NULL,
matchTime bigint NOT NULL DEFAULT EXTRACT(EPOCH FROM current_timestamp),
matchNotes text,
matchBinaryData bytea);

Here’s an example of calling a Functor


GetRatingForParticipants_PostgreSQLImpl *functor = GetRatingForParticipants_PostgreSQLImpl::Alloc();

printf("Get the most recent rating for all participants\n");

printf("Enter the primary ID of the game (int): ");
gets(inputStr);
functor->gameDbId.primaryKey = atoi(inputStr);

printf("Enter the secondary ID of the game (int): ");
gets(inputStr);
functor->gameDbId.secondaryKey = atoi(inputStr);

/// Puts this functor on the processing queue. It will process sometime later in a thread.
rankingServer.PushFunctor(functor);

This is all part of my plan to write a complete lobby server, of which ranking and match tracking is one component.

Categories
Game Development

Function threading for database calls

I’m working on adding a ranking server and a lobby server to RakNet. One of the problems this entails is that database calls are blocking. It’s possible that the ranking server or lobby server will be doing other things at the same time (such as running the game), especially for smaller games. This means that database calls have to be run in a thread.

I spent a few hours setting up the ranking server calls to execute in a thread by copying the input parameters, doing the function, copying the output, and calling the result back in the main thread. But this was really a pain in the butt. It’s not the first time I’ve had to have done it. And I’ll have to do it again, but much more so, for the lobby server.

This morning when I woke up I thought of a solution that so far is turning out well. Create one thread that takes a stream of functors, does processing on those functors, and when done calls a function in the main thread via a callback. The functor itself will have as member variables what was formerly the input and output parameter lists. This avoids the necessity of copying parameters. And since it is up to the user to derive the functor, they can add contextual user data or extend it as they wish. (As opposed to passing a void* and worrying about allocation).

Here it is, leaving out the implementation


/// FunctionThread takes a stream of classes that implement a processing function,
/// processes them in a thread, and calls a callback with the result.
/// It's a useful way to call blocking functions that you do not want to block,
/// such as file writes and database operations.
class FunctionThread
{
public:
FunctionThread();
~FunctionThread();

/// Starts the thread up.
void StartThread(void);

/// Stop processing. Will also call FunctorResultHandler callbacks
/// with /a wasCancelled set to true.
/// \param[in] blockOnCurrentProcessing Wait for the current
/// processing to finish?
void StopThread(bool blockOnCurrentProcessing);

/// Add a functor to the incoming stream of functors
void Push(Functor *functor, void *context);

/// Call FunctorResultHandler callbacks
/// Normally you would call this once per update cycle, although you
/// do not have to.
void CallResultHandlers(void);
}

/// A functor is a single unit of processing to send to the Function thread.
/// Derive from it, add your data, and implement the processing function.
class Functor
{
public:
virtual void Process(void)=0;
/// Called from FunctionThread::CallResultHandlers with wasCancelled
/// false OR Called from FunctionThread::StopThread or
/// FunctionThread::~FunctionThread with wasCancelled true
virtual void HandleResult(bool wasCancelled, void *context) {};
};

Here’s one of the smaller functors. It gets a list of players given a game database id.


/// Given a particular game, get ratings for all players
class GetRatingForPlayers_Functor : public Functor
{
public:
/// Identifies in the game in the database with a primary
/// and secondary integer key. User defined.
RankingServerDBInterface::EntityDBID gameDbId;

// OUTPUT
/// List of players and rankings for a given gameId
DataStructures::List < rankingserverdbinterface::RatedEntity > outputList;
};

Before I had to

1. Add member functions for every operation
2. Add a callback for every member function
3. Copy the input parameters for every operation and store them in an input list
4. Copy the result of the operation and store them in an output list.
5. Create a thread to do the functionality
6. Create interfaces for every operation, that actually does the work in the database

Now I can
1. Create a functor for every operation
2. Implement the functor
3. Pass the functor to the FunctionThread instance

Another nice thing about this is I can reuse the same FunctionThread for both the ranking server and the lobby server. The old way would have created a thread for each of those classes.