Author: rakkar

I heard from 3 unrelated sources today that Blizzard is on a mad hiring spree. I’ve never worked there but from what I’ve heard

A. They pay hourly, 9 hours a day max, with 1 hour of that overtime
B. They pay up to 40% of your yearly pay as a bonus (not sure if this is true, or how true it is, or who it is true for)
C. The work is easy (at least on WoW) because you are tweaking a game that’s already done.

They are only a couple of miles from where I live too. The problem is I don’t see anything on their job board I am well qualified for. I have too much experience for their network programmer position. Tools I could do, but I don’t know C# (though I could learn quickly enough). Also, I’m sort of ambivalent about working on tools, since that is most of what I’ve done in my career.

Looking over their listings makes me think of what kind of job I would like to do. In my opinion here’s how things rank:

Most to least enjoyable

Graphics – You get all the attention from the big wigs
API – You get to design systems
AI – See above
Gameplay – Fun to do a good job and make a good game, but can be high pressure
Audio – Easy to do, low pressure
Physics – Moderately hard, low credit and visibility
Networking – Very hard, doing a good job means people don’t complain.
Maintaining systems – Who likes guessing at bugs caused by other people?

Best pay to worst pay:

Graphics – Highly specialized, high demand
Networking – Highly specialized, critical in games that do MP
AI – Everyone thinks they can do it, but if you have experience you can get a higher price
Physics – Many middleware libraries drives down price
Tools – Slightly specialized
Gameplay – Not specialized
Maintaining systems – Grunt work

Difficulty

Networking – Takes less knowledge than graphics, but much more thought and bugs are harder to fix. Everything is asynchronous.
Graphics – Requires lots of up-to-date knowledge, great math
API – Doing a good job is hard, which is why so many do it badly
Maintaining systems – It’s all about experience at reading other’s code, which is hard
Gameplay – Average
AI – It’s not hard, it’s just a matter of designing what you want before you write it
Physics – Lots of APIs out there, just integration
Audio – Lots of APIs out there, just integration

Probably I would like to do graphics, but my skill isn’t great enough to get a job doing that on its own at the pay I usually get. It would be good to do this in a training capacity again. After that gameplay or AI. There’s too little demand for API authoring, and usually lead programmers take that upon themselves. Speaking of which, I’ve gotten 3 interview offers for lead programmer over the last few months. Galactic Melee really helps on this point. Maybe one of these days soon I’ll be moving up on the industry.

I am having a very difficult time developing on the PS3.

If the PS3 is not hooked up from the day before, I spend on average between 15 minutes and 2 hours getting it running again. Today I spent an entire day, and it still does not work correctly. I probably spent 20% of my time dealing with hardware or setup issues preventing me from doing my work.

There’s 3 things I think they need to work on

1. The causes of failures are often seemingly unrelated to the results of failures. Either A has to be done before B, or if you do A, B will not work, and one would not normally suspect that A and B are related. I’ve been thinking a lot about the analogy of how a chain is only as strong as its weakest link, and in this analogy the PS3 has a very long chain, with many potential failure points. It is actually similar to the difference between gaming on consoles and the PC. On the consoles you plug in the game and it just works. On the PC there are many reasons why a game would not work. Your video card is too old. You did not reboot after uploading drivers. You have some particular combination of drivers the game does not like. If any of these things go wrong, usually you just get a crash with some obscure message that does not necessarily indicate why things went wrong. The PS3 is like that, with some factors variable or semi-random.

2. Errors due to your failure to take some precondition often do not show up until much later, and the manner in which they show up often do not reflect the precondition. It’s like bad pointers when running in release mode. Or put more simply, you try to start it, and it says “Fubar because of 0x8327283” and it turns out it was because you didn’t hold down the reset button long enough when you turned off the machine the day before (that’s a made up example, but if it happened I would not be surprised).

3. On the API, I think a lot more attention should have been given to documentation and generally making things clear. Especially given the name and resources behind it. Or maybe I’m spoiled given the relative high quality of the documentation for the Windows API and DirectX.

I’ve never heard of such a job, but someone ought to make up the job “API Usability Engineer.” If they did I would apply. I would even help Sony if they asked me. I think I have relatively high standards of documentation and what construes acceptable quality for public APIs. A few things come to mind immediately:

* If B relies on A, and there is no reason not to do A, then B should do A implicitly. I can think of some cases immediately where this ought to be but is not true on the PS3
* If 95% of the users would do A the same way, and if that assumption is made B can be written much more simply, provide both A and B.
* If B will fail because A is not done, B should indicate this immediately, and not crash or cause problems later in apparently unrelated code.
* Documentation should cross reference itself, so that if A is related to B…n then B…n is linked to on the same page.
* Failures should not be silent
* All non-trivial functionality should be documented.
* Documentation should not just repeat the name of a function, except in the most trivial cases.
* Samples should be as simple and modular as possible, with an eye towards clarity over repetition or cleverness. By modular I don’t mean just focusing on that one feature, but writing the feature in such a way that the user could copy/paste it and use it with minimal problems
* Error messages should accurately reflect why something went wrong at the API level, not why something went wrong because of some internal reason. Error messages should always indicate how to fix the problem, not just state what the problem is.
* It’s worth spending a day of your time to safe a few minutes of an end-user’s time, because you only have to spend that day once but each end-user will have to spend that time over and over again.

I could think of more but I’m tired and frustrated right now. RakNet is not perfect, so if one wanted to bash me back they could. But I think it reflects these standards more than your typical library and my users sometimes write to tell me how much they appreciate this (And I appreciate the compliments!)

Man what a crunch.

Here’s the capabilities of what it can do on the PC:

Commands:
(~) Quit
(A) Login
(B) Set TitleId
(C) Logoff
(D) Register an account
(E) Retrieve password recovery question from handle
(F) Retrieve password from password recovery question
(G) Update account
(H) Change handle
(I) Download friends
(J) Send add friend invite
(K) Accept add friend invite
(L) Decline add friend invite
(M) Get friends list (print them)
(O) Download ignore list
(P) Add to or update ignore list
(Q) Remove from ignore list
(R) Get ignore list (print it)
(S) Download emails
(T) Send email
(U) Delete email
(V) Update email status
(W) Read emails (print it)
(X) Send IM
(Y) Create room
(Z) Download rooms
(0) Get All rooms (print them)
(1) Leave room
(2) Join room
(3) Room chat
(4) Room invite
(5) Set ready to play status
(6) Kick room member
(7) Set room invite only
(8) Set room allow spectators
(9) Change room slots
(!) Grant moderator
(@) Start game
(#) Quick match
($) Cancel Quick match

It’s all about 15,000 lines of code (probably a quarter of that copy/paste hehe) written mostly over 6 or 7 days. It’s late so I’m making a lot of mistakes.

Tomorrow I will document the headers, test it, and move some files around. Then I can release it as the first beta.

This lobby server is turning out to be much more involved than I originally thought. I have to write every query between 6 to 11 times.

For example, to add a friend to the friends list for a user I have to:

1. Query the user for the friend
2. Validate the inputs on the client (we are connected, etc)
3. Serialize the inputs and send them to the server
4. Deserialize the inputs on the server
5. Check the inputs for sanity (don’t add yourself as a friend, you are connected, …). If bad, reply back.
6. Serialize the inputs again in database format, and send to the DB thread.
7. Deserialize the output from the db thread
8. Parse the output from the db thread (doing the actual work of maintaining the internal friends list).
9. Serialize the reply back to the client.
10. Deserialize the reply from the server on the client.
11. Parse the reply (modifying our friends list), and call the output callback for the user.

That’s not even counting that this relies on another previous query to ask/authorize a friend invitation.

This would be even harder and more involved if I had not yet already wrote and tested the DB queries in 3 other projects!

So I did 9 client commands in 2 days for about 3000 lines of code. I have like 50-60 more commands 🙁

The good part is the only painful commands left are the ignore list (which is easier than friends) and sending emails. Friends was probably the hardest to do.

I’ve sort of skipped per-user permissions for now, and moderator commands. Those will also be very involved and painful.

I feel pretty bad. For two weeks I have been trying to get reflective floors working in Gamebryo. At first I tried just rendering from a different camera angle. It never really worked. So I next looked into portals. But after a couple of days where I actually figured them out, I emailed support to clarify some questions and they said that portals were just for culling, and were not true portals. So then I found a sample where they have reflection. But the code was split among 3 files. So I extracted the part I wanted out but it didn’t work. It took me a couple of days to get a pink texture floor. Then a week to get a texture that moved with the camera, but was distorted and totally wrong. Then I updated the code after not looking at reflection for a couple of days and I couldn’t even get the texture to update anymore.

So I threw in the towel and told the boss to give it to someone else with more graphics experience.

* EDIT * Gamebryo support was helpful through this process. This shouldn’t be taken as a knock against Gamebryo, just my own lack of graphics knowledge. * END EDIT *

While the world doesn’t need another graphics engine, these experiences motivate me to continue work on Rak3D. There’s two features I want to have that I haven’t seen before:

1. Rather than letting users program most features themselves via low-level constructs, Rak3D would include every feature I can imagine, in modular classes. So reflection would be one line of code, the shader would be a multi-part super-shader that does normal, parallax, light mapping, etc. This would be organized into a two-level system, low and high. The low level system is typical of what you see in 3D engines right now. The high level system is like RakNet’s plugins, where they expose direct features. What I like about this is stuff just works, there is no debugging needed.

2. A big problem I have is stuff just doesn’t show up and I have absolutely no idea why (as above). I want to include an interactive on-screen debugger where I can look at my scene heirachy as a tree, which will indicate with colors what is visible. And I can select any object and get full details on it, esp. why it is not visible on screen, or why it is using a default or missing texture.

I think the RakNet website is too programmer-centric. If you look at other sites like FMOD they don’t have a single word about their technical superiority, but instead list their features. To remedy this I’ve made a RakNet Major Features Video. It took me about 4 hours to record, since I restarted many times. The first successful recording was 40 minutes long, I cut it by half twice. The current video blows through the features very fast, but I think is more exciting for it.

I’ve finished writing a ranking server for RakNet. It is a C++ interface to a PostgreSQL database that stores a table that contains a list of matches between two participants. It’s very general – participants, games, scores, ratings, are just numbers. You can store a text string and arbitrary binary data for each match. It uses my functor system so calls are asynchronous.

There are currently 4 functors:
SubmitMatch – Submits the match results between a pair of participants.
GetRatingForParticipant – Get the most recent rating for a single participants.
GetRatingForParticipants – Get the most recent rating for a list of participants.
GetHistoryForParticipant – Get all the matches and the data therein for a particular participant.

I store IDs as a pair of unsigned integers. The intent is that the primary integer can represent a foreign key. The secondary integer can represent a sub-type. In Galactic Melee for example I store rankings for both players and squads, so the secondary integer would differentiate between these.

The class is designed in such a way as to be easily extensible while still easy to use. Generally speaking the more extensible something is, the harder it is to use because there are more pieces involved, so you have a lot more to learn. This is true of Gamebryo for example, which has good division of code but then requires a lot of steps to do anything. I wanted to avoid this, so went to a lot of extra work to make as much automatic as possible.

One nice thing is you can add more queries without changing the original source code. This is because each query is just a derived class.

The current table format is:

CREATE TABLE rankingServer (
rowId serial UNIQUE,
participantADbId_primaryKey integer NOT NULL,
participantADbId_secondaryKey integer NOT NULL,
participantBDbId_primaryKey integer NOT NULL,
participantBDbId_secondaryKey integer NOT NULL,
participantAScore real,
participantBScore integer,
participantAOldRating real,
participantANewRating real NOT NULL,
participantBOldRating real,
participantBNewRating real NOT NULL,
gameDbId_primaryKey integer NOT NULL,
gameDbId_secondaryKey integer NOT NULL,
matchTime bigint NOT NULL DEFAULT EXTRACT(EPOCH FROM current_timestamp),
matchNotes text,
matchBinaryData bytea);

Here’s an example of calling a Functor


GetRatingForParticipants_PostgreSQLImpl *functor = GetRatingForParticipants_PostgreSQLImpl::Alloc();

/// Puts this functor on the processing queue. It will process sometime later in a thread.
rankingServer.PushFunctor(functor);


This is all part of my plan to write a complete lobby server, of which ranking and match tracking is one component.

I’m working on adding a ranking server and a lobby server to RakNet. One of the problems this entails is that database calls are blocking. It’s possible that the ranking server or lobby server will be doing other things at the same time (such as running the game), especially for smaller games. This means that database calls have to be run in a thread.

I spent a few hours setting up the ranking server calls to execute in a thread by copying the input parameters, doing the function, copying the output, and calling the result back in the main thread. But this was really a pain in the butt. It’s not the first time I’ve had to have done it. And I’ll have to do it again, but much more so, for the lobby server.

This morning when I woke up I thought of a solution that so far is turning out well. Create one thread that takes a stream of functors, does processing on those functors, and when done calls a function in the main thread via a callback. The functor itself will have as member variables what was formerly the input and output parameter lists. This avoids the necessity of copying parameters. And since it is up to the user to derive the functor, they can add contextual user data or extend it as they wish. (As opposed to passing a void* and worrying about allocation).

Here it is, leaving out the implementation


/// FunctionThread takes a stream of classes that implement a processing function,
/// processes them in a thread, and calls a callback with the result.
/// It's a useful way to call blocking functions that you do not want to block,
/// such as file writes and database operations.
class FunctionThread
{
public:
FunctionThread();
~FunctionThread();

/// A functor is a single unit of processing to send to the Function thread.
/// Derive from it, add your data, and implement the processing function.
class Functor
{
public:
virtual void Process(void)=0;
/// Called from FunctionThread::CallResultHandlers with wasCancelled
/// false OR Called from FunctionThread::StopThread or
/// FunctionThread::~FunctionThread with wasCancelled true
virtual void HandleResult(bool wasCancelled, void *context) {};
};


Here’s one of the smaller functors. It gets a list of players given a game database id.


/// Given a particular game, get ratings for all players
class GetRatingForPlayers_Functor : public Functor
{
public:
/// Identifies in the game in the database with a primary
/// and secondary integer key. User defined.
RankingServerDBInterface::EntityDBID gameDbId;

// OUTPUT
/// List of players and rankings for a given gameId
DataStructures::List < rankingserverdbinterface::RatedEntity > outputList;
};


Before I had to

1. Add member functions for every operation
2. Add a callback for every member function
3. Copy the input parameters for every operation and store them in an input list
4. Copy the result of the operation and store them in an output list.
5. Create a thread to do the functionality
6. Create interfaces for every operation, that actually does the work in the database

Now I can
1. Create a functor for every operation
2. Implement the functor
3. Pass the functor to the FunctionThread instance

Another nice thing about this is I can reuse the same FunctionThread for both the ranking server and the lobby server. The old way would have created a thread for each of those classes.