Category: Game Development

In the game industry if a game ships badly and doesn’t sell in the first two weeks, it’s all over. The only game that has ever recovered from a bad release that I know of is Anarchy Online and that was only because it came out early enough when massively multiplayer games were still novel. But that is the exception, not the rule. How many games have you played that sucked, and you tried them again a year later just to see if a patch made them better?

Giving a bad impression in the middleware market is not quite such a death-sentence, because people expect middleware to improve and change over time. However, a bugged release is still going to cause you pain for a long time.

Negative posts that are no longer true

When I search for “RakNet vs. X” where X is one of my competitors, I sometimes see posts “RakNet is good, but X does this better.” Three years ago when the post was written, that was true, but people reading up on RakNet wouldn’t know things have changed since then. Even my Gamedev review is based on like 1.0 of RakNet. It’s positive (thank god!) but it’s totally inapplicable to the current release. Imagine if it had been negative, all things currently equal?

Spawn competition

The original reason I wrote RakNet was because I was disenchanted with HawkNL. I respect what the author has done, but I essentially put him out of business if the forum is anything to go by. Usually I won’t link to my competitors so I don’t improve their Google rank, but HawkNL is so long gone I don’t even care. Arguably, I even put Microsoft’s DirectPlay out of business, as they shut down a few years after RakNet was created as well. Mabye not THE reason, but perhaps the straw that broke the camel’s back.

Old releases scattered around

Sometimes people take it upon themselves to write wrappers for RakNet, or mirror it. If the wrapper is just for a release that lacks features, that is OK. But if the release has bugs, then so does the wrapper, and unlike your own releases, the wrapper is out of your control and keeps on giving you a bad name.

Disgruntled users that won’t come back anytime soon

The first graphics library I ever used, before I even graduated college, was Power Render. The documentation, support, and distribution wasn’t very good, and I paid like $250 for it, which was a LOT of money for me at the time. I’m sure it’s much better now after 8 years or so. But even now, if I’m going to look for a graphics library, Power Render is going to be last on the list because I don’t know if I like the other libraries or not, but I know I once had a bad experience with Power Render. If Power Render has just lacked features and otherwise worked, I would have missed those features, but wouldn’t have been mad or felt like I wasted money.

Key point

My biggest mistake ever regarding bad releases was when adding flow control. Flow control is tremendously hard to get working right, unless you’ve done it before it’s hard to understand just how difficult and error prone it is. Bad flow control causes lag and spikes, and because this issue was so hard to get working right under every internet condition, every type of game, every operating system for about 2 years I got occasional complaints about these kinds of issues. In my opinion this is the reason I have any low-cost competitors at all. Disgruntled users that had bad flow control under some case that went to my 3rd place competitor (which doesn’t have flow control, and is otherwise 4 years behind me in development) and had their issues fixed. It’s better not to have a feature at all than one that doesn’t work right in all cases and is well-tested.

So the key point is, be very careful about testing, and don’t release with bugs.

Last week when I was looking for web design companies to make a new website for RakNet, I sent out about 10 requests for bids. The request was well-written, complete, and professional. However, I only got 5 responses. At roughly the same time, I contacted 3 marketing firms to help market RakNet, and didn’t get any responses.

Imagine if, instead of Jenkins Software, Microsoft or Sony sent that same sales request? How many replies would they have gotten?

This is clout, and its based on how much money the other side thinks you have, and how much they think they can gain from dealing with you. It’s like a game almost, where you have to have a certain clout level to talk to certain NPCs.

Connections are the cheat codes to give yourself clout with certain NPCs.

Yesterday I said to a friend, “Friends and connections are what this industry is about I think.” In fact some business activities require so much clout that it is nearly impossible to achieve them without connections, or the clout that connections have already provided you:

1. Become a licensed console developer
2. Sell to a company that makes over a few million in sales a year
3. Purchase major advertising
4. Get mentioned in major game industry news reports
5. Hire top-tier talent
6. Partner with other companies to form bundling deals
7. Get outside funding

And if I wrote a list of the 7 activities that make a middleware company most likely to succeed, what would that be?

Succeeding with connections is not a matter of making false friends of hope of getting something in return. It’s about being able to work with people you trust, and having those people trust you in return. Big deals require trust and 90% of our daily interactions is based the human factor.

Don’t burn your bridges, make an effort to know people, give good customer support, help people who ask for help sincerely, and keep in contact with the people you do know.

When I originally released RakNet I charged $2,000 per application. Later, I charged $0 (free) to increase market share because nobody was buying it at $2,000. Later, as it improved I charged $4,000. I got some sales there, so raised my price to $5,000. All the while I was wondering at the greed of my competitors that charged hundreds of thousands for more restrictive licensing terms. I was competing on price because I could afford to do so, and while my prices were good for an individual at the time, they weren’t enough to hire web designers, support, testers, etc. so I was endlessly stuck as a one man operation. I even read back then on Joel on Software about startups “Don’t compete on price” but didn’t listen.

What I didn’t understand at the time is while there was a HUGE difference between $2000 and $10,000 to me as an individual, to a developer that is the SAME AMOUNT OF MONEY – if your software helps them, they will license it either way, and it’s just as hard to sell to them. This is even more true with a publisher. You could charge basically what it would cost to write, and they would still license your software as long as it saves them time.

The hard part is:

1. Get them to know about your product in the first place
2. Convince them your product is worth using
3. Get through the non-programmers and legal to actually get money into your bank account

Because I undercharged originally, I short-changed my company money licensees would have been willing to pay. And since I was charging one-man prices, I never grew beyond a one man operation.

You might ask, “So what, as long as the quality is the same?” I did, and for programmers it doesn’t matter, because they are looking at your code, not your website. This will get some sales at smaller companies. At larger companies, programmers usually aren’t the ones writing the checks. Before signing any big deal, HR, lawyers, and producers are going to look at your site. And as I found out the hard way, all it takes is for a lawyer to say “This guy’s site sucks, so we don’t even want to deal with him” to stop a deal.

Also, by not financially growing your company, your competitors that do get the funds / clout to do direct marketing, and bundling, and get their software on the front page of gaming sites. I on the other hand have to sell to smaller customer base, many of whom already have my competitor’s product through a bundle. It’s like how I bought Windows Vista with my laptop, something I would have never done if I had the choice. Right now I have so little clout that even some advertisers won’t return my calls.

Because the quality is so good, RakNet actually has #2 marketshare right now, and I’m proud of that. But I only recently started charging realistic prices, and have to play catch-up.

Bit of advice based on what I’m going through now:

If you ever write middleware, make sure your contract includes that you can list the customer on the customer page of your website. It’s easy to get when selling your middleware, but quite hard to get retroactively. Right now a huge game is coming out using RakNet, and I’m not sure if I can list them as a customer which is very frustrating.

I hear that all other middleware companies also force the developers to list the middleware library logos on their splash screen. That’s another thing I previously had optional, and in practice has only happened once that I’m aware of. I’ve probably cheated myself out of a lot of free marketing and sales because of that. Make it another requirement.

I have another deal in the works and just delayed it to ask for this, but it’s important enough I’d rather lose the deal than get a deal without it.

It uses the layered service provider system I wrote about. For any UDP based application (actually with one line of code change it work with TCP too) it will add the desired attributes to the connection.

One big snag is how to implement an incoming bandwidth throttle. There are 3 ways data can come in, nonblocking, blocking, or with IO completion ports. Nonblocking is the easiest, I just stick the data into a buffer and return it when the user later calls recv or recvfrom again. Blocking is harder. I would have to put an infinite loop in the recvfrom call and not return until the required latency has elapsed. However, you can break out of the Windows version of recvfrom with a signaled event, and I have no idea how I’d hook into that event. IO completion ports is similar to nonblocking, but the buffer has to be added to from a worker thread elsewhere. If I can’t figure this out I’d have to drop the incoming bandwidth throttle. This would have been easier if I had implemented it at the NDIS level, although it would have been harder in other ways.

Another problem is the layered transport service provided by windows is DLL and registry based. So if you were to kill the program with end process, the DLL would not be uninstalled automatically when the process shuts down. If you were to then delete the program without uninstalling it that DLL would be stuck in your system, loading up the QoS emulator when you don’t want it. I’m not sure if the DLL can unregister itself in its own startup call. If I can, I will check for the program installation in startup, and have the DLL unregister itself if it looks like the program was uninstalled.

I’m not sure all this will work on Vista either. There was some crap in the Windows SDK documentation about security and various extra things you have to do in the installer, the registry, and the program itself. I only scanned over it in passing while looking for info on the Winsock debug/trace DLL system. I hope there’s code there I can just copy/paste.

This program has been very complicated and difficult to write, partly because I’ve never done anything like it before. It requires knowledge of threads, IO completion ports, sockets and socket properties, the registry, installation settings and permissions, interprocess communication, hooking in and debugging DLLs called by the system, and complicated associations between LSPs, the high level user-code, other LSPs, and the low level transport service provider. That article by Microsoft states “You will probably find that implementing an LSP is no longer a daunting and time-consuming task if you simply extend the layered sample” but if they consider that to be easy I’d hate to see what they consider hard.

Thanks to Jason King for help designing the GUI!

I was reading http://dewitters.koonsolo.com/gameloop.html where the author, Koen Witters, proposes a fixed-update game loop with a variable update render loop using interpolation to find out between-frame render positions. It’s a well written article and his solution is good for single player games.


const int TICKS_PER_SECOND = 25;
const int SKIP_TICKS = 1000 / TICKS_PER_SECOND;
const int MAX_FRAMESKIP = 5;

loops = 0;
while( GetTickCount() > next_game_tick && loops < MAX_FRAMESKIP) { update_game(); next_game_tick += SKIP_TICKS; loops++; } interpolation = float( GetTickCount() + SKIP_TICKS - next_game_tick ) / float( SKIP_TICKS ); display_game( interpolation ); }

However, this has problems with online games. In networking, you often need to extrapolate given a prior position and some amount of time to the current time. For example, I may get a position packet 208 milliseconds ago. If I were to update every 40 milliseconds (25 ticks per second), I would have 8 milliseconds leftover. This moves the problem of left-over ticks to your game objects.

You can either calculate partial ticks, in which case your game objects are slightly out of synch with the fixed-step simulation, or you can ignore them, and just use the least number of ticks you can fit in. If you get frequent updates this is OK, but for objects that are not updated frequently things can get out of place, and a lot faster than 2.4336088699783431e-13.

My solution is variable length game updates with variable length frame updates, with a frequency limit on game updates.

Variable length updates with variable length frame updates:


// Do not skip more than one second (due to debugging)
const int MIN_UPDATE_TIME=1000;
// Do not update more than every 10 milliseconds
const int MAX_UPDATE_TIME=10;
// How much time to render forward
float renderExtrapolationTime;

renderInterpolationTime = GetTimeMS() - lastGameUpdate;
display_game( renderExtrapolationTime );
elapsedGameTime=GetTimeMS()-lastGameUpdate;
}


One problem I had a couple of years ago was how to write an accurate internet simulator feature for RakNet. What I have in there now is what you find in your average games – a user can intentionally add latency to outgoing messages and can randomly drop packets to simulate packetloss. But this is not accurate and not very full-featured.

Here’s a better design:

Maximum outgoing bandwidth – Represents your network card’s ability to send data through the wire.
Maximum incoming bandwidth – As above, but for incoming data
One-way trip time – The minimum time a datagram would take to get to a remote system; the base speed of the network
Per-message additional trip time – On the internet, many people are sending messages, so there will be some delay between each of your messages as other messages goes out. So this number represents the rest of the internet using the router’s bandwidth too. The effect to the sender is that as you flood the connection, your ping goes up. This is what actually happens and is how flow control is accomplished in modern reliable packet implementations (the old way used sliding windows which controlled throughput based on how fast the other system acked your messages).
Maximum router queue – This represents how much average memory each router has. Go beyond this and you start getting packetloss.
Randomly dropped packets – Due to noise on the wire
Out of order packets – As above

The problem with hardcoding this in the user level network layer is that you have to implement a control interface (changable at runtime), you have to do it on both systems, and it can be complex to implement in every circumstance. So I’m writing a layered transport service provider. This is sort of like RakNet’s plugin system, but for Winsock, and it gives you the ability to override various network events with a procedure table. This is stored as a system DLL, and installed and uninstalled by an EXE. I will have this EXE double as a control panel and will use interprocess communication to have the EXE control the invoked DLL.

By the way, to say this is complicated is understating the issue. It’s about 1000 lines of code for the installer, 10,000 lines of code to properly create a plugin that does nothing but pass the data through. You have to understand overlapped events, IO completion ports, support for versions of winsock, an entirely different set of code to install the driver on Vista (plus permission sets), you have to use .def files for export, and more. The article that I’m going by is 8,500 words long (about 20 pages) and takes hours of study to understand well enough to do anything.

After going through all this I think Shunra is justified in charging $500 for their simulator. The solution is hard to write. However, Shunra doesn’t support loopback last I contacted them. My implementation does, so I think I can sell mine to gave developers and others that want a loopback solution. Plus I can put a nice GUI on it, so you can control what is going on at runtime.

I was updating the manual for RakNet and was reminded of how many data structures I’ve written for it.

From the manual:

DS_BinarySearchTree.h – Binary search tree, and an AVL balanced binary search tree.
DS_BPlusTree.h – BPlus tree for fast lookup, delete, and insert.
DS_BytePool.h – Returns data blocks at certain size thresholds to reduce memory fragmentation.
DS_ByteQueue.h – A queue specialized for reading and writing bytes.
DS_Heap.h – Heap data structure, includes both minheap and maxheap.
DS_HuffmanEncodingTree.h – Huffman encoding tree, used to find the minimal bitwise representation given a frequency table.
DS_HuffmanEncodingTreeFactory.h – Creates instances of the Huffman encoding tree.
DS_HuffmanEncodingTreeNode.h – Node in the Huffman encoding tree.
DS_LinkedList.h – Standard linked list.
DS_List.h – Dynamic array (sometimes improperly called a vector). Also doubles as a stack.
DS_Map.h – (Associative array) Ordered list with an per-element sort key.
DS_MemoryPool.h – Allocate and free reused instances of a fixed size structure, used to reduce memory fragmentation.
DS_OrderedChannelHeap.h – Maxheap which returns a node based on the relative weight of the node’s associated channel. Used for task scheduling with priorities.
DS_OrderedList.h – List ordered by an arbitrary key via quicksort.
DS_Queue.h – Standard queue implemented with an array
DS_QueueLinkedList.h – Standard queue implemented with a linked list
DS_RangeList.h – Stores a list of numerical values, and when the values are sequential, represents them as a range rather than individual elements. Useful when storing many values that are usually sequential.
DS_Table.h – Table with columns and rows, and operations on that table.
DS_Tree.h – Noncyclic graph
DS_WeightedGraph.h – Graph with weighted edges, used for routing via Dijkstra’s algorithm

Data structures is one of the few classes in college that has a significant real-world bearing. It’s one of the cornerstones of good design and optimization.

A few times I’ve been asked why I don’t just use STL.

1. STL can cause linker errors
2. STL uses exceptions.
3. STL doesn’t have every feature I need.
4. I don’t know how well STL is optimized, but I know exactly how well my code is optimized.
5. In college I was confused for months before I figured out when people said ‘vector’ they meant ‘dynamic array.’ It became a pet-peeve of mine.
6. In college I started writing my own data structures before I knew STL existed. None of my classes ever covered it And once I started I had momentum to keep going.
5. It’s something I find interesting and enjoy writing.

Aside from STL, of the game companies I’ve worked at, I’ve never seen any of them with as sophisticated a set of data structures as what I have in RakNet. When you need to process and track 30,000 messages a second you need heavy duty data structures like my memory-pool optimized B+ tree. Being able to support 30,000 messages vs. your standard game’s 20-30 messages a second is part of the polish you get with a dedicated mature library.

Back when I started RakNet 5 years ago I was talking to a marketing guy. He something like “You’ll get a sale here and there, but to really be successful you have to approach the companies directly.” My reply was “That’s fine, but I want to add these features first before I start having big companies look at it.”

I didn’t even get a sale here and there, so maybe I was right, but I did go a bit overboard. Not until last week really did I sit down, ask myself “What feature should I add next to be more competitive?” and just couldn’t think of anything.

It’s time to switch to marketing, and hopefully this time it won’t take 5 years to reach my destination.

The first thing I can think to do is partnerships. 9 months ago the tech director at SOE told me if they didn’t already know me, they would have just used Quazal, because Quazal is a featured partner of Gamebryo. But the funny thing Gamebryo doesn’t have networking. The partnership is essentially an advertisement to look at Quazal if you need a networking solution. I’m not complaining, but pointing out the importance of partnerships and alliances. People will go with what they think works, and a featured partner is more likely to work than some library on the net you’ve never worked with before.

As much as I’d like to, I don’t have the connections or clout (yet) to get RakNet into Gamebryo. But where I do have a better chance is in the open source and small game community. If the competitor’s forums are any indication, RakNet is pretty much dominant in this market, even if it’s a market that doesn’t pay very well. But more users is always better, one can always monetize later.

Step 1: Ally more closely with Ogre 3D.

Ogre 3D is a great library not just for its features but because it has a massive userbase. Any forum post gets a response nearly immediately, and that is good not just for its own sake but the support level generated by that draws even more users. I’ve had a good relationship with that community from Galactic Melee so it’s a first draw to get more eyes looking at RakNet.

In that regard, yesterday I wrote a sample that uses Ogre 3D along with RakNet to demonstrate how to do graphical network interpolation code. The users seem to appreciate it. Some people in my own forum have been asking how to do what I did there so it solves two problems.

Step 2: Irrlicht

Irrlicht seems to be the next big graphics engine. I used it back when I was writing a web based shooter. I like it better than Ogre because the style reminds me more of RakNet. I contacted the author of Irrlicht, and he agreed to add RakNet to the toolset page if I were to do add native support, which I will do in a similar fashion. I’ll do this in the upcoming week

Step 3: Partner with smaller commercial game engines

Most game engines that I searched for already had networking. While I think a dedicated library like RakNet will be more advanced than what a game engine will offer as networking on the side, it’s probably a lost cause in most cases. However, I did find a few game engines that didn’t already have networking. Maybe none of my emails will get a response, but if they do this can be a good additional revenue stream as well as benefit for said game engine. This will require a lot more work since I’d actually be learning and supporting new engines, but is also a good opportunity.

Step 4: Pursue connections

(Coincidentally) I had lunch with a very well-connected friend today and he offered to help me get my foot in the door in several big opportunities. I won’t say what, since it may not pan out anyway, but it’s really great having friends to help you out or even just discuss ideas over.

The major money comes from friends and connections. Tip to the new guys: Never burn your bridges and make an effort to make friends with people. You never know who will help you out in the future, and of course you get to return the favor.

Anyway, I’m optimistic about the next 6 months. If RakNet can get a few big partners I think things can finally take off.

I spent an entire day on Sunday thinking about the architecture on how to write a generic system to create, destroy, serialize, and control scope over networked objects in C++. It’s a very hard problem to solve because you know nothing about the game objects themselves, or what complexities the end-users might introduce with their specific architecture.

In the end I decided on 3 classes. One class, ReplicaManager2, is the base plugin for the system. All it does is essentially serialization, deserialization, transmission, and bookkeeping. Stuff you wouldn’t normally care about. I have the user pass a class factory that generates connection instances, one per connection. This controls high-level and non-object specific functionality such as how to create object instances. Lastly, there is the base class Replica2, which game objects should derive from.

The main reason I did it this way, rather than putting everything in the ReplicaManager2 plugin (as I did in its predecessor) was so that the functionality can be easily overridden. This should solve a problem I had with the old system, where I ended up passing a ton of data to the callbacks in order to force it to get it to do what I wanted. By assuming users will override functions, this also gives me the ability to make assumptions and thus much more elaborate functions.

Improvements it has over the old system are:

1. Relay across systems actually works correctly. So you could have a server cluster, with n clients on each server, and everything would just get sent out properly. It automatically deals with cyclical connection graphs up to a depth of 1 away from the source node.

2. Defaults to polling for changes. Each tick, if an object is now constructable, or in scope, and it was not in the last tick (or vice-versa) the object is created or destroyed appropriately. The object is also serialized every n milliseconds, and if it serializes differently under the same context, the newly serialized value is transmitted. This is easier to use than event-based updates (which are also supported) so the user has to write less code.

3. Downloads construct all objects first, then deserializes them, and in the order they were originally created. This is likely what a user would do were they to hand-write the download system, and maximizes the chance that pointers and other dependencies will be valid on deserialization. This also allows you to have objects that cross-reference each other when deserializing, each both have already been created anyway.

4. It’s easier to modify events and more information is given about what triggered those events. For example, it’s now trivial to send using different reliability types based on the context of what is being sent, something that was hard to do before.

5. Changes are easier to initiate on the client, especially construction, which before would have taken custom game code.

Here’s a video that goes over the system. I start out too fast because I was tired of starting over when I made mistakes, but later on it gets better.
http://www.rakkarsoft.com/raknet/manual/ReplicaManager2.html