Threading and Massive Multiplayer Games

[David_S_Intel4]

NetDevil mentioned in his Introductory post that he may have threading questions related to the development of Massive Multiplayer Games, see message, copied below.

Anyone else out there also have experience in this area?

NetDevil, please ask away . . .Your very own version of stump the band!

--d

NetDevil wrote:

I am a senior programmer at NetDevil; we make massive multiplayer games. As you can imagine, we have a lot of threading issues to worry about on our servers! We're currently working on a game due to be released at the end of 2004-ish, tentatively called Auto Assault, to be published under NCSoft. To make things fun, this game uses a realtime Physics engine with a full vehicle model (suspensions, gear ratios...). Weeeee! Ready for those questions? :)

[netdevil]

Hehe, I'm not ignoring you, I've just been busy what with E3 recently behind us now ;).

The general architecture behind our most troublesome (logical) server is that there are a number of "sectors" that run independent of each other physically. Under most circumstances there aren't more than 10 players in each of these instances, but the physics is pretty intense nonetheless, because we're taking more of a "mass carnage" approach than most MMORPGs you might be familiar with... so we have a large enemy AI to Player ratio, and the AI not only has to do tactical/flocking/group AI, but also physics for vehicular or creature movement, pathfinding, weapons targeting, etc. We have an advantage in scalability, since we can always "just add" more physical servers and distribute additional sectors to it, but by the same token we have to handle quite a few players/sectors on each machine to maintain our desired cost-to-user ratios.

The most fundamental question we asked ourselves, then, was the primary threading model. We have DirectPlay running it's own little thread pool, but otherwise, we had to figure out - should we split the work into threads based on task (like AI, Pathfinding, Physics, etc), or by sector (sector 1, sector 2, etc). We reached our conclusion fairly quickly, but I would love to hear any input from the gallery :)

To make things a little more fun, our third-party physics engine *must* kick off that sector's physics at least once every 33 milliseconds or it runs the very high risk of either crashing or "exploding" the physics (producing very abnormal physical results).

-Dale

[Intel_C_Intel]

I have a site dedicated to creating a scalable C API that uses state-of-the-are lock-free algos:

AppCore Framework Site

A super-scalable Winsock Socket Peer C API using IOCP and Overlapped I/O will be built upon it.

The Socket Peer Framework is going to be designed to scale up to tens-of-thousands of connections. There will be build in support for socket groups, socket friends, socket worlds, ect...

Perfect for a Game Server that eat's tens-of-thousands of connections for breakfast!

;)

You thoughts?

[ClayB]

Dale -

I've been pondering this for a bit. I'm not exactly sure how a "sector" figures into things. Is it a physical domain (e.g., level in dungeon, room in castle, forest), or a logical grouping of a limited number of things (vehicles, characters, weapons, treasure, etc.)? You mention being able to add more servers and attach additional sectors to them. While this may not smack of how threading can be used, it seems to me that it would be fairly straightforward to spawn a process in this instance, rather than a thread. So, I think I would go with threads dealing with each sector separately.

Besides the scalability advantage, I would think this method simplifies programming since all functionality is contained within the sector computations. The interactions between elements that control the sector (AI, Pathfinding, Physics, etc.) would all be doing so amongst each other and have no need to interact outside the sector (assuming my interpretation of your description is correct).

-- clay

[netdevil]

Hey Clay,

A sector is a both physical and a logical collection. Items in one sector are 100% contained within that sector and can never leave, excluding the players who can transfer via an entirely different server. It's also physical, denoting an area about 1280m on a side.

There are a sizeable number of shared resources, however, that prevent them from being different processes in an effective manner (although we do have multiple sector service processes running, one per machine).

I actually came to the same conclusion you did, each sector in a thread makes sense. The only big caveat is the 33ms timer... where physics must kick off as precisely as possible at 33ms intervals. So the choices are to either "cap" the other subroutines and force them to end when the time is up (AI, et al), or to have physics run in a timer-based signalled thread, independently. Both solutions present big problems, though!

-Dale

[o_neuendorf]

Hi!

I've never done Massive Multiplayer Games, but quite a bit of massively parallel physical simulation. Problems seem to be similar. Our solution was always to use a domain decomposition (probably what you call "sector"). The golden rule is: as few coupling as possible. Sometimes we had to modify or completely change the algorithm to reach that goal.
The definition of a domain depends on the problem. But the lest important thing is the "task" to perform.

Regards,
Olaf