At the moment, those of us working on the development of Epitaph are doing so with ‘live ammo’. It’s one the reasons I don’t like having too many players around – actual players make it difficult to be cavalier with whether or not the game is working. The nature of development and the language we use here is such that sometimes things break as a result of new features. LPC is a tremendously flexible language, and the virtual machine that the driver provides is a pretty forgiving beast. However, it’s not perfect and sometimes the smallest additions can stop other things from working, or even loading. Working with live ammo means I can leave that kind of thing broken until we next reboot, or until I next need that object myself (whichever is sooner). It’s just easier for us.
When the game opens to players though, that’s no longer appropriate – so I wanted to talk a little about how I’m planning for new features and bug-fixes to be handled on Epitaph when having players is the norm (hopefully!).
In my experience, MUDs are particularly bad for having a good regime in place for this. Many MUD engines require a full reboot if any changes have been made to anything particularly core, and those reboots are scheduled as and when features are completed or when it’s just time for the driver to reboot. Those that allow you to fix or change code in a running game (as with LPC) subtly encourage you to work directly on the active code. While that’s perfectly fine if all you’re doing is coding rooms in your own work-area, it becomes increasingly more perilous as you descend into the deep, dark places of the mudlib .
Still, when working with multiple developers, it’s hard to do anything other than work on the live game unless a formal process is in place. Software tools can go some way towards making collaboration less painful (revision control, for example, is all but mandatory), but in the end those are technical tools to address a social problem. The problem is that developers in these circumstances consider ‘temporary crashes’ a necessary evil of development. After a while, players too begin to simply accept that these things happen – when an error occurs in the game, someone will invariably say ‘maybe they just changed something’ rather than ‘why is this game suddenly causing runtimes all over the place?’
In especially egregious cases, developers with no-one who can really scold them for it may become so cavalier about changes that they don’t even test to see if their changes will even compile properly. They’ll make what they think is a foolproof change, and they’ll mess up the syntax and not even notice. They’ll think ‘That problem I identified earlier will be fixed for when the object is next updated’, but never update it themselves to double-check. Then they log off…
Partially this is all a result of the usual developer attitudes of MUD development, and partially it’s a resourcing, policy and architecture issue. How else are you *supposed* to do it?
Well, look at any commercial MMO that has long term, active development – they never just change code arbitrarily in your client as they’re developing it. Instead, they have a more sophisticated pipeline model – developers work with their own sandboxed versions of the game (there are probably several segments of the pipe here, but I don’t recall ever hearing the process described) before they push a ‘testing’ version onto one of the test servers. When the changes have been properly explored and stress tested, the testing version becomes a ‘release candidate’, and shortly after that the release candidate is pushed onto the real game as a patch. The result is a much more ‘professional’ experience – you aren’t expected to simply shrug off a broken game.
The difficulty here is that it requires three things to be in place for this to work. You need the resources to run a ‘development’ version of your MUD, you need a policy that *requires* all developers to write new code on the development sandbox, and you need an architecture in place that allows for changes in the development version to be pushed easily into the live game. If you don’t have all three of these, then all you *can* do is work with live ammo.
For the purposes of most people, having a culture where people are simply accepting of errors is the easiest solution. But, there is another aspect to adopting a system like formal patching… it makes for a more exciting community, and it makes for a better packaged game.
A new patch for Warcraft is exciting, and it’s exciting for the entire period between it being deployed on a test realm to some time after it’s made live in the real game. A new patch is a source of discussion, a focus for anger and despair, and a promise of things to come. It’s a sign that you have something to which you can look forward. If you deliver features as you develop them, you get a general low-level buzz of interest as people notice the new addition, but what you lose is the impact of having a large chunk of things ready at once. Think of it this way – as a child you probably got more presents over a year than you got at Christmas or your birthday, but which do you remember best? Which did you and your friends talk about most?
Anticipation is a powerful tool, and you neglect it at your peril. Patches, provided you publicise them ahead of time, are a great way of generating anticipation of new features. Before the achievements were made live on Discworld, I made a news post saying ‘achievements are coming, get ready’. What happened from that point is that people started to generate excitement around a coming new feature. I’m not saying that was the only reason it ended up being so popular, but it was one of the reasons so much attention was directed to the system when it went live shortly afterwards.
Imagine then that you have twenty quests ready to go into the game over the course of six months. If you add them in one at a time then even if you post about it, all people will say is ‘a new quest? Yeah, cool’. If however you announce the ‘questing patch’ and then deliver them all at once, people will be talking about the upcoming patch with greater intensity and anticipation. That’s a great energy to have running through your game even if it does mean that the people who finished writing their quests first have to wait until the patch date to see people enjoy them.
So, patches are exciting – but they also have a second benefit for your game experience in that they allow you to package and theme new content in a way that greatly improves how it is delivered.
One of the difficult issues in game development is that content takes much longer to create than it does to consume. Genua took us four years to bring into the game, and I’d say it got ‘consumed’ within a few weeks. People had seen the areas, done the quests, played with the new game systems, and then went ‘yeah, that’s all fine’ and went back to optimising xp/hr. As a reward to effort ratio for developers, it’s soul-destroying.
However, if you have a formal patching regime you can be more cunning in how you package up new content. Let’s take one simple example – a long running game which is about to get a new area, and a new faction system. An area gets consumed within a matter of days, but if you have co-ordinated the development of these two features so that the new area is also highly integrated with the factions, you can ‘control’ the rate of consumption. You can lock off areas until enough faction standing has been accumulated, or you can block some parts of the area forever based on competing faction allegiances. That’s something you can only do if you have the development of the systems working side by side, and that in turn is only possible if the developer culture isn’t about drip-feeding things into the game as they are ‘ready’. The former encourages collaboration to increase impact, the latter encourages self-contained, isolated systems.
The neat thing about packaging is that it can work for almost any combination of game features – you can almost always think up a theme that is sufficiently all-encompassing that you can link up the major additions. If you add in an extensive pottery system, a new high level instance and a goat herding subgame, you can package that up easily by having the pottery system provide components for the instance and the goat herding, and have the instance feedback both rare raw components for the pottery, and perhaps epic goats for the herding system.
One of the real dangers here is that you make your patches *too* big and *too* all-encompassing. Patches are exciting because they are relatively unusual, but not so unusual that they are an annual event *if you’re lucky*. Patches should be small enough to produce regularly (say, every six months) but also large enough to get people salivating.
In any case, the end result is that you get finer grained control over how players consume the content you put forward. Player fun must always be centre most in your mind of course, but it’s possible to increase the effort/reward ratio without simply adding grinds for the sake of grinds. It’s all about how you present it.
Lots of benefits then come from a formal patching system, but it does require a lot of back-end to make it work, and you have to be willing to sacrifice a little ‘agility’ because of the need to co-ordinate deployment.
Step one in setting up this back-end is to actually have a development sandbox. For most MUDs, this will be as simple as having another version of the MUD running off of a common code base, but ideally you want it on an entirely different computer or virtualisation shard. You want it so that if someone puts an infinite loop in their code then it doesn’t matter except to other developers, and so you want it to be held as far as possible from the ‘live’ game as you can.
Step two is to *make* people use this sandbox. Having a sandbox by itself is a cost in the cost/benefit analysis developers make, and humans are ruthless cost/benefit optimisers. You can’t make it optional, because a patching system requires everyone to be working from the same page. If someone makes a single change to the live code, then it is no longer possible to simply push a patch into the game because the patch will override the change that was made. Everyone has to be doing *all* their development on the development shard .
Step three is to make it a ‘pres butan’ task to deploy a new patch. It shouldn’t be an awful, dreadful chore to push the patch onto the live game. It should be almost entirely automated to reduce both errors in following the process, and to make it something that isn’t dependant on someone being able to make the time to do it. So much in MUD development doesn’t happen, or happens slowly, because people have real life ‘shit to do’, and you don’t want that to be the reason your patches are delayed. If patches aren’t being pushed into the game quickly enough then it makes your game seem static to players, and will frustrate the hell out of your developers.
If you’re willing to make the sacrifices and put in place a technical architecture to support the process, patches are an easy way to really harden your development processes, improve the effort/reward ratio for developers, generate community buzz, and just look like a more professional game entirely. We’re certainly going to be doing this for Epitaph when we become version 1.0.
 As an example of this, LPC has a programming structure called a ‘class’ (which is really just a C/C++ struct). If you add a new field to this class and update an object, the object will fail to load if a differently sized version of the class is in an inherit or, more painfully, in the general include that defines it. Thus, even if an object is not using the class at all, it may fail to load because of different sizing just because an object along the inherit chain has an #include.
 And not only am I guilty of everything I’m going to say here, I knew *at the time* I was guilty and just didn’t care very much.
 When doing the achievements on Discworld, the handler touched on almost every core system in the game, and was undergoing huge amounts of development. The result of this was that breaking the handler (which happened often) could result in any related piece of code breaking too. On a game with 100ish players online at any one time.
 The set of ‘people who are professional software engineers’ and the set of ‘people who want to run a game’ don’t overlap particularly.
 There’s also a more agile ‘hotfix’ system, but even that doesn’t work by changing code in the live game until it’s been written and tested somewhere else.
 If you’re one of those people so invested in a game that a +3% buff to your periodic damage is a source of excitement.
 I’ve been one of those people.
 Here, ‘present’ is being defined as anything extra that was given to you for no reason other than people being nice to you.
 At the same time, one of the things that makes a MUD more intimate than an MMO is that you can get to know the developers, and developers are part of the social fabric. You want them chatting to players, because that limits the ‘us versus them’ mentality. There then needs to be some kind of bridge between the development sandbox and the live game. Perhaps you simply share talker channels between them.
 Although you’ll want to limit it so that it needs someone at the top of the org chart to be able to pres the butan at all.
 Imagine if the process was ‘Get a list from RCS of all the files that have been changed, then copy them by hand to the live game, making sure at all points not to over-write a change that has been made there’