The infrastructure work we’ve been doing is starting to converge towards its end-point – I’ve already written at length on all of this, so I won’t beat a dead horse by repeating anything about it save to say ‘see this shit here? See that empty hook? Yeah, it’s empty because this shit is off it’.

We’re now at the point where the most of the remaining significant hurdles have been cleared, and we’re juddering towards the time where Epitaph Black Ops is closed off to players and the live Epitaph takes over. We can’t do it just yet because we’re missing two significant elements – one is shared bug-reports across the live and dev shards, and the other is cross-MUD chatting.

For the latter we have intermud, but that is a creaky, archaic infrastructure that really doesn’t meet our needs. For one thing, it’s entirely separated from the main talker channels, and that’s not the experience I want for players. The intention of the dev server is not to fracture Epitaph’s various individuals into ‘us’ versus ‘them’ – it’s just a professionality requirement to separate out the experimental code from the ‘live’ game. It makes it a little lonely though when players are on one MUD and developers are on another, so Eek’s architecture is maturing towards the point where we can actually have exchanges like this:

[newbie chatter] (1240 MHz) Drakkos*: Hi guys!
[newbie chatter] (1240 MHz) Karras: Oh god, here he is.
[newbie chatter] (1240 MHz) Drakkos*: I sure am! What’s the happy haps?
[newbie chatter] (1240 MHz) Karras: Killing me won’t bring her back, you know.

The little * symbol indicates that the chat comes from another MUD[1], but the effect is otherwise seamless. We can see what’s chatted on Epitaph and Epitaph can see what’s chatted on Epitaph Black Ops. Thus, while we’re remote we’re not out of the loop, and we’re not setting ourselves up as aloof demigods cavorting on our own private Olympus. It’s all good.

Shared bug-reports is the second of the significant elements that need to be addressed. My experience in trying to set that up last week was… unpleasant[2]. At the moment we have two separate mysql installs tracking bugs on the dev shard and the live shard – that is obviously unworkable, we need to have them both work from the same data store. I’ve been vaguely considering modernising our bug handling system too – it’s kinda creaky, written in LPC and filled with issues I’m likely never going to get around to fixing. However, it seems that all of the bug management systems out there are pretty awful (although I did like the look of FogBugz, even if it is a commercial service) and it seems like it would be incredibly difficult to get any to work with the heavy integration we require.

The possibilities for integration is by far the biggest element for offloading onto external systems. Eek’s password routines let us write our own authentication plug-ins for WordPress, phpBB and Mediawiki, and that’s step one of integration. Step two is making the content meaningfully available within the MUD. Hamlet@Epitaph had previously written us an XML parser in LPC, and Turvity@Epitaph has roped this into service for an LPC RSS parser. This has given us some awesome opportunities for integration – as long as something has an RSS/ATOM feed (like the blog, boards and mediawiki have) we can pull in an RSS feed, parse it out and then display it within the MUD. That’s working currently for the blog, but there was some awkwardness in getting it to work for the board (specifically, in terms of authenticating restricted content with an RSS feed without potentially torching any semblance of security). That’s solved now though. We can read stuff in and it’s All Good.

Writing stuff out can be a little trickier. The blog is easy because we can just set up an email address to which we can mail our blog posts (which has been done) and then from within the MUD we just send an email out into the big bad world. That’s been setup and so we now have pretty much 1 for 1 functionality within the MUD as we previously did for our LPC blog engine.

Being able to write board posts from within the MUD is quite important, because it’s part of the regular workflow of many of us. I hardly ever write a board post through the LPC web interface, I almost always write it on the MUD. I can live without that functionality, but I don’t really want to in the long term. It’s not just me that would be affected if we couldn’t post to the boards from within the MUD – we have a number of game systems (such as the error tracker) that post regular updates to creator boards. We don’t want to lose that permanently, but we can do without it for a bit. It’s important, but not mandatory.

The bug system though is different – that’s critical functionality. The sad truth is any barrier you put between you and the bug tracking system means less bugs will be fixed. The cost in maintainance is already very high[3], and if you make it higher by adding awkwardness to the process, it’s just going to be harder and harder for developers to gnaw through their straps and get on with it. Similarly, any barrier you put between a player and a bug report is a roll of the dice on whether or not a bug gets reported at all.

Our error system is currently pretty great in the information it gives about each bug – not perfect, but pretty great. When a bug is reported it pulls in all kinds of useful information such as the filename of the room the player is in and the output of the last runtime error they encountered. The command used to report the bug files it in all the right places with all the right tags (so we can easily get all the typos in all the helpfiles, as an example). When bugs are fixed, that information is communicated back to the player, who also has access to each of the bug-reports they made previously. We have bug tracking handlers that regularly post stats on which directories are the most crudded up, and the delta between closed and opened bugs.  It’s a good deal more professional than even a lot of commercial software shops can boast.

The problem for us in moving to an external bug system is that information is all but mandatory, and it all has to be accessible within the MUD. It’s not appropriate, in my view, to expect people to lodge formal reports in an external tool like bugzilla or FogBugz. If that can be an *extra* route, it’s fine – but it can’t be the only route or many bugs simply won’t be reported. The in-game bug and typo commands have to stay.

That means that when those commands are used, they need to take in all the information that we’d normally get and then offload that into the external service. FogBugz offers some of this via HTTP posting, but there are subtleties to the way we do things that aren’t easily managed.

Within the MUD, we need to be able to get access to those reports for fixing – FogBugz offers RSS feeds which would go a long way to allowing for that, but we run into the same authentication issues we had for the boards except with less obvious solutions. Bugs often contain sensitive information, and we need a way to be able to have authenticated feeds that are shared with the MUD rather than with individual users. We’ve solved that for the boards by using a dedicated account with an IP whitelist, but that was awkward to set up and I don’t know if that’s possible to do with an external bug tracking tool.

Finally, we need players to be able to see what happens to their bugs – they need to get informs when bugs are fixed, be notified of comments added to them, and then we need to be able to get stats on their fixed to denied ratios (for… reasons). So not only do we need to be able to read the data back into the MUD from an external service, we need to be able to do in in various complicated ways.

I don’t know if we’re going to work out how to do all that. We may just go with what we have except make all the MUDs share a database.  This may be one of those areas where the costs outweigh the benefits.

The decision to externalise core game features is a balancing act. You have to weigh up the effort it is going to take to reskin the interfaces to suit your theme, the extra code you need to write in order to integrate the tools properly into your game, the risk associated with increased points of failure[4] and the integration features that you may eventually simply have to live without.  Against that you assess the increased features that you get for free and the likelihood that the maintenance burden on you is removed,  You throw all that in a pot, stir it around and then eventually from that comes the decision as to whether or not it’s worthwhile.


[3] Costs in terms of time, effort and morale. It’s just not fun to fix bugs for the most part. Some people enjoy it, but I am not really one of them.
[4] we don’t need to worry about one server going down now. We need to worry about two