Well, on Friday we are going hit the two week mark in terms of how long we’ve been open. It’s been crazy busy as we’ve had players on ‘in anger’ for the first time. We’ve had players before of course, but this is the first time they’ve not had to be mindful of the fact there was a player purge to come in the future. Now begins the painstaking alchemy of trying to maximise the conversion of time into fun, and that’s what game development is all about.
I’ve made mention a few times of just how big a code base Epitaph is. In a previous postI discussed it a little and talked about what it means to be a million LOC project. The pertinent line for today’s purposes was taken from another blog post, which said:
- A million lines of code will typically have 100,000 bugs pre-test. Best-in-class organizations will ship with around 1k bugs still lurking. The rest of us will do worse by an order of magnitude.
Well, we ain’t best of class as far as that goes it has to be said. Partially because:
- A million lines of code will occupy 67 people (including testers, tech writers, developers, etc) for 40 months, or 223 person-years.
We haven’t had 67 people working on Epitaph to that extent – we haven’t even had a single person working on it full time. Our ‘person years’ is a fraction of that 223, but I couldn’t really guess at what it is. A tenth would be optimistic I would think. I’ve certainly put a full time job’s worth into it, but that had to be done *around* my real job and I am but ONE MAN.
Assuming we are a typical million LOC project, that means we have about 100k bugs lurking in our recesses. That’s not hugely out of expectation – general software metrics suggest between five and ten bugs in every 100 new lines of code. The number of bugs increases as the complexity and size of the code expands. There are parts of Epitaph that are so simple as to be bug free. There are other parts so complex and that interact so constantly with other complex systems that they’ll likely have a good deal more than the average.
Within Epitaph, part of it is not just the code itself but in how the code interacts – if you think of 100 complex lines of code, it might have ten bugs in it when we ‘ship’. Another complex 100 lines may have ten bugs. That’s just internally. The real subtle things occur when those two systems interact, and those bugs can feed on themselves. A bug in one part causes a bug in the other, and that’s a ‘new’ bug that comes in purely from an interaction.
With all that said, I’m pleased with how few genuinely show stopping bugs there are. There have been some of course, and we’ve squashed them as quickly as we can. Since we went live, we’ve applied four patches, and we’re going to apply another reasonably soon (see http://epitaphonline.co.uk/patches.c for our patch list to date). Each of these is a collection of fixed bugs, new content, implemented ideas and some structural changes we’ve wanted to make. We’ve been stressing a lot of the game that hasn’t been well stressed before. That’s leading to a breaking point for some systems where we realise ‘to fix this we need to work with code that is really painful to do anything with. It would be easier to rewrite’. In the process, we introduce new bugs – another software engineering ‘rule of thumb’ is for every two bugs you fix, you’ll introduce another one. Bug fixing is then a real life Zeno’s paradox – you can always trend towards zero but you’ll never get there.
Sometimes though introducing bugs is better than the alternative. It’s all about how many bugs a system will generate in the long term, and that in turn is a function of complexity as outlined above. I’m going to do something I rarely do on the blog and actually show you some in game code. This is code that we inherited from the DW lib we used as a base. It’s the code makes handles exchanging goods for money in a shop:
Just… look at it. LOOK AT IT. Look at what you needed to do in order to remove some money from a player and give them an item in return.
That was the reason behind our rewrite of money – the reports that were coming in started to veer dangerously into the territory that meant we had to work with the money code. That’s basically a dozen interacting objects all with that level of micromanagement and complexity. So, we rewrote it instead:
That’s it now – and of that code, only one line relates to money itself:
pl->transfer_money (cost, query_current_money_area(), query_safe());
A lot of that simplification is due to the lost of ‘game features’ or ‘code flexibility’, but I’ve spoken about that previously too. This was flexibility that only ever had a cost to us. It made every piece of money handling code several times more awkward to do. It involved a whole pile of micromanaging to do what should be a simple action. Sure, it was neat you could spend a pound on a 47p item and get 53p in change back. It was neat that then you’d only be able to give money in supported denominations – if you had a 50p and a 1p, you could give someone 50p, 51p or 1p and nothing else unless they could make change. That’s all very clever, and it’s all very realistic. But in game terms, all people want is to know how much money they have and how much things cost. Now money is just an integer that gets turned into a string for manipulation purposes, and everything just got a whole lot easier.
Sure, in the process we introduced new bugs, but we’re fixing them, and quickly. Since we released, I make an entry in a spreadsheet of how many bugs in various categories are open, and how many are closed at the end of of a ‘bugfix’ day. On day one of epitaph, we had one open report. Today, at the time of writing, we have 97 open reports. Since open day, we have closed over four hundred reports. That suggests two things – one is that we’re only just denting the 100k or so bugreps that we likely have overall. The second thing is – we’re doing pretty well. We have fixed (or denied) 80% of the bugreps that have come to us already. Over our entire lifetime on Epitaph, we have closed 2900 bugreps. Our close rate to date is a touch under 97%.
This is a bit of self congratulation about how well we’re handling the incoming reports, but that’s not to rest on our laurels. The game is still buggy, full of typos and unbalanced in a number of areas. Every patch that becomes a little less true, but it’s still true *now*. We’ll keep working on that, and every patch we can be a little more confident that people are getting the fun play experience we wanted them to have.
But, in this post I have actually veered away from the main point I wanted to make. Some people have been chatting on the radio saying how they feel as if they’re making some small contribution to indie game development by submitting bugs. That’s not true – they’re making a *huge* contribution. Every single bug that is reported is a bug that will be fixed. An individual report by itself may not meaningfully shift Epitaph from ‘broken’ to ‘working’, but the accumulation of bug reports is doing wonders.
The thing is – we as developers can’t *really* test the game. It *works* for us. A game system is never put in unless it works (more or less) flawlessly. The developer is a terrible person to determine what counts as flawless though. If you use a different syntax, approach it in a different way, come at it after using another command we hadn’t used previously, the cracks start to show. The developer testing, unless there’s more discipline in the process than any of us really have the time to put into it, is really just a test of the ‘ideal path’ through a system. It needs real people with no inbuilt expectation of how a system should work to really find out the ways in which it just doesn’t.
So, a huge thank you to everyone for your help in reporting bugs so far, and I hope you continue on with it until we are entirely bug free. It may be discouraging as a developer to see a bug count creep ever higher every time you look, but we all *know* how important it is that the bugs are reported and the bugs are resolved. Not just bugs though – unfair features, difficulty curves that are unreasonable, general balancing issues. Your part in identifying all of these and helping us fix them is immense.
Perhaps the greatest thing about this kind of indie game development is that we’re all part of a collaboration here. You’re not sou-less consumers of content served up by distant developers. In many ways players and developers together are shaping this into the game that we want. None of us, developers included, will get this *all* their own way. Working together, with give and take and with a wary eye on the implications of changes, Epitaph is gradually becoming a communal property in which everyone has a real claim of ownership.
 YES I AM A MASSIVE NERD GO AWAY
 Fittingly, about money.
 Which we always give a reason for.
 if it is at all possible – as you’ll see with new money, we’ll even rewrite entire game systems to fix issues!
 ANY DAY NOW