The Process of a Patch

With the recent confusion of the patch release, I have sought the experience of someone to help us understand and to explain to all of us Civfanatics just what goes into creating a patch. This will be based on assuming a usual publisher-development relationship and not a self-publishing developer (like Valve), nor a developer with a lot of independence.

Patch Decision: Simply put, first off the process starts with the decision to make a patch. That is a publisher decision, not a developer decision. Much of the time, however, it's an automatic decision. Making the first patch for a game is pretty much always automatic. Generally speaking, the likelihood of patch development being approved drops as ^a) more time passes since the game's release; and ^b) add-ons come out for the game.

Development: Once the decision is made to make a patch, the normal development process starts. A list of issues to fix is compiled and the programmers begin writing the code. Builds are given to some of the QA people who test these fixes and identify new problems, which are sent to the programmers to write the code, therefore repeating the cycle. This continues until either the development team is satisfied that the patch is ready or the deadline comes. While patches generally have a more flexible schedule than full products (for which the release date is usually set many months before being ready), deadlines still exist

Just like in development of normal software, patch development also has lockdowns, usually. A lockdown (also known as feature freeze) is when the patch is said to be feature-complete. No more fixes and/or features will be added, barring critical discoveries, and the rest of the development time is spent on testing and perfecting/re-fixing the existing fixes/features. This then continues until a release candidate is made.

QA (Quality Assurance): Here it's important to make a note on QA. There is not a single QA department. There are at least two, sometimes more (as in the case with Civ4). First, there's internal QA. These are people who are employees of the developer. They typically get the most versions to test. Their primary focus is on making sure that the things that were supposed to be fixed are indeed fixed. Internal QA will usually create special circumstances, if needed, to test these existing fixes.
.
A possible second level of QA is the testers taken from the outside public, not employees of either company. Generally these are volunteers, so it's hard to say much about their testing methodologies, as they will vary from one group to another and differ between people in each group. Such groups are typically in direct contact with the developers.

The final level of QA is the publisher QA. In addition to checking the existing fixes, they do hardware testing, trying the patch out on different hardware configurations to make sure that it hasn't broken, for example, the game on a certain brand of video cards. Publisher QA may also test for installer issues and such. Typically the people from publisher QA are not specialized on the particular game and instead deal with a wider array.

Release Candidate: So, when the developer makes a release candidate, it means that it's a version of the patch that may be ready for release. The version is submitted to the publisher (text for localization, if necessary, is submitted earlier, at around lockdown time), and then the developer waits on the publisher's word. The publisher may reject the patch, identifying new issues that the developer has to address, in which case the process repeats itself with a new release candidate. If the patch is approved, then it's ready. This can take some time if the publisher has other games to deal with at the moment or other high-priority matters.

Patch Approval: However, when a patch is approved, it doesn't mean the developer can release it. Releasing a patch is also the publisher's prerogative most of the time, so it'll be out when the publisher posts it to the relevant websites.

At some point, a public beta version may be made available but it's not a move that many publishers would approve of. Hence, it's something that usually happens with independent developers, Stardock being a good example.

As you can see above, the full process of creating a patch is very complex and time consuming. It’s not something that can be done very quickly. Even if it is completed and deemed ready for release, it’s still up to the publisher on when to release it.

Methos said:

QA (Quality Assurance): Here it's important to make a note on QA. There is not a single QA department. There are at least two, sometimes more (as in the case with Civ4). First, there's internal QA. These are people who are employees of the developer. They typically get the most versions to test. Their primary focus is on making sure that the things that were supposed to be fixed are indeed fixed. Internal QA will usually create special circumstances, if needed, to test these existing fixes.
.
A possible second level of QA is the testers taken from the outside public, not employees of either company. Generally these are volunteers, so it's hard to say much about their testing methodologies, as they will vary from one group to another and differ between people in each group. Such groups are typically in direct contact with the developers.

The final level of QA is the publisher QA. In addition to checking the existing fixes, they do hardware testing, trying the patch out on different hardware configurations to make sure that it hasn't broken, for example, the game on a certain brand of video cards. Publisher QA may also test for installer issues and such. Typically the people from publisher QA are not specialized on the particular game and instead deal with a wider array.

Click to expand...

It would take quite a leap of faith IMO to believe that the last two patches have been tested in any capacity. The botched combat animations in 3.03 and the lopsided AI resource trading in the latest patch only took about half an hour of actual playing to discover, something that obviously never happened before release.

Methos, I appreciate the information.
Many people have no clue how the process works... they just expect the patch, they want it right now, and they expect it to be perfect. It's not that easy... or the game would have been perfect when it was released (or closer to it).
I'm thankful when companies patch games... and when individuals also make an effort to improve things. It's not always that way.

I have given Firaxis the benefit of the doubt on many occasions, but I cannot really see anyway the last two patches could have been cleared by remotely competant QA. I accept that some small bugs may slip through QA, but not ones that become blatant within ten minutes of any game.

3.03, in its released condition, cannot have been tested. If a single person had played even 10 minutes of a game, they'd have spotted the combat animations were broken. I do not believe three levels of QA missed that. I suspect that since 3.03 was a botched and hasty attempt to make BtS Vista compliant, it bypassed any kind of game testing.

3.13 could conceivably have undergone some testing, but only very minimal - certainly no efforts to play the game under various conditions. The culture display bug, while cosmetic, wouldn't have got through observant testing. The errors in AI resource trading also make it implausible any extensive in game testing was carried out. The dead colonies bug could get through poor testing where minimal variation on the conditions were carried out.

Vaguely interesting as it is Methos, the information you post merely increases my bafflement at the buggy condition of the last two patches. How could serious bugs that were blatant in 10 minutes, and (largely) fixable in a matter of hours by Bhruic, get through a multi stage QA process such as you described which lasted over a month?

Good article, Methos.

Methos said:

Development: Once the decision is made to make a patch, the normal development process starts. A list of issues to fix is compiled

Click to expand...

Stop here.... isn't this normally its own step?

And, it could potentially take quite a bit of time. If the Publisher approves unlimited programming resources, sure, this step is pretty minimalized. However, given the reality of limited resources, the "list of issues" needs to have some effort at a ballpark time/effort required, and the list needs to be ranked according to desirability of each fix. Cross-referencing these two matrixes is necessary in order to make some kind of an intelligent decision on what is to get fixed, and what simply is outside the scope of the patch.

Wodan

Wodan said:

Good article, Methos.


Stop here.... isn't this normally its own step?

And, it could potentially take quite a bit of time. If the Publisher approves unlimited programming resources, sure, this step is pretty minimalized. However, given the reality of limited resources, the "list of issues" needs to have some effort at a ballpark time/effort required, and the list needs to be ranked according to desirability of each fix. Cross-referencing these two matrixes is necessary in order to make some kind of an intelligent decision on what is to get fixed, and what simply is outside the scope of the patch.

Wodan

Click to expand...

There is a full Software Development Lifecycle, and yes, that is part of it. There is a short version of the SDLC: Indentification, Analysis, Coding, Testing. (That's just from memory - I'll have to look at one of my old textbooks for the short version.)


1 - Identification. In the case of a patch, it would be the need to patch the game. (This is generally the easiest step, but it's not the programmers who decide it. It's usually the producer, or even the publisher who says there needs to be a patch or expansion in XYZ amount of time).

2 - Feasibility. This is both in cost, and technical issues. Is there a glitch that's really difficult to fix? If so, will it run into other projects? Maybe Civ4 only has another year in it's lifecycle (lifecycle being patches at this stage) until they move on to the next game. If it's say, 3 months, and they've got a glitch that may take 4 months, then it probably won't get fixed, unless they can get an extension. Remember, the publisher wants to make money, too.

3 - Analysis. When making a piece of software, this is entity/relationship diagrams, state transition diagrams, and so on. For a patch, it might involve looking over code and seeing where it broke.

4 - Design. Design is "almost programming", usually prototypes and looking at your "wishlist" (musts/wants). If the bug is in their "must fix" list, it'll get fixed. If it's in their "wants list", then they may or may not fix it depending on if they have time for it, or finished their "must fix" list. This "wishlist" is what you hear about often in the developer chats. Sometimes a work-around might be drawn up, too.

5 - Coding. This is the fun part! Now, good quality assurance dictates that any change in code must be well documented, especially when it comes to bug fixes.

6 - Testing. This is done during, and after coding. In fact, steps 3 through 6 might be repeated many times during the project.

7 - Implementation/Maintainence. This is where the game is played (installed on the player's machine). On "real world" projects, this step is a bit more complicated, involving several types of implementation, training of users. Of course, one of the most important part of this phase is documentation, at least on the end user end. Maintainence is more patches, which loops back as far as step one.

Here's an example of what something might look like:

1. Indentification - Many users are flooding this fan site called CFC about an issue with video cards causing glitchy graphics.

2. Feasibility - Yes, this might be time consuming, but it's also a show-stopper, and the publisher really wants those annoying calls to stop.

3. Analysis - Which video cards are causing this problem to happen? Is there anything in the software engine that's causing it, or can fix it? Is it related to video card drivers? Experiment! What would be an acceptable percentage of video cards to be fixed for the patch?

4. Design - What's the best way to fix this problem? What things need to be coded/re-coded?

5. Coding - The programmers fix 90% of cards, which is "acceptable" for a major patch. (ok, maybe 10% are oddball video cards that would be too hard to code for, or might break existing cards that were just fixed).

6. Testing - Now, does it really work as advertised? Test it on several major cards that were having the problem.

7. Implementation/Maintainence - Release the patch (assumes all testing and coding are complete).

Firaxis isn't Blizzard Entertainment. Firaxis has more than one game to take care of.

One of the biggest pitfalls with changing existing code is testing unrelated things (ie. regression testing) that might have been unintentionally affected. It is very easy to focus on just what was changed and miss something unrelated. Full regression testing is very time consuming and costly.

Just because the deadline approaches doesn't mean the bugs stop being identified. Some are old bugs just identified and some are new ones the testers have found. The deadline will only allow so much to be done. Do you extend the deadline (again)? Their bosses aren't any more understanding than yours are about things not being done on time.

Eventually, someone is going to have to put their foot down and say "no more". Otherwise it will never get finished. Only what about the remaining bugs? Some get prioritized out. Some get quick fixes that there is no time to test perfectly.

Guess what, the QA folks have deadlines too! Only as time runs out, the testing is always the part the gets cut. It is extremely rare for the bosses/users to say "Hey, are sure you have enough time in the plan for testing?". It much more likely to hear "That's too long, cut some time out." One guess where the time comes out of.

I don't think that this is something limited to software development either. Anyone who has ever had a job of any kind should recognize the general pattern here. The details may be different but nobody has the enough time to do everything that needs to be done and few of us can do it perfect the first time, every time.

I seem to have drifted a little there. Anyway, there's my essay on "Why bugs happen".

Denniz said:

One of the biggest pitfalls with changing existing code is testing unrelated things (ie. regression testing) that might have been unintentionally affected.

Click to expand...

As well as trying to understand what the previous programmer did (if it was there code), or even trying to remember what the code did in the first place when you haven't touched it for 2 years.

I really have to agree with theKurgen and MrCynical. Regardless of what you believe the process is, and what various stages are, patch 3.03 cannot have been tested. To quote MrCynical:

If a single person had played even 10 minutes of a game, they'd have spotted the combat animations were broken. I do not believe three levels of QA missed that.

I worry that this topic is a selfish attempt to show empathy and understanding towards any staff at Firaxis that may read the forums and could possibly give details about the progress of patches, since the last staff member "left" because he asked for opinions and didn't like the result.

tempacc said:

I really have to agree with theKurgen and MrCynical. Regardless of what you believe the process is, and what various stages are, patch 3.03 cannot have been tested. To quote MrCynical:

If a single person had played even 10 minutes of a game, they'd have spotted the combat animations were broken. I do not believe three levels of QA missed that.

I worry that this topic is a selfish attempt to show empathy and understanding towards any staff at Firaxis that may read the forums and could possibly give details about the progress of patches, since the last staff member "left" because he asked for opinions and didn't like the result.

Click to expand...

However, animations may not be a top priority fix. Something that seems like a simple fix might in fact, be complex.


The programming process you see in this thread goes for any software (game or software application), not just one company.

Look at it this way - would you rather have a glitchy graphic, or an exploit that allows someone to cheat against you in multiplayer, or some other bug that potentially crashes the game?

Combat animations do have a workaround, too - turning them off. A bug that does not crash the game, cause an exploit, or in any way is not a showstopper and has a work-around isn't going to get top priority.

Or, to use a real-world example, let's say you have an application that, when the user saves data to the database, they see a screen with a disk icon that spins. If in an update, the save icon stops spinning (which is a bug), but there's some data corruption bug as well, which bug do you think will get fixed first? The data corruption one.

Chieftess said:

A bug that does not crash the game, cause an exploit, or in any way is not a showstopper and has a work-around isn't going to get top priority.

Click to expand...

Patch 3.03 introduced bugs that made the game crash. Even things as simple as clicking a link in the Civilopedia made this happen.

tempacc said:

I worry that this topic is a selfish attempt to show empathy and understanding towards any staff at Firaxis ....

Click to expand...

I noticed that a lot of our members questioned how it was that Solver and Bhruic could create an unofficial patch in mere days and Firaxis couldn't. I was hoping this article would explain why it takes companies (not just Firaxis) so much longer.

Methos said:

I noticed that a lot of our members questioned how it was that Solver and Bhruic could create an unofficial patch in mere days and Firaxis couldn't. I was hoping this article would explain why it takes companies (not just Firaxis) so much longer.

Click to expand...

The reason a patch is even needed is because a huge part of your process (the 3 layers of testing) clearly didn't happen at all. If it did, 3.03 and 3.13 obviously wouldn't have even been released. Why would you release a patch if the testers reported back that it completely screwed parts of the game? Why would it even be possible to release a patch 'built off a bad branch of code' they had lying around? For a modern software company it just seems amazing.

I agree that the testing process should be improved. However, regardless of that, one thing that may have happened is that late fixes caused the critical bugs you guys are mentioning. Read above in Methos' and Chieftess' explanations. You'll see how there is opportunity for fixes which occur after much of the testing has occurred. In an ideal world, this should kick the entire testing process back to step 1. However, limited resources or desire to get the patch out would have almost certainly abbreviated the testing of the late fix (along with the opportunity to discover unintended side-effects of the fix).

Wodan

It really is very simple.

Either you throw enough money at the bugs for them to go away.

Or you don't.

It is very clear that Firaxis has decided they've got all the money out of this particular cash cow, and that there's no reason to stay behind just to satisfy the customer.

Let's all buy Civ 5 and make that decision right...

Methos said:

I noticed that a lot of our members questioned how it was that Solver and Bhruic could create an unofficial patch in mere days and Firaxis couldn't. I was hoping this article would explain why it takes companies (not just Firaxis) so much longer.

Click to expand...

With Alexman continue to visit (yet not post) here do you think that's a good sign for another BTS patch? No doubt there are those who work for Firaxis like Alexman who would like to fix and improve the game if given the chance.

Hey, blame me while you're at it. I'm visiting but not posting

Smidlee said:

With Alexman continue to visit (yet not post) here do you think that's a good sign for another BTS patch?

Click to expand...

Maybe Alex is just like the rest of us, perusing the forums looking for pointers. In regards to the patch, who knows. I hope so.

Methos said:

Maybe Alex is just like the rest of us, perusing the forums looking for pointers. In regards to the patch, who knows. I hope so.

Click to expand...

I think that alexman ( and the rest of Civ IV ) are working in the real patch, for two reasons:
- IMHO the 3.13 patch shows that that the grand plan was to make a bigger patch and that the work was halted prematurely ( some things that were changed were not related with the game fixes ... but ofcourse that is only my opinion, and I only know the programming I learned in Chemistry degree.... so take it with a big grain of salt )
- The only time I saw alexman in the forums after that little 3.13 changelog incident he was browsing the BtS bug subforum....

About the OP:

Nice post, and IMHO a necessary one.... but I have to agree that it looks that the 3.13 patch didn't had a proper QA ( IMHO , the fixes were tested seperately properly , but the ingame testing was a little rushed ( maybe a quick game or two.... ) ). Hopefully a good patch will be released in a proper time....