Beta Testing
by George

In General

Beta testing is perhaps the most important phase of any video game's development. Whether you've spent a month or a few years on your adventure, if you don't have it beta tested properly, all that work could be ruined by one simple little gameplay killer. If you understand the importance of beta testing, then read this tutorial carefully and take it to heart. If you don't understand the importance of beta testing, then read this tutorial carefully and take it to heart. I cannot stress enough how important the beta testing phase is.

When should beta testing begin? The short answer is after alpha testing is finished. When is alpha testing finished? That's easy! That's when you think your game is ready for release. Your game is the best you can get it and you're sure there is no more you can do and you feel happy about releasing it. Now it's time for beta testing. This seems to be a difficult concept for new Level Builders (and a few seasoned Level Builders) to get their heads around. Trust me, your game does not go for beta testing until you are convinced it's finished. If it's not finished, you're still in the alpha testing stages and if you employ your beta testers now they will be wasting their time and your game will not be beta tested properly.

When a game moves from alpha testing to beta testing, the only changes you should make would be to fix bugs, gameplay errors and textural errors. You should never build new areas or new gameplay or new puzzles during or after beta testing. Again I stress, do not begin beta testing until your game is FINISHED. Beta testing is purely to iron out problems in finished games. If you ignore this advice the chances are you will release a buggy game. Beta testers take their job seriously and if games are released with bugs it reflects on them and the job they did. If those bugs are down to the Level Builder changing things during or after beta testing and the game is released with bugs you will never get beta testers to help you again. Beta testers will invest days and even weeks of their personal time to help you with your game. Respect that and don't let them down.

Alpha Testing

Don't cut corners and skip this step. Yes, I know, you're excited about your game and you want to show it off and that's marvellous and wonderful and admirable and quite understandable, but your beta testers are not there to do your job for you. If you upload an appallingly buggy game that hasn't been properly alpha tested, don't be surprised if your beta testers don't do a good job and don't volunteer later when you actually need beta testers. Alpha testing is your job, not your beta testers, and it's extremely important. I consider my alpha testing to be complete when I have a package I think is ready for release. In other words, I can find nothing to change or fix and in my heart I know my game is the best I can make it and I feel it's ready for release. Now I begin beta testing.

During alpha testing, it is easy to move Lara around and forget to put her back, place temporary triggers to open a door and then forget to remove them and a few other little things, so after I've packaged up my game for beta testing guess what? Yes, I then test the beta package from start to finish using only the files I will be uploading for my beta testers. This is my final Alpha test. If this goes well and all the level jumps work properly and I successfully hit a finish trigger, only then is the game uploaded and the download link sent out to the beta testers.

The Beta Testers

It's important to have experienced beta testers on your team. Oh sure, everyone who plays custom levels considers themselves an expert on beta testing - one only has to read the reviews to realise that - but the truth is most players haven't a clue what's involved. Even with the best intentions, an inexperienced beta testing team isn't going to do the job for you. Like any field in life, beta testing is a skill which must be learned. Beta testers like Raymond, Dutchy, and Gerty, probably the most experienced custom level beta testers on the planet, didn't learn their profession overnight. It takes a lot of experience and hard work to be a good beta tester.

Your beta testing team must be motivated and nothing demotivates beta testers more than a lack of communication or lack of respect and appreciation for the work they're doing. Communication is very important. Your team must be kept updated by email and preferably have a private lounge at a forum where they can openly discuss the levels being beta tested and where there can be free communication with the level builder. Beta testing is a commitment and carries responsibility. Beta testing isn't a game so take it seriously.

When you first contact your beta testers with details of when you will be starting beta testing, ensure you can keep your word. Don't tell them you'll be starting beta testing at the weekend and then make them wait two weeks because your level isn't finished. Your beta testers have lives, they have families, they have jobs and they have other commitments. Chances are they are making sacrifices elsewhere in their lives to schedule you in. Don't expect them to come back if you let them down because you can't keep your word. Give them plenty of notice. They may have to plan carefully to schedule you into their diaries. They may have to take time off other things or stay home on a particular day, so give them plenty of warning and make sure you don't let them down.

Beta Testing

Messing with your trle folder when you are about to start beta testing is a stupid idea. I once knew a level builder who moved his game files out of his trle folder and started building the WAD and texture set for his next game just as beta testing was beginning. In other words, he was just going through the motions to bolster his ego and thought his level was so good the beta testers would find nothing. He had already decided to release his alpha tested version regardless of whatever the beta testers found. That is the kind of behaviour and lack of respect I would expect from children.

Your job as a level builder isn't done and won't be done until after beta testing and your game is released. Beta testing is perhaps the most labour intensive part of the whole level building job so don't think you can relax now, or worse, think you can install a fresh Level Editor so you can begin work on a new project. If the beta testers find out you think your masterpiece is flawless and you're installing a fresh Level Editor because you don't expect to make any changes during or after beta testing and you're simply stroking your ego to look cool, your beta testers will dump you and rightly so. Make sure you value them and respect their work.

A beta tester's role is primarily to find problems, therefore any suggestions regarding atmosphere, music, texturing or any other non problematic items are only brought to your attention for your consideration.

For a beta test to be successful, your testers should have never played your level before. As new players, they will go everywhere and search everywhere and get stuck everywhere. This is good! And the results can be surprising! However, once your testers have played your level, their effectiveness as beta testers is greatly diminished because they now know which routes to take and where all the pickups and baddies are. This is another good reason why you should never have your level tested by your beta testers before it is finished.

When folks get stuck while beta testing, reply to them via PM or email. Don't put the solution up on forum boards where everyone can read it. I don't even recommend using spoilers because beta testers can be quite nosy! This is important when testing gameplay. If, for example, you have 4 beta testers and they all get stuck in the same spot, you now know you have a serious gameplay problem. If only one of them gets stuck and three of them pass that area easily, you know you don't have a gameplay problem. However, if when the first tester gets stuck you post the solution on the board, all the other beta testers will read it and, therefore, won't get stuck because they know the solution. Then you will likely release your game with a serious gameplay problem. Yes, situations where all your players get stuck in the same place is not 'challenging' or 'good level building design' it is a serious gameplay problem and either your game was not beta tested properly, you ignored the advice of your beta testers, or you don't understand what good gameplay is. I can't stress this point enough. If all your beta testers get stuck at the same point, so will everyone else who plays your game. This is a startling indicator that your gameplay needs fixing. Don't leave it as it is, change it. Sure, you didn't get stuck there, but you built the level and you knew the route and what to do before you even started. Learn to 'listen' to the gameplay as your beta testers are playing through for the first time. The idea is to iron out frustrations and keep your gameplay flowing and your game entertaining. If beta testers start showing frustration, tune in, especially if they all show frustration at the same place.

When changes are made to levels during or after beta testing, the game will require another full testing. Even if a game has been beta tested half a dozen times, if you change it again, it will require another testing. Once beta testing is over and your level has been retested prior to release, do not make any further changes whatsoever. I learned this lesson the hard way. The night before Imprisoned Spirits 3 was released I turned a set of double doors round as a last minute adjustment. That's all I did, turn a set of double doors around and then I released the game without testing it again. Guess what? Folks could get in behind those double doors and get stuck because I'd turned them around. I had to take the game offline until I found out what the problem was, get that level retested and then upload it again. Trust me, it is better to release a game you know works rather than risk tampering with it after beta testing.

Believe me, I know what I'm talking about. Just prior to releasing The Jurassic Stone, which was all packaged up and ready to upload, I found myself with some time as I'd lost trle.net's FTP address and was waiting for an email and so I decided to have another run through the game. Well, I found a place where Lara did the jig. I'd been over the place a hundred times and missed it and none of the beta testers had found it either. It was unlikely anyone playing the game would find it, but I figured I'd change it anyway. So I flattened one broken floor tile, and touched up the texture, saved it, outputted the WAD, converted the level and put the new TR4 file into the upload package. Now, I only changed one floor tile right? No need to retest the whole level just for that eh? It was the second level too so I would have to replay the entire game. Not necessary for one tile and one texture, is it? I took a deep breath and told myself to test it again. If you change something, you must test the level. But I felt lazy. 'Oh, it's only one silly floor tile! How is that going to affect the game?' But I retested it anyway. When I got to the second level where I'd made the change, I suddenly discovered that a flame emitter which was triggered by a flyby camera didn't work and so a door didn't open. I was stunned. What happened?

I opened the editor and checked. Would you believe it? That tile I'd changed had a flyby cam heavy trigger on it and when I'd flattened the tile, the flyby camera way overhead had moved off the tile and so it didn't heavy trigger the flame emitter. That's right, flyby cams can move off floor tiles if you break them or flatten them. I broke out in a cold sweat. What if I'd just uploaded the game without testing it again? After all, it was only one floor tile, right? No one would have been able to finish the game. It would have been a disaster. The moral of the story is, if beta testing is over, do not change anything. At least you know folks will be able to finish the game. Much better they do a jig than find themselves stuck in front of a door that won't open, don't you agree? Still not convinced? Try releasing a level with serious gameplay bugs after a good beta testing and see how much you and your beta testers enjoy the experience.

Do not use DOZY during beta tests, even if your beta testers complain. DOZY is buggy and if beta testers use it they are really of no use to you. How can you test for gameplay if your beta testers are getting stuck and using DOZY to get past areas?

Remember, beta testers are only valuable if they've never played your level before so it's important to ensure your game is the absolute best it can be so as not to waste this opportunity. If beta testers help you with alpha testing or you let them run around in it while you're building, their beta testing results will be horrendously devalued because they know where to go and what to do and it will be very easy for them to run past problems. The first time fresh beta testers play your level they will get everywhere and into places you wouldn't believe and this is where they will prove to be the most valuable. Even Crystal Dynamics got this wrong with Underworld. If you really must mess with a beta tested game, why not find a fresh beta testing team to test it again before release? I guess it all boils down to how much you care about the people who play your games. Do you really want folks to get stuck in your games? If you don't care about the players, why are you building levels?

Making Changes after Beta Testing

It can be a good idea to wait until Beta testing is over before actually going to work on making changes. One thing you should never do is make dozens of changes and upload dozens of new beta packages to be retested as you go along. That will just simply not work and you will drive your beta testers away. They are busy and they will not wish to replay your game a dozen times because you keep uploading new packages. If you are only half way through beta testing and there is loads to fix, it is not okay to upload a new package. It is not sensible nor is it reasonable. Beta testers are busy people with real lives. Finish beta testing and then make the changes before even thinking about uploading it again for the next phase of beta testing. Every time your beta testers replay your level their value as beta testers diminishes greatly. Their first time through is your most valuable testing time. The pre launch test, discussed later in this tutorial, should be only their second time through your adventure.

Another reason for not making any changes until after beta testing is finished is that you may change an area only to have to change it back again as beta testing progresses. Do this too many times and you'll soon lose track of things. As the beta test progresses, write things down. Take comprehensive notes on everything and make a list of things you intend to change. When beta testing is complete, you can then go through your list making changes systematically and tick them off as you go. This keeps things simple and prevents you from making numerous changes early on during beta testing only to have to go back and try to remember what you've done because you need to undo changes or adjust them further. It is easy to lose control if you work this way, particularly if you have a large adventure of many levels.

Yet another reason for not making changes until after beta testing is over is because changes can affect your gameplay without you realising it. If you make dozens of changes here and there in a haphazard fashion, it is easy to make mistakes and forget what you've already changed and gameplay killers can creep into your work unawares. If you have a written list that you're ticking off as you go along and make the changes in a controlled orderly fashion after beta testing, it is much easier to keep control of things.

Pre Release Final Beta Test

We are now at the most crucial part of a Level's development - the second beta test. Release is not far away. Emotions can be extremely intense and conflicting during this phase, particularly for Level Builders giving birth to their first baby. This is also where some beta testers and level builders take ridiculous risks without realising what they are doing.

The intensity of the emotions around this time have to be experienced to be believed. There is nothing like it. For a level builder, especially someone releasing their first or second game, the excitement can be almost unbearable. They will also be exhausted (this doesn't apply of course to rubbish levels thrown together in a few days), and it can be terrifying. Oh yes, releasing your baby to the world can be very scary. What if it's full of bugs? What if thousands of people download it and they all get stuck? What if I've messed up somewhere? What if everyone hates it? These are all very real and conflicting emotions which can be overpowering, which is why good beta testing is so important. Releasing a properly tested level which you KNOW works, is settling and calming, leaving you to enjoy your release without tossing and turning all night worrying about your game having serious gameplay bugs.

Just as with the first beta test, there are a few guidelines if you want to have a successful pre launch beta test. Firstly, beta testing is OVER. Get that clear. This pre launch test is only to test the changes and that the game still works and everyone can successfully hit a finish trigger. Beta testing is over. From this point on it is not advisable to make any changes whatsoever, even if you find little problems like stretched textures. This is not a time to build new rooms, change puzzles or add new objects. The only thing you should change at this stage would be gameplay killers missed during beta testing or introduced while making changes. If you do have to make necessary changes, the whole package will have to be tested again. If you ignore this advice, trust me, you are playing with fire.

Additionally, if things are not going according to plan and your level starts messing up, this is not the time to decide to split the level or remake areas. The game has been beta tested so the problems have only come about due to your changes since beta testing began. Find the cause of the problems and look for the easy fix. It is much safer to simply delete stuff at this stage than to try rebuilding anything. Even if your testers find a shortcut at this stage, don't start remodelling the room. Why not just stick spikes there to cut it off? It is always the easy option you're looking for at this stage.

Above all, enjoy your beta testing and enjoy your level releases.