Regression Testing for Beginners: What, Why, When, How?

Regression testing is one of the most interesting topics to talk about. People talk about it all the time (but strangely, I’ve never talked about it here on AskTester :D). Some love it, some hate it but people seem to do it in any testing projects.

The reason is simple: regression testing too important to be missed out.

Even though regression testing is common and people do it a lot, people do not know how to do it right, or worse they have never heard about it.

Anyway, in this post today, I’ll share with you everything you need to know about regression testing so that you can get started with it or improve your regression testing.

Let’s get started!

What is regression testing?

There are several definitions, but here’s the one I like:

Regression testing is the process of testing changes to computer programs to make sure that the older programming still works with the new changes.

(Source: http://searchsoftwarequality.techtarget.com/definition/regression-testing)

and here’s another definition, from my reliable friend Wiki:

Regression testing is a type of software testing which verifies that software which was previously developed and tested still performs the same way after it was changed or interfaced with other software.

Simply put, a regression test is a test to make sure that in your software what worked in the past still works at the moment.

But why we need regression testing?

Software these days is too big and complicated to make it right for the first time. It’s not practical for developers to build one big build including all the features and release it to customers. Instead, developers build small builds and each builds just covers a certain feature of the software. As you can imagine, the later build will also be “bigger” than previous builds because features in the previous builds will also be brought into the later builds.

They keep doing that until having a final build with all the features completes and then release to the customer…like this:

Most of the time, developers often have to deal with “builds” (or “releases” or “sprints” depends on what shop you’re in). Developers will basically do two things in a build:

#1: Write code to implement new features for that build

#2: Make sure that writing code for new features does not break previously working features

While most of the developers can handle the task #1 of writing new code to implement a feature nicely and easily, developers often overlook the task #2 of making sure new code not breaking old code.

Things are perfect until developers start making some code changes….

Here’s how developers reason:

“Common man, I just added/removed one line of code. Things will be alright”

“I only made a minor change here..and there”

Not to mention, some developers overrate themselves.

Don’t get me wrong, I’m not saying that developers are telling lies (even though some do), in many cases, developers are not fully aware of the impact of their code to the entire system. It’s not really because of developers’ competency, but more often it’s because the system is too big and complicated to draw a clear line of what’s impacted, what’s not.

As the saying goes “what can go wrong will go wrong”, so in order to avoid unwanted impact and side-effects of code changes, every build needs to be tested to verify new features works and old features work as well…and we call that test “regression test“.

When to do regression testing?

A regression test is often performed whenever there’s a change in the system, this includes:

• Whenever a new feature is implemented
• Whenever an old feature is modified/changed
• Whenever a bug is fixed in the system
• Whenever environment is changed
• Any other changes you are aware of

However, in practice, testers do not have to run regression test for any changes. A team needs to evaluate the risk and see what changes worth the effort of regression testing.

But regression test is not a silver bullet. There are pros and cons of regression testing too:

Pros of regression testing:

• Help detect unexpected defects caused due to changes newly added to the system.
• Reduce the number of defects escaped to end-user
• More importantly, help release products with confidence.

Cons of regression testing:

While regression test is crucial and considered as a must-have in any software testing life cycle, there are cons as well. There are different reasons, but it boils down to this:

COST. 

I’ll explain:

• This is straight-forward, re-running your whole test suite of test cases build after build is time, resource consuming testing activity. This is especially true if you have thousands of test cases need to be re-run manually or if your team can make build frequently. Even if your regression test is performed by running automated tests, it still costs your team time, tools and skilled resource to automate the test. Of course, you can have good ROI with automation, but it’s still a cost.
• Maintaining regression test suite which is getting bigger and bigger over time. Every time there’s a new feature is added or modified, test cases will be added or updated accordingly. In the long run, your regression test suite will become bigger which keeping them updated is challenging too
• Traceability (e.g: know what to test, what not), since the regression test suite, if you do not do a good job of tracking/tracing, you will easy miss test cases or cannot track your coverage.

Besides the cost, there’s another negative side of regression testing. Even though I’m a professional tester and I have performed a lot of regression tests, I have to confess this: Regression testing is one of the least interesting testing activities out there. In other words, doing regression testing build after build is bored to death. I have never seen a tester jumping for joy at regression testing in my career so far.

Sorry guys, but that’s the brutal fact.

Running the same hundred of test cases build after build where most of the time the test case will pass the test case is not fun at all. As a result, you get careless…and this is dangerous. Once you are indifferent and not paying 100% attention to your work, it’s easy for you to miss a step or worse you may miss bugs…that’s bad.

Solutions? Automate your regression test. Actually, regression is a perfect type of test to be automated. Here’s why:

• The tests are repetitive
• The tests mostly are checks
• The tests are stable

When it comes to regression testing, if implemented properly, the machine tends to do a better job (and faster) than the human does.

Think about the scenario where you have all your regression test cases automated, so whenever you have a new build,  you kick off your automation test and you then can get the test result in next couple of hours instead of 1-2 days if the tests are executed by testers. Also, if your regression test suite is big, you can have it executed overnight while you sleep and you can get the result next morning.

How cool is that? Needless to say, machine is very precise in the way it runs the test and the best part: it never complains that regression test is boring 😀

By having your regression test suite automated, you are now having more time for other testing activities like exploratory testing, test reporting, defect reporting, test planning, etc

Of course, test automation has its own problem and it deserves a separate post for it, but trust me, if you can have your regression test automated, you are maximizing the benefit of it.

Pro tips:

• Frequently update your regression test suite. Whenever you have a new test case or a change request, you need to update your test cases accordingly.
• Add fixed/closed bugs as part of regression test suite. Review your bug database and add them into your regression test suite if there are no test cases covered that scenario of the bug
• Categorize and prioritize your test cases so that you know where important test cases of what components need to be run first. You can categorize them into Critical, High, Low or number them 1, 2, 3 or whatever you like.

How to do regression testing?

While every project is different, here’s the basic regression testing process:

Step #1: Make a regression test suite

Disregard of the projects or the tools you are having, having a regression test suite is the prerequisite.

What is regression test suite? Regression test suite is a collection of test cases you have already written them for your products. If you don’t have time for detailed test cases, your regression test suite can be a check-list of the test ideas you want to test. Basically, regression test suite will consist of all the tests you plan to re-run to serve one purpose: make sure what worked still works.

Once you have your regression test suite, your next steps are:

• Categorize your test cases into components or features of your system (Ex: Login, Create Order, Delete Order, Reporting, etc)
• Prioritize your test cases based on the importance of the test cases (Ex: Critical, High, Low, etc). The highest priority test case will be the test cases cover the most important features of the system and need to be executed first.

By categorizing and prioritizing the test cases in your regression test suite, you can always group your test case and know what test cases need to be run first,  what tests can be run later when you don’t have enough time or it doesn’t make sense for full regression test.

Step #2: Run them

Now you already had your regression test suite in place, so whenever you have a new build, you then kick off your regression test, right?

Wait! You will run them but before doing that, ask yourself the following questions:

• Is your build good enough for further testing?

Why do we need to know that? Here’s the thing:

What would happen if you are planning your regression test suite and assign tasks to team members to run the regression test, but at the end of the day, you realize that one of the most important features of the system is entirely broken that it doesn’t make sense to run any other tests. Now you and your team wasted time to test the failed build.

So, before running regression test suite, make sure that the build is good enough for further testing. You can achieve this by performing a build acceptance test, smoke test or sanity test. A smoke test is often a small subset of your regression test suite consisting of critical scenarios of the system under test. This test is often a short one (often takes 1-2 hours to perform). If any of the test case in the smoke test is failed, you can claim the build is not good enough for further testing and the notify developers/managers accordingly. This can save you and your team a lot of time.

• Is full regression test required?

While full regression is a nice-to-have, in some cases it’s not necessary to run all your test cases if you know the changes are very isolated and you know exactly what features/component will be affected and what not. By knowing this, you can filter your test cases and select the right test cases to be regressed. Similarly, in some context where you don’t have a lot of time for full regression, you now can know what test cases you need to run first and what test cases you can leave out.

Once you have a list of test cases need to be regressed, you now can start the test. If you already had your tests automated, you now can kick off the test and go and make some coffee 😀

Pro tips:

If you are using a spreadsheet to track your regression test result, make sure you have the following columns in your regression test suite:

• Test result: Passed/Failed/Inconclusive
• Note: Where you can take note or the ID/description of the defect if the test is failed
• Build record: Where you can take note what build version the test is executed against

Step #3: Collect test result and report

Once you finish your regression test, the final step is to review the test result before sending out the report to managers. Basically, you need to review the followings:

• Make sure “Failed” test cases linked with bug ID or at least having some notes describing while the test is failed
• Make sure “Not run” or “Inconclusive” test results having notes described the reason accordingly
• Keep eyes on unusual tests with all “Passed” or all “Failed”

Some FAQs about regression test:

What’s the objective of regression test?

Like I mentioned above, regression testing is an activity of re-running test cases to make sure new changes do not break working features. With that said, the objective of regression is not to explore and detect new bugs. The objective of regression testing is to confirm what used to work still works now. In other words, the regression test is expected to have passed results.

When performing regression, should I follow the steps strictly or deviate from the steps?

There’s no set-in-stone rule about how to run regression test suite. Theoretically, you need to follow the regression test strictly steps by steps when running them. However, following steps-by-steps is boring and not so effective in finding new defects where most of the time the tests are passed. So, the solution is to follow the steps but give you time to deviate from the steps to explore a bit more of the feature.

Can regression testing be automated?

Of course, yes. In many cases, it’s recommended to automate your regression test to make your regression test efficient and to give you more time to perform further exploratory testing and detect problems.

How much is regression testing enough?

I don’t know you but it’s impossible to always achieve 100% your regression test unless your regression test is small or run by machines. Sometimes you need to run all your test cases, sometimes you just need to run *important test cases only*. This is especially true if you have limited resource to perform the whole regression test. So, be smart and select your test cases wisely.

My team does not have time for regression test, what should I do?

This is not something uncommon in the testing world. In some companies, test team is often put at the end of software project life cycle and as a result, testing is often put in a state of not having enough time to run the test cases. There are two choices for you:

1. Change your project process
2. Live with what you have

While changing your company process or asking for more time is the best way to do, but most of the time, testers are often not in the position to make such changes. I’m not saying that you cannot change, but it’s tough and requires a lot of effort and conditions to make things happened. What you are ended up living with what you have in hands.

So, what you do if you don’t have much time left to test? Here’s suggestion:

• Select the most important tests to regress. Taking advantage of the category and priority of your test cases, you should be able to sort out the most critical test cases to run the tests.
• Be honest and keep managers informed. This makes test reporting so important. Tell managers what tests you have run, what not and its risks. Don’t try to impress the managers here, just be realistic.
• Automate your regression test. If you have skilled resource, try to automate your regression test. While recommending you to automate your test while you don’t have a lot of time at hands sounds like counter-intuitive advice, but trust me, if you are having a big regression test (and keep increasing) and most of them are stable tests, regression test can do help you a lot of in the long run.

Final thoughts

Regression testing is one of the most powerful testing activities…and just like anything that is powerful, it can help you strengthen your testing and make better software but it can also be very costly. So, when you do it, it’s recommended to focus on the value (e.g. what you will benefit) from that activity. Also, if possible, consider to automate it to get the most out of it.

Don’t do regression testing just for its own sake, do it for reasons.