Pop quiz:

What’s the most common question from your boss/manager?

….

I know you have your own answer but I’m pretty sure these are the most common ones:

How much you tested it?

Did you test enough?

I don’t know you or how many years in testing but for me those questions are really challenging.

You can write quality test cases. You can find critical bugs. You can execute test cases pretty fast. But when it comes to measuring your test coverage… it’s a headache.

You don’t want answers like “Yeah, I tested a lot”, “Well, I think I tested it enough”.

You also don’t want to say “I don’t know” either.

As a tester, you not only write test cases, find defects but also provide the information to stakeholders/your boss and in this case you need to be very clear about the your test coverage so that your boss can make decisions accordingly.

In today’s post, I’d like to share with you some ideas to help you measure your test coverage properly.

What is test coverage?

Simply put:

“Test Coverage is a measure, depicting how many possible code execution scenarios, and conditions have been translated into your executed test cases.”

In other words, it’s a metric to tell how much you test something.

Let’s take an example of a new car with a one year warranty and you need to test it.

Can you tell how many years will this car last? Even if you find out the duration, can you be 100% sure about its accuracy?

No.

It is because you don’t have information of the future drivers, weather conditions, maintenance and accidents etc. So you cannot design the test cases of the unknown possibilities.

Now that we know that our test cases will never be mature enough to become 100% sure about our product, so does this mean that we should not test at all?

Of course, not. So how will we determine how much of the software are we testing and how can we know how much testing is already done and how much is remaining?

Here’s how:

We will quantify our test coverage by using the given information knowledge and experience. We will create an estimated of 100% coverage as a benchmark, which may or may not be a 100% in actual context. But we will cover as much as we can. There are many ways to do the test coverage which can be used separately or combined depending on the situation.

Here are some of them:

Code coverage:

This coverage will result in a white box testing as we will try to create all the conditions by fulfilling which, we can execute either every path, branch, line of the code. With larger application this cannot be feasible enough strategy for a good coverage which does not miss anything.

Requirement Document:

This one is the most easiest and fastest but this might not be enough most of the times because if the document has something missing you will not be able to cover that part of the system. Secondly, it is very rare in the era of agility, to create a complete and concrete requirement specification because it changes so fast that you can’t risk all your testing efforts on one source.

Related read: How to design effective test case based on requirement

Use-Case Based:

Here we study the Use-Case document to see what features will be the part of the system and what will not be, and try to clear the boundaries of the system as much as possible and then create test-case based on the use-case steps , preconditions and post conditions. Then we can step out of the comfort zone of that system and check how it behaves, which will lead us to create negative test-cases.

All of these approaches can be used independently or combined according to the software system.

Side-note: Please just use those as reference for your test coverage report. You basically can base on what make sense to your project. For example, you can report the test coverage based on coverage of browsers, OS, platforms, devices, etc.

Once we have decided our approach we can now move forward to create a sort of scale through which we are going to measure our test coverage later because you know, the point of this whole hustle is to derive a value against our testing efforts. Let’s suppose that we have chosen code coverage based test coverage and we have 15 set of functions in our software system out which 12 have been tested to date, then our achieved test coverage can be calculated as :

Achieved Test Coverage = (Number of functions tested / Total functions)*100 = 80%

Similarly for a Requirement based coverage and use case based coverage, if we are using a hybrid we can aggregate the score, or give weightage to each kind of approach and then evaluate that into a single score.

What test coverage is used for?

As I promised earlier, I am going to state how this concept will be helpful to you and how can we use it in a better way.

  • Team’s Self Evaluation Baseline: It helps us to evaluate our own performance and effectiveness. We can also keep track of our own activities and see how much testing have we done and how much is left.
  • Quantification of Performed Testing: Our stakeholders will never be convinced about the quality progress until we provide them some numbers and stats. We can achieve it through this.
  • Testing’s Exit Criteria: Although 100% test coverage never means 100% tested but it helps us in defining some threshold after which we can stop a testing iteration. For example, in a regression testing, we can stop executing tests after 100% test coverage.
  • Smooth Testing: Our quality team will see the same picture of the current state of the software and its quality. That’s why; testing will be more organized and less obstructive.
  • Ease in parallel testing and work division: As we have divided our tests strategically, we can easily divide the workload into our team members and they can work parallel.
  • More grip: Defining the test coverage requires the whole QA team to learn any and everything about the system and hence this gives them more control over the complete testing process and strong base. 

Final thought

Remember that there is always a room for improvement. If you want to do better, then it is really important for you to understand the scope of the system before creating the 100% test coverage criteria. You can’t cover a system if you are unaware of its boundaries. Try to use hybrid approaches to be more thorough in your testing.

So implement and practice this. Explore more coverage strategies and impress your team. And hey! Don’t forget to brag about it in the comments below *wink*