As a software test engineer, you may probably hear about smoke testing, sanity testing, retesting and regression testing. Very likely, many of them are testing activities that you often do on a daily basis. However, for junior testers (and even for experienced testers) distinguishing these terms can be a real challenge.
In many cases, you can still do the job right even if you can’t tell the difference. However, as a professional tester, knowing what you do and why you do things is always the right thing to do.
In this post today, I would like to help you clarify the difference between smoke testing, sanity testing, retesting and regression testing to clear all the confusion you may have…and move forward.
Alright, let’s do it.
I’ll go into details right below, but before that, here’s a quick definition of each terms:
- Smoke Testing: We do smoke test whenever we receive a new build that is relative unstable to ensure that the critical functionalities of the AUT working fine. The idea of this test is to find critical problems as soon as possible and reject the build so that we don’t waste time to test further (the broken build).
- Sanity Testing: We do sanity test whenever we receive a new build that is relative stable to conduct to identify the dependent missing functionalities. In other words, you will validate the major functionalities of the system in a more detailed way.
Both sanity tests and smoke tests are ways to avoid wasting time and effort by quickly determining whether an application is too flawed to merit any rigorous testing.
- Retesting: We do retesting for a feature/function which had defects (bugs) & those bugs have been fixed recently.
- Regression Testing: We do regression testing for AUT when there are changes including new features / bug fixed to make sure no impact to the current existing functionalities.
To have better understanding, let me introduce you the table of comparison as below:
| Smoke Testing | Sanity Testing | Regression Testing | Re-testing |
|---|---|---|---|
| Smoke testing is executed to determine if critical functionalities of AUT are working fine | Sanity testing is executed to determine if the section of AUT is still working as design after some minor changes or bug fixes | Regression testing is executed to confirm whether a recent program or code change has not adversely affected existing features | Retesting is executed to confirm the test cases that failed in the final execution are passing after the defects are fixed |
| The purpose of smoke testing is to verify the “stability” of the system in order to proceed with more rigorous testing | The purpose of sanity testing is to verify the “rationality” of the system in order to proceed with more rigorous testing | The purpose of regression testing is that new changes should not have any side effects to existing functionalities | Re-testing is done on the basis of the defect fixes |
| Defect verification is not the part of smoke testing | Defect verification is not the part of sanity testing | Defect verification is not the part of regression testing | Defect verification is the part of re-testing |
| Smoke testing is executed before regression test | Sanity testing is executed before regression testing, after smoke testing | Based on the project and availability of resources, regression testing can be carried out parallel with Re-testing | – Re-testing is done before we start sanity testing – Priority of re-testing is also higher than regression testing, so it is carried out before regression testing |
| Smoke test can be excuted manually or automatically | Sanity test often is executed manually | You can do automation for regression testing, manual testing could be expensive and time consuming | You cannot automate the test cases for re-testing |
| Smoke testing is subset of Regression testing | Sanity testing is subset of Acceptance testing | Regression testing is only done when there is any modification or changes become mandatory in existing project | Re-testing executes a defect with same data and the same environment with different inputs with new build |
| Test cases of smoke test are part of regression testing, only cover critical functionalities | Sanity Test can be performed without test cases but domain knowledge is required | Test cases for regression testing can be obtained from the functional specification, user tutorials and manuals, and defect reports in regards to corrected problems | Test cases for re-testing can be re-used from previous executed test ca |
Wrap-up
Hopefully after reading this article, you will have the basic knowledge of smoke testing, sanity testing, retesting and regression testing. Like I said at the beginning, it’s ok if you can’t tell the difference among those terms. Terminologies are confusing all the time. However, by knowing what you do and call it by its right name is how a professional tester should do.
Like it? Don’t like it? Share your comments below.
Hi aLong Manh,
Thank you for sharing your knowledge about testing. But after read your article, something is seemly that not clear with me. So could you please give me example about type of this test in basic function? I think it will clearer than just only theory.
Hope that will have your reply soon 🙂
Hi Tai,
Firstly, I want to thank you for your interest in this topic!
To answer your question, I will give some example to make it easier to understand:
– Retesting: also is called bug verification. That means when a bug is fixed, we testers have to re-verify the bug to see if it is actually fix or not, by following the old steps, or the new steps that may reproduce the bug.
Example: you found BUG-001. Now BUG-001 is fixed, you have to check it. This action is called retesting or bug verification.
– Regression test: running a set of test cases that covers all functionalities of the AUT. After so many changes are made into the AUT, we may have to run the test cases to ensure the whole system is ok. At that time, we have to run the set of test cases for some or all functionalities.
Example: Yahoo! has many functions that are Login, Logout, Register New Account, Create New Email, Send Emails, Delete Emails, etc. The test cases for these functions are around 4000. Now we have to run these test cases to ensure that Yahoo! system is stable. Depending on the situation, we may run test cases for Create, Send and Delete Emails. Or We will have to run all of the 4000 test cases. That’s called Regression Test
– Smoke test: running a set of test cases of critical functionalities of AUT, which ensures that the major functionalities of the application are working fine.
Example: Yahoo! has some critical functions that are Login, Logout, Register New Account, Create New Email, Send Emails, Delete Emails. And the test cases for these function are just a few for each functions.
– Sanity test: is performed when development team needs to know quick state of the product after they have done changes in the code or there is some controlled code change in a feature to fix any critical issue, and stringent release time-frame does not allow complete regression testing.
Example: in a project there are five modules like login page, home page, user detail page, new user creation, and task creation etc. So we have the bug in login page like on login page username field accepts the less than six alpha-numeric characters which are against the requirements as in requirements it is specified that username should not be below than six characters but as username accepts the less than six characters it is the bug.
So now the bug is reported by the testing team to the developer team to fix it. When the developing team fixes the bug and passed it to testing team than the testing team checks the other modules of the application means checks that fix bug does not affect the functionality of the other modules but keep one point always in mind that testing team only checks the extreme functionality of the modules, do not go deep to test the details because of the short time so this is the sanity testing.
Hope you find my answer useful!
Best regards,
LM
Thanks dude,Great article for testers .
Hi. That is great article, thank you!
p.s. i think there is mistake in sentence in the table of comparison “The purpose of smoke testing is to verify the “rationality” of the system in order to proceed with more rigorous testing”. It should say “The purpose of sanity testing…”
Hi,
On the “re-testing” column you wrote at bottom most row that “Test cases for re-testing cannot be obtained before start testing” I think this is wrong since if you are re testing a bug fix you can use test cases that were intended to test that module of the AUT, moreover… you can prepare edge test cases so to know whether the module was fixed properly and will not dive any problems in the near future. That is my humble opinion.
Hi Yulia, Yonata,
You’re correct. I have made correction accordingly.
Thanks for pointing it out.
Cheers
Hi lang manh,
your explanation is very much helpfl but can you also explain the order of performing for these 4 test . for example – i have performed my retest – then after wat test types should i follow to successully complete my testing .
hi,
sanity testing is a subset of regression testing
smoke testing is a subset of acceptance testing