How Good Is Your Software QA Process?

Some of the things you should consider to improve your software QA processes significantly:

• If you measure your team’s performance by the number of bugs raised by the team, they will raise a bug for every misplaced pixel. Modifying this to use the number of bugs above a specified severity level will get your misplaced pixels classified as urgent.
• Using the number of bugs tested or cleared as a KPI tends to encourage testers to be less thorough and focus on checking that the bug as described is fixed without touching anything that might affect their statistics.
• Any metric you use in your software QA & testing process will be heavily dependent on the context – a tester working with a developer who unit tests thoroughly, makes sure his integration is clean, and checks that he’s doing the right thing, will not find many bugs in that developer’s work. A project that’s had its timeline cut to meet an external deadline will generate more bugs and have more bugs reach the customer than one that’s been well-planned and executed without major time or resource stress.
• Not one metric can capture the complexities of software testing. Who is doing a better job: a tester who works through a complex new feature and finds a dozen of severe bugs in a week, or the one who works through a complex new feature and finds one catastrophic bug in that week? The answer is neither: both testers are finding the right information, but the information they find differs because they’re working in different areas with different developers.

Software testing is a very important part of product development. A primary purpose of the QA process is to detect software failures so that defects may be discovered and corrected. Testing cannot establish that a product functions properly under all conditions, but only that it does not function properly under specific conditions. The scope of software QA & testing process often includes the examination of code as well as the execution of that code in various environments and conditions as well as examining the aspects of code: does it do what it is supposed to do and does it do what it needs to do. In the current culture of software development, a software testing organization may be separate from the development team.

Whilst ratios are always useful and may be viewed as an expedient and simple way to arrive at an estimate, there is a significant risk when one organization uses another organization’s ratios and then applies these to their project and their QA processes without regard to differences in technology, software quality assurance process maturity, and skill levels.

Part of the problem with identifying a reasonably accurate ratio is not just the reasons above but also the lack of data collected by organizations on this information. Thus, for many, it is not an exact figure that is sought but an assurance that the testing estimates are reasonably correct and there is a degree of confidence that the software testing can be undertaken within the project time frame with the staff and budget allocated to this function.

Testing is a process rather than a single activity. The QA process starts with test planning, then designing test cases, preparing for the execution, and evaluating the status until the test closure. So, we can divide the activities within the fundamental quality assurance process into the following basic steps:
1) Planning and Control
2) Analysis and Design
3) Implementation and Execution
4) Evaluating exit criteria and Reporting
5) Test Closure activities

Proper documentation is a crucial part of your software QA processes. Test plan outlines the common strategy that will be applied to test an application, the resources that will be used, the test environment in which testing will be performed, and the schedule of testing activities along with the limitations. Typically, Software Quality Assurance Team Lead is in charge of writing a Test Plan.

A test case is a complexity of inputs, series of steps, and conditions that can be used during the QA process. The key point of this activity is to find out whether the software is successful in terms of its functionality and other aspects. There are various types of test cases such as logical, functional, error, negative test cases, physical test cases, UI test cases, etc. that could be used for the execution of smoke testing or regression testing.

Checklist – is a list of tests that should be run in a definite procedure. It helps to understand if testing is fully run and how many of those tests failed. It also helps to better structure the testing process, by listing all the features that need to be tested. Test order in the checklist may be strict as well as random.

Availability of the test documentation types (test plan, test cases, checklists) on the project depends on the budget, QA team size, development process, etc.

Test management tools are used to store information on how testing is to be done, plan testing activities, and report the status of software quality assurance activities. Test management tools are very useful for smoke testing as well as regression testing when the execution test set should be defined. The tools have different approaches to testing and thus have different sets of features. Generally, they are used to maintain and plan manual testing, run or gather execution data from automated tests, manage multiple environments, and enter information about found defects. Test management tools offer the prospect of streamlining the QA process and allow quick access to data analysis, collaborative tools, and easy communication across multiple project teams. Many test management tools incorporate the requirements helping to streamline the test case design. Tracking of defects and project tasks are done within one application to further simplify the testing.

A bug tracking system or defect tracking system is a software application that keeps track of reported software bugs in software development projects and it helps to better structure your software QA process. It may be regarded as a type of issue tracking system.

Many bug tracking systems, such as those used by most open-source software projects, allow end-users to enter bug reports directly. Other systems are used only internally in a company or organization doing software development. Typically bug tracking systems are integrated with other project management software.

A bug tracking system is usually a necessary component of a professional software development infrastructure, and consistent use of a bug or issue tracking system is considered one of the “hallmarks of a good software team”.

Isolated Test Environment is an exact copy from your production environment but with only the bare minimum needed to test in it. It is also in a separate network and the isolation aspect means that this separate network can not connect to the production network and vice versa. Because the networks can’t see each other, you can use the same IP addresses in both environments and this, of course, means that you don’t have to customize your servers in the test environment. There is no chance that a test user will accidentally contact a production database or mail server.
Ideally, all testing activities should be done in the isolated test environment as to not affect the development team and ongoing development process.

Monitoring the health and uptime of your web server or web host is an important part of the software QA process. You want to make sure your website is always available to your users, and that your website isn’t slow. Server monitoring tools allow you to gather and analyze data about your web server.
Part of the problem is that different operating systems, browsers, connection speeds, devices, and locations mean that any number of combinations could be interacting with your application. Local testing is useful but fails to identify the vast amount of variables that users can encounter when trying to use your app.

End-to-end testing is a software testing methodology to test an application flow from start to end. The purpose of this testing is to simulate the real user scenario and test the system and its components to make sure it integrates well with other elements and provides the necessary level of data integrity.

It is performed from start to finish under real-world scenarios, like communication of the application with hardware, network, database, and other applications.

The main reason for carrying out this testing is to determine various dependencies of an application as well as ensuring that accurate information is communicated between various system components. It is usually performed after the completion of functional and system testing of any application.

Software testing can be done by a dedicated Software Quality Assurance team that can be on-site or external (outsourced or outstaffed). Depending on the project-specific and budget, testing also can be provided by stakeholders, managers, developers.
If the product is available for the users, beta-testers can be involved in the testing activities as well.

A test report is an organized summary of testing objectives, activities, and results. It is created and used to help stakeholders (product manager, analysts, testing team, and developers) understand product quality and decide whether a product, feature or defect resolution is on track for release. Test reporting should help you understand the achieved value of testing.