Some time ago, in another article, we told you about the importance of Automated Testing in our company, the architecture behind this strategy and the adopted technologies with their advantages.
In this post we want to share how we implemented a Tests Automation Strategy for a migration project from an Undocumented Monolithic Solution (Legacy) to a Microservices one. Based on this empirical assessment we want to describe the main benefits of Automating Tests, the required investment, the quantitative results we obtained and the comparison with Manual Testing during a product reengineering process.
Costs and Benefits of Automation
Today, the software industry is trying to accelerate the development cycles to obtain deliverables in a short time while maximizing the return of the investments. Although, day to day, new techniques, technologies and methodologies arise to speed up these cycles, Traditional Tests have become a bottleneck for companies. This has resulted in the industry adoption of** "Time-boxed Tests"** putting in risk the quality of software products. For this reason, software companies need to discard traditional testing techniques and move towards a strategy that reduces testing times without compromising product quality.
The benefits of an **Automation Strategy **are evident not only for the companies but also for products and end clients. Nevertheless, the initial investment required to automate an entire Suite of Tests is substantial, the return of the investment takes its time, and it has a direct relation with the number of executions of the suite. For that reason, the Test Automation Strategy is recommended for long-term projects that apply Incremental Development Methodologies. On the contrary, this strategy is inadvisable when a project adopts a Development Methodology whose deliveries are only a few and whose Applications do not evolve incrementally. In these last scenarios, it would be necessary to devote a lot of time to automate tests and then this suite would be run just a few times along the project life, making the initial investment rarely recouped.
Organizing the Tests Suite
On the assumption that the Development Cycles that use Agile Methodologies are short, it was necessary to assess the implementation of a strategy related to continuous Automatic Functional Tests. Since Regression Suites are good candidates to be automated, our decision was to implement a Hybrid Testing Strategy where Automation was the main focus. Manual Tests were only applied in particular cases where automation was not suitable. For this challenge, we focused on the users’ experience to divide tests into three categories according to their importance for users and the affected population in case of failure.
Automation began with the most critical cases, Smoke Tests. This action was later extended in accordance with severity criteria. To assess the importance of each scenario in the user’s experience, we evaluated with the log of reported defects, both in the legacy solution and in similar applications.
The Team Training
Apart from the considerations regarding the structure and organization of testing suites, the training of the team working with the applied technologies is an important point for the migration from manual to automated testing. As regards the people involved in this project, we faced two different situations: a person with more than eight years of experience in manual testing but without any automation knowledge; and, another person without any previous work experience. In the first case, we strongly relied on this person’s knowledge of quality assurance. This fact was highly valuable for prioritizing and optimizing test cases to obtain efficient and time-saving tests. Meanwhile, this person became qualified in the new technologies during the 10 months that took covering the 100 percent of automated test cases. For the second person, he did not count with any previous knowledge in quality assurance and for that reason the approach was different. His training in the adopted technologies was prioritized, delegating the technical knowledge on quality to the other team members. Once this person obtained the required maturity to automate Test Cases, we started with his qualification in quality assurance. In 5 months, this person managed to contribute actively in the Automation of Test Cases.
The Obtained Results
Per release, before starting this Test Automation Project, 5 people were required to run the full Test Cycle and it took them 2 weeks to complete it. After 9 months of work, the Manual Testing suite was automated and **only 1 person could complete it in a single day. ** The formula for the accumulated cost of tests per release number C(i) (Formula 1) can be calculated adding up the cost of test design (CD), the cost of implementation (CI), the accumulated cost of test execution (CE(i)), and accumulated cost of test suite maintenance (CM(i -1)).
If we analyze the accumulated cost of the Manual Tests and the Automated ones, we can see that by the sixth release the cost of both suites equals. From this moment onwards, we clearly identify the cost of Manual Tests exceeds the cost of the Automated ones.
Another characteristic that is worth mentioning is the evolution of the Automated Test Suite in time. As displayed in the following figure, a month after initiating the automation, the Smoke Test Suite was completely automated. By running this suite automatically in the CI (Continuous Improvement) Pipeline and before promoting the application to the testing environment, several critical defects were early detected. After 4 months, the Sanity Tests Suite was completely automated. This allowed the automatic detection of major defects when deploying the app on the testing environment. Finally, **9 months after the automation, the Regression Tests Suite was also completely automated. **Since this suite currently takes 4 hours to be completed by itself, it is executed every night. This facilitates the daily identification of defects that were introduced the day before in the application.
Since in this project it was difficult to build synthetic testing data, real database information was used for this automation process. Consequently, as shown in the figure below **the test data is highly dependent on time **(dates).
As time goes by, it is necessary to update the testing database with real data from the production environment. In the figure above, notice the drop in test cases that is illustrated by the end of July 2021 when this update was performed. Also, see the increase in the number of failed test cases by July 12th which may be caused by an integration problem in the application. With this report, defects could be found and resolved in less than 24 hours.
Conclusion
The greater performance of the Automated Tests Suite is obtained when executing them frequently. This can be managed when integrating it with the CI Pipeline with the objective of having constant feedback of the quality of the product during the development process. On the other hand, it is very important to train the involved teams in quality assurance techniques that can reduce the time dedicated to the Manual Tests and be concentrated in a greater understanding of the business rules. Relying on Automated Testing mitigates the high staff turnover present in the software industry. In addition, it reduces its direct impact on the quality of the project due to the transfer of business knowledge to the automated suite of tests. This organization asset is integrated to the suite avoiding this information to be solely present in those who participate in projects.
Migue
Archt
Archt