Sberbank is a key lender to the national economy and a leader in the deposit marketplace. Its services are used by over 97.8 million active customers in 21 countries around the world. This financial institution has the largest branch network in Russia (around 15,000 points of service) and its foreign network consists of subsidiary banks, branches and representative offices operating in the UK, USA, CIS, Central and Eastern Europe, India, China, and other countries.

Use of Disparate Testing Tools in Growing Organization

Prior to the centralized implementation of the Micro Focus Application Lifecycle Management (ALM/Quality Center) platform at Sberbank, information about defects in information system releases was gathered using various tools. This made managing patches very complicated. A considerable number of defects were not promptly resolved and remained in the systems even after their deployment into production.

“To improve software quality, it was necessary to detect as many defects as possible before acceptance testing, and to reduce the number of defects at the production stage,” says Mikhail Vorotyagin, Director of Quality Assurance at SAP, Sberbank (from 2014–2018 he headed the Department of Testing Organization at Sberbank-Technology).

Many different tools were used to manage the testing process. Centralized management used to be carried out with IBM Rational Jazz, but its performance began to decrease noticeably as the load increased due to growing numbers of teams and test objects. This led to teams beginning to independently install RedMine, Jira, ALM/Quality Center and other products. The summary report was generated manually, which took around six hours. During this time, part of the information would become out-of-date and had to be updated.

The Sberbank-Technology Quality Department was established in April 2014. It includes five functional testing departments, management of stress testing and automation, management of test environments and a business unit for testing organization. At first, three specialists worked in the department but, over time, the team has grown and now has 25 employees. They are responsible for replicating practices and providing effective testing and defect management tools with a uniform standard of quality management settings.

ALM/Quality Center Performs Best in Detailed TCO Evaluation

The selection of a centralized suite of development, testing and maintenance tools began at Sberbank in mid-2014. At that time, the software development and testing were carried out following the waterfall approach which was the standard procedure for Sberbank, known as the “Development Factory”. Later, another standard Sberbank technique started to be used – the end-to-end production process (EEPP).

“We needed a management tool that could be flexibly integrated with the testing and load testing automation tools used at the time; a tool that was convenient to use, provided object tracking, and all the necessary functions in one place,” explains Pavel Masyutin, former Chief Engineer for maintenance testing tools at Sberbank-Technology. This would help to provide full visibility into the testing process.

When choosing a platform for centralized testing management, five products were considered. They were both proprietary (ALM/Quality Center, Jazz, Microsoft TFS) and open source (Zephyr, among others). The company’s specialists compiled a list of the main tasks performed by a software tester during the day. Integration capabilities with other tools and the total cost of ownership for five years were evaluated for each product, taking into account the cost of licenses, server maintenance, and other requirements. As a result, a rating model for the total cost of ownership (TCO) was created. It contained the most important functional and economic indicators. Having evaluated the products under consideration using this model, the experts settled on ALM/Quality Center, since this toolkit was the best across the board.

Voluntary Transition to ALM/Quality Center

“Among the advantages of ALM/Quality Center that are important for us are the ability to maintain lists of software requirements and test scripts using a single tool, and the provision of a single, consolidated reporting database and convenient, customizable dashboards to visualize it,” continues Pavel Masyutin. “It is important to note that ALM/Quality Center enables us to maintain a unified defect registration database using unified rules and common attributes.”

At that time, project management was carried out using Micro Focus Project and Portfolio Management; testing management was carried out using ALM/Quality Center tools; and incident data from releases transferred to operation were extracted from Micro Focus Service Manager. All the information was consolidated and analyzed to understand and evaluate the current situation.

“Another advantage of ALM/Quality Center is the flexibility to support various types of testing,” adds Pavel Masyutin. “It is possible to manage the settings centrally through top-level templates.”

The transition to ALM/Quality Center, which began in mid-2015, was gradual. “It was carried out on an exclusively voluntary basis – the teams themselves decided at what point they should switch to the new tool,” recalls Mikhail Vorotyagin. Some of the teams switched to ALM/Quality Center quickly, while others continued to work with Jira. A standard ALM/Quality Center module was used to synchronize them, which allowed defect information to flow from one system to the other. At the same time, uniform testing rules were being created and replicated.

The number of ALM/Quality Center licenses has also grown gradually and now they number 1,200. This is the largest ALM/Quality Center deployment in Europe. Micro Focus Business Process Testing licenses were purchased to test CRM applications. Functional and load testing quality control is organized using Micro Focus LoadRunner Enterprise.

ALM/Quality Center Supports Move towards DevOps

Development of Fast Waterfall, Agile, and DevOps methodologies began in 2016. Part of the implementation of DevOps involved changing testing by dividing it into two separate functions: engineering (development of automated tests) and expert (design of complex E2E tests for business logic). Development, testing and operation teams began to unite as single-product teams. In connection with the transition to flexible development based on DevOps, it was decided in April 2018 to disband the unified quality management structure of Sberbank-Technology and transfer these functions to product teams. This would speed up decision-making and reduce the time needed for creating new releases.

ALM/Quality Center is currently used for functional and automated testing. Various tools are used for testing: Micro Focus UFT One, JMeter, Selenium, Cucumber, TestComplete, SOAP UI. The SberQuality portal consolidates information about test launches, the number of defects, and the division into manual and automated tests. It is also where various KPI tests are published. Reports on the current status of the releases are prepared based on all this data. On average, around 40,000 tests are performed every day. Of these, only 1.5– 2,000 are performed manually, and the rest are automated. The maximum number of parallel ALM/Quality Center users during peak periods reached 1,100 people. Sberbank is currently deploying a pilot project to implement a Micro Focus ALM Octane DevOps toolkit. The product has already received good reviews from the developers and some teams are currently ready to use all its capabilities.

Reduced Reporting Time and Increased User Satisfaction

According to Mikhail Vorotyagin, the transition to ALM/Quality Center has enabled centralized management and the unification of development standards, including standardizing the work of the functional testing and automation teams. Attribute information, objects status, defects criticality level, and attribute defect composition is now consolidated and systematized. The centralized storage of test scripts means there is no dependency on individuals, so no loss of operating time when an employee leaves, for example.

“ALM/Quality Center has increased testing process productivity. As a result, reporting time was reduced from 6–8 to 1–1.5 hours,” says Pavel Masyutin. “Thanks to synchronization with Jira, the overall satisfaction of the testing teams has increased: it became easier and more convenient for them to work and the number of routine manual operations decreased.”

A total of 392 teams use ALM/Quality Center and 7,971 requests for changes in the releases were implemented using this toolkit. Over 750 different test environments were used in the tests. 135,320 defects were detected before being put into production, which corresponds to 87% of the total number. After acceptance testing, 67% fewer critical defects were found and the proportion of critical defects decreased from 20% to 5%. Meanwhile, automated testing (40%) and automatic test launches (65%) began to be used more often.

Active Collaboration with ALM/Quality Center R&D Teams

Sberbank is the largest ALM/Quality Center customer. Bank experts are collaborating directly with ALM/Quality Center developers and are participating in beta testing programs for new releases and versions. This makes a significant contribution to the development of the toolkit. In particular, they have initiated a number of improvements to ALM/Quality Center brought about by the scale and specifics of the application of this solution in their organization.

“In the near future, we plan to achieve smooth ALM/Quality Center operation. We want to establish tracking indicators that reflect its reliability and availability in order to minimize the downtime,” said Fedor Radzievsky, leading technology expert (from 2014–2018, he held the position of chief engineer in the department of load testing and automation at Sberbank-Technology). Similar actions will be taken in relation to other testing tools. This is due to the fact that the release cycles are shorter and the cost of system downtime has increased significantly.

In addition, a pilot project for the implementation of ALM Octane is being investigated (teams will be asked to switch to this product in the event of its successful completion) and a number of pilots for new products are also being conducted to synchronize information on various test environments.

Options for transferring to a dynamic private cloud-hosted infrastructure for conducting tests are also being considered. This would involve not just the infrastructure, but also the entire set of tools with the required support.