Challenge

Banking customers love mobile technology and web services. They love being able to bank easily, from anywhere, at any time. They love services like balance transfer and bill pay. They love completing complex financial transactions with a single screen tap.

Consumers’ love affair with banking apps puts the banks themselves in a high-pressure situation, however: they’re in a constant race to launch new software. “Banks win customers by continually offering new application functionality and feature sets,” notes a Senior Manager who runs the Performance Testing and Service Virtualization department at one of the United States’ top four banks. “It’s very competitive.”

Application development cycles are therefore more fast-paced than ever – and if that wasn’t enough, applications are increasingly complex. “At one time, most applications ran as independent pieces of software,” the Manager says. “That’s changed today. Many applications need to connect with other horizontal servicing systems and with applications hosted with external vendors.”

One thing remains unchanged, of course: the applications have to work. “Banking customers have a low tolerance for applications issues and a high expectation for performance,” the Manager notes. So despite faster development and more complex applications, applications testing teams have no choice. They have to keep up.

Fortunately, this Test Manager’s bank found a way to bridge the gap between the ever-increasing demands of its business and its testing capabilities: it implemented Service Virtualization (SV) software – a solution that allows its performance testing group to simulate even complex production environments quickly, easily, and cost-effectively. It also allows the Virtualization team to support the needs of other Functional Testing and Development teams to increase their time to market.

Automated Testing and QA Solutions Vital – But Challenges Remained

SV is not the first Micro Focus tool the bank embraced within its QA group. It uses Micro Focus Applications Lifecycle Management (ALM) software to manage test cases, requirements, and defect logs. Micro Focus Performance Center (PC) is the bank’s enterprise solution for performance testing, with Performance Center software functioning as its performance testing tool. Micro Focus Diagnostics and Micro Focus SiteScope software are also deployed within the bank’s testing environments as monitoring tools, to help QA further validate application performance and scalability.

The bank has used many of these tools for years – in some cases, over 16 years – and today they are the foundation of the bank’s enterprise test center, enabling the bank’s applications development and QA processes to achieve a high degree of sophistication and discipline. This, in turn, allows development and QA to become more responsive to business requests. Within recent months, for example, the bank has transitioned much of its development processes from waterfall to agile, something that wouldn’t be possible without centralized visibility into requirements and test status.

Despite these advances, however, the bank was still challenged to keep up with testing demand – particularly for applications that integrate with services from other companies. Some of the bank’s software, for example, exchanges data with another financial institution’s mainframes. Testing those applications has been an ongoing challenge and fraught with delays. “They give us access to their mainframe for testing,” the Manager explains, “but only 6 to 10 hours a week, and all during non-business hours.”

These limitations made it difficult for the Manager’s team to complete its performance testing in a timely fashion – and the only alternative, standing up a duplicate environment, is cost prohibitive. “It would probably cost us as high as $3 million per month,” the Manager recalls.

So historically, the performance testing team and engineering teams improvised. Some teams built simulated web services using open source software. Others deployed off-the-shelf software, but there was no standard enterprise solution.

These stop-gap measures helped, but they were also time-consuming and resource-intensive. It often took people up to 50 hours to build simulators, an effort that typically added 2 weeks to their testing cycles. And the simulators were often limited in functionality, which left the bank vulnerable to the possibility that it would release software with unknown performance issues.

Solution

Required Functionality, Ease-of-Use, Lower Cost

Then Micro Focus introduced the bank to Service Virtualization software, and the Manager saw the value and decided it was time to implement a robust, enterprise service virtualization solution.

Before committing to SV, however, the bank did its due diligence. It compared the Micro Focus solution to other available products, particularly CA LISA, because some of the performance testers had experience with that tool. The bank ultimately concluded that SV was the best available service virtualization solution. First, SV supports the service virtualization functionality the bank’s performance testers needed. These virtualized services can be executed as part of performance test scenarios and can emulate performance degradation as well as the functionality. A second critical consideration was the solution’s ease-of-use. “The skill set required to use CA LISA is greater than for the SV software,” the Manager notes. SV software can be supported by the PT team skillsets and also be integrated with ALM and Performance Center software.

And finally, SV also costs less than CA LISA. “We realized that SV delivers comparable functionality at a lower price,” says the Manager. “That made our business case.”

Micro Focus Professional Services Ensures Proper Configuration

After the bank committed to SV software as its service virtualization solution, it engaged Micro Focus Professional Services to help with the implementation. “We consider them to be some of the best talent in the industry.” And while the Manager’s team is itself a highly competent testing organization, it made sense to leverage Micro Focus’s expertise to design and build its SV platform.”

Micro Focus delivered on that trust: the platform it built – 25 VMware virtualized servers running on HPE ProLiant BL620c Server Blades and HPE ProLiant BL460c Gen8 Server Blades – has since proven an excellent fit for the bank’s virtualized performance testing workload. “Within a short time of implementing the platform, we were simulating well over 100 services on it. But even at that level of use, we had bandwidth left over for growth, without having to invest in additional infrastructure.” It helped that throughout the evaluation process Micro Focus continued to improve the performance of SV so that now the infrastructure we specified can support twice the initial volume anticipated. This was an added bonus since interest in this platform as a service for other areas has exploded.

The Manager estimates that his team of 53 testers will handle around 350 test engagements in the next year, representing around 100,000 to 150,000 hours’ worth of performance testing.

Results

Flexible Solution Supports Multiple Use Cases

Because SV doesn’t have a steep learning curve, the bank’s performance test team adopted the solution quickly, leveraging the solution in several different ways.

In some cases, the testers work with the bank’s vendor partners to simulate their services’ functionality. For this category of use case, the testers create virtual replicas of third-party services within SV and then test how the bank’s applications perform when they interact with those services.

In other cases, it’s impractical to replicate a third-party’s applications environment. In these instances, the bank takes a black box approach: it uses SV to create simulators that perform as stipulated by the bank’s contracts with its vendors. This approach effectively minimizes the bank’s exposure to risk. It allows the bank to validate, to its own satisfaction, that its applications meet its performance requirements. Then, when the applications go into production, the bank’s vendors are responsible for ensuring their services meet their respective contract terms and service level agreements.

Another way the bank now relies heavily on SV is for component level performance testing. This is particularly important for testing real-time messaging (RTM) web services. “We use Micro Focus LoadRunner within our SV simulators to validate the performance of applications at RTM levels,” the Manager explains. This supports the bank’s agile development methodology because the performance tests can be conducted before upstream applications that leverage the RTM components have been fully coded.

A fourth use case addresses applications that pull from large, shared databases. “With SV, we can simulate large data sets, or environments that draw from multiple databases, more easily and quickly than if we had to replicate the data sets themselves,” the Manager says.

Reduced Testing Times – Better Testing Outcomes

Since leveraging SV, the bank has achieved a number of significant improvements in its performance testing capabilities and outcomes.

Instead of the 50 hours once required to build simulators, with SV software, testers can build them in around 4 hours: an efficiency gain of around 80%. This, in turn, allows the performance testing team to shave weeks from the time required to validate application performance and frees them to do other tasks, including increasing test coverage and quality.

SV also enables the bank to optimize how it allocates its testing resources. Before, it needed performance engineers to build its simulators. SV, however, is easy enough to use that performance testers can build simulators themselves. This frees the engineers to focus on analytics and other high-value tasks. It also enables the bank to lower its performance testing costs by around 60%, because it no longer has to allocate to testing tasks the engineers’ time, which is more expensive.

The performance testing team is also more responsive thanks to SV. “Because we can create simulators more easily and quickly, we can respond more quickly when developers send us code,” the Manager says. “It’s a better fit for agile development.”

Perhaps most important, however, is how SV has improved the efficacy of the bank’s performance testing – without incurring the costs associated with standing up physical copies of critical services platforms. “We’re catching performance issues that we couldn’t identify with the simulation tools we used before SV,” the Manager states.

In one notable example, the Manager’s team re-tested an application using SV and caught an issue that it had previously missed. The team first tested the application before it implemented SV; the application seemed to run exactly as intended using an open source simulation tool. Then, shortly after the application was launched, the bank decided to add some new functionality, and when the performance testing team ran the new version in an SV simulator, it discovered an issue. Under certain conditions, the application’s response time slowed from an acceptable three seconds to around 25 seconds, this was masked by leveraging the other tool.

“We were lucky because the application was still new. Its user base wasn’t very large,” the Manager explains. “But eventually it would have been noticeable. Application response times would have slowed to levels that our customers would have found frustrating. This is unacceptable in the banking market where customers can move their business at will.”

Instead, the bank’s developers were able to re-code the software to fix the issue – without any impact to the bank’s customers. “We wouldn’t have caught this with the simulators we were using before we switch to SV,” the Manager says.

Experiences like that demonstrate clearly that the bank made the right decision when it adopted SV. “Our applications today are more complex than ever,” the Manager concludes. “The pace of development cycles keeps speeding up. But with Service Virtualization, we not only keep up, we improve our testing capabilities. We keep getting better.”