The Testing Challenge: Cost

The pressures of time and cost are constant barriers to effective implementation. These pressures can be offset, for example, by spending more money to reduce testing time. Adding to this inherent testing challenge is the increasing complexity of integrated processing systems. But if we invest in tools which increase productivity, we can deliver more quickly and be better prepared to deal effectively with increasing complexity. In our latest article in the ‘Testing Challenge’ series we look at the role of cost in testing payment systems and provide some ideas about how to manage it.

The deployment and operation of payment system technology and the accurate testing of systems is critical to business success in financial services. Whether it’s a major quarterly waterfall software release or this week’s Agile update, bug-free systems are cost critical. News travels fast when something goes wrong, and if it does you must act quickly to minimize costs and reputational damage. Everyone from the CEO downwards recognizes how important this is, so why is it so difficult to get right?

The testing challenge

Let’s focus on costs. In the broadest terms, the following are cost sources for system testing:

Test Tools

• Purchase or development
• Maintenance
• Upgrades

Personnel Time

• Development and maintenance of tests
• Execution of Tests          
• Tool Training

Let’s consider the two components of system testing (new capability testing and regression testing) and how their cost demands often rise over time.

Testing new capabilities

In its simplest form, we are merely testing new capabilities of an existing system using a single, existing test tool (let’s call it the “base” test tool). However, over time, the reality grows more complex, leading to higher costs:

• New capabilities often mean adding whole systems into the financial operations mix, which may not be testable with the existing test tool.
• Even within the base test tool, new processing system functions often require new capabilities.

Over time, many organizations wind up with an “organically” grown mix of different test tools provided by external parties, and other testing capabilities developed internally. Maintenance and custom development costs for external test tools, often from multiple vendors, are not easily controllable. The personnel costs of maintaining expertise in multiple tools and developing multiple internal systems are limitless. Personnel costs rise if any test tool is developed internally, since it has to be tested before new capabilities are introduced. Not to mention that any negative test results may imply that the testing tool itself is at fault.

In addition to the increased cost of executing tests for all new systems and system capabilities, the growing complexity of system architecture considerably increases the time needed for system integration testing. Test tools typically simulate a single member of a back and forth message exchange. System integration testing with this type of test tool typically requires that all other systems used in the test message flow are “live”, that is, the systems not under test must be the actual hardware and software system. System maintenance, software problems and configuration problems in any one system disables the integration testing of them all. And having data out of sync between systems causes major headaches.

Regression testing

In its simplest form, regression testing is simply an ever-growing set of tests performed on a single system using a single test tool designed to test that system. A basic set of tests derived from the original system implementation provides the core test cases and new ones are added as each new set of capabilities in a release are added. But, over time, similar challenges arise in regression testing to those in new capability testing.

While regression avoids having to execute the new tests for the release, it requires testing of systems that are not updated. And the complexity of systems integration testing is compounded in regression testing.  All systems have to be involved in regression systems integration, and the aforementioned problems of availability and accuracy are compounded. Superimposing the complexity of system integration testing upon the increasing numbers of unit and integration regression tests leads to an unmanageable regression effort.

Summary of Cost Impacts

The costs of the different types of testing discussed above can be summarized as follows:

Test Tools

Purchase or development

• New systems require the purchase or development of new test tools.
• New capabilities for existing test tools require the purchase of additional components from external vendors, or the development of internal tools

Maintenance

• Purchased test tools incur re-licensing costs.
• Internally developed test tools require personnel costs to be kept up to date.

Personnel Time

 Development and maintenance of tests

• Creation of new tests on multiple testing platforms
• Update of old tests on multiple testing platforms
• Creation of test data on multiple testing platforms

Test Project Execution

• Management of data on multiple testing platforms
• Execution operations on multiple testing platforms
• Analysis of test results on multiple testing platforms

Integration Testing

• Maintaining availability of all “live” systems while integration testing each system
• Management of data consistency cross all platforms

Tool Training

• Maintaining the understanding of all of test tools either by individuals or interfacing teams
• Maintenance of tool understanding regardless of personnel turnover
• Obtaining expertise in legacy systems and test tools

A Cost-Controlled Approach

While a certain amount of cost in all these areas is a fact of testing life, costs can be controlled as long as a single test environment is sufficient for all test demands. However, when multiple separate systems are part of the overall solution, the cost of multiple tools to test all systems fully and the inability of a single tool to simulate all systems reduces control. The way to regain control in this instance is a single, highly flexible test tool which provides the following cost advantages:

• One source of purchase and maintenance cost
• One process for support management
• Centralized test development and maintenance interface
• Centralized test data management
• Centralized test execution and result analysis functions
• The ability to simulate all systems in an integration test except the system under test
• Training on a single system up front and during personnel turnover

Our experience has led us to develop a test platform which enables organisations to simulate a real world transaction across legacy systems, multi-step transactions, APIs, web services and tokenisation. This helps to reduce cost and removed the barriers to effective implementation.