The Next Generation of Testing Tools for Exchanges

The modern society expects quality and reliability from the modern exchange platforms. Thorough verification of exchange technology is not possible without adequate software testing tools. Understanding their evolution can help deliver rapidly evolving advanced exchange systems.

Test automation tools allow simulating connected automated systems, running tasks in an unattended mode and generating load as in real-life market environment. The process of selecting a tool to test a trading or post trade system usually starts with looking at generic testing solutions widely used in other industries. No one has been fired for using standard commercial solutions, but the problem with most of them is that they are targeted at desktop and web interfaces. However, integration with trading APIs usually requires developing connectivity and session libraries as well as a framework that would allow to uniformly define message contents and expected results. Essentially, one will need to build one testing tool on top of another.

Exchange testing tools evolution

The are a few specialized testing tools available that already support FIX, FAST, ITCH, binary exchange protocols and have expandable architectures. The user specifies what to send into the market and what the expected result from every gateway should be. This process is called active testing. If the tool fits the purpose and the QA team is capable enough, thousands of test cases to validate most of the exchange platform’s functionality under test can be prepared and maintained.

The execution of huge regression libraries produces massive amounts of data. Additional insights can be obtained from analyzing these large data sets. The next evolutionary step in test tools is passive testing where one uses non-intrusive test tools that capture all messages generated in the system, normalize them, and store them for processing. Passive testing tools can help  obtain valuable results about the system and the test library coverage. We also found that passive test tools are invaluable in supporting client on-boarding and certification. Clients connect to the exchange platform and perform the necessary steps, while the tool captures these activities and produces pertinent client self-certification reports.

One can create a good functional testing tool and a scalable load testing tool in order to generate hundreds of thousands of messages per second and stress one’s platform to its limits.

However, there is another area in between - intermittent defects that do not manifest themselves every time the tests are performed, potential black swans.

Such defects are at the confluence of the functional and non-functional testing. To trigger them, one needs to repeat the tests many times and under load, and they can be easily missed in the enormously large load testing logs. These defects might not have severe technical consequences, though.

Ideal passive testing tool

What characteristics should a passive tool have to capture defects at the confluence of functional and non-functional testing? It should be scalable, stable, have minimal impact on the system under test; it should collect all events from all components in the system; it should efficiently capture all of the data for analysis; it should notify the user when something goes wrong; it should support the ability of the user to configure scenarios that trigger such a notification; finally, when something indeed goes wrong, the user should be able to restore the transactions around the event.

An ideal passive testing tool for exchanges functions exactly like a production monitoring / market surveillance solution.

Production Monitoring System

A good active testing tool should be scalable, stable, connected to many test environments, and easily configurable. It should be able to validate data consistency across various feeds, store the audit trail, and be capable of executing complex scenarios.

Ideal active testing tool

The above is clearly an algo-trading system. Instead of running Excel in and getting Excel out, connect something that behaves like a market participant with an algo engine.

Production algo-trading engine

Is that it? Can the test tools of the next generation be just algo-trading and market-surveillance platforms? Not really. There are two distinct reasons for that. First, imagine trying to fit several algo engines and an additional monitoring solution into your budget. It will substantially increase the cost of testing. The other limitation in using ordinary real-life systems is that their intention is to do what is right. However, the purpose of testing is to find what is wrong. Testing tools of the next generation will be very much alike production systems, but deformed and mutated to decrease the hardware footprint and to enable all possible permutations.

In a more distant future, there might be test tools based on machine-learning techniques. However, they can be enabled only if the test tools of the next generation are scalable and mutable. As singularity is not here yet, human intelligence is still required to create and operate test tools and large regression libraries. The best part of any testing solution is the human mind behind it.