Innovation in Quality Assurance - What is the Impact on Trading Technology?

This post outlines some ideas on how particular changes in quality assurance might affect the trading technology. There are lots of discussions on Internet-of-Things, Cloud and Web. However, there are other things in software verification and they are a little bit closer to our area - exchange and brokerage platforms.

These are some interesting QA trends:
- FrAgile process
- Crowd-sourced testing
- Formal methodologies
- Cognitive technology

Agile methods are very popular in start-up and web-industry. Agile methods do not necessarily result in better products, but they can make software developers much happier. Please refer to the great opening keynote address during Google test automation conference GTAC 2011 – Test is dead, to learn about enhanced version of agile – FrAgile development process.

As Reid Hoffman said: “If you are not embarrassed by the first version of your product, you've launched too late”. Test ideas first to find the right product, before building the product right. Have many users by the time your competitors will have a well tested product.

Large user pool can help you to use Crowd-testing. Give your product to users and collect immediate feedback on your software. To do it efficiently, you would need to rely on sophisticated code instrumentation and data collection. What is the point of applying crowdsourcing if you are not capable of efficiently processing its output?

But can we really use FrAgile process and Crowd-testing in finance? Can we allow an algo going berserk to burn money and face prosecution? Maybe what we need is the very opposite to agile?

There are areas like transport, medicine, nuclear power that have more rigorous controls. Quality assurance methods evolve there. Advances in model checking allowed PROVING as opposed to ASSUMING that something will work. There was a post in the TABB Forum a few months ago questioning why are we not using mathematics and computer science similar to what NASA uses to design safe autopilot? Is not it embarrassing that we have outages in the financial markets at the same time when spacecrafts are gliding over the outskirts of the solar system?

We can invest heavily into formal methods to have more stable financial software. But, the thing to remember is that despite of all the formal methods the New Horizons probe had an outage that knocked it out of contact with Earth for three days. To expect that your systems will break is the only way to improve the quality of your software.

So what's next? Can we learn something from other industries? Be both faster and safer. To understand how to do it lets look into the Doomsday scenario. Lets talk about people who had severe problems with technology - Battlestar Galactica and Sarah Connor.

The only reason the characters were able to make it until the end of the second season is that they were relying on the very same technology that was trying to kill them. They were protected by systems having the same level of sophistication as those they were struggling with. This is the mentality that we should adapt. When we use a complex platform, we need to build software to test our software. And you do not want your risk control and test instruments to be inferior to what will hit you. Having a good robot on your side is the only way to survive the robot apocalypse.

So, what's the impact on the trading technology? Developing testing instruments and monitoring tools should has the same priority as the trading platform itself. It is possible to use agile methods and continuous integration if your system is designed with testability and risk control in mind. At the very least you can deploy, start, restart and configure it automatically. Also, it is good to be able to shutdown the system if necessary. Do not expect formal artefacts from the agile process. Have a parallel stream instead to develop the necessary test harness. Software engineer in test approach proposed by Google is more or less building software to test software.

We can also use formal verification, theorem proving, static analysis and other things. But, the software will break anyway. The absence of adequate monitoring and kill switches is what turns a problem into a disaster.

What about audit and regulatory requirements? Develop passive testing tools. They do not generate any messages themselves. They capture the traffic and store it for analysis. Passive testing tools can gather all the evidence you need.

Ensure that your trading system can co-exist with the tools. Passive testing tools can also serve as both surveillance and client on-boarding systems. You can use them for the crowd-sourcing as well. If your test environment is opened to others and you have a strong passive test capability you'll be able to get a lot of valuable feedback without even asking for it.

Lets consider testing trading algos. We can deploy one or several scalable matching engines to serve as market simulators. We can deploy competing algos to act as market participants. They can both replay some data and at the same time mimic a realistic market impact. In addition, we can have latency and front-running proxies for better tests diversity.

It is possible to take the next version of the algo and put it to the test. However, what is the point of running it once? Markets are not deterministic. Good simulated markets are not deterministic either. We will get a slightly different result every time we run the tests. You need to repeat your tests multiple times to assess the quality and efficiency of your software.

This is where everything comes into place:
1) Efficient restart and testability to enable unattended test execution
2) Machine-readable specifications to synchronize all the tools and formally iterate the possible tests
3) Passive testing capabilities and potentially market surveillance to capture and store all the relevant metrics (both P&L and system related ones)

You can run as many tests as you like and pass your system through a wide range of scenarios. But, we all have limited time and also limited hardware. It is worth mentioning now two more quality assurance trends. To select efficient limited subset of tests from an infinite number of possibilities, cognitive technology and mutation testing are there to help. It is possible to use them to find the optimal test scenario subset for your trading platform. They can also help you in enhancing the test set along the changes in your system and in surrounding markets.