How many patches are you sitting on?

You could be forgiven for thinking this should be a rhetorical question – the answer is surely none? However, such is the scale and complexity of managing FICC voice trading systems, with all their interdependencies and interoperability requirements, that the reality is very much the opposite.

The story of how IT departments are fighting a constant stream of security patches, bug fixes, and software upgrades is well known. Indeed, it even has its own ‘day’ – “Patch Tuesday” being the day when Microsoft pushes out its patches.  

However, not all patches are created equal. While some may be a critical security patch, others may be a bug fix for a rarely used feature. So how do you make sure that the you’re focusing your attention on the areas that really do matter?

Security is invariably going to come at the top of the list for banks and for good reason.

In February last year, the U.S. Securities Exchange Commission (SEC) updated its guidance to public companies for disclosure of cybersecurity risks and incidents. The SEC’s interpretation essentially created a new regulatory disclosure category for cybersecurity incidents.  It is a similar picture in the UK, where the Financial Conduct Authority (FCA) includes cybersecurity in its regulatory compliance agenda and outlines specific expectations for disclosure of incidents, and in Singapore where the Monetary Authority of Singapore (MAS) has taken decisive action towards placing cybersecurity at the top of its agenda by setting up an international advisory panel and appointing its first Chief Cyber Security Officer to drive regulatory standards compliance for the financial services market.

So we all know we need to prioritise security patches. That presumably is a given. 

But even then, installing a patch isn’t a straight forward job. Such is the complexity of the systems that a fix in one place can end up causing a critical break elsewhere. Prioritising the patches is a start, but it doesn’t go far enough. 

You can’t simply flow patches across the system and expect everything to be ok. They have to be tested, certified, deployed – and all over the weekend when the trading floor is closed.

The very nature of the highly regulated, compliance-bound trading floor is such that the infrastructure behind it has to be tested to within an inch of its life. 

In many cases, a bank will have multiple pre-production environments, taking weeks or even months for any changes to move into the production environment – and even then they can still be plagued by last minute problems.

The solution, until now has been for all this to happen first in the lab, and then only be rolled out into the live trading floor over the weekend, with the Ops team having to quite literally walk the floor to check every voice trading Turret and Dealerboard.

But compounding things, as Standard Chartered put it when speaking at the DevOps Enterprise Summit last year, “Operations have increasingly become a bottleneck and it can still take weeks of a manual process to provision an environment.”

 If you estimate that one person can on average manually test about 50 turrets over the weekend, and that even a mid-sized trading floor can have as many as 2,000 turrets, that is a lot of weekend overtime.  And it’s overtime that is incurred at least once a month as new security patches are deployed.

Banks’ operations teams are faced with a stark reality – such is the scale of the task that they have little option but to delay deploying security patches and do them in a once-a-month weekend blitz, exposing their organisation to risk. And that’s not to even think about all the other bug fixes and upgrades which are being deprioritised due to the onerous and costly manual testing process.

However, there is another way. By automating testing, patches can quickly and safely move from the lab into the production environment, without any need for teams to spend their weekend walking-the-floor. 

By augmenting the “walk the floor” process with automation, banks’ operations teams can reduce the test time from hours to minutes, enabling repeated testing not only of turret features, but also validate the audio quality for each call for MiFID II compliance.

This equates to a 90 percent annual time saving and $1.3 million in annual costs saved for a typical FICCs trading floor.

But this isn’t just about saving time and money. It’s about cutting the cost of downtime by patching faster and smarter, and it’s about accelerating value so you get more from your voice trading system and don’t leave software upgrades languishing in the lab until someone has time to get round to them.