IBM employee fingered as culprit in massive DBS outage

A seven hour outage because of an error ina routine repair job sounds outrageos but thats the way banking systems and the realted IT infrastruture are. A seemingly small incident/bug can paralyse an entire bank's network of ATMs,Branches & Online banking.

Not all banking systems are created equal ...

Many banks worldwide trust their ATM and POS operations to fault tolerant systems that are designed to run flawlessly despite any hardware or software component failure - but conventional mainframes are not built that way.

As a further consequence, such fault tolerant systems are also built for very simple online repair, without requiring elaborate procedures to be followed. This reduces greatly the risk of human error - which obviously had hit DBS Bank badly in this case. 

Using fault-tolerant hardware is probably the best option, but its higher costs would be justifiable only if banks are able to ascribe a $ value to the cost of such outages, including reputation loss.

A quicker and more pragmatic solution could be the adoption of a "4-eye" process, in which the two eyes of the 'maker' (the IBM employee in this case) are supplemented by two more eyes of a 'checker', who could be a bank employee.

Although it is by no means 100% foolproof, experience with another Tier-1 bank assures that such a process drastically reduces the chances of occurrence of a DBS-type of event.

According to a study performed by Standish Group in 2008, the average cost of downtime for ATM operations is 3,600 US$ per minute, and for POS operations it is 4,700 US$ per minute. Multiplying these by 420 minutes of downtime results in a financial loss of roughly 3.5 million dollars, which is more than an adequate fault tolerant system would cost. As DBS is certainly a bit larger than the "average bank", their losses may have been even higher.

By the way, reputation loss is not considered in the above figures.

The suggested "four eyes" procedure may prove to be not so practical in real life, as it requires additional bank employees who get trained in hardware maintenance. They would have little practical experience compared to the vendor's technicians, as those deal much more often with related incidents because they serve a number of different customers. So when in doubt, the opinion of the vendor's technician is likely to prevail anyway ...