This year’s Sibos had lots of interesting ups and downs (read my blog at Sibos Weblog to find out more) but one that stood out was the comment from Citigroup’s Tom Abraham that the financial models we use to justify IT investments are fundamentally flawed. Tom summed it up by saying: "We have these elaborate and sophisticated financial models with ROI, IRR and NPV but the margin of error that leads to decisions is substantial. At the end of the day, you are taking a bet."

In the 21st century, surely we cannot still be making multi-million dollar bets?

I entered the IT industry in the 1980’s believing that technology was the future. Well, it was the future then and still is now. During my first years in the industry, the main issue we came up against was the lack of understanding of technology by the users. The business community was typically very hands-off and had little or no understanding or, it has to be said, interest in dealing with technology.

Back then, technology was originally introduced because it delivered obvious benefits in the automation of manual processes. The efficiencies of taking humans out of the process became a major focal point in most firms.

The cost benefit analysis was pretty easy. Ten clerks performing administration of accounts costing $25,000 a head can be replaced by a small data processor costing $300,000 with a payback in less than eighteen months.

However, even then, it was not that simple. I remember talking to a group of bankers in the mid-1990’s about data warehousing. The simple fact was that data warehouses were viewed as expensive. So I asked the question of the vendor: "How much does your system typically save a bank?" After a dumb silence of about ten minutes, and a major consultation between the CEO and his marketing head, the answer came back "We don’t know".

They did not know because no-one really measured the savings. They justified the investments but did not measure the returns once the investment was signed-off.

The result was that that particular vendor invested in a major exercise to quantify the value of the solutions they delivered. That also sounds easy but it was not. Quantifying the value that solutions deliver is never as easy as it sounds.

For example, there is the original pre-implementation dream. This is the work of fiction known as a Cost-Benefit Analysis (CBA) that ensures for every $1 invested at least $2 is saved or generated. Savings are in terms of headcount – or these days, systems consolidated and maintenance and legacy $’s reduced – and income is found through a sudden flock of customers through the door because they love the way the new technology gives them better service.

So far, so good.

Having done all that work and generated the dream, the management team sign on the contract and everything is then forgotten. That is the reason why most technology firms “don’t know” the value of the systems they sell, because no-one measures the value once it is delivered. The reason no-one measures the value is that it can often be quite scary. The vendor does not want to measure the value in case the solution does not deliver. The client often does not measure the value because the ones who cost-justify the systems are also the ones with their heads on the block if it all goes wrong.

That has to change, because we can no longer afford to continue making bets. We need more certainty, if nothing else to generate credibility and confidence in the technology function and remove uncertainty and risk. Therefore, the biggest way to lock-down future ROI in technology investments is to create a track record of measuring the margin of error made in calculating the ROI for historical technology investments. For many, that can only start today because they have no track record of measurements.

The question then is what should you measure?

I believe there are four things that should be measured when identifying the value of technology, or for that matter any other investments. Money, staff, customers and risk.

Monetary measurements are the obvious ones – how much did we spend; did it save us more than we spent; did it generate any revenue; how much in total?

Although monetary measurements are obvious, it still amazes me how few organisations actually measure that one. Go back to almost any major technology programme two years down the line and ask: "What’s the ROI on this then?" and typically the answer comes back: "We think it’s a lot, but cannot work out whether the revenue generated is down to the new technology or changes in the market or the fact that diddlysquat bank slipped up on that banana skin." That just goes to show that no-one really understood the financial models in the first place and certainly did not track them through.

If you cannot measure money effectively, just think how difficult it is to measure the other things – staff, customers and risk.

Staff measurements should identify how the technology implementation improved staff morale or otherwise, through tracking employee retention, productivity and satisfaction. Customer measurements should track the costs of acquisition, retention, maintenance, cross-sell and advocacy. Risk should measure the ability of the bank to adapt to change, implement compliance on time, respond to new competitive pressures and avoid industry exposures or reputation impacts.

All of these areas – money, staff, customers and risk – are assisted or hindered by technology. All of these areas provide ROI and payback if you implement the right technologies. All of these areas are what you should be measuring before, during and after making major technology decisions – both as a bank and as a provider to a bank.

Why? Because, as a bank, it will help you to get more decisions right in the future and take less bets. As a service provider, it will make it easier for you to sell more solutions to your clients because you will have a proven track record of delivering value.

The question then is whether to measure all of these areas or whether to pick out the key areas that represent the most value and return on investment for what you are trying to achieve. This brings in the final dimension of value, which is to measure the value based upon the specifics of each and every implementation.

Every implementation’s benefits analysis will be different – both before and after the project. Therefore, although you can calculate value and return, the calculation is not a one-size-fits-all. Some financial institutions have already improved their operations through automation and re-engineering, whilst some have not. Some financial institutions already have sophisticated value measurement systems in place which link key performance indicators to overall objectives, whilst some have not. Some financial institutions have manual processes which are less automated. Every institution is different, both in their measurement capabilities, financial sophistication and automation.

Therefore, there is a great importance in measuring the right parameters that are specific to the situation. These 'situational' measures may even vary within organisations, by function, line of business and geography. The key is to link the technology benefits with the specific business metrics impacted in that functional, line of business, geographic application.

The result is that different models apply to different business types, sizes, organisation structures and efficiency levels. The most important factor is that, overall, you understand how to measure – and what to measure – to determine the value and return on technology investments before, during and after the project delivery.

Today, most firms tend to only measure and analyse the value and return enough to get the senior executive vice president to sign-off on the deal. The result is, as Tom Abraham said: "We have these sophisticated financial models but you are taking a bet." Tomorrow, taking bets will not cut to the chase, so quit gambling and start working out how to measure the right things.

Chris Skinner is a director of TowerGroup and founder of ShapingTomorrow.com.
Web links: www.towergroup.com and www.shapingtomorrow.com
Author's email: Chris Skinner