Little Big Data

Big Data is one of the trends of the moment, with beautiful infographic representations of everything from social networking, video uploads, web searching and app purchases. These are, no doubt, important tools in shaping our strategies in payments. However, this is not new.  The challenges of big data as defined by Doug Laney, the father of Big Data in 2001, are the 3 Vs - Volume, Velocity, and Variety – and his main example was e-commerce.

Well the challenge has only grown bigger, every system processing payments has grown in all of Doug Laney’s metrics. Every company now has its own ‘Big Data’ challenges – a large amount of data, with millions of transactions added daily and an increasing number of channels it is receiving it from.

The challenges are how to use this data to benefit the business. Increasingly, real-time analysis of this data becomes important – to track trends for availability, fraud and risk. These are all vital factors for ensuring a system delivers its maximum return on investment, where margins are increasingly being squeezed by competition and regulation. The challenge is focusing this massive amount of data into something that can be acted upon. Modern dashboard design certainly helps this – the key to efficiency being the right data at the right time.

The off-line analysis of this data is also key, the trends both in traditional business intelligence methodologies  and the tangential Big Data analysis are all key to ensuring profitability for the future.

The surprise is that many organisations just throw this data away. Even data warehouse data is often truncated such that information that could be used in analyses is lost forever.

So, whilst the challenges and scales of big data sets with its Terabytes of data do not seem immediately relevant to individual companies in the payments chain,  I believe they are.

The question is – what is your Little Big Data strategy?  Are you using what you have to its best benefit to the business?