Three Steps for Success in Unlocking Big Data's Potential

In my last blog post, “The Future is Sooner and Stranger Than You Think” I had talked about the characteristics of Big Data in terms of the 4 Vs (Volume, Velocity, Variety, Value), the 5 Ms (Monitoring, Mapping, Management, Middleware, Measurement). Today Big Data is inescapable in the Financial Services industry, with increasing volumes of customer and transactional information flowing through organizations. The FS sector is one of the most data-driven industries, and analysts estimate that somewhere between 80 and 90% of the data that exists within a bank’s datacentre is not analysed – data from call logs, weblogs, emails and documents. In addition, there are other relatively new sources of unstructured data that continuously spew digital exhaust and contribute to what’s known as ‘Big Data’– blogs, online news, weather, Twitter, YouTube and Facebook to name a few.

There is a potential goldmine of information in both types of data source, and the combination of data from traditional sources combined with data from new and unknown sources presents financial services institutions (FSIs) with new opportunities not only to comply with stringent regulatory requirements, but also to derive valuable information and insight for business advantage. The cost of missing out on these benefits is high.  In a recent study, “From Overload to Impact:  An Industry Scorecard on Big Data Business Challenges,” C-level FSI executives said that their companies were unable to realise an average of 12% of additional revenue per year due to an inability to draw actionable insight just from the data their organization collects.

The second key tenet of Big Data is the industrialization and commercialization of a slew of technologies such as Hadoop, Hive, Sqoop, Flume, Pig and Mahout, which enable businesses to sift through massive quantities of unstructured and structured content. The potential benefits are tremendous, from responding to regulatory pressures faster, better and cheaper, to the creation of targeted products, services and marketing campaigns. At a consumer level utilising Big Data can improve loyalty. For example, should a customer make a large purchase and reach their credit limit, they may migrate to a competing card provider for their next purchase. However, if the bank records a status update from the customer announcing their desire for a new product, it can prepare and pre-approve a loan for this purchase, subsequently decreasing the likelihood that the customer will visit a competing bank for their next purchase.

Fully grasping Big Data is a big undertaking, best achieved by assessing your organization’s current position and prioritising which challenge should be tackled first to meet your stated goal. There are three steps that FSIs can take to begin to manage and leverage Big Data successfully:

Step 1: Targeting a Specific Problem to Solve

Depending on the organization, the following areas are strong candidates for taking advantage of Big Data:

• Next Best Offer – Banks can use predictive analytics on a combination of data to create a series of targeted offers for customers, and make these offers available in real time at the next point of customer interaction
• Pricing Management – Banks, Capital Markets institutions and insurance firms can use information from both types of data source to price products for individual customers, taking into account risk, capital, cost allocations, transfer pricing and a multitude of additional dimensions
• Payments Analytics – Banks can offer value-added services to their merchant customers by analysing payments cleared by them and alerting them of opportunities if they see certain patterns emerge
• Fraud, AML, Trader and Broker Compliance – All categories of fraud suffer from over-zealous software that generates a high number of false positives. These result in a significant operational problem as they need to be analysed manually. Tuning the software to reduce the number of these alerts results in the opposite problem, with real fraud going unreported. Using both sources of seemingly unrelated data offers the potential to catch fraudulent activities earlier than current methods allow. In the case of internal fraud monitoring, trader and broker compliance software can monitor trading activity coupled with additional data points from sources such as social media, SMS and emails, and create a graph analysis that traditional tools are unable to provide, in order to detect any patterns.
• Reputational Risk – All institutions worry about their reputation and getting feedback on newly adopted policies and newly launched products. Most of the ’noise’ around a brand comes from new data sources, and Big Data technologies can be very effective at quickly gathering  this information for analysis.
• Credit Risk – Valuations for credit risk are very computationally intensive. The number of risk factors that need to be modeled are in the 1,000s, with each risk factor taking 1,000s of stochastic paths. This problem is traditionally solved using very large grids of compute nodes. Big Data technologies offer a significantly cheaper alternative, with a huge number of loosely coupled Hadoop nodes that these computations can be offloaded onto.

Step 2: Deciding on a Data Strategy

The essence of Big Data is bringing disparate sources of data together – some structured, some not, some persistent, some streaming – all at the right instant of time. Unlike traditional data management where the structure of the data is decided upon its arrival - “schema on write”- Big Data mandates the realization of metadata at time of consumption - “schema on read”. This creates a new series of challenges in determining not only which data to persist and where, but also how to locate the persistent data when it is needed, all in real-time.

It is critical to identify sources of structured data within an institution, and to unify these sources. Their metadata is well-known and bringing this data into a unified data model is a good first step. Identifying additional data needs based on the use cases identified in Step 1 will bring this second source of data closer to the point of realization. Figuring out where to persist this data will depend on the use case.

Step 3: Applications – Build or Buy

For the identified use cases, applications can be built in-house or bought. A significant number of vendors supply packages that can generate Next Best Offer or Identify Fraud. The ones that stand out are the ones that seamlessly merge the traditional world of well-known data, analytics and visualization with the new world of seemingly innumerable data sources, utilising Big Data technologies to generate new insights.

Ultimately, Big Data is here to stay, and FSIs that embrace its potential and outline a viable strategy, as well as understand and build a solid analytical foundation, are well positioned to make the most of it.

I would love to hear from you about your plans to harness big data for executing your business strategies.