Data is the new gold. But are we managing it to its full potential?

Data management is a critical issue for financial institutions: the challenges associated with it are vast and multifaceted. A group of industry experts from Nomura, BNY (ex-BNY Mellon), and Xceptor explored these challenges at a recent AFME OPTIC conference in London discussing some ways financial institutions can unlock the true value of their data to make it “gold”. 

The dark side of data, or "dark data" and its challenges 

While much data is accessible nowadays, significant challenges arise from dealing with granular, unstructured, or fragmented data spread across various silos within firms. As AI-driven systems become more prevalent, particularly in trading and decision-making contexts, the need to incorporate these neglected data sets becomes more pressing. 

Ling Ling Lo, Global Head, Data Strategy and Transformation, and EMEA Chief Data Officer from Nomura, brought attention to the treasure trove of dark data – the data that is unstructured, unloved, and underused. This kind of data, while difficult to access and process, is key to unlocking the next generation of trading and AI models.  Underutilised and often overlooked, it isn't typically harnessed by financial institutions but holds immense potential for advanced applications like AI and machine learning.  

However, the problem lies in its format and availability. Traditional data related, for instance, to trading and client information is readily available and accessible due to regulatory demands. On the contrary, dark data such as unstructured or fragmented data found in documents, meetings, or conversations and chats - remains largely untapped. This is because most institutions rely on aggregated data, which is easier to manage but less effective for AI-driven analysis.  

Therefore, firms need to look beyond structured data and start tapping into more unstructured, fragmented, and under-analysed sources. A significantly greater volume of granular data could be leveraged for AI-driven processes, which would have a substantial impact. For banks, for instance, the ability to exploit these hidden datasets offers a major competitive edge, especially in developing sharper predictive algorithms. 

The new gold it is  

It is crucial to have access to the right data. Archie Jones, SVP Data and AI Ethics at BNY, emphasised the value of artificial intelligence (AI) and machine learning (ML) in enhancing commercial outcomes through data. With financial services now having access to vast amounts of quality data, the key challenge is refining it for specific, profitable applications.  

And what’s key - the potential of generative AI to transform unstructured data into usable insights could be a game changer. Especially in capital markets where granularity and real-time insights are invaluable. 

The panel also mentioned the role of AI in automating data processes. For instance, AI can perform sentiment analysis on communications, helping firms derive valuable information from conversations and interactions. Generative AI was particularly noted for its ability to summarise and process complex documents quickly, offering potential time and cost savings. 

However, alongside these advancements, there are acknowledged risks, such as the possibility of AI models producing inaccurate outputs (referred to as "hallucinations") when they lack a solid data foundation. To mitigate this, Dan Reid, Chief Technology Officer at Xceptor, mentioned the importance of Retrieval Augmented Generation (RAG), which allows AI models to pull from specific, proprietary documents, ensuring more reliable and accurate results. Bringing solid governance to processes around AI and data is also essential to maintain accuracy and reliability. 

Cloud and Metadata: transforming data from expense to value  

We’re living in a time when we are witnessing a transformative shift towards cloud adoption in financial services. As many of us, including myself, remember, back in 2016-2017, banks were very hesitant to embrace cloud technology, but today we are seeing a significant shift with the rise of roles like Chief Cloud Officer, reflecting the growing importance of cloud strategies in the financial sector. This shows the critical role cloud solutions now play in enhancing operational efficiency and innovation across banking institutions. Cloud is a big enabler when it comes to data. For example, the shift to cloud platforms has been revolutionary in resolving legacy data issues. Cloud offers the scalability and flexibility banks need to optimise their data models and make legacy data actionable, transforming data from a burdensome cost centre into a valuable asset. 

What’s key, as Xceptor’s CTO mentioned, you need engineers to have access to the data. With on-premises systems, extracting data from legacy applications often took a considerable amount of time, with engineering teams needing potentially "decades" to make data model changes or access deeply buried data. In contrast, moving data to the cloud has greatly accelerated these processes, allowing institutions to extract and monetise their data much faster due to the cloud's flexibility and advanced capabilities. This shift has made a substantial difference in how quickly firms can access and utilise their data.  

In the new age, as the speaker from Nomura mentioned, it is also the metadata that is becoming of higher importance than your data itself. Looking at metadata, tagging, and working with these continuously (as retrospective approach does not work) makes a difference.  

Trust in data, trust in AI is the new horizon  

Trust in data, alongside trust in AI, represents the next frontier in financial services, a complex, multifaceted topic on its own. Ling Ling Lo of Nomura emphasised the urgent need for organisation to start collecting "quality data" to build your own models effectively, and the importance of using large language models (LLMs) to convert your unstructured data into structured formats. The central questions remain: how can we unlock the true value of data and begin leveraging it to its fullest potential? Generative AI (GenAI) might just be the answer if we approach data governance and AI governance thoroughly. In any case, the race to master data management in capital markets in the age of AI is far from over - we're rather at the beginning of the journey.