How Data Lifecycle Management can help eliminate Dark Data

Data Lifecycle Management is a concept that has been around for a while and the benefits have been mentioned in detail in every article.

However, Data Lifecycle Management (DLM) has not been used to understand and eliminate dark data.  And this article attempts to spin a new view on how DLM can help in this area, with great business benefits.

But, before we start, a quick brief about dark data.

Gartner defines dark data as the information assets organizations collect, process and store during regular business activities, but generally fail to use for other purposes. Some examples of dark data are correspondence with customers (Email, chat logs, letter), social media posts, call center recordings, surveys and forms and surveillance video footage. Most organisations retain a humongous amount of dark data. In Splunk’s global research survey of more than 1,300 business and IT decision makers, 60 percent of respondents reported that half or more of their organization’s data is considered dark. A full one-third of respondents reported this amount to be 75 percent or more.

Dark data is a product of the information overload that organisations produce and source and eventually store, more so in these times of cheaper storage options and with the hope that this data will be valuable in the future. However, most organisations fail to use even a small fraction of this dark data as the metadata labels for such data are not documented making this data unretrievable. This results in accelerated storage costs and missed opportunities to create business outcomes.

Retaining Dark data for years may also contravene regulatory guidelines on data retention, leading to penalties and fines. Which is why dark data is a bane than a boon to organisations.

And this is where Data Lifecycle Management steps in. Creating a strategy and process to classify dark data helps organisations gain value from such data which yields several benefits to business.

So, how can DLM help in managing and eliminating dark data?

Data Lifecycle Management addresses the problem of dark data through a proactive approach. It helps organisations understand its information through application of policies. DLM helps with data discovery, data classification, data encryption, obfuscation and disposition which allows organisations to dispose data that has no business use.

DLM creates business value out of data and enables organisations to manage access to data changes over time. DLM creates policies and processes to attach relevancy to information and get rid of redundant and duplicate data.

With this understanding of DLM, let’s now look at the approach that needs to be taken to manage dark data so as to make it valuable to business.

The best DLM approach to deal with dark data is to find scalable ways and means to manage it and reduce the amount of unknown and unclassified data in the ecosystem by improving the efficacy of the Data Discovery/Creation and Classification processes. This also means an all-encompassing three-dimensional approach is required to manage dark data covering the following dimensions:

• People: Evaluate and make requisite changes in the data driven culture across the organization. This can range identifying new roles and responsibilities in the Data Governance organisation to implementing various methods to improve data literacy and usage, through trainings, certifications, upskilling programs for data management, storytelling sessions, community support groups, creating data champions/evangelists and more such activities. 
• Processes: Evaluate the AS-IS processes implemented as part of the DLM solution and leverage new-age structure, architecture, and automation to reinvent the processes. 
• Technology: Ensure the right fit tools and architectures are in place, which effectively support the need for growing data volumes, velocity, and variety. Automation is a valuable lever that must be used and made available as part of each DLM process.

 People

An effective data driven culture is an essential prerequisite to manage dark date wherein its essential to:

• Establish a responsibility/accountability matrix to clearly designate stakeholders – their roles and responsibilities – this should be consensus driven and have organization wide acceptability. KRA’s need to be in place for effective measurement and a robust rewards and recognition program to boost interest and accountability.
• Conceptualize and implement a process for regular deletion/purging of unused/non reliable data, in the process also achieve elimination of data redundancy.
•  As part of the data services organization, an empowered and focused cross functional team should work to arrest any proliferation of dark data.
•  

Processes

• Ubiquitously institutionalize DLM based on Standard operating Proced8ures (SOPs) and principles wherein the DLM team should focus on:
• Clear policies and processes across the data lifecycle – from data capture to data retire/deletion. A scalable and robust Data Classification process (by type, sensitivity, format and storage)
• Implement tools driven workflows on data consumption and its audit - preventing under usage of captured and stored data.
• Set up review boards/steering committees for regular review of data usage patterns, user feedback and identify data sets which are redundant – fit for deletion/purge.

Most of these focus areas will be part of the Key Result Areas (KRAs) of a mature data governance program.

Technology

Ensuring that the right set of tools and architectures are weaved in the data landscape can aid the DLM initiative to effectively implement the goals and objectives for minimizing if not eliminating dark data. The enablers here are:

• Deploy Best in class but right-fit architectures and associated data management tools – eliminate silos and redundancies which can lead to dark data creation.
• Invest in Generative AI & ML solutions that have the capability to identify and tag dark data and suggest appropriate remedial actions.
• Improve data governance and DLM control aided by appropriate set of tools.

The benefits of using DLM to eliminate dark data cannot be stressed enough. Right from creating business benefits, with positive impact on key balance sheet KPIs like revenue, growth, costs, and others, leveraging dark data helps organisations forge into the future through innovation, hyper personalization, customer delight and regulatory compliance. It’s a very strong lever in an organisation’s growth and innovation journey. Dark data turns information into an asset and it’s a critical lever for the future.

This article has been co-authored by Shirish Kulkarni.