Beyond the Monolithic Era – Data as a Product

While organisations have typically survived in data silos for years and with companies having access to more data than ever before in the digital world, data is now recognized as a vital enterprise asset. Thus, there is a scramble to give data its due importance and thus the world has witnessed a surge in Data Management programs, some small and others massive, and many disruptive as well. And in pursuit of automated intelligent decision making and creating a robust ecosystem for facing continuously evolving regulatory laws, history has seen a transition from Data warehouse to the more recent cloud-based Data Lakes, the motive for all of which is to empower business users with impactful trend analysis, improve their data usage experience and enable quick business decisions based on data.

A paradigm shift has thus occurred in the approach to the way data is managed with Data Products and Data as a Product emerging as new-age terms in the world of Data Management as it leads to meaningful outcomes like increased data usage through democratization, encouraging self-service in data usage, improved data quality to enable accurate business outcomes and decisions.

 However, like most data programs, there are a few pre-requisites for this new concept to make inroads and be successful. The most important aspect of any data program is Executive Sponsorship. Sustained and a well thought through business sponsorship is the key to success of a Data Management program; and ensuring executive support necessitates Periodic Briefing on the program and challenges, an involved participation in Change Management and demonstrating the need for a broad and deep consensus for data improvementa a top down push for consensus across all vested teams.

Periodic Briefing - This implies continuous monitoring of key business metrics and communication back to the concerned authorities in a bid to demonstrate success, show value and in essence garner continued support from executives and staff. Metrics help to measure and track the effectiveness of the program and can lead to providing the support that is vital to the programs adoption, scalability and success However, this is more easily said than done. An essential step to take care is what is to be measured, how is it to be tracked, what are the dependencies, who does it need to be reported to, what is the impact and so on.

And this is where a lot of the large Data Management programs go for a toss. Often in programs spanning multiple departments and functions, the ability to focus on key metrics, as it pertains to a particular business functions or area, is lost. And this is primarily due to a lack of ownership and drive, since what we have are domain owners who are not necessarily entrenched or well versed in business affairs and just wear the data hat independent of business workings.

We have personally witnessed large enterprise scale Data Governance programs aligned to enterprise overhauls, but which haven’t seen the light of the day due to the sheer massive scale and lack of business acumen.

Change Management - Change management helps solve business issues by aligning both people and processes to strategic initiatives that help an organization achieve its business vision. Data governance is the oversight of the enterprise data which drives the business. It encompasses the business framework, the processes and policies surrounding the data, and the day-to-day management of that data. When an organization recognizes that it has data quality issues and wants to establish data governance, they struggle to kick start the initiative since employees are reluctant to participate. and as much as one may try to apply the tenets of Organizational Change Management, often they fail and the reason here is very simple. Attempting to get a buy-in across the organization and multitude of people is a mammoth task; the challenges, constraints, culture also is seen as vastly varying across departments and functions. Thus, one is left with an overwhelmingly stupendous task of education and consensus building but due to the tweaks that are required in the approach, it gets into a viscous circle.

But there are options – Small, Controlled Projects

Experience says that projects that are smaller in size, that can be controlled easily and that are considered viable for realizing value earlier are the ones that need to be planned and executed. There could be multiple such small projects all leading up to a larger data transformation program at the enterprise level. The success of each of these smaller projects provide valuable lessons to other project teams and with a well-integrated plan they will all come together cohesively.

And it is now time to  move from a monolithic data platform where the hosting and ownership of multi domains lies with a single platform towards a domain-oriented data platform and ownership. This is quite similar to the microservices architecture systems that were decomposed into distributed services, built around business domain capabilities. One needs to think of a specific business function owning and serving their data sets for access and consumption within the enterprise[SK1] 

Needless to say, these in all probability could be source centric. Take the case of a customer data domain and let’s hypothetically consider two functions that primarily deal with this domain.

1. Sales function through a CRM 
2. The after delivery and maintenance through  an ERP

Going by our rationale, it should ideally be in the realms of both the above functions to deal with the sanctity of customer data. So, here’s what a logical and practical approach would be, identify all Governance roles like Data Owners, Stewards, and Data Architects within the sales function who will own the data, identify and address all anomalies in the data through a defined set of processes and technologies. This will ensure that the data emanating out of the Sales function is sacrosanct in all respects since it is coming from the people who understand it best. Any further enrichment can then be carried out by the ERP functions.

This approach 1 resonates with Gartner’s predicament which says, ‘Big data is set to move out of the spotlight in the near future, with 70 per cent of all organisations to take up small and wide data instead by 2025’.

Thus, the essence is to take up a smaller subset of data, which can be managed well and offer useful insights. And this is where Data as a product concept steps in.

Data As a Product –

Assigning data ownership and strategy to business domain owners raises concerns around accessibility, usability, security, and harmonization of data and that’s where the application of product thinking comes into picture and Data as a Product has become the new buzzword in most industries today.

So, what exactly is Data as a product? Well, it’s about applying product thinking to datasets available in an organization. Product thinking is where a unit or a business function considers their data assets as their products and the rest of the enterprise as their customers. And that implies ensuring that the data is discoverable, secure, explorable, trustworthy etc. And thus comes the concept of Data as a Product (Daap) , the consequence of applying product thinking to data assets.

There are a few core principles that apply to a DaaP concept:

Reusability - By treating your data as a product, you will be creating a solid foundation of core datasets that are used across the organization.  These datasets can be reused multiple times for ad hoc aggregations in different teams –actually a high adoption of these datasets is a sign of maturity and can be a key indicator that can be used to measure the success of transformation programs.

Accessibility - DaaP ensures easy accessibility of the data and with sufficient metadata surrounding it, where business is empowered to take the required decisions ·      

Self-Describing - With a fair number of metadata and standardized nomenclatures, the datasets become inter-operable. ·     

Trustworthy - By far the largest characteristic out of this concept, checking data quality regularly and automatically is a must to fulfil the trustworthy characteristic of data as a product. And owners of the datasets need to act in accordance with the results of the checks to increase the trustworthiness of the data

While Daap is a new concept, it is fast gaining momentum in the data management world, especially due to its various benefits like empowering data users by reducing complexity out of data as data sets are broken down into bite sized hunks that can be easily managed and governed, improving data literacy in the organization and also improving the quality of the data. Organizations are looking towards bringing in this concept in their strategy and have embarked on the journey of DaaP.

 In our next article we will delve deeper into the concept of Data as a Product and how we can apply the concept of DaaP to arrive at a Data Mesh architecture, a distributed data architecture, under centralized governance and standardization for interoperability, enabled by a shared and harmonized self-serve data infrastructure.

Authors

Parvathy Menon & Sushama Diwekar