Today’s Abundance of Data Can Potentially be Leveraged for Better Decision-making
The world of data has exploded in recent decades – both in terms of the quantity of data inundating organizations as well as its complexity. Using these data for analyses and in service of data-informed business questions is necessary to accelerate efficiency and keep up with fast-paced and dynamic supply networks. In this new reality, traditional, decentralized data management strategies have become outdated. At the same time, when accessible, it is clear that data-informed decisions enhance value and provide a high return on ROI. While digitalization of supply chains enhances opportunities for business intelligence to valuably drive decision-making, achieving those ends requires a modern data architecture and robust data management processes. Part One of this series here describes the problems that modern enterprises face to effectively and securely store and manage large amounts of data in service of all the advantages delivered by analytics-driven business insights.[1] These challenges set the stage for the benefits provided by democratized, self-service analytics platforms.
Modern Data Architecture is Needed to Realize the Potential of
Self-Service Data-driven Decision-making
The core of organizational effectiveness,[2] decision-making requires supply chain practitioners to be well informed. Business decisions made based on the best available information can significantly improve company and network performance. To that end, data analytics and business intelligence provide clarity around ambiguous inputs and market drivers to give decision-makers context within which questions at hand are situated. The ROI on data-driven decision-making and businesses is evident[3], but common challenges inherent in traditional data storage and management among supply chain organizations necessitate modern data architectures and data management processes. Accessing, owning, and creating their own BI metrics and reports is transformative for many business stakeholders and an effective approach to disseminating informed decision-making throughout a company. A thoughtful, centralized data architecture combined with democratized, self-service solutions provides immense value. A broad expanse of users across companies are able to use data insights and predictions to valuably inform business processes and decisions, advance products, and ultimately, deliver better and more efficient service to customers.
Technological development and resulting challenges to legacy data management strategies are the catalysts for modernizing approaches to data architecture. Simply put, with a modern data architecture and self-service model, the power of data becomes more prevalent, relevant, and valuable. As non-technical business users are increasingly data literate and data dependent, they need accurate, fast, and easy access to metrics and visualizations to take informed actions and gain advantages in their industries.
Supply Chain Management Needs to Integrate ‘Mountains’ of Data from Many Sources
Companies within supply chain networks not only generate mountains of data themselves through their own internal systems (e.g., finance, human resources, operations, and sales), they are also continuously integrating data pulled from a multitude of external systems, including suppliers’, customers’, and third-party providers’ data/systems and public data sources. The below image is a visual representation of the many possible sources of data (focused on logistics data in this particular diagram) that a data-driven enterprise can incorporate to make the best possible decisions to add value and maximize ROI.
Supply chain management, by nature, is a discipline steeped in interdependence among functions within the enterprise and across a multitude of companies. Data solutions attempt to integrate data from many different sources in data warehousing projects. Effective business intelligence requires coordination of internal sources and streamlined integration with network/trading partners’ data as well.
Challenges Created by Decentralized Data Storage
The sheer volume of data available to an enterprise and diversity of sources often lead to decentralized storage, data formats, architectures, and applications. Users, addressing various problems as they arise, create spreadsheets of their own pivot tables, descriptive statistics, and departmental metrics and reports. Often saved on local drives, these documents are snapshots of data tables, creating multiple copies of information. As data is refreshed from the plethora of sources feeding into the system, these decentralized spreadsheets and reports are typically not automatically updated in the various places individuals have them stored. It’s a familiar challenge to find data inconsistencies throughout organizations when copies of data files are not systematically updated concurrently and kept in sync.
Similarly, dispersed reports throughout an organization cannot be monitored for data quality, accuracy of calculations, or security. It is easy to see how data stewardship and governance issues can arise in decentralized data storage schemas. Clients or partners can be sent reports that are not coordinated and validated by consistent governance protocols. Users could be making flawed strategy decisions based on outdated, inconsistent, and invalid data.
Data Entropy Impedes the Realization of Value from the Mountains of Data Available
The corollary to this phenomenon in the physical world, entropy, is a fitting metaphor. Entropy is “a measure of the disorder of a system… The more disordered a system and higher the entropy, the less of a system’s energy is available to do work.”[4] Systems’ state of order decreases over time and it requires more energy to restore order than to increase disorder. This applies to data as well. Disordered data throughout an organization and its supply chain network limit the value that the ever-increasing quantity of data can add to the system. All the time and energy required to deal with data entropy is not available to solve real business problems.
The format of the data itself is also relevant to how it can be stored, and the challenges that companies face in ingesting it so that it can be used effectively. We have previously discussed data structures in ChainLink’s 2019 article, It’s All About the Data[5]. Databases with tables designed of columns and rows are primarily configured to house structured data. Unstructured and semi-structured data are, by nature, more decentralized and more challenging to digitize and manage. Digitization of unstructured and semi-structured data is a substantial undertaking, adding complexity and additional processing is necessary before it can provide value and insights.
Decentralized, disordered data storage is an inefficient approach to
fostering an analytics advantage within a company or network.
Decentralized, disordered data storage limits a company or network’s opportunities to realize an analytics advantage. Establishing a single source of truth reduces these mistakes by reducing disparate and redundant reports throughout organizations and eventually, eliminates the need for teams to download spreadsheets and create their own analyses without common governance standards.
Siloed Data and Siloed Functions/Activities Inhibit Collaboration
The existence of ‘silos’ is a common phenomenon referenced within many organizations. Operating in silos refers to the independent, inadequately coordinated activities of various teams and domains within enterprises and across supply networks. Though not as fundamental as a law of thermodynamics, the proclivity of teams to get focused on their own departmental functions is very natural and is often reinforced by functionally-focused metrics/KPIs to measure success and award compensation. Siloed activities can stifle collaboration and cross-functionality, ultimately limiting efficiency and the ability to deliver value to customers and partners.[6] The right incentives and organizational dynamics are key to breaking down silos. But providing the right tools and accurate and secure data is also important. Overcoming identification with and focus on single-function goals above the larger, long-term vision requires incorporating context. Teams need tools and data to gain that context; a broader view across the enterprise and supply chain.
Legacy data architectures do not facilitate open access to data resources across teams as data management is similarly susceptible to these segmented processes. Users often do not have access to the data from other departments that they would need to draw the most valuable insights. Even if they do get access from another department, it is commonly not integrated with their own data and analytics or with data from other departments and external data. When data storage and analytics, reporting, and visualizations are segregated, or their functions kept isolated from business decision-makers within an organization, departmental goals can supersede those of the company and limit collaborative capacity and application of context to broader, cross-domain questions.
Shortage of Analytical Expertise Often Limits an Organization’s Insights and Decision-making Capacity
Many non-technical supply chain operators are not familiar with complex data analyses and tools. Data querying, extraction, and report-building for non-IT users can be tedious and difficult, if not, impossible. In some environments, it requires being able to program in statistical/analytical programming languages, such as Python or SQL. Furthermore, many data users want to access and apply metrics to solve business problems – not build reports. In legacy data storage structures and management solutions, data engineers and other technical specialists are relegated to fielding data requests from teams enterprise-wide. This bottleneck is extremely inefficient. Exacerbating this challenge is the exponential increase in data in recent decades produced by the enterprise itself, as well as data absorbed by the organization, and the increasing appetite for access to that data across the organization.
The value of the insights drawn from business intelligence is limited to the workload capacity of technical users to provide departments data and reports. Furthermore, to a significant extent, the ability of the business to draw valuable, data-informed insights relies on IT teams, as non-SMEs, to interpret business metrics. Users’ responsiveness and flexibility to act on the problems at hand are constrained by the IT bottleneck. Providing data access to general business users fosters a sense of ownership and increases transparency. In today’s world of data, non-technical employees are familiar with health data on their watches and checking the analytics on their social media and LinkedIn. The wider culture is increasingly data-oriented and supply chain organizations are well-served by leveraging this orientation throughout their enterprises.
Self-service and Democratization, Enabled by a Modern Data Architecture, Address Many Common Data Challenges
The challenges described through Part One of this series are multi-dimensional, and each may independently have multiple solutions. Because analytics in the hands of business users is so valuable, they will not stop creating their own spreadsheets until they are given tools by which they can accomplish most of the things they are achieving within those spreadsheets. Self-service analytics and data democratization transformations can provide these tools, addressing several of these challenges, and providing opportunities to optimize governance standards and cultivate data literacy and a data-driven culture within an enterprise or supply network within a modern data architecture.
Self-service analytics allows for non-technical users to access data from a centralized source of truth to more quickly, easily, and accurately serve the business and clients. AI and natural language processing are accelerating companies’ transformation to data architectures that facilitate these more democratized approaches to analytics and in many cases, users do not need to have any coding skills to access the information they need without putting in IT tickets and waiting their turn.
Data governance is at the heart of digital transformation issues as it relates to data access within and among supply chain network organizations. Democratization introduces the ability for more users to access and customize information and analytics. Guardrails and guidelines on access to sensitive data are key. Supporting users with training and resources in employing a self-service application and fostering a data-driven culture are also critical. Stay tuned for Part Two of this series, exploring how self-service approaches provide solutions to the above challenges of legacy data architectures. The next article will discuss how the realms of governance and security, as well as data literacy and culture in a democratized data system, are critical to consider in successfully democratizing data to derive valuable insights from the plethora of data produced by supply chain networks.
[1] Analytics Advantage – Part 1A: Democratization: What is the Analytics Advantage and What Keeps Companies From Realizing It? ChainLink Research. 2021. — Return to article text above
[2] Steps to Effective Decision-Making in Organizations. Gallup. 2020. — Return to article text above
[3] A fascinating behind-the-scenes look at data analytics. The Center of Applied Data Science. 2022. — Return to article text above
[4] Entropy. Openstax.org. Rice University. 2023. — Return to article text above
[5] It’s All About the Data. ChainLink Research. 2019. — Return to article text above
[6] Sleep, Stefan & Gala, Prachi & Harrison, Dana E., 2023. “Removing silos to enable data-driven decisions: The importance of marketing and IT knowledge, cooperation, and information quality,” Journal of Business Research, Elsevier, vol. 156(C). — Return to article text above