The Data Detectives

Solving Inter-Enterprise Integration Issues


Bad data is usually the biggest challenge to integrating systems. It can bring not only sytems but also the whole enterprise to its knees. That is why leading per-formers like Wal-Mart and Dell are obsessed with the quality of their business information, and put so much effort into cleaning up their data.


Data: The Foundation for the Fourth Dimension

Since the dawn of business, timely and accurate data has been critical to the success of enterprises. In the last 30 years, and especially in the last 10 years, there has been an incredible explosion in the amount and variety of digitized data that is available (see Figure 1).

Figure 1 – Universe of Data Sources

The integration and standardization of all this data provides the foundation for the emerging “fourth dimension” (Read The Fourth Dimension). The vision is grand, and the availability of the data is only the first step. There is much more work to be done.

The Challenge of Interenterprise

In the 80’s and 90’s, the primary systems integration challenge was tying together all the disparate systems within the enterprise. If you think that was hard, the rise of the Virtual Corporation and the Federated business model has created even bigger challenges—tying together all of the companies in the Federation. Not only are the data quality issues exponentially larger, but there are other major challenges:

  • Huge numbers of connections
  • Huge volumes of transactions
  • Expanding list of policy compliance requirements
  • Expanding definition of the business events that we care about
  • Avalanche of sensory data
  • Substantial change management issues
  • Security challenges when connecting to thousands of trading partners and data sources
  • Lack of good clean data at trading partners and other sources

This last point is particularly pernicious.

The Dirty Data “Disease”

Text Box: Types of Data Dirt • Unnormalized data within a system: e.g. “HP” vs. “Hewlett-Packard” vs. “Hewlett Packard” vs. “Hewlett Packard, Inc.” • Unnormalized between systems: Even if you’ve cleaned up one system, it uses different convention than next system. • Bad Data: e.g. typos (“Gewlett-Packard”); or failure to follow process: checkout clerk bar-code scans one flavor of yogurt and keys X5, instead of individually bar-coding the 5 different flavors that were actually sold. Bad data is usually the biggest challenge to integrating systems. It can bring not only sytems but also the whole enterprise to its knees (see side bar “Types of Data Dirt”). When the data is wrong, late, or incomplete, mistakes are made, performance deteriorates and money is lost. That is why leading per-formers like Wal-Mart and Dell are obsessed with the quality of their business information, and put so much effort into cleaning up their data. Thankfully, there are some emerging strategies for dealing with the new realities, not only for fixing data problems, but also to help with some of the other interenterprise challenges listed above (change management, security, etc.).

Network Application Platforms

In particular, there is a good case for using a networked application provider—if you pick the right one. Examples of providers include Viewlocity, GT Nexus, RiverOne, Viacore, e2Open, One Network, Ketera, Descartes, and others. While each of these has different focus in terms of the functionality provided and the industries served, they each provide a common platform that many companies can plug into. The best of these providers bring a wealth of experience and tools. They’ve gone through the process of bringing individual trading partners on-board thousands of times. Because of this, they have amassed experience, developed methodologies, and built tools for rapid on-boarding, as well as for on-going monitoring and management of the network of partners.

The Network Effect

There can be a powerful “network effect”—tremendous economies of scale in the on-boarding process—if many of your trading partners are already connected to a particular platform provider. This means a significant portion of the work of integrating those trading partners may have already been done. The portion of work remaining to be done depends on the amount of additional per-trading-partner customization required for the data and processes being connected. For example, transportation processes and data—such as RFQ, bidding, load tendering, documentation—are fairly standardized across trading partners, with usually only modest variations. That is one of the reasons for the success of transportation platforms, such as GT Nexus. Other processes, such as design collaboration, may require a higher degree of per-trading-partner customization.

Networked Application Requirements

To provide a complete integration capability, the network application vendors must excel within these three dimensions:

  • Project/change management
  • Data quality issues
  • Monitoring

Project and Change Management

It’s human nature to resist change, and that’s equally true when you ask (or tell) a trading partner to integrate their systems with yours. To varying degrees, trading partners may resist at the strategic level (i.e. upper management is not convinced of the need) and at the execution level (any effort can be sabotaged if the individual employees feel threatened or decide they don’t want to do it). A good network application provider has gone through this process so many times, they know all the objections and how to deal with human nature. They’ve developed capabilities in:

  • Coaching and Training

    – they have startup and training down to a science. They know how to come in and explain the project, describe benefits, allay fears, describe what is expected of the partners in terms of business processes changes, due dates, data quality metrics, incorporation into SLAs, etc. Through experience and time they’ve condensed the necessary information into startup guides and mature e-learning modules, that in some cases can be done in less than one hour of the trading partner’s time. This minimizes the number of face-to-face meetings required and speeds startup.

  • Methodologies and


    – such as project planning, test plans and monitoring, project status tracking, documentation, etc.

  • Systems Integration Teams

    – they have teams that can help trading partners with the actual work. For example, GT Nexus[1] (a logistics application service provider) has their own team of EAI engineers who are able to translate trading partners’ proprietary formats into GT Nexus’ internal standards without the assistance of the partner’s IT resources.

Figure 2 – Example of Project Status Tracking

Figure 2 shows an example of a project status tracking system used to track the status of each on-boarding step and activity for each trading partner. In this way, all parties know their assignments and can see their progress.

In spite of all this vendor support, integration success depends upon a win for the trading partner. Lack of benefit or the suppliers was one of the reasons for the failure of many suppliers to adopt Internet-based industry exchanges. One major retailer uses GT Nexus to connect their suppliers. They discovered that this platform cut down on the time it took their supplier to create documentation. For example, it used to take 30-60 minutes to create an invoice packing list and book a carrier, now it takes about 10 minutes. This kind of benefit is critical to the rapid adoption of the system by trading partners.

Data Quality Issues

As mentioned above, some of the toughest trading partner integration issues are the data quality problems. Here are examples of problems that can occur with data from 3PLs (Third Party Logistics service providers):

  • Timeliness

    – e.g. ASNs (Advanced Ship Notice) that are not sent until several days after an item is shipped

  • Completeness and/or duplication

    – there may be multiple incremental ASNs, each with partial information. A shipping document might lack the bill-of-lading number or be missing the vessel/voyage/flight number.

  • Incorrect data

    – this could be wrong SKU’s, differences between the Master Bill of Lading vs. the House Bill of Lading, etc.

  • Incorrect formatting

  • Unnormalized data

    – 3PLs often use many codes for a customer, changing over time

Data quality issues are the norm, not the exception. Therefore, the networked application platform provider must provide complete capabilities to cleanse, normalize, and compensate for bad data. This should include, for example, mapping and aliases or synonym capability—by customer, by division or other arbitrary unit, and by data field—to get all data into a common internal format and representation. Also rich and intelligent capability to normalize data formats – i.e. move elements around, get rid of leading zeros, etc.

The more battle-tested the tools are, usually the better the results. Some include sophisticated fuzzy logic (e.g. these two customers have similar addresses, they may be the same customer).

Figure 3 – Example of Mapping Partner’s Location Codes into Common Code

Figure 3 shows one example of normalization. In this case, GT Nexus has their common internal code for indicating a specific location. Trading partners (suppliers, customers, carriers, 3PLs, etc.) all use different location codes. In some cases the same partner uses several different codes. This system allows complete mapping of these codes.

In another example, the bill-of-lading number (BL#) might have a different format in the 3PL’s bill-of-lading instruction vs. the Carriers bill-of-lading vs. the Carrier 315 (EDI “Ocean Shipment Status” message) – all for the same shipment. GT Nexus’ system knows how to normalize all of these into a common format.

From these examples, it is apparent these data cleansing tools are context-specific – in other words the method, rules, and specifics vary from industry to industry and application to application. There is tremendous domain knowledge, gained over a long period of experience, that goes into creating very sophisticated tools and capabilities for addressing these data issues and for the whole on-boarding process (e.g. security). Evaluation of the platform providers’ domain knowledge should be one of the key criteria in selecting the best provider for your needs.

On-going Monitoring Tools and Capabilities

Some of the most important action happens once a trading partner is connected. On-going monitoring is critical.

  • 7/24 Automated Monitoring

    The platform must provide 7/24 automated monitoring of all traffic to detect problems and if possible automatically fix them, else alert someone immediately.

  • 7/24 Manual Monitoring and Correction

    There needs to be a 7/24 human staff to monitor, repair, and reprocess messages as needed. This may include contacting the IT department of the affected trading partners to fix immediate problems

  • Continual Performance Improvement

    Longer term, recurring data quality problems must be identified and addressed in an ongoing way.

Figure 4 – Example Trading Partner Data Quality Dashboard

To illustrate the last point, systems like GT Nexus provide the ability monitor overall data quality performance of trading partners over time (correctness, timeliness, completeness, etc). When the same platform is used for A) the integration of trading partners and for B) the actual execution, then it can track the timeliness of each trading partners’ messages relative to the actual events on the ground and highlight those whose latency is out-of-tolerance.

Enabling New Business Models

Networked application platforms seem to be our best hope for tackling multi-enterprise integration and data quality challenges. By doing this well, they enable the next generation of business models to function effectively and take us to the next level of performance. An inter-connected web of 3rd party platforms, with core expertise at solving these thorny integration issues, is not only becoming more and more of a reality, it is becoming more and more of a necessity. These networked application platforms are becoming an essential part of the data universe foundation for 21st century business. As always, make sure to pick your partners well!

[1] There’s a good case study about how GT Nexus helped Crowley Maritime at /research/detail.cfm?guid=D4070690-8004-6D99-1CA6-0C876628C6AC

Scroll to Top