Finding the Best Data in the Era of Big Data

MercuryGate Blog - Data, Analytics, Metrics & KPIs Category
As much as modern transportation relies on oil to move freight, today’s logistics operations are equally dependent on data. Many experts have declared that data is ‘the oil’ that drives the new digital economy. In many ways, this analogy of data as the new oil is an apt one. But, unlike oil, the reservoir of data is expanding every day with a vast amount of it going untapped and underused. Like oil, most of the data flowing into enterprise systems is initially in very raw or ‘crude’ form – unorganized and unstructured. It’s up to each enterprise to harness the data, to clean it, and make it ready for use.
Data Accuracy Still Eludes Most Shippers
In a recently released American Shipper, just 5 percent of all shippers stated that their organizations have extremely accurate data. More surprising is that 20% of shippers have no idea how accurate their data is. Shippers are not alone in this struggle. According to a 2017 Experian report on global data management, 94 percent of US organizations are facing data governance challenges.
The truth is that accurate data is a prerequisite for generating real value through analyzing large sets of data. Big data faces three key challenges of data quality, according to Li Cai and Yangyong Zhu in their article published in the Data Science Journal.
These data quality challenges include:
  1. The diversity of data sources – There are abundant data types and complex data structures, which increases the difficulty of data integration.
  2. Data volume is tremendous – It is often very difficult to judge data quality within a reasonable amount of time.
  3. Data change very fast – In other words, the “timeliness” of data is very short, which in turn requires greater processing power from technology.
Unless addressed, these data challenges can lead to an increasing amount of poor data being fed into shipping software, which is then used for data analysis and leads to misguided decisions, delays, increased costs, and compliance and regulatory penalties.
Creating Clarity and Insights from Data Mayhem
The best transportation operations are based on precision and timing, and the same is true of the data they use to drive their decision making. While many decisions rely on gut or instinct, they need to be supported by accurate, timely, and clean data that can be trusted.
One of the first steps to take in building a data quality program is deploying a central tool to capture all the various sources of data. For transportation operations, that means investing in a transportation management system (TMS). Ideally, it also means a single TMS that pulls in transportation data from across the enterprise. The idea here is to break down siloes. The more fragmented the data sets, the more chance there is the data is faulty, incomplete, and prone to error.
With a TMS in place, the next step is to integrate other systems from across the supply chain. This might include an enterprise resource planning (ERP) system, where orders are maintained, and a warehouse management system (WMS). The goal is to have a complete view of the logistics data that can inform transportation decisions, which will reduce the touchpoints of the data and the potential for errors.
Also, it’s important that data has a purpose. Logistics leaders should identify the questions they want answered, and then build the data sources and data quality initiative to help answer those questions. This will ensure there is a focus on the data that matters most. Finally, a business intelligence tool will be needed to mine the data, translate it into insights, and deliver information via visual dashboards to drive operational improvements.
Being data-centric or data-driven is no longer a mere talking point. With so much emphasis on more-informed decision making, shippers need to take steps to make sure their data is helping them make the best decisions. Shippers are focused on increasing efficiency, reducing freight costs, and improving visibility, and the right data will help them achieve these goals. The steady flow of data will only continue to flood into every aspect of business. Transportation organizations can gain a competitive advantage by implanting the right technology and data management processes today.
Cai, L. & Zhu, Y., (2015). The Challenges of Data Quality and Data Quality Assessment in the Big Data Era. Data Science Journal. 14, p.2. DOI:
Wedel, M. & Kannan P.K., (2016). Marketing Analytics for Data-Rich Environments. Journal of Marketing: AMA/MSI. 80, p. 97–121

Request a MercuryGate Demo to Get Started

Share on email
Share on facebook
Share on twitter
Share on linkedin

Leave a Reply

Your email address will not be published. Required fields are marked *