Restoring a Culture of Data Quality
Data quality is not just an IT problem, it is a business problem. This may not be a pleasant realization for executives who have other things to think about and expect someone on their team to be taking care of such issues. But when your company is attempting to cultivate relationships with customers, shareholders, regulators, and auditors, and you cannot deliver on promises because your data does not back you up, your data problem is now everyone’s problem.
Reconciling bad data is becoming increasingly relevant as new methods for analyzing extremely large data sets are being developed to reveal patterns, trends, and insights. These methods are commonly referred to as big data analytics, and the technology is coming whether your company’s infrastructure is ready for it or not. Fortunately, organizations that prepare in advance will be well-positioned to take advantage of these advances in technology.
But what is it that qualifies data as being good or bad, and what can you as a company do to get your hands around this growing problem? There are several criteria that need to be considered but one thing is clear: cleaning your data now will save you time and money later.
What Causes Poor Data Quality
Data quality refers to the accuracy, completeness, timeliness, and consistency of the data used in your organization. There are many ways that bad data can enter your system, but in general they fall into two major categories:
Human Error
Simple human error is the source of the majority of data quality issues. A large part of human error is attributable to a lack of standardization at the data entry point, which leads to employees recording data inconsistently. This is quite common in organizations that have multiple people from multiple locations entering data.
More traditional examples of human error are caused by poor data entry. This can include misspellings, typos, and abbreviations as some of the most common data entry problems. All of these factors contribute to a lack of data conformity and completeness.
Employees are not the only ones that can enter data incorrectly. Data entered by customers, prospects, or others outside of your organization can create similar challenges. It is not uncommon for customers to misinterpret the intent of a field or enter information differently than expected.
System Inaccuracies
Limitations of technology also give rise to issues with data quality. Organizations rely on multiple systems and software platforms to run their operations, and if those systems do not integrate properly multiple versions of “truth” will create compatibility errors.
Any time you change systems and data is migrated to the new platform, there is an inherent risk of diminished data quality. Data can get lost or corrupted in the transfer. Since most system migrations rely on humans to perform the final review, this is another opportunity for additional human error.
Similarly, platform updates are another occasion that data quality problems can originate. While the original implementers of a platform typically understand how your data is structured, newer developers may lack that insight and not realize the impact of platform updates on your data.
Lastly, some software has inherent limitations due to oversights in its development. Because organizations record such a large amount of complex data, software platforms that are not reliable have increased opportunities for system errors that introduce data quality problems.
The Cost of Bad Data
Recent research has shown that organizations believe that poor data quality is responsible for an average of $15 million per year in losses. But how can this loss be related to a company’s current data structure? To put things in perspective, a useful rule to follow when assessing the impact of your data quality is the 1-10-100 rule, which was first proposed by George Labovitz and Yu Sang Chang in the early 1990s.
There are 3 phases in 1-10-100 rule, each of which explains the cost of maintaining data quality. In the “prevention” phase, $1 represents the cost to verify accurate data at the point of capture. This is the simplest and least expensive way of collecting data and validating its accuracy. This is the obviously the ideal solution for businesses as it means data was gathered correctly the first time.
In the “correction” phase, we see that the initial $1 rises exponentially to $10. This $10 represents the increased cost that an incorrect datapoint will have on your business down the line. Without an effective prevention method, dirty data can start to have a serious impact on business efficiency, and companies wanting to resolve the issue will find themselves paying much more to backtrack and correct bad data than if they would had they validated their data when it was first collected.
Lastly, in the “failure” phase, we see that the cost of bad data increases tenfold to $100. This $100 represents the amount companies will pay for doing nothing about their poor data, which leads to misinformed business decisions and lost opportunities.
The costs outlined in the 1-10-100 rule are obviously qualitative, but this rule-of-thumb is an effective representation of importance of validating your data immediately rather than gathering first then cleaning it later. The 1-10-100 is a solid representation of how much costs increase the longer you leave dirty data in your database.
Business Impacts of Bad Data
Beyond the significant value loss, bad data has numerous far-reaching impacts on an organization. With so many ways that data quality can be impacted, it is likely that your company suffers from some degree of bad data. Small issues may not affect your business drastically, but if problems are pervasive or critical, the consequences can be significant.
Decision-Making & Strategy
Your business decisions are only as good as the data on which they are based. Only good quality data can be relied upon to inform the right decisions for your business. Without supporting data your overall strategy may be based on false assumptions, leading to improper execution of strategy. A particularly dangerous aspect of poor data quality is the false sense of security it can impart. Unaddressed data errors can blind you to problems in your business, allowing those problems to grow.
Productivity
Poor data quality can significantly reduce productivity, create inefficiencies, and increase operational costs. In a 2019 survey of data scientists, 73.5% of respondents spent 25% or more of their time managing, cleaning, and/or labeling data. Given that the median salary of entry-level data scientists is $95,000, this “data janitor work” represents a significant financial outlay, and is a waste of data scientists’ skills.
Organizational Mistrust
Managing, and even simply tolerating, bad data can have a significant impact on employee morale. Employees who were hired for high-skill work are not likely to be gratified with manual data cleanup. Meanwhile, the frustration of dealing with inaccurate, incomplete, or inconsistent data makes work more difficult and less satisfying.
Further, when your data is inconsistent between systems, your company is dealing with multiple sources of “truth.” This leads to teams not agreeing on which systems are correct and reliable, making it more difficult to align employees with shared objectives.
Reputation
The damage done by bad data is not limited to internal impacts. Quality issues with customer data impacts your customers as well. Billing errors and poor customer experiences can be frustrating for customers. While the impact of these errors will be felt in your customer service department, another challenge presents itself when feedback from those frustrated customers begins to materialize.
An organization’s relationships with shareholders, regulators, and auditors is also impacted by data quality. Reliable data allows an organization to build trust with those individuals and entities that are tasked with ensuring proper governance and performance of the organization.
Developing a Data Hygiene Strategy
Data quality will not improve without a concentrated effort. The only way to avoid the damage bad data can cause is to proactively address existing errors in your data, prevent future errors from being introduced, and change your organization’s culture surrounding data.
Invest in Data Quality Management
As evidenced by the 1-10-100 rule, in order to minimize costs, organizations have to make an investment in data quality management. Depending on the extent of your data problems, you may be able to find a data quality software solution that meets your needs, or it may be necessary to outsource the work to external consultants. Data quality software can audit your database to locate issues and provide recommendations on their resolution.
Focus on Data Integration
According to a 2019 study, companies with more than 1,000 employees leverage an average of more than 200 distinct applications, with each employee accessing an average of 9.5 on a regular basis. With so many access points, companies have an immense amount of data flowing into the organization from multiple sources at any given time.
Proper data integration allows for the combination of data from these multiple sources, allowing the user to clean, reconcile, and restructure the data according to company standards. Many businesses find that using a third-party software that specializes in data integration is much more streamlined than creating an in-house solution.
Prevent Poor Data Quality
Beyond improving data quality and fixing existing errors, companies must make efforts to prevent poor data quality in the first place. Data quality software can help catch new errors as they are entered into your system, which is far more effective than locating and resolving errors later.
If your organization has not developed data entry guidelines or if existing guidelines have become outdated, it is wise to review data quality issues to see if revised rules can be created to help prevent incomplete records from being created in the future.
Change Your Data Quality Culture
For many organizations, improving data quality comes down to changing the data culture of your business. The reality is that a commitment to data quality starts with leadership. If your entire team is not bought-in on improving data quality, you are going to continue having problems. To get support from your leadership team and other stakeholders, focus on outlining the business case for data quality improvement.
Lucasys Addresses Data Deficiencies
As more and more companies recognize the importance of data quality, they are looking for software that can implement advanced solutions to meet their needs. Forward-thinking organizations recognize that reliable data impacts every business area, and those that are building new data quality initiatives look for robust tools that recognize the importance of the data lifecycle.
Lucasys software works with your existing systems to ensure data quality is preserved at the point of input. Built-in validations and data quality controls ensure that your data remains reliable over time. Lucasys professional services team members work together with you to perform a data quality audit, remediate deficiencies, and establish the systems and processes needed to maintain a culture of data quality.
To learn more about how Lucasys services and solutions empower organizations to maintain a culture of data quality, schedule a demo or contact us today!