Information quality

Organisations have been increasingly aware of the cost of low quality information. Data reflecting poorly the status of a customer have been increasingly hurting the Businesses. Moreover the capture of information has been frequently ineffective, thus leading to poor business process performance or lost opportunities.

Many organizations are running projects to: • Standardize data coming from different systems • Cleanse data before using them for analytical purposes • harmonise their customer data and develop the customer holistic view or 360 degree view.

Methodologies

Methodologies to address the issue of information quality in a holistic approach have been proposed (see TIQM, TDQM by MIT). Other analyses focus on the databases and their quality (quality of structure, data definition, data values and how accurately they represent the real world entity types) (see Jack Olson – Data Accuracy Dimension).

Software tools

In parallel to the development of information & data quality assessment methods, a software market has been maturing to support the efforts of Businesses to analyse and improve their information – the data quality market. Software tools combining data profiling, standardizing, matching & merging & cleansing functionality have been increasingly in demand. A consolidation of this niche market took place in the recent years. A comprehensive review and functionality classification of various relevant solutions is given by Infoimpact (caution: some reviews are outdated).

Information quality cannot be an afterthought. If that happens, data quality is low and any efforts to improve it do not yield the expected outcome. It has been recognized that information quality improvement is like any other quality improvement project. You have to consider the organization, the introduction of new processes, new roles (e.g. information steward) & people involved in these quality related processes, as well as IT systems and relevant functionality. Tools are not enough to make it happen. A culture change is needed.

There is a gap between proposed information quality methodologies (even narrowed down to the data profiling methodology) and data quality tools functionality. Limitations of data quality tools functionality, vis-a-vis proposed holistic information quality methodologies, can be easily identified. For example:

metadata apply not only in databases, but also in the application and presentation layer of system. For example a business rule embedded in the application logic may aim to apply a specific domain definition (e.g. a range of values on a field). Moreover business rules are often documented in print form.
presentation quality or information usability is not assessed by data quality tools
the organizational aspects of information quality initiatives are not supported

As information flows throughout the information value chain of the Enterprise, it has to be checked at certain critical points, in order to avoid the diffusion of defective data. These critical points can be:

at the data entry or the data import from another system
at the operational systems integration layer, where data validation & transformation takes place (e.g. in an Enterprise Service Bus – ESB implementation)
at a reference system which holds master data for a set of entity types (a CRM system holds master data for the customer entity type)
at the staging area in a data warehouse infrastructure

Moreover feedback on information quality should be provided at all points at which a user retrieves and evaluates information (e.g. information retrieved by a certain business role, a report produced by the data warehouse).

Relevant trends

A growing market called CDI (customer data integration) has been maturing in order to support the increased demand for improved knowledge on the Customer.