Critical Evaluation of Validation Rules Automated Extraction from Data

• Main Author: David Pejcoch
• Format: Article
• Language: English
• Published: Czech Society of Systems Integration 2014-10-01
• Series: Journal of Systems Integration
• Subjects: Data Quality; Data Validation; validation rules; Canonical Data Model; 4FT-Miner; GUHA; Drools; business rules
• Online Access: http://www.si-journal.org/index.php/JSI/article/viewFile/212/162

The goal of this article is to critically evaluate a possibility of automatic extraction of such kind of rules which could be later used within a Data Quality Management process for validation of records newly incoming to Information System. For practical demonstration the 4FT-Miner procedure implemented in LISpMiner System was chosen. A motivation for this task is the potential simplification of projects focused on Data Quality Management. Initially, this article is going to critically evaluate a possibility of fully automated extraction with the aim to identify strengths and weaknesses of this approach in comparison to its alternative, when at least some a priori knowledge is available. As a result of practical implementation, this article provides design of recommended process which would be used as a guideline for future projects. Also the question of how to store and maintain extracted rules and how to integrate them with existing tools supporting Data Quality Management is discussed