A system proposal for automated data cleaning environment

One of the great challenges to obtaining knowledge from data sources is to ensure consistency and non-duplication of stored information. Many techniques have been proposed to minimize the work cost and to allow data to be analyzed and properly corrected. However, there are still other essential asp...

Full description

Bibliographic Details
Main Authors: Carlos Roberto Valêncio, Toni Jardini, Victor Hugo Penhalves Martins, Angelo Cesar Colombini, Marcio Zamboti Fortes
Format: Article
Language:English
Published: Institute of Technology and Education Galileo da Amazônia 2020-10-01
Series:ITEGAM-JETIA
Online Access:https://itegam-jetia.org/journal/index.php/jetia/article/view/685
id doaj-d28671a35b034abdb2188caa580b77f5
record_format Article
spelling doaj-d28671a35b034abdb2188caa580b77f52020-11-25T04:03:17ZengInstitute of Technology and Education Galileo da AmazôniaITEGAM-JETIA2447-02282020-10-0162510.5935/jetia.v6i25.685A system proposal for automated data cleaning environmentCarlos Roberto Valêncio0Toni Jardini1Victor Hugo Penhalves Martins2Angelo Cesar Colombini3Marcio Zamboti Fortes4São Paulo State University - UNESP. São Jose do Rio Preto–São Paulo, BrazilSão Paulo State University - UNESP. São Jose do Rio Preto–São Paulo, BrazilSão Paulo State University - UNESP. São Jose do Rio Preto–São Paulo, BrazilFederal University of Fluminense - UFF. Niterói–Rio de Janeiro, BrazilFederal University of Fluminense - UFF. Niterói–Rio de Janeiro, Brazil One of the great challenges to obtaining knowledge from data sources is to ensure consistency and non-duplication of stored information. Many techniques have been proposed to minimize the work cost and to allow data to be analyzed and properly corrected. However, there are still other essential aspects for the success of data cleaning process that involve many technological areas: performance, semantic and autonomy of the process. Against this backdrop, we developed an automated configurable data cleaning environment based on training and physical-semantic data similarity, aiming to provide a more efficient and extensible tool for performing information correction which covers problems not yet explored such as semantic and autonomy of the cleaning implementation process. The developed work has, among its objectives, the reduction of user interaction in the process of analyzing and correcting data inconsistencies and duplications. With a properly calibrated environment, the efficiency is significant, covering approximately 90% of inconsistencies in the database, with a 0% percentage of false-positive cases. Approaches were also demonstrated to show that besides detecting and treating information inconsistencies and duplication of positive cases, they also addressed cases of detected false-positives and the negative impacts they may have on the data cleaning process, whether manual or automated, which is not yet widely discussed in literature. The most significant contribution of this work refers to the developed tool that, without user interaction, is automatically able to analyze and eliminate 90% of the inconsistencies and duplications of information contained in a database, with no occurrence of false-positives. The results of the tests proved the effectiveness of all the developed features, relevant to each module of the proposed architecture. In several scenarios the experiments demonstrated the effectiveness of the tool. https://itegam-jetia.org/journal/index.php/jetia/article/view/685
collection DOAJ
language English
format Article
sources DOAJ
author Carlos Roberto Valêncio
Toni Jardini
Victor Hugo Penhalves Martins
Angelo Cesar Colombini
Marcio Zamboti Fortes
spellingShingle Carlos Roberto Valêncio
Toni Jardini
Victor Hugo Penhalves Martins
Angelo Cesar Colombini
Marcio Zamboti Fortes
A system proposal for automated data cleaning environment
ITEGAM-JETIA
author_facet Carlos Roberto Valêncio
Toni Jardini
Victor Hugo Penhalves Martins
Angelo Cesar Colombini
Marcio Zamboti Fortes
author_sort Carlos Roberto Valêncio
title A system proposal for automated data cleaning environment
title_short A system proposal for automated data cleaning environment
title_full A system proposal for automated data cleaning environment
title_fullStr A system proposal for automated data cleaning environment
title_full_unstemmed A system proposal for automated data cleaning environment
title_sort system proposal for automated data cleaning environment
publisher Institute of Technology and Education Galileo da Amazônia
series ITEGAM-JETIA
issn 2447-0228
publishDate 2020-10-01
description One of the great challenges to obtaining knowledge from data sources is to ensure consistency and non-duplication of stored information. Many techniques have been proposed to minimize the work cost and to allow data to be analyzed and properly corrected. However, there are still other essential aspects for the success of data cleaning process that involve many technological areas: performance, semantic and autonomy of the process. Against this backdrop, we developed an automated configurable data cleaning environment based on training and physical-semantic data similarity, aiming to provide a more efficient and extensible tool for performing information correction which covers problems not yet explored such as semantic and autonomy of the cleaning implementation process. The developed work has, among its objectives, the reduction of user interaction in the process of analyzing and correcting data inconsistencies and duplications. With a properly calibrated environment, the efficiency is significant, covering approximately 90% of inconsistencies in the database, with a 0% percentage of false-positive cases. Approaches were also demonstrated to show that besides detecting and treating information inconsistencies and duplication of positive cases, they also addressed cases of detected false-positives and the negative impacts they may have on the data cleaning process, whether manual or automated, which is not yet widely discussed in literature. The most significant contribution of this work refers to the developed tool that, without user interaction, is automatically able to analyze and eliminate 90% of the inconsistencies and duplications of information contained in a database, with no occurrence of false-positives. The results of the tests proved the effectiveness of all the developed features, relevant to each module of the proposed architecture. In several scenarios the experiments demonstrated the effectiveness of the tool.
url https://itegam-jetia.org/journal/index.php/jetia/article/view/685
work_keys_str_mv AT carlosrobertovalencio asystemproposalforautomateddatacleaningenvironment
AT tonijardini asystemproposalforautomateddatacleaningenvironment
AT victorhugopenhalvesmartins asystemproposalforautomateddatacleaningenvironment
AT angelocesarcolombini asystemproposalforautomateddatacleaningenvironment
AT marciozambotifortes asystemproposalforautomateddatacleaningenvironment
AT carlosrobertovalencio systemproposalforautomateddatacleaningenvironment
AT tonijardini systemproposalforautomateddatacleaningenvironment
AT victorhugopenhalvesmartins systemproposalforautomateddatacleaningenvironment
AT angelocesarcolombini systemproposalforautomateddatacleaningenvironment
AT marciozambotifortes systemproposalforautomateddatacleaningenvironment
_version_ 1724440829562454016