A system proposal for automated data cleaning environment
One of the great challenges to obtaining knowledge from data sources is to ensure consistency and non-duplication of stored information. Many techniques have been proposed to minimize the work cost and to allow data to be analyzed and properly corrected. However, there are still other essential asp...
Main Authors: | , , , , |
---|---|
Format: | Article |
Language: | English |
Published: |
Institute of Technology and Education Galileo da Amazônia
2020-10-01
|
Series: | ITEGAM-JETIA |
Online Access: | https://itegam-jetia.org/journal/index.php/jetia/article/view/685 |
id |
doaj-d28671a35b034abdb2188caa580b77f5 |
---|---|
record_format |
Article |
spelling |
doaj-d28671a35b034abdb2188caa580b77f52020-11-25T04:03:17ZengInstitute of Technology and Education Galileo da AmazôniaITEGAM-JETIA2447-02282020-10-0162510.5935/jetia.v6i25.685A system proposal for automated data cleaning environmentCarlos Roberto Valêncio0Toni Jardini1Victor Hugo Penhalves Martins2Angelo Cesar Colombini3Marcio Zamboti Fortes4São Paulo State University - UNESP. São Jose do Rio Preto–São Paulo, BrazilSão Paulo State University - UNESP. São Jose do Rio Preto–São Paulo, BrazilSão Paulo State University - UNESP. São Jose do Rio Preto–São Paulo, BrazilFederal University of Fluminense - UFF. Niterói–Rio de Janeiro, BrazilFederal University of Fluminense - UFF. Niterói–Rio de Janeiro, Brazil One of the great challenges to obtaining knowledge from data sources is to ensure consistency and non-duplication of stored information. Many techniques have been proposed to minimize the work cost and to allow data to be analyzed and properly corrected. However, there are still other essential aspects for the success of data cleaning process that involve many technological areas: performance, semantic and autonomy of the process. Against this backdrop, we developed an automated configurable data cleaning environment based on training and physical-semantic data similarity, aiming to provide a more efficient and extensible tool for performing information correction which covers problems not yet explored such as semantic and autonomy of the cleaning implementation process. The developed work has, among its objectives, the reduction of user interaction in the process of analyzing and correcting data inconsistencies and duplications. With a properly calibrated environment, the efficiency is significant, covering approximately 90% of inconsistencies in the database, with a 0% percentage of false-positive cases. Approaches were also demonstrated to show that besides detecting and treating information inconsistencies and duplication of positive cases, they also addressed cases of detected false-positives and the negative impacts they may have on the data cleaning process, whether manual or automated, which is not yet widely discussed in literature. The most significant contribution of this work refers to the developed tool that, without user interaction, is automatically able to analyze and eliminate 90% of the inconsistencies and duplications of information contained in a database, with no occurrence of false-positives. The results of the tests proved the effectiveness of all the developed features, relevant to each module of the proposed architecture. In several scenarios the experiments demonstrated the effectiveness of the tool. https://itegam-jetia.org/journal/index.php/jetia/article/view/685 |
collection |
DOAJ |
language |
English |
format |
Article |
sources |
DOAJ |
author |
Carlos Roberto Valêncio Toni Jardini Victor Hugo Penhalves Martins Angelo Cesar Colombini Marcio Zamboti Fortes |
spellingShingle |
Carlos Roberto Valêncio Toni Jardini Victor Hugo Penhalves Martins Angelo Cesar Colombini Marcio Zamboti Fortes A system proposal for automated data cleaning environment ITEGAM-JETIA |
author_facet |
Carlos Roberto Valêncio Toni Jardini Victor Hugo Penhalves Martins Angelo Cesar Colombini Marcio Zamboti Fortes |
author_sort |
Carlos Roberto Valêncio |
title |
A system proposal for automated data cleaning environment |
title_short |
A system proposal for automated data cleaning environment |
title_full |
A system proposal for automated data cleaning environment |
title_fullStr |
A system proposal for automated data cleaning environment |
title_full_unstemmed |
A system proposal for automated data cleaning environment |
title_sort |
system proposal for automated data cleaning environment |
publisher |
Institute of Technology and Education Galileo da Amazônia |
series |
ITEGAM-JETIA |
issn |
2447-0228 |
publishDate |
2020-10-01 |
description |
One of the great challenges to obtaining knowledge from data sources is to ensure consistency and non-duplication of stored information. Many techniques have been proposed to minimize the work cost and to allow data to be analyzed and properly corrected. However, there are still other essential aspects for the success of data cleaning process that involve many technological areas: performance, semantic and autonomy of the process. Against this backdrop, we developed an automated configurable data cleaning environment based on training and physical-semantic data similarity, aiming to provide a more efficient and extensible tool for performing information correction which covers problems not yet explored such as semantic and autonomy of the cleaning implementation process. The developed work has, among its objectives, the reduction of user interaction in the process of analyzing and correcting data inconsistencies and duplications. With a properly calibrated environment, the efficiency is significant, covering approximately 90% of inconsistencies in the database, with a 0% percentage of false-positive cases. Approaches were also demonstrated to show that besides detecting and treating information inconsistencies and duplication of positive cases, they also addressed cases of detected false-positives and the negative impacts they may have on the data cleaning process, whether manual or automated, which is not yet widely discussed in literature. The most significant contribution of this work refers to the developed tool that, without user interaction, is automatically able to analyze and eliminate 90% of the inconsistencies and duplications of information contained in a database, with no occurrence of false-positives. The results of the tests proved the effectiveness of all the developed features, relevant to each module of the proposed architecture. In several scenarios the experiments demonstrated the effectiveness of the tool.
|
url |
https://itegam-jetia.org/journal/index.php/jetia/article/view/685 |
work_keys_str_mv |
AT carlosrobertovalencio asystemproposalforautomateddatacleaningenvironment AT tonijardini asystemproposalforautomateddatacleaningenvironment AT victorhugopenhalvesmartins asystemproposalforautomateddatacleaningenvironment AT angelocesarcolombini asystemproposalforautomateddatacleaningenvironment AT marciozambotifortes asystemproposalforautomateddatacleaningenvironment AT carlosrobertovalencio systemproposalforautomateddatacleaningenvironment AT tonijardini systemproposalforautomateddatacleaningenvironment AT victorhugopenhalvesmartins systemproposalforautomateddatacleaningenvironment AT angelocesarcolombini systemproposalforautomateddatacleaningenvironment AT marciozambotifortes systemproposalforautomateddatacleaningenvironment |
_version_ |
1724440829562454016 |