A Novel Deduplication Technique Using an Evolutionary Approach

Abstract: The process which identifies the records that refers to the same entity in data storage is known as record deduplication. UDD can effectively identify duplicates of different web databases. Initially from the non duplicate record set, the two different classifiers, a Weighted WCSS is used for deduplication. The approach joins different pieces of attribute with similarity function extracted from the data content to produce a deduplication function that able to identify presence of replicas. The main goal of this paper is to promote a method that finds a proper combination of the proper pieces of attribute with likeness function. In the existing approaches optimization is less. The proposed system has adapted the firefly algorithm for record duplication. The structure distributes many similarities with evolutionary computation techniques such as GP and SVM based approaches. The experimental result was done by comparing the proposed algorithm with other two existing approaches machine algorithm.

Keywords: Deduplication, genetic algorithm, optimization, firefly, data mining