A New Automated Big Data Partitioning Approach to Improve Condensation Methods Performance

The enormous amount of structured and unstructured data produced in many fields leads to the era of big data. These data make the existing mining algorithms ineffective to process it. Therefore, the data reduction techniques are principally utilized prior to applying data mining algorithms. The instance selection is one of the promising reduction techniques advocated to reduce the size-volume of training dataset via selecting most relevant instances. However, the traditional instance selection methods suffer from the scalability of data, due to memory limitations. Recent approaches proposed to partition the training dataset into subsets and apply instance selection methods to individual subsets. Most of these approaches are based on a random partitioning, which negatively affects the performance of the instance selection methods, especially for a high number of subsets. In this work, we propose a new partitioning approach called automated overlapped distance-based partitioning. Our approach assigns the instances to the subsets regarding the distance. The instances can be assigned to two subsets based on a defined threshold. We implement and test experimentally the proposed approach using six standard datasets and the CNN method, a standard instance-selection condensation method. The results demonstrate that our approach is better than current random approaches in terms of the reduction rate and effectiveness criteria. Moreover, our approach is able to maintain a high reduction rate and effectiveness results when the numbers of subsets is increasing.

• Publication year

  2018

• Venue

  International Computer Engineering Conference

• Publication date

  2018-12-01

• Fields of study

  Computer Science

• Identifiers
  DOI 10.1109/ICENCO.2018.8636145
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Improving instance selection methods for big data classification

  2017influential reference
• Three new instance selection methods based on local sets: A comparative study with several approaches from a bi-objective perspective

  2015cited by this paper
• Big Data: A Survey

  2014cited by this paper
• Big Data: A Revolution That Will Transform How We Live, Work, and Think

  2013cited by this paper
• Big data: The next frontier for innovation, competition, and productivity

  2011cited by this paper
• Understanding Big Data: Analytics for Enterprise Class Hadoop and Streaming Data

  2011cited by this paper
• Stratified prototype selection based on a steady-state memetic algorithm: a study of scalability

  2010influential reference
• Fast Nearest Neighbor Condensation for Large Data Sets Classification

  2007cited by this paper
• Adaptive Prototype Learning Algorithms: Theoretical and Experimental Studies

  2006cited by this paper
• Advances in Instance Selection for Instance-Based Learning Algorithms

  2002cited by this paper
• An incremental prototype set building technique

  2002cited by this paper
• Instance Selection and Construction for Data Mining

  2001cited by this paper
• Reduction Techniques for Instance-Based Learning Algorithms

  2000cited by this paper
• The Reduced Nearest Neighbor Rule

  1998cited by this paper
• The self-organizing map

  1990cited by this paper
• Two Modifications of CNN

  1976cited by this paper
• Asymptotic Properties of Nearest Neighbor Rules Using Edited Data

  1972cited by this paper
• Nearest neighbor pattern classification

  1967cited by this paper

• No citing papers are available for this paper.