Good data preparation is a key prerequisite to successful data mining. Conventional wisdom suggests that data preparation takes about 60 to 80% of the time involved in a data mining exercise. There have been good reviews of the problems associated with data preparation. However the data preprocessing is a crucial step used for variety of data warehousing and mining. Real world data is noisy and can often suffer from corruptions or incomplete values that may impact the models created from the data. Accuracy of any mining algorithm greatly depends on the input datasets. In this paper we describe a novel idea of predicting the missing values in the dataset by a well known principle of Maximum likelihood EM (Expectation Maximization). After doing implementing and applying the EM filter, the dataset is completed with the estimated values, based on the well known principle of expected maximization of attribute instance. We demonstrate the efficacy of the approach on real data sets as a preprocessing step.

1. Sameer S. Prabhune, Dr. S. R. Sathe "Reconstruction of a Complete Dataset from an IncompleteDataset by Expectation Maximization Technique", International Journal of Computer Science and Network Security, VOL. 10 No. 11, November 2010
2. Data Preparation for Data Mining, D Pyle, 1999, Morgan Kaufmann Inc. , ISBN 1-55860-529-0.
3. S. Parthsarthy and C. C. Aggarwal, "On the Use of Conceptual Reconstruction for Mining Massively Incomplete Data Sets, "IEEE Trans. Knowledge and Data Eng. , pp. 1512-1521,2003.
4. J. Quinlan, C4. 5: Programs for Machine Learning, San Mateo, Calif. : Morgan Kaufmann, 1993.
5. S. Mehta,S. Parthsarthy and H. Yang " Toward Unsupervised correlation preserving discretization", IEEE Trans. Knowledge and Data Eng. ,pp 1174- 1185 ,2005.
6. Ian H. Witten and Eibe Frank , "Data Mining: Practical Machine Learning Tools and Techniques" Second Edition, Morgan Kaufmann Publishers. ISBN:81-312-0050-
7. R. Little, D. Rubin. Statastical Analysis with Missing Data. Ch. 8 , pp 164-172,Wiley Series in Probability and Statistics, 2002.
8. UCI Machine Learning Repository,
9. Jiawei Han and Micheline Kamber "Data Mining Concepts and techniques "
10. M. richardson and P. Domingos. Mining Knowledge –sharing sites for viral marketing.
11. Data Mining Leading Edge: Insurance & Banking, D Romano in Proceedings of Knowledge Discovery and Data Mining, Unicom, Brunel University, 1997.
12. Real-world Data is Dirty: Data Cleansing and the Merge/Purge Problem, M A Hernandez and S J Stolfo, Data Mining and Knowledge Discovery 2,p1-31, 1998.

Data Mining Data Preprocessing Missing Data