A study on the selection error rate of classification algorithms evaluated by k-fold cross validation.

• Main Authors: Chiao-YingLin, 林巧盈
• Other Authors: Tzu-Tsung Wong
• Format: Others
• Language: zh-TW
• Published: 2014
• Online Access: http://ndltd.ncl.edu.tw/handle/23699989925707105417

碩士 === 國立成功大學 === 資訊管理研究所 === 102 === The performance of a classification algorithm is generally evaluated by K-fold cross validation to find the one that has the highest accuracy. Then the model induced from all available data by the best classification algorithm, called full sample model, is used for prediction and interpretation. Since there are no extra data to evaluate the full sample model resulting from the best algorithm, its prediction accuracy can be less than the accuracy of the full sample model induced by the other classification algorithm, and this is called a selection error. This study designs an experiment to calculate and estimate the selection error rate, and attempts to propose a new model for reducing selection error rate. The classification algorithms considered in this study are decision tree, naïve Bayesian classifier, logistic regression, and support vector machine. The experimental results on 30 data sets show that the actual and estimated selection error rates can be greatly different in several cases. The new model that has the median accuracy can reduce the selection error rate without sacrificing the prediction accuracy.