On Decompositions of Decision Function Quality Measure

• Main Author: V.M. Nedel’ko
• Format: Article
• Language: English
• Published: Irkutsk State University 2020-09-01
• Series: Известия Иркутского государственного университета: Серия "Математика"
• Subjects: machine learning; bias-variance decomposition; decision function complexity
• Online Access: http://mathizv.isu.ru/en/article/file?id=1353

A comparative analysis of two approaches to the decomposition of quality criterion of decision functions is carried out. The first approach is the bias-variance decomposition. This is the most well-known decomposition that is used in analyzing the quality of decision function construction methods, in particular for justifying some ensemble methods. This usually assumes a monotonous dependence of the bias and variance on the complexity. Recent studies show that this is not always true. The second approach (G.S. Lbov, N.G. Startseva, 1989) is a decomposition into a measure of adequacy and a measure of statistical stability (robustness). The idea of the approach is to decompose the prediction error into approximation error and statistical error. In this paper we propose a method of statistical estimation of the components of both decompositions on real data. We compare the dependencies of these components on the complexity of the decision function. Non-normalized margin is used as a general measure of complexity. The results of the study and the experiments on UCI data show significant qualitative similarities in behavior of the bias and the adequacy measure and between the variance and the statistical stability measure. At the same time, there is a fundamental difference between the considered decompositions, in particular, with increasing complexity, the measure of adequacy cannot increase, while the bias first decreases, but at high enough values of complexity usually starts to grow.