A Tunable Loss Function for Robust Classification: Calibration, Landscape, and Generalization

• Main Authors: Cava, J.K (Author), Dasarathy, G. (Author), Diaz, M. (Author), Kairouz, P. (Author), Sankar, L. (Author), Sypherd, T. (Author)
• Format: Article
• Language: English
• Published: Institute of Electrical and Electronics Engineers Inc. 2022
• Subjects: α α-loss; Arimoto conditional entropy; Calibration; Classification (of information); Classification algorithm; Classification algorithms; classification-calibration; Classification-calibration; Conditional entropy; Entropy; Generalisation; generalization; Logistics; -loss; Neural networks; Noise measurement; Noise measurements; Optimisations; Optimization; Privacy; Quasi convexity; robustness; Robustness; strictly local quasi-convexity; Strictly local quasi-convexity
• Online Access: View Fulltext in Publisher

 We introduce a tunable loss function called α-loss, parameterized by α ∈ (0,∞], which interpolates between the exponential loss (α = 1/2), the log-loss (α = 1), and the 0-1 loss (α = ∞), for the machine learning setting of classification. Theoretically, we illustrate a fundamental connection between α-loss and Arimoto conditional entropy, verify the classificationcalibration of α-loss in order to demonstrate asymptotic optimality via Rademacher complexity generalization techniques, and build-upon a notion called strictly local quasi-convexity in order to quantitatively characterize the optimization landscape of α-loss. Practically, we perform class imbalance, robustness, and classification experiments on benchmark image datasets using convolutional-neural-networks. Our main practical conclusion is that certain tasks may benefit from tuning α-loss away from logloss (α = 1), and to this end we provide simple heuristics for the practitioner. In particular, navigating the α hyperparameter can readily provide superior model robustness to label flips (α > 1) and sensitivity to imbalanced classes (α < 1). IEEE