Support Vector Machine Model Selection Using Strangeness

Model Selection is the task of choosing the best model for a particular data analysis task. It generally makes a compromise between fit with the data and the complexity of the model. Currently the most popular techniques used by practitioners are Cross-Validation (CV) and Leave-One-Out (LOO). In this study we concentrated on the Support Vector Machine (SVM)(Boser et al., 1992) model. Recently, Ozögür-Akyüz et al.(In Press), following on work by Ozögür et al.(2008), show that selecting a model whose hyperplane achieves the maximum separation from a test point obtains comparable error rates to those found by selecting the SVM model through CV. In other words, while methods such as CV involve finding one SVM model (together with its optimal parameters) that minimises the CV error, Ozögür-Akyüz et al.(In Press) keep all of the models generated during the model selection stage and make predictions according to the model whose hyperplane achieves the maximum separation from a test point. The main advantage of this approach is the computational saving when compared to CV or LOO. However, their method is only applicable to large margin classifiers like SVMs. We continue this line of research, but rather than using the distance of each test point from the hyperplane we explore the idea of using the nonconformity measure (Vovk et al., 2005; Shafer & Vovk, 2008) of a test sample to a particular label set. The nonconformity measure is a function that evaluates how ‘strange’a prediction is according to the different possibilities available. The notion of nonconformity has been proposed in the on-line learning framework of conformal …