Class Separability in Spaces Reduced By Feature Selection

We investigated the geometrical complexity of several high-dimensional, small sample classification problems and its changes due to two popular feature selection procedures, forward feature selection (FFS) and Linear Programming Support Vector Machine (LPSVM). We found that both procedures are able to transform the problems to spaces of very low dimensionality, where class separability is improved over that in the original space. The study shows that geometrical complexities are effective tools for comparing different feature selection methods in aspects relevant to classification accuracy, yet independent of particular classifier choices.