Analysis and Comparison of the Impact of Statistical Feature Reduction for Classification using Machine Learning

Abstract

This study aims to analyze the impact of statistical feature reduction on classification performance and to identify the optimal hyperparameter configurations for predicting medical conditions. Three machine learning algorithms—Support Vector Machine (SVM), Decision Tree (DT) and K-Nearest Neighbors (KNN) were compared using the Healthcare Risk Factors Dataset. Two filter-based feature selection methods, the Kruskal–Wallis test (KW) and Mutual Information (MI), were employed to generate a full feature set and feature subsets at the 25th, 50th, and 75th percentiles. Model performance was evaluated using 5 repeats of 5-fold Stratified K-Fold Cross-Validation, integrated with systematic hyperparameter tuning. The experimental results demonstrated that SVM achieved the highest performance using the full feature set (C = 300), yielding an accuracy of 0.9214 ± 0.0005, precision of 0.9212 ± 0.0005, recall of 0.9214 ± 0.0005, and an F1-Score of 0.9210 ± 0.0005. This accuracy significantly outperformed DT (0.8752 ± 0.0028) and KNN (0.7916 ± 0.0015). Furthermore, reducing the feature set to the 75th percentile using the KW method maintained a high accuracy of 0.9165 ± 0.0006, representing a marginal decline of only 0.49%. These findings indicate that SVM is highly suitable for disease diagnosis within this dataset. Moreover, statistical feature reduction effectively minimizes computational complexity without a significant loss in accuracy.