Model Selection Stability under Limited Training Samples: An Empirical Evaluation across Multiple Datasets

ผู้แต่ง

  • Wannaporn Sarapugdi Program in Management of Digital Innovation, Faculty of Science and Technology, Rajabhat Maha Sarakham University, Maha Sarakham 44000, Thailand https://orcid.org/0009-0003-4739-0755

DOI:

https://doi.org/10.65205/jcct.2026.e4409

คำสำคัญ:

Limited Training Data, Model-Selection Stability, Repeated Sampling, Selection Regret, Statistical Comparison

บทคัดย่อ

Comparative machine-learning studies often treat the algorithm with the highest observed score as the preferred model, although that choice may depend on the observations available for training. This study examined model-selection stability under limited training samples across six public classification datasets and five algorithms: Logistic Regression, an RBF-kernel SVM, Random Forest, XGBoost, and LightGBM. For each dataset, 30% of the observations were retained as a fixed evaluation set and the remaining 70% formed a development pool. Training fractions of 10%, 20%, 40%, 60%, 80%, and 100% were evaluated in 30 runs per condition. For fractions below 100%, training composition varied across stratified subsets; the 100% condition reused the complete development pool and served as the reference condition. Macro-F1 was the primary model-selection criterion. Dominant winner rate and normalised selection entropy summarized selection stability, while selection regret compared limited-data choices with the full-data reference. Paired Wilcoxon signed-rank tests with Holm correction were used to assess support for the mean-leading algorithm within the experimental protocol. Within-fraction winner agreement was comparatively high in several datasets, but this did not always mean that the same algorithm remained dominant as more training data became available. Landsat changed from Random Forest at 10–20% to LightGBM from 40% onward, whereas Dry Bean changed from Logistic Regression at 10% to SVM-RBF at intermediate fractions and XGBoost at 80–100%. Mean selection regret declined from 0.0097 to 0.0002 for Landsat and from 0.0058 to 0.0008 for Dry Bean between the 10% and 80% conditions. In five limited-data conditions, the mean-leading algorithm was not statistically supported against all competitors after multiplicity correction. The findings show that both training-sample size and composition can change the selected model and the evidence supporting that choice. They describe model-selection stability under training-sample variation conditional on a single fixed evaluation set.

Downloads

Download data is not yet available.

เอกสารอ้างอิง

Alpaydin, E. & Kaynak, C. (1998). Optical Recognition of Handwritten Digits [Data Set]. UCI Machine Learning Repository. https://doi.org/10.24432/c50p49

Alwash, M., Al Hajj, G. S., Grytten, I., & Sandve, G. K. (2025). Meta Simulation Approach for Evaluating Machine Learning Method Selection in Data Limited Settings. Scientific Reports, 15(1), 40766. https://doi.org/10.1038/s41598-025-24627-y

Bornschein, J., Visin, F., & Osindero, S. (2020, July 13-18). Small Data, Big Decisions: Model Selection in the Small-Data Regime. The 37th International Conference on Machine Learning, 119, 1035-1044. PMLR. https://proceedings.mlr.press/v119/bornschein20a.html

Breiman, L. (2001). Random Forests. Machine Learning, 45(1), 5-32. https://doi.org/10.1023/a:1010933404324

Chen, T., & Guestrin, C. (2016, August 13-17). XGBoost: A Scalable Tree Boosting System. The 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 785-794. https://doi.org/10.1145/2939672.2939785

Cortes, C., & Vapnik, V. (1995). Support-Vector Networks. Machine Learning, 20(3), 273–297. https://doi.org/10.1007/bf00994018

Dietterich, T. G. (1998). Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms. Neural Computation, 10(7), 1895-1923. https://doi.org/10.1162/089976698300017197

Holm, S. (1979). A Simple Sequentially Rejective Multiple Test Procedure. Scandinavian Journal of Statistics, 6(2), 65-70.

Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., & Liu, T.-Y. (2017, December 4-9). LightGBM: A Highly Efficient Gradient Boosting Decision Tree. The 31st International Conference on Neural Information Processing Systems, 3149-3157. https://dl.acm.org/doi/10.5555/3294996.3295074

Lohweg, V. (2012). Banknote Authentication [Data Set]. UCI Machine Learning Repository. https://doi.org/10.24432/c55p57

Nadeau, C., & Bengio, Y. (2003). Inference for the Generalization Error. Machine Learning, 52(3), 239–281. https://doi.org/10.1023/a:1024068626366

Rajput, D., Wang, W.-J., & Chen, C.-C. (2023). Evaluation of a Decided Sample Size in Machine Learning Applications. BMC Bioinformatics, 24(1), 48. https://doi.org/10.1186/s12859-023-05156-9

Savvides, R., Mäkelä, J., & Puolamäki, K. (2023). Model Selection with Bootstrap Validation. Statistical Analysis and Data Mining: The ASA Data Science Journal, 16(2), 162–186. https://doi.org/10.1002/sam.11606

Shannon, C. E. (1948). A Mathematical Theory of Communication. Bell System Technical Journal, 27(4), 623-656. https://doi.org/10.1002/j.1538-7305.1948.tb00917.x

Silvey, S., & Liu, J. (2024). Sample Size Requirements for Popular Classification Algorithms in Tabular Clinical Data: Empirical Study. Journal of Medical Internet Research, 26, e60231. https://doi.org/10.2196/60231

Srinivasan, A. (1993). Statlog (Landsat Satellite) [Data Set]. UCI Machine Learning Repository. https://doi.org/10.24432/c55887

UCI Machine Learning Repository. (2019). Rice (Cammeo and Osmancik) [Data Set]. UCI Machine Learning Repository. https://doi.org/10.24432/C5MW4Z.

UCI Machine Learning Repository. (2020a). AI4I 2020 Predictive Maintenance Dataset [Data Set]. UCI Machine Learning Repository. https://doi.org/10.24432/c5hs5c

UCI Machine Learning Repository. (2020b). Dry Bean [Data Set]. UCI Machine Learning Repository. https://doi.org/10.24432/c50s4b

Uddin, S., & Lu, H. (2024). Dataset Meta-Level and Statistical Features Affect Machine Learning Performance. Scientific Reports, 14(1), 1670. https://doi.org/10.1038/s41598-024-51825-x

Wiesenfarth, M., Reinke, A., Landman, B. A., Eisenmann, M., Saiz, L. A., Cardoso, M. J., Maier-Hein, L., & Kopp-Schneider, A. (2021). Methods and Open-Source Toolkit for Analyzing and Visualizing Challenge Results. Scientific Reports, 11(1), 2369. https://doi.org/10.1038/s41598-021-82017-6

Wilcoxon, F. (1945). Individual Comparisons by Ranking Methods. Biometrics Bulletin, 1(6), 80-83. https://doi.org/10.2307/3001968

ดาวน์โหลด

เผยแพร่แล้ว

15-11-2026

รูปแบบการอ้างอิง

Sarapugdi, W. (2026). Model Selection Stability under Limited Training Samples: An Empirical Evaluation across Multiple Datasets. วารสารคอมพิวเตอร์และเทคโนโลยีสร้างสรรค์, 4(3), e4409. https://doi.org/10.65205/jcct.2026.e4409

Similar Articles

You may also start an advanced similarity search for this article.