An Enhanced Bayesian Optimization Method for Tuning of Hybrid and High-dimensional Hyperparameters of CNNs in Brain Tumor Detection and Classification

Abstract

Background and Objective: Bayesian Optimization (BO) is a powerful strategy for optimizing complex black-box functions and is widely used for hyperparameter tuning in machine learning. Traditional BO methods commonly use Gaussian Processes (GPs), Random Forests (RF), or Bayesian Neural Networks (BNNs) as surrogate models. However, they struggle to scale in hybrid and high-dimensional search spaces and often fail to capture the spatial hierarchies required for image-based tasks such as those handled by Convolutional Neural Networks (CNNs). This study aims to overcome these limitations by proposing an Enhanced Bayesian Optimization (EBO) framework specifically designed to optimize CNN hyperparameters for brain tumor detection and classification using MRI data. Methods: A Bayesian Convolutional Neural Network (BCNN) is introduced as a novel surrogate model to address the hybrid and high-dimensional hyperparameter search spaces of CNNs. Its performance is benchmarked against GP, RF, and BNN, each paired with five acquisition functions: Expected Improvement (EI), Upper Confidence Bound (UCB), Probability Improvement (PI), Entropy Search (ES), and Knowledge Gradient (KG). Experiments on two MRI datasets - binary (tumor vs. non-tumor) and three-class (glioma, meningioma, pituitary), show BCNN consistently outperforms other surrogates. To further improve validation accuracy, the two best acquisition functions are hybridized with Bayesian CNN to form the EBO framework. Results: The Bayesian CNN surrogate outperformed GP, RF, and BNN across acquisition functions, with ES and KG showing the best mean performance. The proposed hybrid BCNN_ES+KG (EBO) achieved the highest validation accuracies of 97.0% (Dataset D1) and 92.13% (Dataset D2), surpassing single acquisition functions. Using the optimized hyperparameters, Optimized_CNN 1 reached 98.0% accuracy and Optimized_CNN 2 achieved 95.79% accuracy, both outperforming existing state-of-the-art methods. Conclusions: The proposed EBO framework, using Bayesian CNN as a surrogate model combined with a hybrid ES+KG acquisition strategy, effectively optimizes high-dimensional CNN hyperparameters. The optimized CNNs achieved superior performance, validating the effectiveness and generalizability of EBO for brain tumor detection and classification using MRI data.