Brain Tumor Detection Using Vision Transformer with Multi-Scale Feature Fusion

Abstract

We present a novel approach for automatic brain tumor diagnosis from MRI images using multi-scale feature fusion processes using ViT. The suggested architecture gathers global contextual information using transformer self-attention, while maintaining fine grained details, which are crucial for tumor boundary identification. In this paper we propose a novel feature fusion method that fuses the features from various transformer layers, where the model can take into account multiple levels of feature abstraction simultaneously. The results obtained by our proposed methodology using three public datasets (BTMD, BraTS 2020 and selected TCIA collections) demonstrate that our methodology outperforms the standard CNN-based methods with 97.54% accuracy, 96.9% sensitivity and 98.3% specificity. The design provides a good balance between the feature extraction capabilities and the computational requirements, making it suitable for clinical use. Our results suggest that with appropriate modifications, transformer models can significantly improve the accuracy of brain tumor diagnosis without sacrificing interpretability via attention visualization.