Hybrid AI-Driven Lung Cancer Prediction with Transparent Feature Interpretation Using LIME and SHAP

Abstract

Abstract— On the other hand, lung cancer remains a global health problem, underscoring the crucial role of intelligent predictive analytics in rapidly and accurately identifying the disease at its early stages. This study aims to present an all-embracing approach to predict lung cancer reliably for which we use advanced feature engineering along with model optimisation and explainable learning methodologies. The Lung Cancer Risk Dataset is obtained from Kaggle and also is free to access. The data is preprocessed by removing duplicates, applying label encoding and partitioning the dataset. To select the most discriminative qualities RFE with SVM is used for feature selection. The performance of various ML algorithms such as LR, Gaussian Naive Bayes, SVM, MLP, KNN, DT, RF, and XGBoost are tested based on important classification measures.To get a better power of prediction this is further optimised by the use of Nelder-Mead algorithm with XGBoost. From the experimental results, it is observed that the accuracy of the optimised XGBoost model is 100% which is higher than all the individual models. All of the ensemble learning methods are resilient, as demonstrated by the accuracy of 100% for a Voting Classifier, an ensemble of Gradient Boosting, XGBoost, LightGBM and CatBoost. LIME and SHAP can help to improve model interpretability, by providing insight into the contribution of essential features to ensure reliability in the clinical decision support.