A Transparent Multilayer Ensemble Framework for Cloud Job Failure Prediction Using Explainable AI

Abstract

The efficient utilisation of resources, maximum performance and enhanced fault tolerance in modern cloud data centres require accurate failure prediction of jobs. In this paper, we propose to improve the multilayer ensemble approach to predict cloud task failures using the Google Cluster 2019 dataset. The framework is the combination of several categorisation techniques such as DT, KNN, ANN, Extreme Gradient Boosting and AdaBoost. The models are combined using hard Voting Classifier to enhance the stability and robustness of predictions. Further, a stacking classifier is created with RF, KNN, MLP as the base classifiers and LR as the meta-learner to improve the prediction accuracy. The results obtained by experiments are quite satisfactory. Voting Classifier has 99.98% accuracy and Stacking Classifier has 100% accuracy for forecasting the cloud job outcomes. The predictions are interpreted using XAI techniques like LIME and SHAP, highlighting the importance of the features for transparency and reliability. A Flask based web application is created to incorporate the trained models for real world deployment that allows for secure user signup and signin with SQLite, real time user input and interactive result visualisation. The method results in clear conclusions like “job will complete successfully” or “job failure predicted”. This enables reliable, understandable and easy to use assistance to fail predictions of cloud jobs.