Topology-Aware Generative AI for Fault Diagnosis and Recovery Planning in Cloud Data Centers
Abstract
Cloud data center incidents increasingly span compute, storage, GPU, network, and virtualization layers. A single physical fault can generate hundreds of alerts across layers, and engineers must correlate them, identify the root cause, and plan safe recovery under time pressure. Large language models (LLMs) can reason over heterogeneous operational text, but without grounding in the actual infrastructure they may hallucinate causes and propose recovery actions that violate redundancy, capacity, or dependency constraints.
This study presents a topology-aware generative AI framework for fault diagnosis and recovery planning. Telemetry from five infrastructure domains is normalized and correlated in time, and attached to a knowledge graph that represents the physical and logical topology. A retrieval-augmented LLM receives the blast-radius subgraph of an incident together with retrieved runbooks and prior incidents, and produces ranked root cause hypotheses with cited evidence and a structured recovery plan. Every plan passes through a constraint validator that checks topology consistency, blast radius, capacity and redundancy preservation, dependency order, and policy rules, followed by a dry-run simulation. Rejected plans are returned to the model with the violated constraint for repair. The framework was evaluated on historical and fault-injected incidents against rule-based correlation, an LLM with retrieval only, and an unvalidated LLM planner.
The framework achieved top-1 root cause accuracy of 82-88% and top-3 accuracy of 92-96%, compared with 55-65% top-1 for rule-based correlation and 68-75% for retrieval-only LLM without topology. Median time to recovery decreased by 35-50% relative to manual response. The unsafe-action rate of generated plans was 0-1% after validation, compared with 9-14% for the unvalidated LLM planner.
Conclusions: Grounding generative AI in infrastructure topology and enforcing deterministic validation of its recovery plans allows the benefits of LLM reasoning to be used in data center operations without accepting the risk of unsafe automated actions. The framework provides a replicable pattern for human-supervised, constraint-bounded AI operations.
Article Information
Journal |
International Journal of Science, Research and Technology |
|---|---|
Volume (Issue) |
Vol. 7 No. 1 (2024): International Journal of Science, Research and Technology (IJSRAT) |
DOI |
|
Pages |
11381-11391 |
Published |
February 14, 2024 |
| Copyright |
All rights reserved |
Open Access |
This work is licensed under a Creative Commons Attribution 4.0 International License. |
How to Cite |
Manoj Kumar Reddy Kalakoti (%2024). Topology-Aware Generative AI for Fault Diagnosis and Recovery Planning in Cloud Data Centers. International Journal of Science, Research and Technology , Vol. 7 No. 1 (2024): International Journal of Science, Research and Technology (IJSRAT) , pp. 11381-11391. https://doi.org/10.15662/IJSRAT.2024.0701003 |
References
[2] H.S. Gunawi, M. Hao, R.O. Suminto, A. Laksono, A.D. Satria, J. Adityatama, K.J. Eliazar, (2016). Why does the cloud stop computing? Lessons from hundreds of service outages, in: Proceedings of the Seventh ACM Symposium on Cloud Computing (SoCC), ACM, Santa Clara, CA, 1-16.
[3] M. Jeon, S. Venkataraman, A. Phanishayee, J. Qian, W. Xiao, F. Yang, (2019). Analysis of large-scale multi-tenant GPU clusters for DNN training workloads, in: Proceedings of the 2019 USENIX Annual Technical Conference (ATC), USENIX, Renton, WA, 947-960.
[4] B. Schroeder, G.A. Gibson, (2007). Disk failures in the real world: What does an MTTF of 1,000,000 hours mean to you?, in: Proceedings of the 5th USENIX Conference on File and Storage Technologies (FAST), USENIX, San Jose, CA, 1-16.
[5] B. Schroeder, E. Pinheiro, W.-D. Weber, (2009). DRAM errors in the wild: A large-scale field study, in: Proceedings of the Eleventh International Joint Conference on Measurement and Modeling of Computer Systems (SIGMETRICS), ACM, Seattle, WA, 193-204.
[6] M. Ma, W. Lin, D. Pan, P. Wang, (2020). Self-adaptive root cause diagnosis for large-scale microservice architecture, IEEE Trans. Serv. Comput. 15 (3) 1399-1410.
[7] Y. Dang, Q. Lin, P. Huang, (2019). AIOps: Real-world challenges and research innovations, in: Proceedings of the 41st International Conference on Software Engineering: Companion Proceedings (ICSE-Companion), IEEE, Montreal, Canada, 4-5.
[8] T. Ahmed, S. Ghosh, C. Bansal, T. Zimmermann, X. Zhang, S. Rajmohan, (2023). Recommending root-cause and mitigation steps for cloud incidents using large language models, in: Proceedings of the 45th International Conference on Software Engineering (ICSE), IEEE, Melbourne, Australia, 1737-1749.
[9] P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, et al., (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks, in: Advances in Neural Information Processing Systems 33 (NeurIPS), Curran Associates, Red Hook, NY, 9459-9474.
[10] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, Y. Cao, (2023). ReAct: Synergizing reasoning and acting in language models, in: The Eleventh International Conference on Learning Representations (ICLR), Kigali, Rwanda.
[11] K. Valmeekam, M. Marquez, A. Olmo, S. Sreedharan, S. Kambhampati, (2023). PlanBench: An extensible benchmark for evaluating large language models on planning and reasoning about change, in: Advances in Neural Information Processing Systems 36 (NeurIPS), Curran Associates, Red Hook, NY.
[12] B. Beyer, C. Jones, J. Petoff, N.R. Murphy, (2016). Site Reliability Engineering: How Google Runs Production Systems, O'Reilly Media, Sebastopol, CA.
[13] B. Beyer, N.R. Murphy, D.K. Rensin, K. Kawahara, S. Thorne, (2018). The Site Reliability Workbook: Practical Ways to Implement SRE, O'Reilly Media, Sebastopol, CA.