Kunlun Anomaly Troubleshooter: Enabling Kernel-Level Anomaly Detection and Causal Reasoning for Large Model Distributed Inference

Anomaly troubleshooting for large model distributed inference (LMDI) remains a critical challenge. Resolving anomalies such as inference performance degradation or latency jitter in distributed system demands significant manual efforts from domain experts, resulting in extremely time-consuming diagnosis processes with relatively low accuracy. In this paper, we introduce Kunlun Anomaly Troubleshooter (KAT), the first anomaly troubleshooting framework tailored for LMDI. KAT addresses this problem through two core innovations. First, KAT exploits the synchronicity and consistency of GPU workers, innovatively leverages function trace data to precisely detect kernel-level anomalies and associated hardware components at nanosecond resolution. Second, KAT integrates these detection results into a domain-adapted LLM, delivering systematic causal reasoning and natural language interpretation of complex anomaly symptoms. Evaluations conducted in Alibaba Cloud Service production environment indicate that KAT achieves over 0.884 precision and 0.936 recall in anomaly detection, providing detail anomaly insights that significantly narrow down the diagnostic scope and improve both the efficiency and success rate of troubleshooting.

• Publication year

  2025

• Venue

  arXiv.org

• Publication date

  2025-11-08

• Fields of study

  Computer Science

• Identifiers
  DOI 10.48550/arXiv.2511.05978arXiv 2511.05978
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• OmniScience: A Domain-Specialized LLM for Scientific Reasoning and Discovery

  2025cited by this paper
• Trace-based Multi-Dimensional Root Cause Localization of Performance Issues in Microservice Systems

  2024cited by this paper
• ChatTime: A Unified Multimodal Time Series Foundation Model Bridging Numerical and Textual Data

  2024cited by this paper
• Minder: Faulty Machine Detection for Large-scale Distributed Model Training

  2024cited by this paper
• The Llama 3 Herd of Models

  2024cited by this paper
• Language Models Still Struggle to Zero-shot Reason about Time Series

  2024cited by this paper
• Judging LLM-as-a-judge with MT-Bench and Chatbot Arena

  2023cited by this paper
• From Point-wise to Group-wise: A Fast and Accurate Microservice Trace Anomaly Detection Approach

  2023cited by this paper
• Efficient Memory Management for Large Language Model Serving with PagedAttention

  2023cited by this paper
• Automatic Root Cause Analysis via Large Language Models for Cloud Incidents

  2023cited by this paper
• PaLM: Scaling Language Modeling with Pathways

  2022cited by this paper
• Training language models to follow instructions with human feedback

  2022cited by this paper
• TraceCRL: contrastive representation learning for microservice trace analysis

  2022cited by this paper
• Self-Consistency Improves Chain of Thought Reasoning in Language Models

  2022cited by this paper
• Anomaly Detection and Failure Root Cause Analysis in (Micro) Service-Based Cloud Applications: A Survey

  2021cited by this paper
• MicroHECL: High-Efficient Root Cause Localization in Large-Scale Microservice Systems

  2021cited by this paper
• Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks

  2020cited by this paper
• A Systematic Review on Anomaly Detection for Cloud Computing Environments

  2020cited by this paper
• Graph-based trace analysis for microservice architecture understanding and problem diagnosis

  2020cited by this paper
• USAD: UnSupervised Anomaly Detection on Multivariate Time Series

  2020cited by this paper
• Language Models are Few-Shot Learners

  2020cited by this paper
• Evaluating Modern GPU Interconnect: PCIe, NVLink, NV-SLI, NVSwitch and GPUDirect

  2019cited by this paper
• RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers

  2019cited by this paper
• Evaluating the Factual Consistency of Abstractive Text Summarization

  2019cited by this paper
• LogAnomaly: Unsupervised Detection of Sequential and Quantitative Anomalies in Unstructured Logs

  2019cited by this paper
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

  2019cited by this paper
• ROUGE: A Package for Automatic Evaluation of Summaries

  2004cited by this paper
• OpenMP: an industry standard API for shared-memory programming

  1998cited by this paper
• A High-Performance, Portable Implementation of the MPI Message Passing Interface Standard

  1996cited by this paper

• No citing papers are available for this paper.