The growth of large models demands multinode cooperation during training and inference processes. The computing node failures can interrupt these processes, subsequently causing information loss and prolonging the execution time. To reduce the prohibitively large overhead incurred by the computing nodes failures, the accurate prediction of computing node failure is vital, which can help to avert potential large overhead, service interruptions, and negative customer experiences. Existing solutions of computing nodes failure prediction mainly focus on utilizing state-of-the-art time-series models to enhance the performance of computing node failure prediction. However, on the one hand, they could not capture the causal relationship between device over-utilization and node failures; On the other hand, they fail to extract the complex spatial-temporal cascading correlations among computing node failure events. These limits can degrade the performance of computing node failure prediction. To address these above problems, this article makes an effort to focus on designing a continuous-time dynamic graphs-based computing node failures prediction (CTDG-NFP) scheme, to accurately predict in dynamic cluster environments. Specifically, the CTDG-NFP scheme first designs a novel multiple-dimensional feature-biased neighbor sampling method, which jointly considers CPU utilization-biased, memory utilization-biased, temporal-biased and spatial-biased, to sample relevant context. Then, the CTDG-NFP scheme extracts diverse computing node failure motifs by multiple-dimensional feature-biased-based long-short-path walk method and set-based anonymization method. Finally, the CTDG-NFP scheme adopts time encoder to encode these motifs, and thereby extracting the complex spatial-temporal correlations among computing node failure events. On this basis, contrastive learning is adopted to train the computing node failure prediction model. Extensive evaluations with various real-world failure traces demonstrate the CTDG-NFP scheme can achieve superior performance in terms of six widely used performance metrics compared with the SOTA node failure prediction methods.
Huang et al. (Thu,) studied this question.