[{"data":1,"prerenderedAt":896},["ShallowReactive",2],{"content-doc-\u002Fzh\u002Fblogs\u002FAlgorithm-Explained":3,"surround-\u002Fzh\u002Fblogs\u002FAlgorithm-Explained":894},{"_path":4,"_dir":5,"_draft":6,"_partial":6,"_locale":7,"title":8,"description":9,"date":10,"category":11,"author":12,"body":16,"_type":886,"_id":887,"_source":888,"_file":889,"_stem":890,"_extension":891,"coverImage":59,"plainText":892,"authorNames":893},"\u002Fzh\u002Fblogs\u002FAlgorithm-Explained","blogs",false,"","让AI真正“防宕机”：openUBMC 故障预测系统的指标误区澄清与无感运维应用","当前，我们在 openUBMC 中构建了故障预测的完善体系，包含内存、硬盘、光模块、链路的故障预测与诊断技术，进行智能运维决策。","2026\u002F03\u002F19","essentials",[13],{"name":14,"description":15},"张文宇","华为AI算法工程师，统计学博士，深度参与openUBMC算法设计与开发。",{"type":17,"children":18,"toc":866},"root",[19,27,32,37,42,47,52,62,83,94,104,109,113,118,123,148,153,196,311,314,319,326,331,336,346,351,357,362,366,375,380,386,391,396,405,415,428,431,437,442,448,453,556,586,596,602,607,701,706,714,719,729,732,738,743,748,753,759,793,799,850,858,861],{"type":20,"tag":21,"props":22,"children":24},"element","h3",{"id":23},"概要",[25],{"type":26,"value":23},"text",{"type":20,"tag":28,"props":29,"children":30},"p",{},[31],{"type":26,"value":9},{"type":20,"tag":28,"props":33,"children":34},{},[35],{"type":26,"value":36},"在 openUBMC 的智能运维探索中，一个看似平庸的“双50%”指标，背后竟藏着远超随机猜测的算法价值。本文带你穿透数据表象，理解精准率与召回率的真实含金量。",{"type":20,"tag":21,"props":38,"children":40},{"id":39},"故障预测中的指标真相",[41],{"type":26,"value":39},{"type":20,"tag":28,"props":43,"children":44},{},[45],{"type":26,"value":46},"在现代数据中心中， BMC（Baseboard Management Controller） 是服务器的“隐形守护者”。随着 AI 运维（AIOps）的普及，利用 BMC 采集的带外数据（如带内错误事件、电流波动、风扇转速、温度）进行故障预测已成为核心能力。",{"type":20,"tag":28,"props":48,"children":49},{},[50],{"type":26,"value":51},"在阅读相关的技术论文时，我们常看到算法评估指标为精准率 \u002F 覆盖率 \u002F F1分数（Precision\u002FRecall\u002FF1-score）：",{"type":20,"tag":28,"props":53,"children":54},{},[55],{"type":20,"tag":56,"props":57,"children":61},"img",{"alt":58,"src":59,"title":60},"alt text","\u002Fcategory\u002Fblog\u002F%E7%AE%97%E6%B3%95%E7%A7%91%E6%99%AE\u002F%E5%9B%BE%E7%89%871.png","四个模型的精准率 \u002F 覆盖率 \u002F F1 分数指标对比",[],{"type":20,"tag":63,"props":64,"children":65},"blockquote",{},[66],{"type":20,"tag":28,"props":67,"children":68},{},[69],{"type":20,"tag":70,"props":71,"children":72},"em",{},[73,75,81],{"type":26,"value":74},"图片来源：Wang X, Li Y, Chen Y, et al. On workload-aware dram failure prediction in large-scale data centers",{"type":20,"tag":76,"props":77,"children":78},"span",{},[79],{"type":26,"value":80},"C",{"type":26,"value":82},"\u002F\u002F2021 IEEE 39th VLSI Test Symposium (VTS). IEEE, 2021: 1-6.",{"type":20,"tag":28,"props":84,"children":85},{},[86,88],{"type":26,"value":87},"而部分模型的指标甚至都在 50% 以下。这似乎有点不符合我们的直觉：",{"type":20,"tag":89,"props":90,"children":91},"strong",{},[92],{"type":26,"value":93},"就算随机抛一枚均匀硬币做猜测，指标是不是也能轻松达成 50%？为什么模型的指标却如此之低？",{"type":20,"tag":28,"props":95,"children":96},{},[97,99],{"type":26,"value":98},"简单直接的回答是 ",{"type":20,"tag":89,"props":100,"children":101},{},[102],{"type":26,"value":103},"【精准率50% & 覆盖率50%】和【随机抛均匀硬币猜测】不一定等效，甚至通常不等效。",{"type":20,"tag":28,"props":105,"children":106},{},[107],{"type":26,"value":108},"要回答这个问题，我们首先得弄清楚：在故障预测的二分类模型中，这些数字究竟是怎么算出来的。",{"type":20,"tag":110,"props":111,"children":112},"hr",{},[],{"type":20,"tag":21,"props":114,"children":116},{"id":115},"故障预测的二分类模型",[117],{"type":26,"value":115},{"type":20,"tag":28,"props":119,"children":120},{},[121],{"type":26,"value":122},"在 BMC 场景中，二分类模型本质上是在做“预警决策”：",{"type":20,"tag":124,"props":125,"children":126},"ul",{},[127,138],{"type":20,"tag":128,"props":129,"children":130},"li",{},[131,136],{"type":20,"tag":89,"props":132,"children":133},{},[134],{"type":26,"value":135},"正类 (Positive)",{"type":26,"value":137},"：预测设备即将在未来一段时间内发生内存宕机或电源故障。",{"type":20,"tag":128,"props":139,"children":140},{},[141,146],{"type":20,"tag":89,"props":142,"children":143},{},[144],{"type":26,"value":145},"负类 (Negative)",{"type":26,"value":147},"：预测设备运行正常。",{"type":20,"tag":28,"props":149,"children":150},{},[151],{"type":26,"value":152},"基于预测与真实结果的重合情况，产生了四个原子指标：",{"type":20,"tag":124,"props":154,"children":155},{},[156,166,176,186],{"type":20,"tag":128,"props":157,"children":158},{},[159,164],{"type":20,"tag":89,"props":160,"children":161},{},[162],{"type":26,"value":163},"TP (真阳性)",{"type":26,"value":165},"：成功预警了即将发生的故障。",{"type":20,"tag":128,"props":167,"children":168},{},[169,174],{"type":20,"tag":89,"props":170,"children":171},{},[172],{"type":26,"value":173},"FP (假阳性)",{"type":26,"value":175},"：虚警，设备正常却报了警。（备注：误诊）",{"type":20,"tag":128,"props":177,"children":178},{},[179,184],{"type":20,"tag":89,"props":180,"children":181},{},[182],{"type":26,"value":183},"TN (真阴性)",{"type":26,"value":185},"：正确判断设备平安无事。",{"type":20,"tag":128,"props":187,"children":188},{},[189,194],{"type":20,"tag":89,"props":190,"children":191},{},[192],{"type":26,"value":193},"FN (假阴性)",{"type":26,"value":195},"：漏报，设备坏了但模型没发现。（备注：漏诊）",{"type":20,"tag":197,"props":198,"children":199},"table",{},[200,229],{"type":20,"tag":201,"props":202,"children":203},"thead",{},[204],{"type":20,"tag":205,"props":206,"children":207},"tr",{},[208,213,219,224],{"type":20,"tag":209,"props":210,"children":212},"th",{"align":211},"left",[],{"type":20,"tag":209,"props":214,"children":216},{"align":215},"center",[217],{"type":26,"value":218},"实际为正类",{"type":20,"tag":209,"props":220,"children":221},{"align":215},[222],{"type":26,"value":223},"实际为负类",{"type":20,"tag":209,"props":225,"children":226},{"align":215},[227],{"type":26,"value":228},"总和",{"type":20,"tag":230,"props":231,"children":232},"tbody",{},[233,260,286],{"type":20,"tag":205,"props":234,"children":235},{},[236,245,250,255],{"type":20,"tag":237,"props":238,"children":239},"td",{"align":211},[240],{"type":20,"tag":89,"props":241,"children":242},{},[243],{"type":26,"value":244},"预测为正类",{"type":20,"tag":237,"props":246,"children":247},{"align":215},[248],{"type":26,"value":249},"TP (True Positive)",{"type":20,"tag":237,"props":251,"children":252},{"align":215},[253],{"type":26,"value":254},"FP (False Positive)",{"type":20,"tag":237,"props":256,"children":257},{"align":215},[258],{"type":26,"value":259},"预测为正的样本数",{"type":20,"tag":205,"props":261,"children":262},{},[263,271,276,281],{"type":20,"tag":237,"props":264,"children":265},{"align":211},[266],{"type":20,"tag":89,"props":267,"children":268},{},[269],{"type":26,"value":270},"预测为负类",{"type":20,"tag":237,"props":272,"children":273},{"align":215},[274],{"type":26,"value":275},"FN (False Negative)",{"type":20,"tag":237,"props":277,"children":278},{"align":215},[279],{"type":26,"value":280},"TN (True Negative)",{"type":20,"tag":237,"props":282,"children":283},{"align":215},[284],{"type":26,"value":285},"预测为负的样本数",{"type":20,"tag":205,"props":287,"children":288},{},[289,296,301,306],{"type":20,"tag":237,"props":290,"children":291},{"align":211},[292],{"type":20,"tag":89,"props":293,"children":294},{},[295],{"type":26,"value":228},{"type":20,"tag":237,"props":297,"children":298},{"align":215},[299],{"type":26,"value":300},"真实正样本数",{"type":20,"tag":237,"props":302,"children":303},{"align":215},[304],{"type":26,"value":305},"真实负样本数",{"type":20,"tag":237,"props":307,"children":308},{"align":215},[309],{"type":26,"value":310},"样本总数",{"type":20,"tag":110,"props":312,"children":313},{},[],{"type":20,"tag":21,"props":315,"children":317},{"id":316},"核心评估指标的计算方式",[318],{"type":26,"value":316},{"type":20,"tag":320,"props":321,"children":323},"h4",{"id":322},"_1-精准率-precision",[324],{"type":26,"value":325},"1. 精准率 (Precision)",{"type":20,"tag":28,"props":327,"children":328},{},[329],{"type":26,"value":330},"精准率（Precision）：代表“不乱报”。",{"type":20,"tag":28,"props":332,"children":333},{},[334],{"type":26,"value":335},"计算公式为：",{"type":20,"tag":28,"props":337,"children":338},{},[339],{"type":20,"tag":340,"props":341,"children":343},"code",{"className":342},[],[344],{"type":26,"value":345},"Precision = TP \u002F (TP + FP)",{"type":20,"tag":28,"props":347,"children":348},{},[349],{"type":26,"value":350},"它是【正确正样本数】占【预测为正的样本数】的比例，衡量的是在所有“报警”中，有多少是真的？它决定了运维人员的信任度。",{"type":20,"tag":320,"props":352,"children":354},{"id":353},"_2-覆盖率recall",[355],{"type":26,"value":356},"2. 覆盖率（Recall）",{"type":20,"tag":28,"props":358,"children":359},{},[360],{"type":26,"value":361},"覆盖率（Recall）：代表“不漏掉”。",{"type":20,"tag":28,"props":363,"children":364},{},[365],{"type":26,"value":335},{"type":20,"tag":28,"props":367,"children":368},{},[369],{"type":20,"tag":340,"props":370,"children":372},{"className":371},[],[373],{"type":26,"value":374},"Recall = TP \u002F (TP + FN)",{"type":20,"tag":28,"props":376,"children":377},{},[378],{"type":26,"value":379},"它是【正确正样本数】占【真实正样本数】的比例，衡量的是在所有“真实故障”中，有多少被提前抓住了？它决定了系统的安全性。",{"type":20,"tag":320,"props":381,"children":383},{"id":382},"_3-f1分数-f1-score",[384],{"type":26,"value":385},"3. F1分数 (F1-Score)",{"type":20,"tag":28,"props":387,"children":388},{},[389],{"type":26,"value":390},"F1分数 (F1-Score)：精准率和覆盖率的“调和平均数”。",{"type":20,"tag":28,"props":392,"children":393},{},[394],{"type":26,"value":395},"在实际操作中，精准率和覆盖率往往是一对“冤家”：你为了不漏掉故障（提高覆盖率）而扩大报警范围，就不可避免会产生更多误报（降低精准率）。为了衡量两者的整体平衡性，我们引入了 F1分数：",{"type":20,"tag":28,"props":397,"children":398},{},[399],{"type":20,"tag":340,"props":400,"children":402},{"className":401},[],[403],{"type":26,"value":404},"F1 = 2 * (Precision * Recall) \u002F (Precision + Recall)",{"type":20,"tag":28,"props":406,"children":407},{},[408,413],{"type":20,"tag":89,"props":409,"children":410},{},[411],{"type":26,"value":412},"为什么要用调和平均，而不是普通的算术平均？",{"type":26,"value":414},"\n因为调和平均对“极小值”非常敏感。如果你的精准率是 90% 但覆盖率只有 1%（空有精度但几乎漏掉了所有故障），算术平均还有 45.5%，但 F1 分数会迅速跌落至约 2%。这能有效防止模型通过“极端偏科”来刷分。",{"type":20,"tag":63,"props":416,"children":417},{},[418],{"type":20,"tag":28,"props":419,"children":420},{},[421,426],{"type":20,"tag":89,"props":422,"children":423},{},[424],{"type":26,"value":425},"注",{"type":26,"value":427},"：F1 分数与我们开头关心的问题没有必然联系，仅顺便做普及。",{"type":20,"tag":110,"props":429,"children":430},{},[],{"type":20,"tag":21,"props":432,"children":434},{"id":433},"问题分析为什么-50-不等于抛硬币",[435],{"type":26,"value":436},"问题分析：为什么 50% 不等于“抛硬币”？",{"type":20,"tag":28,"props":438,"children":439},{},[440],{"type":26,"value":441},"假设我们监测 1000 台设备，其中只有 10 台会发生故障（即正样本占比仅为 1%）。我们来看看“抛硬币”和“模型预测”的巨大差异：",{"type":20,"tag":320,"props":443,"children":445},{"id":444},"_1-抛硬币的盲目性",[446],{"type":26,"value":447},"1. 抛硬币的“盲目性”",{"type":20,"tag":28,"props":449,"children":450},{},[451],{"type":26,"value":452},"如果通过抛一枚均匀的硬币来决定设备是否有故障（即 50% 的概率猜故障，50% 的概率猜正常）：",{"type":20,"tag":197,"props":454,"children":455},{},[456,479],{"type":20,"tag":201,"props":457,"children":458},{},[459],{"type":20,"tag":205,"props":460,"children":461},{},[462,465,470,475],{"type":20,"tag":209,"props":463,"children":464},{"align":211},[],{"type":20,"tag":209,"props":466,"children":467},{"align":215},[468],{"type":26,"value":469},"实际为正",{"type":20,"tag":209,"props":471,"children":472},{"align":215},[473],{"type":26,"value":474},"实际为负",{"type":20,"tag":209,"props":476,"children":477},{"align":215},[478],{"type":26,"value":228},{"type":20,"tag":230,"props":480,"children":481},{},[482,508,531],{"type":20,"tag":205,"props":483,"children":484},{},[485,493,498,503],{"type":20,"tag":237,"props":486,"children":487},{"align":211},[488],{"type":20,"tag":89,"props":489,"children":490},{},[491],{"type":26,"value":492},"预测为正",{"type":20,"tag":237,"props":494,"children":495},{"align":215},[496],{"type":26,"value":497},"5",{"type":20,"tag":237,"props":499,"children":500},{"align":215},[501],{"type":26,"value":502},"495",{"type":20,"tag":237,"props":504,"children":505},{"align":215},[506],{"type":26,"value":507},"500",{"type":20,"tag":205,"props":509,"children":510},{},[511,519,523,527],{"type":20,"tag":237,"props":512,"children":513},{"align":211},[514],{"type":20,"tag":89,"props":515,"children":516},{},[517],{"type":26,"value":518},"预测为负",{"type":20,"tag":237,"props":520,"children":521},{"align":215},[522],{"type":26,"value":497},{"type":20,"tag":237,"props":524,"children":525},{"align":215},[526],{"type":26,"value":502},{"type":20,"tag":237,"props":528,"children":529},{"align":215},[530],{"type":26,"value":507},{"type":20,"tag":205,"props":532,"children":533},{},[534,541,546,551],{"type":20,"tag":237,"props":535,"children":536},{"align":211},[537],{"type":20,"tag":89,"props":538,"children":539},{},[540],{"type":26,"value":228},{"type":20,"tag":237,"props":542,"children":543},{"align":215},[544],{"type":26,"value":545},"10",{"type":20,"tag":237,"props":547,"children":548},{"align":215},[549],{"type":26,"value":550},"990",{"type":20,"tag":237,"props":552,"children":553},{"align":215},[554],{"type":26,"value":555},"1000",{"type":20,"tag":124,"props":557,"children":558},{},[559,569],{"type":20,"tag":128,"props":560,"children":561},{},[562,567],{"type":20,"tag":89,"props":563,"children":564},{},[565],{"type":26,"value":566},"覆盖率",{"type":26,"value":568},"：10 台故障设备中，因为运气猜中 5 台，覆盖率 = 50%。",{"type":20,"tag":128,"props":570,"children":571},{},[572,577,579,584],{"type":20,"tag":89,"props":573,"children":574},{},[575],{"type":26,"value":576},"精准率",{"type":26,"value":578},"：一共猜了 500 台设备有故障，但其中只有 5 台是真的。精准率 = 5 \u002F 500 = ",{"type":20,"tag":89,"props":580,"children":581},{},[582],{"type":26,"value":583},"1%",{"type":26,"value":585},"。",{"type":20,"tag":28,"props":587,"children":588},{},[589,594],{"type":20,"tag":89,"props":590,"children":591},{},[592],{"type":26,"value":593},"结论",{"type":26,"value":595},"：在正样本占比低时，抛硬币虽然能拿到 50% 的覆盖率，但精准率会低得离谱。",{"type":20,"tag":320,"props":597,"children":599},{"id":598},"_2-50-精准率-50-覆盖率模型的含金量",[600],{"type":26,"value":601},"2. “50% 精准率 + 50% 覆盖率”模型的含金量",{"type":20,"tag":28,"props":603,"children":604},{},[605],{"type":26,"value":606},"如果一个算法能跑出 50% 精准率 + 50% 覆盖率：",{"type":20,"tag":197,"props":608,"children":609},{},[610,631],{"type":20,"tag":201,"props":611,"children":612},{},[613],{"type":20,"tag":205,"props":614,"children":615},{},[616,619,623,627],{"type":20,"tag":209,"props":617,"children":618},{"align":211},[],{"type":20,"tag":209,"props":620,"children":621},{"align":215},[622],{"type":26,"value":469},{"type":20,"tag":209,"props":624,"children":625},{"align":215},[626],{"type":26,"value":474},{"type":20,"tag":209,"props":628,"children":629},{"align":215},[630],{"type":26,"value":228},{"type":20,"tag":230,"props":632,"children":633},{},[634,656,679],{"type":20,"tag":205,"props":635,"children":636},{},[637,644,648,652],{"type":20,"tag":237,"props":638,"children":639},{"align":211},[640],{"type":20,"tag":89,"props":641,"children":642},{},[643],{"type":26,"value":492},{"type":20,"tag":237,"props":645,"children":646},{"align":215},[647],{"type":26,"value":497},{"type":20,"tag":237,"props":649,"children":650},{"align":215},[651],{"type":26,"value":497},{"type":20,"tag":237,"props":653,"children":654},{"align":215},[655],{"type":26,"value":545},{"type":20,"tag":205,"props":657,"children":658},{},[659,666,670,675],{"type":20,"tag":237,"props":660,"children":661},{"align":211},[662],{"type":20,"tag":89,"props":663,"children":664},{},[665],{"type":26,"value":518},{"type":20,"tag":237,"props":667,"children":668},{"align":215},[669],{"type":26,"value":497},{"type":20,"tag":237,"props":671,"children":672},{"align":215},[673],{"type":26,"value":674},"985",{"type":20,"tag":237,"props":676,"children":677},{"align":215},[678],{"type":26,"value":550},{"type":20,"tag":205,"props":680,"children":681},{},[682,689,693,697],{"type":20,"tag":237,"props":683,"children":684},{"align":211},[685],{"type":20,"tag":89,"props":686,"children":687},{},[688],{"type":26,"value":228},{"type":20,"tag":237,"props":690,"children":691},{"align":215},[692],{"type":26,"value":545},{"type":20,"tag":237,"props":694,"children":695},{"align":215},[696],{"type":26,"value":550},{"type":20,"tag":237,"props":698,"children":699},{"align":215},[700],{"type":26,"value":555},{"type":20,"tag":28,"props":702,"children":703},{},[704],{"type":26,"value":705},"这意味着报出的 10 个故障里，有 5 个是真的；同时抓住了全场 10 个故障中的 5 个。",{"type":20,"tag":28,"props":707,"children":708},{},[709],{"type":20,"tag":89,"props":710,"children":711},{},[712],{"type":26,"value":713},"对比抛硬币 1% 的精准率，这个算法的预测能力是抛硬币的 50 倍；预测为正的总样本数为 10，意味着这个算法比随机猜测更加谨慎！",{"type":20,"tag":28,"props":715,"children":716},{},[717],{"type":26,"value":718},"这就是为什么在故障预测（通常是极端不平衡数据）的论文中，精准率和覆盖率即便都在 50% 左右，也往往意味着模型捕捉到了极其深刻的特征规律，而非随机运气。",{"type":20,"tag":28,"props":720,"children":721},{},[722,727],{"type":20,"tag":89,"props":723,"children":724},{},[725],{"type":26,"value":726},"额外思考",{"type":26,"value":728},"：我们的举例假设是正样本占比低，这符合故障预测数据“正常样本多、故障样本少”的普遍规律。如果正负样本比例均衡，或者正样本占比高于负样本时，与随机抛硬币相比的结果会如何？欢迎补充结论！",{"type":20,"tag":110,"props":730,"children":731},{},[],{"type":20,"tag":21,"props":733,"children":735},{"id":734},"进阶思考在-openubmc-中如何根据运维成本调整预测策略",[736],{"type":26,"value":737},"进阶思考：在 openUBMC 中，如何根据运维成本调整预测策略？",{"type":20,"tag":28,"props":739,"children":740},{},[741],{"type":26,"value":742},"在正样本比例低的数据中，精准率 50% 和覆盖率 50% 虽然远优于随机猜测，但未必是“最优决策”。模型指标的取舍，本质上是一场关于“运维成本”的博弈。",{"type":20,"tag":28,"props":744,"children":745},{},[746],{"type":26,"value":747},"在 openUBMC 的实际工程中，最典型的应用场景莫过于 DDR 内存故障预测。现代服务器内存均支持 ECC（纠错码）机制。当内存位翻转在纠错能力范围内时，系统会上报可纠正错误（Correctable Error, CE），此时业务运行不受影响；然而，一旦错误模式超过纠错上限（如同一条带内多位翻转），则会演变为不可纠正错误（Uncorrectable Error, UCE），直接触发系统致命异常（如 Linux 的 Kernel Panic），导致业务瞬间宕机。",{"type":20,"tag":28,"props":749,"children":750},{},[751],{"type":26,"value":752},"业界研究表明，CE 是 UCE 的重要前兆，尤其是高频 CE、多列 CE 等特定模式对 UCE 具有极强的启示作用。基于这些 CE 特征构建的二分类模型，其指标取舍直接决定了运维的经济性：",{"type":20,"tag":320,"props":754,"children":756},{"id":755},"_1-追求高精准率",[757],{"type":26,"value":758},"1. 追求高精准率",{"type":20,"tag":63,"props":760,"children":761},{},[762,783],{"type":20,"tag":28,"props":763,"children":764},{},[765,770,772,781],{"type":20,"tag":89,"props":766,"children":767},{},[768],{"type":26,"value":769},"运维动作",{"type":26,"value":771},"：在",{"type":20,"tag":773,"props":774,"children":778},"a",{"href":775,"rel":776},"https:\u002F\u002Fwww.openubmc.cn\u002Fzh\u002Fblogs\u002FUBMC-AI",[777],"nofollow",[779],{"type":26,"value":780},"上篇文章（openUBMC：基于BMC的AI故障预测系统构建及应用）",{"type":26,"value":782},"中，我们提到故障预测最常见的应用方式是与业务联合，在业务调度间隙替换即将失效的服务器，从而实现业务无损。在追求高精准率时，运维仅在模型极度确信故障将发生时，才通知现场备件更换。",{"type":20,"tag":28,"props":784,"children":785},{},[786,791],{"type":20,"tag":89,"props":787,"children":788},{},[789],{"type":26,"value":790},"成本考量",{"type":26,"value":792},"：对于超大规模集群（如 10 万节点以上），如果精准率过低，每日产生的“虚假预警”将导致运维团队陷入无穷无尽的无效排查中，甚至导致备件库存非正常耗尽。此时，我们宁愿覆盖率只有 30%（只抓最明显的故障），也要保证精准率在 80% 以上，以确保每一次下场维修都是“有的放矢”。",{"type":20,"tag":320,"props":794,"children":796},{"id":795},"_2-追求高覆盖率",[797],{"type":26,"value":798},"2. 追求高覆盖率",{"type":20,"tag":63,"props":800,"children":801},{},[802,817,826],{"type":20,"tag":28,"props":803,"children":804},{},[805,809,810,815],{"type":20,"tag":89,"props":806,"children":807},{},[808],{"type":26,"value":769},{"type":26,"value":771},{"type":20,"tag":773,"props":811,"children":813},{"href":775,"rel":812},[777],[814],{"type":26,"value":780},{"type":26,"value":816},"中，我们提到我们会将故障预测与芯片 RAS 结合，利用芯片的冗余修复能力，实现硬件故障无感修复。在追求高覆盖率时，一旦预测到 UCE 风险，运维会 (a) 尽早更换内存条, 或 (b) 直接由BMC侧调用芯片的RAS能力修复部件即将失效的区域。",{"type":20,"tag":28,"props":818,"children":819},{},[820,824],{"type":20,"tag":89,"props":821,"children":822},{},[823],{"type":26,"value":790},{"type":26,"value":825},"：",{"type":20,"tag":827,"props":828,"children":829},"ol",{},[830,840],{"type":20,"tag":128,"props":831,"children":832},{},[833,838],{"type":20,"tag":89,"props":834,"children":835},{},[836],{"type":26,"value":837},"更换内存条",{"type":26,"value":839},"：如果承载的是核心数据库业务，宕机一分钟的损失远超一根内存条的成本。此时，即便模型精准率仅为 30%（即 10 次报警中仅 3 次是真的），但只要覆盖率达到 80%，就能避免绝大部分非计划停机，这种“过杀”策略在经济上是完全划算的。",{"type":20,"tag":128,"props":841,"children":842},{},[843,848],{"type":20,"tag":89,"props":844,"children":845},{},[846],{"type":26,"value":847},"调用芯片的 RAS 能力",{"type":26,"value":849},"：这种方案在不中断业务的情况下对硬件故障进行细粒度的修复，不需要隔离整服务器，即使精准率较低导致误隔离，也仅仅是影响部件冗余资源的使用，对于业务运行几乎无成本损失。",{"type":20,"tag":28,"props":851,"children":852},{},[853],{"type":20,"tag":56,"props":854,"children":857},{"alt":58,"src":855,"title":856},"\u002Fcategory\u002Fblog\u002F%E7%AE%97%E6%B3%95%E7%A7%91%E6%99%AE\u002F%E5%9B%BE5_%E5%89%AF%E6%9C%AC.png","故障预测与芯片RAS结合，对于业务运行几乎无成本损失，通常为模型追求高覆盖率的优秀运维方案",[],{"type":20,"tag":110,"props":859,"children":860},{},[],{"type":20,"tag":28,"props":862,"children":863},{},[864],{"type":26,"value":865},"在 BMC 故障预测的二分类世界里，没有绝对完美的指标，只有最契合业务场景的平衡点。真正能让 AI 赋能运维的，是我们在深入理解 CE 与 UCE 的物理特性、衡量业务宕机与人工成本后，对精准率与覆盖率进行的最后那次温情调优。",{"title":7,"searchDepth":867,"depth":867,"links":868},4,[869,871,872,873,878,882],{"id":23,"depth":870,"text":23},3,{"id":39,"depth":870,"text":39},{"id":115,"depth":870,"text":115},{"id":316,"depth":870,"text":316,"children":874},[875,876,877],{"id":322,"depth":867,"text":325},{"id":353,"depth":867,"text":356},{"id":382,"depth":867,"text":385},{"id":433,"depth":870,"text":436,"children":879},[880,881],{"id":444,"depth":867,"text":447},{"id":598,"depth":867,"text":601},{"id":734,"depth":870,"text":737,"children":883},[884,885],{"id":755,"depth":867,"text":758},{"id":795,"depth":867,"text":798},"markdown","content:zh:blogs:Algorithm Explained.md","content","zh\u002Fblogs\u002FAlgorithm Explained.md","zh\u002Fblogs\u002FAlgorithm Explained","md","概要 当前，我们在 openUBMC 中构建了故障预测的完善体系，包含内存、硬盘、光模块、链路的故障预测与诊断技术，进行智能运维决策。 在 openUBMC 的智能运维探索中，一个看似平庸的“双50%”指标，背后竟藏着远超随机猜测的算法价值。本文带你穿透数据表象，理解精准率与召回率的真实含金量。 故障预测中的指标真相 在现代数据中心中， BMC（Baseboard Management Controller） 是服务器的“隐形守护者”。随着 AI 运维（AIOps）的普及，利用 BMC 采集的带外数据（如带内错误事件、电流波动、风扇转速、温度）进行故障预测已成为核心能力。 在阅读相关的技术论文时，我们常看到算法评估指标为精准率 \u002F 覆盖率 \u002F F1分数（Precision\u002FRecall\u002FF1-score）：  图片来源：Wang X, Li Y, Chen Y, et al. On workload-aware dram failure prediction in large-scale data centers C \u002F\u002F2021 IEEE 39th VLSI Test Symposium (VTS). IEEE, 2021: 1-6. 而部分模型的指标甚至都在 50% 以下。这似乎有点不符合我们的直觉： 就算随机抛一枚均匀硬币做猜测，指标是不是也能轻松达成 50%？为什么模型的指标却如此之低？ 简单直接的回答是  【精准率50% & 覆盖率50%】和【随机抛均匀硬币猜测】不一定等效，甚至通常不等效。 要回答这个问题，我们首先得弄清楚：在故障预测的二分类模型中，这些数字究竟是怎么算出来的。  故障预测的二分类模型 在 BMC 场景中，二分类模型本质上是在做“预警决策”： 正类 (Positive) ：预测设备即将在未来一段时间内发生内存宕机或电源故障。 负类 (Negative) ：预测设备运行正常。 基于预测与真实结果的重合情况，产生了四个原子指标： TP (真阳性) ：成功预警了即将发生的故障。 FP (假阳性) ：虚警，设备正常却报了警。（备注：误诊） TN (真阴性) ：正确判断设备平安无事。 FN (假阴性) ：漏报，设备坏了但模型没发现。（备注：漏诊）  实际为正类 实际为负类 总和 预测为正类 TP (True Positive) FP (False Positive) 预测为正的样本数 预测为负类 FN (False Negative) TN (True Negative) 预测为负的样本数 总和 真实正样本数 真实负样本数 样本总数  核心评估指标的计算方式 1. 精准率 (Precision) 精准率（Precision）：代表“不乱报”。 计算公式为： Precision = TP \u002F (TP + FP) 它是【正确正样本数】占【预测为正的样本数】的比例，衡量的是在所有“报警”中，有多少是真的？它决定了运维人员的信任度。 2. 覆盖率（Recall） 覆盖率（Recall）：代表“不漏掉”。 计算公式为： Recall = TP \u002F (TP + FN) 它是【正确正样本数】占【真实正样本数】的比例，衡量的是在所有“真实故障”中，有多少被提前抓住了？它决定了系统的安全性。 3. F1分数 (F1-Score) F1分数 (F1-Score)：精准率和覆盖率的“调和平均数”。 在实际操作中，精准率和覆盖率往往是一对“冤家”：你为了不漏掉故障（提高覆盖率）而扩大报警范围，就不可避免会产生更多误报（降低精准率）。为了衡量两者的整体平衡性，我们引入了 F1分数： F1 = 2 * (Precision * Recall) \u002F (Precision + Recall) 为什么要用调和平均，而不是普通的算术平均？ \n因为调和平均对“极小值”非常敏感。如果你的精准率是 90% 但覆盖率只有 1%（空有精度但几乎漏掉了所有故障），算术平均还有 45.5%，但 F1 分数会迅速跌落至约 2%。这能有效防止模型通过“极端偏科”来刷分。 注 ：F1 分数与我们开头关心的问题没有必然联系，仅顺便做普及。  问题分析：为什么 50% 不等于“抛硬币”？ 假设我们监测 1000 台设备，其中只有 10 台会发生故障（即正样本占比仅为 1%）。我们来看看“抛硬币”和“模型预测”的巨大差异： 1. 抛硬币的“盲目性” 如果通过抛一枚均匀的硬币来决定设备是否有故障（即 50% 的概率猜故障，50% 的概率猜正常）：  实际为正 实际为负 总和 预测为正 5 495 500 预测为负 5 495 500 总和 10 990 1000 覆盖率 ：10 台故障设备中，因为运气猜中 5 台，覆盖率 = 50%。 精准率 ：一共猜了 500 台设备有故障，但其中只有 5 台是真的。精准率 = 5 \u002F 500 =  1% 。 结论 ：在正样本占比低时，抛硬币虽然能拿到 50% 的覆盖率，但精准率会低得离谱。 2. “50% 精准率 + 50% 覆盖率”模型的含金量 如果一个算法能跑出 50% 精准率 + 50% 覆盖率：  实际为正 实际为负 总和 预测为正 5 5 10 预测为负 5 985 990 总和 10 990 1000 这意味着报出的 10 个故障里，有 5 个是真的；同时抓住了全场 10 个故障中的 5 个。 对比抛硬币 1% 的精准率，这个算法的预测能力是抛硬币的 50 倍；预测为正的总样本数为 10，意味着这个算法比随机猜测更加谨慎！ 这就是为什么在故障预测（通常是极端不平衡数据）的论文中，精准率和覆盖率即便都在 50% 左右，也往往意味着模型捕捉到了极其深刻的特征规律，而非随机运气。 额外思考 ：我们的举例假设是正样本占比低，这符合故障预测数据“正常样本多、故障样本少”的普遍规律。如果正负样本比例均衡，或者正样本占比高于负样本时，与随机抛硬币相比的结果会如何？欢迎补充结论！  进阶思考：在 openUBMC 中，如何根据运维成本调整预测策略？ 在正样本比例低的数据中，精准率 50% 和覆盖率 50% 虽然远优于随机猜测，但未必是“最优决策”。模型指标的取舍，本质上是一场关于“运维成本”的博弈。 在 openUBMC 的实际工程中，最典型的应用场景莫过于 DDR 内存故障预测。现代服务器内存均支持 ECC（纠错码）机制。当内存位翻转在纠错能力范围内时，系统会上报可纠正错误（Correctable Error, CE），此时业务运行不受影响；然而，一旦错误模式超过纠错上限（如同一条带内多位翻转），则会演变为不可纠正错误（Uncorrectable Error, UCE），直接触发系统致命异常（如 Linux 的 Kernel Panic），导致业务瞬间宕机。 业界研究表明，CE 是 UCE 的重要前兆，尤其是高频 CE、多列 CE 等特定模式对 UCE 具有极强的启示作用。基于这些 CE 特征构建的二分类模型，其指标取舍直接决定了运维的经济性： 1. 追求高精准率 运维动作 ：在 上篇文章（openUBMC：基于BMC的AI故障预测系统构建及应用） 中，我们提到故障预测最常见的应用方式是与业务联合，在业务调度间隙替换即将失效的服务器，从而实现业务无损。在追求高精准率时，运维仅在模型极度确信故障将发生时，才通知现场备件更换。 成本考量 ：对于超大规模集群（如 10 万节点以上），如果精准率过低，每日产生的“虚假预警”将导致运维团队陷入无穷无尽的无效排查中，甚至导致备件库存非正常耗尽。此时，我们宁愿覆盖率只有 30%（只抓最明显的故障），也要保证精准率在 80% 以上，以确保每一次下场维修都是“有的放矢”。 2. 追求高覆盖率 运维动作 ：在 上篇文章（openUBMC：基于BMC的AI故障预测系统构建及应用） 中，我们提到我们会将故障预测与芯片 RAS 结合，利用芯片的冗余修复能力，实现硬件故障无感修复。在追求高覆盖率时，一旦预测到 UCE 风险，运维会 (a) 尽早更换内存条, 或 (b) 直接由BMC侧调用芯片的RAS能力修复部件即将失效的区域。 成本考量 ： 更换内存条 ：如果承载的是核心数据库业务，宕机一分钟的损失远超一根内存条的成本。此时，即便模型精准率仅为 30%（即 10 次报警中仅 3 次是真的），但只要覆盖率达到 80%，就能避免绝大部分非计划停机，这种“过杀”策略在经济上是完全划算的。 调用芯片的 RAS 能力 ：这种方案在不中断业务的情况下对硬件故障进行细粒度的修复，不需要隔离整服务器，即使精准率较低导致误隔离，也仅仅是影响部件冗余资源的使用，对于业务运行几乎无成本损失。   在 BMC 故障预测的二分类世界里，没有绝对完美的指标，只有最契合业务场景的平衡点。真正能让 AI 赋能运维的，是我们在深入理解 CE 与 UCE 的物理特性、衡量业务宕机与人工成本后，对精准率与覆盖率进行的最后那次温情调优。",[14],[895,895],null,1781619863854]