Files
pangolin/server/internal/scheduler/detect/rules.go
wangjia 88757b2ac4 feat(detect): 判定引擎 signals+rules+streak (tsk_OYEiDCzM9_0Y)
新建 server/internal/scheduler/detect/ 包,实现任务 15D:

## 核心模块

**lifecycle.go**
- 定义 LifecycleService Go interface(ListNodes / TransitionStatus / SetWeight /
  BumpVersion / GetLoad / GetLoadHistory),供 #5 真实实现,内含乐观锁语义(0 行
  受影响即本周期放弃,幂等可重入)
- 提供 MockLifecycle 内存 stub(SetConflict / Events 等测试辅助方法)

**signals.go**
- computeSignals:读 SnapshotsByNode,归一化为 NodeSignal
  - domesticFailISPs:按 ISP 聚合(stripPrefix "3rd-" 合并 15B/15C 同 ISP 视角),
    L3 失败权重最高(L3 挂即记该 ISP 失败,即使 L1/L2 通),无数据 vantage 不计分母
  - overseasOK:境外对照点全部 L1 通为真
  - trafficDropPct:15min 前后半段在线数降幅百分比
- ProbeSnapshotter interface(解耦 probe.Store,便于 mock)

**rules.go**
- DetectConfig:全量阈值常量(DomesticFailNumerator/Denominator=2/3、
  SuspectStreakMin=2、TrafficDropThreshold=80%、TrafficBaselineMin=20、
  ConfirmedStreakMin=6、RecoverStreakMin=2、SuspectWeight=10)
- isSuspectTriggered:整数算术避免浮点误差(fail×den >= total×num)
- isFault / effectiveSuspectStreakMin / isTrafficWarning 规则助手

**streak.go**
- StreakStore:Redis hash detect:streak:{node}(fail_streak / suspect_streak /
  recover_streak),TTL 2h(宽于 6×5min=30min),进程重启后自动恢复

**engine.go**
- Engine.Tick(ctx):列举 up/blocked_suspect 节点,按节点依序运行五条规则
- Rule 1 suspect:≥2/3 ISP 失败 + 境外正常 → 连续 2 周期 → up→blocked_suspect,
  weight→10,写 probe:freq:{node}=60(1min 提频标记),BumpVersion
- Rule 2 流量预警:trafficDrop≥80% + 基线≥20 → suspect 门槛放宽至 1 周期
- Rule 3 confirmed:blocked_suspect 连续 6 周期 → blocked_confirmed → down(跳过
  draining),入队 detect:replace:queue(JSON: nodeId + replacementUuid)
- Rule 4 recover:suspect 期间境内连续 2 周期恢复 → up(weight 交 15E warmup)
- Rule 5 fault:境内+境外同时失败 → 不迁移,调 Notifier.NotifyFault(log stub)
- Notifier interface + LogNotifier stub(15G 就绪前输出 slog.Warn)

**engine_test.go**(19 个测试,全部通过)
- 表驱动覆盖:恰好 2/3 失败、1/3 失败不触发、9min vs 10min streak、
  流量预警 1 周期即 suspect、suspect 第 6 周期 confirmed、境内外同挂 fault 不迁移、
  恢复 2 周期回 up、3rd- 前缀合并、L3 覆盖 L1/L2
- 乐观锁冲突:SetConflict → 放弃本周期 → 无副作用 → 解冲突后正常重试
- 进程重启:同一 miniredis,新 Engine 读取已存在 streak 继续计数不误清零
- 产出验证:replace queue JSON 结构、probe:freq 标记、weight 变更、事件 detail 字段

**migrations/000010_node_blocked_statuses.{up,down}.sql**
- 扩展 nodes.status ENUM 加入 blocked_suspect / blocked_confirmed

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-13 20:15:13 +08:00

102 lines
3.8 KiB
Go
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
package detect
// DetectConfig holds all tunable thresholds for the detection engine.
// All fields are filled with production defaults by DefaultConfig().
// 15F (hot-reload) may overwrite fields at runtime; the Engine reads via a
// pointer so updates take effect on the next Tick without a restart.
type DetectConfig struct {
// DomesticFailNumerator and DomesticFailDenominator define the ISP-failure
// fraction that triggers the suspect rule.
// Default: 2/3 (two out of every three domestic ISPs must fail).
DomesticFailNumerator int // 2
DomesticFailDenominator int // 3
// SuspectStreakMin is the number of consecutive failing cycles required to
// transition a node from "up" to "blocked_suspect".
// Default: 2 cycles (10 min at the 5-min tick period).
SuspectStreakMin int
// TrafficDropThreshold is the minimum percentage drop in online-connection
// count over the 15-minute window that activates the traffic-warning rule.
// Default: 80.0 %.
TrafficDropThreshold float64
// TrafficBaselineMin is the minimum current online-connection count for
// the traffic-warning relaxation to apply. Below this threshold the node
// may simply be idle rather than experiencing user flight.
// Default: 20 connections.
TrafficBaselineMin int
// ConfirmedStreakMin is the number of consecutive cycles that a node must
// spend in "blocked_suspect" before being promoted to "blocked_confirmed".
// Default: 6 cycles (30 min).
ConfirmedStreakMin int
// RecoverStreakMin is the number of consecutive passing cycles while in
// "blocked_suspect" required to recover the node back to "up".
// Default: 2 cycles (10 min).
RecoverStreakMin int
// SuspectWeight is the routing weight applied when a node first enters
// "blocked_suspect", reducing traffic directed to it.
// Default: 10.
SuspectWeight int
}
// DefaultConfig returns a DetectConfig pre-filled with production defaults.
func DefaultConfig() DetectConfig {
return DetectConfig{
DomesticFailNumerator: 2,
DomesticFailDenominator: 3,
SuspectStreakMin: 2,
TrafficDropThreshold: 80.0,
TrafficBaselineMin: 20,
ConfirmedStreakMin: 6,
RecoverStreakMin: 2,
SuspectWeight: 10,
}
}
// isSuspectTriggered reports whether the domestic ISP-failure rate meets or
// exceeds the configured fraction (default ≥ 2/3).
//
// Integer arithmetic avoids floating-point rounding:
//
// fail/total >= num/den ↔ fail × den >= total × num
func (cfg *DetectConfig) isSuspectTriggered(sig NodeSignal) bool {
if sig.DomesticTotalISPs == 0 {
return false // no domestic probe data this cycle; cannot judge
}
return sig.DomesticFailISPs*cfg.DomesticFailDenominator >=
sig.DomesticTotalISPs*cfg.DomesticFailNumerator
}
// isFault reports whether the node appears to have a local outage rather than
// a domestic censorship event.
//
// Condition: domestic probes failing AND overseas probes also failing.
// When both sides are down the most likely cause is a node-level failure
// (hardware, network, crashed process) rather than GFW interference.
//
// The fault rule takes precedence over all other rules and must be evaluated
// first in the processing loop.
func isFault(sig NodeSignal) bool {
return sig.DomesticFailISPs > 0 && sig.OverseasHasData && !sig.OverseasOK
}
// effectiveSuspectStreakMin returns the cycle threshold required to enter
// "blocked_suspect", relaxed to 1 when the traffic-warning rule is active.
func (cfg *DetectConfig) effectiveSuspectStreakMin(sig NodeSignal) int {
if cfg.isTrafficWarning(sig) {
return 1
}
return cfg.SuspectStreakMin
}
// isTrafficWarning reports whether the 15-minute traffic-drop signal exceeds
// the threshold and the baseline is large enough to be meaningful.
func (cfg *DetectConfig) isTrafficWarning(sig NodeSignal) bool {
return sig.TrafficDropPct >= cfg.TrafficDropThreshold &&
sig.TrafficBaseline >= cfg.TrafficBaselineMin
}