88757b2ac4
新建 server/internal/scheduler/detect/ 包,实现任务 15D: ## 核心模块 **lifecycle.go** - 定义 LifecycleService Go interface(ListNodes / TransitionStatus / SetWeight / BumpVersion / GetLoad / GetLoadHistory),供 #5 真实实现,内含乐观锁语义(0 行 受影响即本周期放弃,幂等可重入) - 提供 MockLifecycle 内存 stub(SetConflict / Events 等测试辅助方法) **signals.go** - computeSignals:读 SnapshotsByNode,归一化为 NodeSignal - domesticFailISPs:按 ISP 聚合(stripPrefix "3rd-" 合并 15B/15C 同 ISP 视角), L3 失败权重最高(L3 挂即记该 ISP 失败,即使 L1/L2 通),无数据 vantage 不计分母 - overseasOK:境外对照点全部 L1 通为真 - trafficDropPct:15min 前后半段在线数降幅百分比 - ProbeSnapshotter interface(解耦 probe.Store,便于 mock) **rules.go** - DetectConfig:全量阈值常量(DomesticFailNumerator/Denominator=2/3、 SuspectStreakMin=2、TrafficDropThreshold=80%、TrafficBaselineMin=20、 ConfirmedStreakMin=6、RecoverStreakMin=2、SuspectWeight=10) - isSuspectTriggered:整数算术避免浮点误差(fail×den >= total×num) - isFault / effectiveSuspectStreakMin / isTrafficWarning 规则助手 **streak.go** - StreakStore:Redis hash detect:streak:{node}(fail_streak / suspect_streak / recover_streak),TTL 2h(宽于 6×5min=30min),进程重启后自动恢复 **engine.go** - Engine.Tick(ctx):列举 up/blocked_suspect 节点,按节点依序运行五条规则 - Rule 1 suspect:≥2/3 ISP 失败 + 境外正常 → 连续 2 周期 → up→blocked_suspect, weight→10,写 probe:freq:{node}=60(1min 提频标记),BumpVersion - Rule 2 流量预警:trafficDrop≥80% + 基线≥20 → suspect 门槛放宽至 1 周期 - Rule 3 confirmed:blocked_suspect 连续 6 周期 → blocked_confirmed → down(跳过 draining),入队 detect:replace:queue(JSON: nodeId + replacementUuid) - Rule 4 recover:suspect 期间境内连续 2 周期恢复 → up(weight 交 15E warmup) - Rule 5 fault:境内+境外同时失败 → 不迁移,调 Notifier.NotifyFault(log stub) - Notifier interface + LogNotifier stub(15G 就绪前输出 slog.Warn) **engine_test.go**(19 个测试,全部通过) - 表驱动覆盖:恰好 2/3 失败、1/3 失败不触发、9min vs 10min streak、 流量预警 1 周期即 suspect、suspect 第 6 周期 confirmed、境内外同挂 fault 不迁移、 恢复 2 周期回 up、3rd- 前缀合并、L3 覆盖 L1/L2 - 乐观锁冲突:SetConflict → 放弃本周期 → 无副作用 → 解冲突后正常重试 - 进程重启:同一 miniredis,新 Engine 读取已存在 streak 继续计数不误清零 - 产出验证:replace queue JSON 结构、probe:freq 标记、weight 变更、事件 detail 字段 **migrations/000010_node_blocked_statuses.{up,down}.sql** - 扩展 nodes.status ENUM 加入 blocked_suspect / blocked_confirmed Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
102 lines
3.8 KiB
Go
102 lines
3.8 KiB
Go
package detect
|
||
|
||
// DetectConfig holds all tunable thresholds for the detection engine.
|
||
// All fields are filled with production defaults by DefaultConfig().
|
||
// 15F (hot-reload) may overwrite fields at runtime; the Engine reads via a
|
||
// pointer so updates take effect on the next Tick without a restart.
|
||
type DetectConfig struct {
|
||
// DomesticFailNumerator and DomesticFailDenominator define the ISP-failure
|
||
// fraction that triggers the suspect rule.
|
||
// Default: 2/3 (two out of every three domestic ISPs must fail).
|
||
DomesticFailNumerator int // 2
|
||
DomesticFailDenominator int // 3
|
||
|
||
// SuspectStreakMin is the number of consecutive failing cycles required to
|
||
// transition a node from "up" to "blocked_suspect".
|
||
// Default: 2 cycles (10 min at the 5-min tick period).
|
||
SuspectStreakMin int
|
||
|
||
// TrafficDropThreshold is the minimum percentage drop in online-connection
|
||
// count over the 15-minute window that activates the traffic-warning rule.
|
||
// Default: 80.0 %.
|
||
TrafficDropThreshold float64
|
||
|
||
// TrafficBaselineMin is the minimum current online-connection count for
|
||
// the traffic-warning relaxation to apply. Below this threshold the node
|
||
// may simply be idle rather than experiencing user flight.
|
||
// Default: 20 connections.
|
||
TrafficBaselineMin int
|
||
|
||
// ConfirmedStreakMin is the number of consecutive cycles that a node must
|
||
// spend in "blocked_suspect" before being promoted to "blocked_confirmed".
|
||
// Default: 6 cycles (30 min).
|
||
ConfirmedStreakMin int
|
||
|
||
// RecoverStreakMin is the number of consecutive passing cycles while in
|
||
// "blocked_suspect" required to recover the node back to "up".
|
||
// Default: 2 cycles (10 min).
|
||
RecoverStreakMin int
|
||
|
||
// SuspectWeight is the routing weight applied when a node first enters
|
||
// "blocked_suspect", reducing traffic directed to it.
|
||
// Default: 10.
|
||
SuspectWeight int
|
||
}
|
||
|
||
// DefaultConfig returns a DetectConfig pre-filled with production defaults.
|
||
func DefaultConfig() DetectConfig {
|
||
return DetectConfig{
|
||
DomesticFailNumerator: 2,
|
||
DomesticFailDenominator: 3,
|
||
SuspectStreakMin: 2,
|
||
TrafficDropThreshold: 80.0,
|
||
TrafficBaselineMin: 20,
|
||
ConfirmedStreakMin: 6,
|
||
RecoverStreakMin: 2,
|
||
SuspectWeight: 10,
|
||
}
|
||
}
|
||
|
||
// isSuspectTriggered reports whether the domestic ISP-failure rate meets or
|
||
// exceeds the configured fraction (default ≥ 2/3).
|
||
//
|
||
// Integer arithmetic avoids floating-point rounding:
|
||
//
|
||
// fail/total >= num/den ↔ fail × den >= total × num
|
||
func (cfg *DetectConfig) isSuspectTriggered(sig NodeSignal) bool {
|
||
if sig.DomesticTotalISPs == 0 {
|
||
return false // no domestic probe data this cycle; cannot judge
|
||
}
|
||
return sig.DomesticFailISPs*cfg.DomesticFailDenominator >=
|
||
sig.DomesticTotalISPs*cfg.DomesticFailNumerator
|
||
}
|
||
|
||
// isFault reports whether the node appears to have a local outage rather than
|
||
// a domestic censorship event.
|
||
//
|
||
// Condition: domestic probes failing AND overseas probes also failing.
|
||
// When both sides are down the most likely cause is a node-level failure
|
||
// (hardware, network, crashed process) rather than GFW interference.
|
||
//
|
||
// The fault rule takes precedence over all other rules and must be evaluated
|
||
// first in the processing loop.
|
||
func isFault(sig NodeSignal) bool {
|
||
return sig.DomesticFailISPs > 0 && sig.OverseasHasData && !sig.OverseasOK
|
||
}
|
||
|
||
// effectiveSuspectStreakMin returns the cycle threshold required to enter
|
||
// "blocked_suspect", relaxed to 1 when the traffic-warning rule is active.
|
||
func (cfg *DetectConfig) effectiveSuspectStreakMin(sig NodeSignal) int {
|
||
if cfg.isTrafficWarning(sig) {
|
||
return 1
|
||
}
|
||
return cfg.SuspectStreakMin
|
||
}
|
||
|
||
// isTrafficWarning reports whether the 15-minute traffic-drop signal exceeds
|
||
// the threshold and the baseline is large enough to be meaningful.
|
||
func (cfg *DetectConfig) isTrafficWarning(sig NodeSignal) bool {
|
||
return sig.TrafficDropPct >= cfg.TrafficDropThreshold &&
|
||
sig.TrafficBaseline >= cfg.TrafficBaselineMin
|
||
}
|