wangjia
e7b37d7719
ci-pangolin / Lint — shellcheck (pull_request) Successful in 9s
ci-pangolin / OpenAPI Sync Check (pull_request) Successful in 23s
ci-pangolin / Redline Scan — 脱敏 (UI 文案) (pull_request) Successful in 5s
ci-pangolin / Flutter — analyze + test (pull_request) Failing after 37s
ci-pangolin / Portable SQL — 可移植性 (mysql/sqlite) (pull_request) Successful in 6s
ci-pangolin / Codegen Drift — token 生成物未漂移 (pull_request) Successful in 4s
ci-pangolin / Go — build + test (pull_request) Successful in 19s
ci-pangolin / E2E Smoke — L4 进程级端到端 (pull_request) Successful in 17s
ci-pangolin / Go — integration (mysql/redis testcontainers) (pull_request) Successful in 4m16s
ci-pangolin / Golden — 视觉回归 (components + auth) (pull_request) Successful in 15s
ci-pangolin / Lint — shellcheck (push) Successful in 5s
ci-pangolin / OpenAPI Sync Check (push) Successful in 23s
ci-pangolin / Redline Scan — 脱敏 (UI 文案) (push) Successful in 6s
ci-pangolin / Flutter — analyze + test (push) Failing after 15s
ci-pangolin / Portable SQL — 可移植性 (mysql/sqlite) (push) Successful in 5s
ci-pangolin / Codegen Drift — token 生成物未漂移 (push) Successful in 5s
ci-pangolin / Go — build + test (push) Successful in 7s
ci-pangolin / E2E Smoke — L4 进程级端到端 (push) Successful in 19s
ci-pangolin / Go — integration (mysql/redis testcontainers) (push) Failing after 4m29s
ci-pangolin / Golden — 视觉回归 (components + auth) (push) Successful in 13s
CI 整条吊在一台 mac,关机/掉线即全停且无人知道。新增
deploy/runner/runner-deadman.sh(仿 bootstrap/monitor/deadman-watch.sh):
放另一台常在线主机,每 3 分钟查 gitea Actions API 里 mac-pangolin-2 的 status,
连续 offline 到阈值 → Telegram 告警,恢复也通知;API 取不到时跳过不误报。
响应字段已对照真 gitea API 核验({"runners":[{"name":...,"status":"online"}]}),
shellcheck 干净,实跑验证 online→不告警、缺失→进告警分支。
docs/ci-runner.md 增「韧性」节(告警部署 + cron/launchd)并标注持久化激活仍待
本机手动跑(pkill/launchctl,Claude 跑不了);验证段 job 数 9→10。
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
74 lines
3.2 KiB
Bash
Executable File
74 lines
3.2 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# runner-deadman —— CI runner 死信守望。CI 整条吊在一台 mac(mac-pangolin-2)上,
|
|
# mac 一关/掉线 CI 全停且无人知道。本脚本放在【另一台常在线主机】上,周期查 gitea
|
|
# Actions API 里该 runner 的 status,连续 offline 到阈值 → Telegram 告警;恢复也通知。
|
|
#
|
|
# 仿 deploy/bootstrap/monitor/deadman-watch.sh(同一发信/去抖范式),探活对象从
|
|
# 「TCP 端口」换成「gitea runner 在线状态」。
|
|
#
|
|
# 配置 /etc/pangolin-runner-deadman.env:
|
|
# GITEA_URL=http://127.0.0.1:13000 # gitea 基址(经 relay 或直连)
|
|
# GITEA_TOKEN=... # gitea 读写 key(Bitwarden「gitea 读写key」)
|
|
# GITEA_REPO=wangjia/pangolin # owner/repo
|
|
# RUNNER_NAME=mac-pangolin-2 # 要守望的 runner 名
|
|
# TELEGRAM_BOT_TOKEN=... TELEGRAM_CHAT_ID=...
|
|
# FAIL_THRESHOLD=2 # 连续几次 offline 才告警(去抖)
|
|
#
|
|
# 在该常在线主机上每 3 分钟跑一次(cron,用户态即可):
|
|
# */3 * * * * /usr/local/bin/runner-deadman >/dev/null 2>&1
|
|
# (mac 宿主可改用 launchd LaunchAgent,见 docs/ci-runner.md「韧性」节。)
|
|
set -uo pipefail
|
|
|
|
ENV_FILE=/etc/pangolin-runner-deadman.env
|
|
[ -r "$ENV_FILE" ] || { echo "缺少 $ENV_FILE" >&2; exit 1; }
|
|
# shellcheck disable=SC1090
|
|
. "$ENV_FILE"
|
|
: "${FAIL_THRESHOLD:=2}"
|
|
: "${GITEA_REPO:=wangjia/pangolin}"
|
|
: "${RUNNER_NAME:=mac-pangolin-2}"
|
|
STATE_DIR="${STATE_DIR:-$HOME/.local/state/pangolin-runner-deadman}"
|
|
mkdir -p "$STATE_DIR"
|
|
statef="${STATE_DIR}/${RUNNER_NAME}.fail"
|
|
|
|
send_tg() {
|
|
local text="$1"
|
|
[ -n "${TELEGRAM_BOT_TOKEN:-}" ] && [ -n "${TELEGRAM_CHAT_ID:-}" ] || { echo "$text"; return; }
|
|
curl -fsS --max-time 15 \
|
|
"https://api.telegram.org/bot${TELEGRAM_BOT_TOKEN}/sendMessage" \
|
|
--data-urlencode "chat_id=${TELEGRAM_CHAT_ID}" \
|
|
--data-urlencode "text=${text}" >/dev/null 2>&1 || echo "Telegram 发送失败" >&2
|
|
}
|
|
|
|
# 查 runner 状态。响应形如:
|
|
# {"runners":[{"id":6,"name":"mac-pangolin-2","status":"online","busy":false,...}],...}
|
|
resp=$(curl -fsS --max-time 15 -H "Authorization: token ${GITEA_TOKEN}" \
|
|
"${GITEA_URL%/}/api/v1/repos/${GITEA_REPO}/actions/runners" 2>/dev/null) || {
|
|
# API 取不到(监控主机网络抖动 / relay 没起 / gitea 宕):不计入去抖,避免误报。
|
|
echo "runner-deadman: gitea API 不可达,跳过本次(不改状态)" >&2
|
|
exit 0
|
|
}
|
|
|
|
# 提取该 runner 的 status(优先 jq,缺 jq 退回 grep)。
|
|
if command -v jq >/dev/null 2>&1; then
|
|
status=$(printf '%s' "$resp" | jq -r --arg n "$RUNNER_NAME" \
|
|
'.runners[]? | select(.name==$n) | .status' 2>/dev/null)
|
|
else
|
|
status=$(printf '%s' "$resp" \
|
|
| grep -oE "\"name\":\"${RUNNER_NAME}\"[^}]*\"status\":\"[a-z]+\"" \
|
|
| grep -oE '"status":"[a-z]+"' | tail -1 | cut -d'"' -f4)
|
|
fi
|
|
|
|
if [ "$status" = "online" ]; then
|
|
if [ -f "$statef" ]; then
|
|
send_tg "🟢 [CI runner ${RUNNER_NAME}] 恢复在线"
|
|
rm -f "$statef"
|
|
fi
|
|
else
|
|
# status 为 offline / 空(runner 已从 gitea 掉线或不存在)。
|
|
fails=$(( $(cat "$statef" 2>/dev/null || echo 0) + 1 ))
|
|
echo "$fails" > "$statef"
|
|
if [ "$fails" -eq "$FAIL_THRESHOLD" ]; then
|
|
send_tg "🔴 [CI runner ${RUNNER_NAME}] 离线(status=${status:-缺失}),连续 ${fails} 次,CI 已停摆"
|
|
fi
|
|
fi
|