feat(deploy/bootstrap): 节点幂等初始化脚本(加固+pangolin用户+监控+防火墙)
全新 Debian 12 一键初始化,后续新机复用。四块: - 系统:apt 升级+基础包(含 python3-systemd)、时区、swap 补足、sysctl(BBR/fq/句柄/队列) - Pangolin:建 pangolin 非特权系统用户(/var/lib/pangolin, nologin) - 网络安全:ufw 默认拒入站、放行 SSH+443(tcp/udp);fail2ban sshd jail (精简 Debian 无 rsyslog/auth.log → backend=systemd 读 journald) - SSH 加固:root 仅密钥、可选禁密码(仅当已有 authorized_keys 时生效,防锁死) - 监控:pangolin-monitor(systemd timer,CPU/内存/磁盘/进程/出网 → Telegram, 异常+心跳);deadman-watch(EC2 侧主动探活,补"宕机/不上报"告警) 已在 racknerd(107.172.55.251)实跑验证全部步骤通过。密钥/真实配置 gitignore。 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
wangjia
@@ -0,0 +1,3 @@
|
||||
# 真实配置含密钥,不入 git(只提交 *.example 模板)
|
||||
bootstrap.env
|
||||
deadman.env
|
||||
@@ -0,0 +1,68 @@
|
||||
# 节点初始化(bootstrap)
|
||||
|
||||
全新 Debian 12 机器的**幂等**一键初始化。后续每台新机器照同样流程跑即可。
|
||||
覆盖四块:① 系统加固 ② Pangolin 运行环境 ③ Telegram 监控 ④ 网络安全。
|
||||
|
||||
## 一、目标机上跑(本地)
|
||||
|
||||
```bash
|
||||
cp bootstrap.env.example bootstrap.env
|
||||
vi bootstrap.env # 按需改;要监控就填 TELEGRAM_BOT_TOKEN / CHAT_ID
|
||||
sudo bash init.sh
|
||||
```
|
||||
|
||||
## 二、从本机远程跑(已配 ssh 免密别名,推荐)
|
||||
|
||||
```bash
|
||||
# 前提:已 ssh-copy-id,~/.ssh/config 里有该机别名(如 racknerd)
|
||||
rsync -az deploy/bootstrap/ racknerd:/root/bootstrap/
|
||||
ssh racknerd 'cd /root/bootstrap && bash init.sh'
|
||||
```
|
||||
|
||||
`init.sh` 幂等,可反复跑(改了 bootstrap.env 后重跑即生效)。
|
||||
|
||||
## init.sh 做了什么
|
||||
|
||||
| 步 | 内容 |
|
||||
|----|------|
|
||||
| ① 系统 | `apt upgrade` + 基础包(ufw/fail2ban/chrony/curl/jq…);时区;**swap**(不足则建 `/swapfile` 补到 `SWAP_SIZE_MB`);sysctl(**BBR**/fq/文件句柄/连接队列) |
|
||||
| ② 环境 | 建 **`pangolin` 非特权系统用户**(`/var/lib/pangolin`, nologin) |
|
||||
| ④ 防火墙 | `ufw` 默认拒入站;放行 SSH + `443/tcp`(REALITY)+ `443/udp`(Hy2);其余拒绝 |
|
||||
| ④ 暴破 | `fail2ban` sshd jail |
|
||||
| ① SSH | drop-in 加固:`root` 仅密钥、`MaxAuthTries`、可选**禁密码登录** |
|
||||
| ③ 监控 | 装 `pangolin-monitor` + systemd timer,定期经 Telegram 上报 |
|
||||
|
||||
### 🔒 防锁死红线
|
||||
禁用密码登录(`DISABLE_PASSWORD_AUTH=true`)**仅当 `/root/.ssh/authorized_keys` 非空时才生效**;
|
||||
否则自动保留密码登录并告警。所以**先 `ssh-copy-id` 再跑**,或确保公钥已装。
|
||||
|
||||
## 监控(③ 的细节)
|
||||
|
||||
- **节点侧**`pangolin-monitor`(本机 systemd timer,每 `MONITOR_INTERVAL_MIN` 分钟):
|
||||
采 CPU/内存/磁盘、受监控 systemd 单元存活(`MONITOR_UNITS`)、出网连通;
|
||||
**越界/掉线/断网 → 🔴 必报;正常 → 🟢**(心跳是否推送由 `MONITOR_SEND_OK_PULSE` 控制)。
|
||||
- **死信(不上报/宕机)**:节点宕了自己报不了,需在**常在线主机(EC2)**跑 `deadman-watch.sh`
|
||||
主动探活补这层:
|
||||
|
||||
```bash
|
||||
# 在 EC2 上
|
||||
sudo install -m 755 deploy/bootstrap/monitor/deadman-watch.sh /usr/local/bin/deadman-watch
|
||||
sudo cp deploy/bootstrap/deadman.env.example /etc/pangolin-deadman.env && sudo vi /etc/pangolin-deadman.env
|
||||
# 每 3 分钟探一次(用户 cron 即可,无需 root)
|
||||
( crontab -l 2>/dev/null; echo '*/3 * * * * /usr/local/bin/deadman-watch >/dev/null 2>&1' ) | crontab -
|
||||
```
|
||||
|
||||
节点端口连续失败到阈值 → 🔴 告警;恢复 → 🟢 通知。
|
||||
|
||||
## 文件
|
||||
|
||||
| 文件 | 说明 |
|
||||
|------|------|
|
||||
| `init.sh` | 主初始化(幂等) |
|
||||
| `bootstrap.env.example` | 配置模板(SSH/swap/防火墙/Telegram) |
|
||||
| `monitor/pangolin-monitor.sh` | 节点自监控上报 |
|
||||
| `monitor/pangolin-monitor.{service,timer}` | systemd(timer 的 `__INTERVAL__` 由 init.sh 渲染) |
|
||||
| `monitor/deadman-watch.sh` | EC2 侧死信探活 |
|
||||
| `deadman.env.example` | 死信守望配置模板 |
|
||||
|
||||
密钥类(`bootstrap.env`、`/etc/pangolin-monitor.env`、`/etc/pangolin-deadman.env`)**不入 git**。
|
||||
@@ -0,0 +1,37 @@
|
||||
# Pangolin 节点初始化配置(被 init.sh 读取)。
|
||||
# 用法:cp bootstrap.env.example bootstrap.env && 编辑 → 随 init.sh 一起拷到机器上。
|
||||
# 所有项都有合理默认;留空即用默认。敏感值(Telegram token)不要提交进 git。
|
||||
|
||||
# ── SSH 加固 ────────────────────────────────────────────────────────────────
|
||||
SSH_PORT=22
|
||||
# 是否禁用密码登录(仅当 root 已有 authorized_keys 时才会真正生效,否则自动跳过以防锁死)
|
||||
DISABLE_PASSWORD_AUTH=true
|
||||
|
||||
# ── 系统 ────────────────────────────────────────────────────────────────────
|
||||
TIMEZONE=Asia/Shanghai
|
||||
# 目标 swap 总量(MB)。当前 swap 不足且无 /swapfile 时,创建 /swapfile 补足。0=不管。
|
||||
SWAP_SIZE_MB=2048
|
||||
|
||||
# ── 防火墙(放行端口)───────────────────────────────────────────────────────
|
||||
# 数据面:REALITY(443/tcp)+ Hysteria2(443/udp)。多个 tcp/udp 端口用空格分隔。
|
||||
FW_ALLOW_TCP="443"
|
||||
FW_ALLOW_UDP="443"
|
||||
|
||||
# ── 监控(Telegram)──────────────────────────────────────────────────────────
|
||||
# 留空则跳过监控安装。token 找 @BotFather 建;chat_id 找 @userinfobot 取。
|
||||
TELEGRAM_BOT_TOKEN=""
|
||||
TELEGRAM_CHAT_ID=""
|
||||
# 本节点在告警里显示的名字
|
||||
NODE_NAME="racknerd-la"
|
||||
# 上报间隔(分钟)
|
||||
MONITOR_INTERVAL_MIN=5
|
||||
# 阈值(百分比);超过即告警
|
||||
CPU_ALERT_PCT=90
|
||||
MEM_ALERT_PCT=90
|
||||
DISK_ALERT_PCT=85
|
||||
# 要监控存活的 systemd 单元(空格分隔;不存在的会被忽略)
|
||||
MONITOR_UNITS="sing-box pangolin-agent"
|
||||
# 出网连通探测目标
|
||||
MONITOR_NET_PROBE="https://www.cloudflare.com/cdn-cgi/trace"
|
||||
# 周期性"健康"心跳是否也推送(true=每次都报,false=仅异常时报)
|
||||
MONITOR_SEND_OK_PULSE=true
|
||||
@@ -0,0 +1,7 @@
|
||||
# deadman-watch 配置(放常在线主机如 EC2 的 /etc/pangolin-deadman.env)。
|
||||
TELEGRAM_BOT_TOKEN=""
|
||||
TELEGRAM_CHAT_ID=""
|
||||
# 要守望的节点:name=host:port 空格分隔(port 取对外的代理端口,如 REALITY 443)
|
||||
NODES="racknerd-la=107.172.55.251:443"
|
||||
# 连续探测失败几次才告警(去抖动)
|
||||
FAIL_THRESHOLD=2
|
||||
Executable
+230
@@ -0,0 +1,230 @@
|
||||
#!/usr/bin/env bash
|
||||
# init.sh —— Pangolin 节点一键初始化(幂等)。在全新 Debian 12 上以 root 运行一次,
|
||||
# 重复运行安全。涵盖四块:① 系统加固 ② Pangolin 运行环境 ③ Telegram 监控 ④ 网络安全。
|
||||
#
|
||||
# 用法(机器上):
|
||||
# cp bootstrap.env.example bootstrap.env && vi bootstrap.env # 按需改,填 Telegram
|
||||
# sudo bash init.sh
|
||||
#
|
||||
# 远程一条命令(从本机,已配 ssh 免密别名 racknerd):
|
||||
# rsync -az deploy/bootstrap/ racknerd:/root/bootstrap/ && \
|
||||
# ssh racknerd 'cd /root/bootstrap && bash init.sh'
|
||||
#
|
||||
# 设计红线:禁用密码登录前会先确认 root 已有 authorized_keys,否则跳过并告警(防锁死)。
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
# ── 日志 ──────────────────────────────────────────────────────────────────────
|
||||
log() { printf '\033[1;32m[init]\033[0m %s\n' "$*"; }
|
||||
warn() { printf '\033[1;33m[init]\033[0m %s\n' "$*" >&2; }
|
||||
die() { printf '\033[1;31m[init]\033[0m %s\n' "$*" >&2; exit 1; }
|
||||
|
||||
[ "$(id -u)" -eq 0 ] || die "请以 root 运行:sudo bash $0"
|
||||
[ -r /etc/debian_version ] || warn "非 Debian 系发行版,部分步骤可能需调整"
|
||||
|
||||
# ── 配置 ──────────────────────────────────────────────────────────────────────
|
||||
# 默认值;bootstrap.env 覆盖。
|
||||
SSH_PORT=22
|
||||
DISABLE_PASSWORD_AUTH=true
|
||||
TIMEZONE=Asia/Shanghai
|
||||
SWAP_SIZE_MB=2048
|
||||
FW_ALLOW_TCP="443"
|
||||
FW_ALLOW_UDP="443"
|
||||
TELEGRAM_BOT_TOKEN=""
|
||||
TELEGRAM_CHAT_ID=""
|
||||
NODE_NAME="$(hostname)"
|
||||
MONITOR_INTERVAL_MIN=5
|
||||
CPU_ALERT_PCT=90
|
||||
MEM_ALERT_PCT=90
|
||||
DISK_ALERT_PCT=85
|
||||
MONITOR_UNITS="sing-box pangolin-agent"
|
||||
MONITOR_NET_PROBE="https://www.cloudflare.com/cdn-cgi/trace"
|
||||
MONITOR_SEND_OK_PULSE=true
|
||||
|
||||
if [ -f "$SCRIPT_DIR/bootstrap.env" ]; then
|
||||
log "读取配置 $SCRIPT_DIR/bootstrap.env"
|
||||
# shellcheck disable=SC1091
|
||||
. "$SCRIPT_DIR/bootstrap.env"
|
||||
else
|
||||
warn "未找到 bootstrap.env,全部使用默认值(监控将因缺 Telegram 配置而跳过)"
|
||||
fi
|
||||
|
||||
export DEBIAN_FRONTEND=noninteractive
|
||||
|
||||
# ── ① 系统:更新 + 基础包 + 时区 ──────────────────────────────────────────────
|
||||
step_base() {
|
||||
log "[1/8] apt 更新 + 基础工具"
|
||||
apt-get update -q
|
||||
apt-get -y -o Dpkg::Options::=--force-confold upgrade
|
||||
apt-get install -y --no-install-recommends \
|
||||
ufw fail2ban chrony curl jq ca-certificates gnupg lsb-release \
|
||||
iproute2 procps python3-systemd
|
||||
# python3-systemd:精简 Debian 无 rsyslog,fail2ban 走 systemd 后端读 journald 必需。
|
||||
log " 时区 → $TIMEZONE"
|
||||
timedatectl set-timezone "$TIMEZONE" 2>/dev/null || warn "设置时区失败(无 systemd-timedated?)"
|
||||
}
|
||||
|
||||
# ── ① swap(512MB 机器必需)────────────────────────────────────────────────────
|
||||
step_swap() {
|
||||
log "[2/8] swap"
|
||||
if [ "${SWAP_SIZE_MB:-0}" -le 0 ]; then log " SWAP_SIZE_MB=0,跳过"; return; fi
|
||||
local cur; cur="$(free -m | awk '/^Swap:/{print $2}')"
|
||||
if [ "${cur:-0}" -ge "$SWAP_SIZE_MB" ]; then
|
||||
log " 当前 swap ${cur}MB ≥ 目标 ${SWAP_SIZE_MB}MB,跳过"; return
|
||||
fi
|
||||
if [ -f /swapfile ]; then log " /swapfile 已存在,跳过新建"; return; fi
|
||||
local add=$(( SWAP_SIZE_MB - cur ))
|
||||
log " 创建 /swapfile ${add}MB(现有 ${cur}MB → 目标 ${SWAP_SIZE_MB}MB)"
|
||||
fallocate -l "${add}M" /swapfile 2>/dev/null || dd if=/dev/zero of=/swapfile bs=1M count="$add" status=none
|
||||
chmod 600 /swapfile
|
||||
mkswap /swapfile >/dev/null
|
||||
swapon /swapfile
|
||||
grep -q '^/swapfile ' /etc/fstab || echo '/swapfile none swap sw 0 0' >> /etc/fstab
|
||||
}
|
||||
|
||||
# ── ① sysctl 网络调优(BBR 等)────────────────────────────────────────────────
|
||||
step_sysctl() {
|
||||
log "[3/8] sysctl 调优(BBR/fq/文件句柄/连接队列)"
|
||||
cat > /etc/sysctl.d/99-pangolin.conf <<'SYSCTL'
|
||||
# Pangolin 节点网络调优
|
||||
net.core.default_qdisc = fq
|
||||
net.ipv4.tcp_congestion_control = bbr
|
||||
net.core.somaxconn = 4096
|
||||
net.core.netdev_max_backlog = 16384
|
||||
net.ipv4.tcp_fastopen = 3
|
||||
net.ipv4.tcp_max_syn_backlog = 8192
|
||||
fs.file-max = 1048576
|
||||
SYSCTL
|
||||
sysctl -q --system || warn "sysctl 应用有告警(BBR 模块可能需较新内核;Debian 12 内核已内置)"
|
||||
local cc; cc="$(sysctl -n net.ipv4.tcp_congestion_control 2>/dev/null || echo '?')"
|
||||
log " tcp_congestion_control = $cc"
|
||||
}
|
||||
|
||||
# ── ② Pangolin 专用非特权用户 ─────────────────────────────────────────────────
|
||||
step_user() {
|
||||
log "[4/8] 创建 pangolin 系统用户(非特权)"
|
||||
if id -u pangolin >/dev/null 2>&1; then
|
||||
log " 用户 pangolin 已存在,跳过"
|
||||
else
|
||||
useradd --system --create-home --home-dir /var/lib/pangolin \
|
||||
--shell /usr/sbin/nologin pangolin
|
||||
log " 已创建 pangolin(home=/var/lib/pangolin, nologin)"
|
||||
fi
|
||||
install -d -o pangolin -g pangolin -m 750 /var/lib/pangolin
|
||||
}
|
||||
|
||||
# ── ④ 防火墙(ufw)────────────────────────────────────────────────────────────
|
||||
step_firewall() {
|
||||
log "[5/8] ufw 防火墙"
|
||||
ufw default deny incoming >/dev/null
|
||||
ufw default allow outgoing >/dev/null
|
||||
ufw allow "${SSH_PORT}/tcp" >/dev/null
|
||||
for p in $FW_ALLOW_TCP; do [ -n "$p" ] && ufw allow "${p}/tcp" >/dev/null; done
|
||||
for p in $FW_ALLOW_UDP; do [ -n "$p" ] && ufw allow "${p}/udp" >/dev/null; done
|
||||
ufw --force enable >/dev/null
|
||||
log " 放行:SSH ${SSH_PORT}/tcp · TCP[$FW_ALLOW_TCP] · UDP[$FW_ALLOW_UDP],其余 deny"
|
||||
}
|
||||
|
||||
# ── ④ fail2ban(SSH 暴破防护)─────────────────────────────────────────────────
|
||||
step_fail2ban() {
|
||||
log "[6/8] fail2ban(sshd jail)"
|
||||
# 精简 Debian 无 rsyslog/auth.log,SSH 日志只在 journald → 用 systemd 后端读取。
|
||||
cat > /etc/fail2ban/jail.d/pangolin.local <<F2B
|
||||
[DEFAULT]
|
||||
allowipv6 = auto
|
||||
|
||||
[sshd]
|
||||
enabled = true
|
||||
backend = systemd
|
||||
port = ${SSH_PORT}
|
||||
maxretry = 5
|
||||
findtime = 10m
|
||||
bantime = 1h
|
||||
F2B
|
||||
systemctl enable fail2ban >/dev/null 2>&1 || true
|
||||
systemctl restart fail2ban || warn "fail2ban 重启失败"
|
||||
}
|
||||
|
||||
# ── ① SSH 加固(先验证 authorized_keys,防锁死)────────────────────────────────
|
||||
step_ssh() {
|
||||
log "[7/8] SSH 加固"
|
||||
local dropin=/etc/ssh/sshd_config.d/99-pangolin.conf
|
||||
local akf=/root/.ssh/authorized_keys
|
||||
{
|
||||
echo "# Pangolin 加固(由 init.sh 生成)"
|
||||
echo "Port ${SSH_PORT}"
|
||||
echo "PubkeyAuthentication yes"
|
||||
echo "PermitRootLogin prohibit-password"
|
||||
echo "ChallengeResponseAuthentication no"
|
||||
echo "X11Forwarding no"
|
||||
echo "MaxAuthTries 4"
|
||||
if [ "${DISABLE_PASSWORD_AUTH}" = "true" ]; then
|
||||
if [ -s "$akf" ]; then
|
||||
echo "PasswordAuthentication no"
|
||||
else
|
||||
warn " root 无 authorized_keys → 保留密码登录以防锁死(请先 ssh-copy-id 再重跑)"
|
||||
echo "PasswordAuthentication yes"
|
||||
fi
|
||||
else
|
||||
echo "PasswordAuthentication yes"
|
||||
fi
|
||||
} > "$dropin"
|
||||
|
||||
if sshd -t; then
|
||||
systemctl reload ssh 2>/dev/null || systemctl reload sshd 2>/dev/null || warn "sshd reload 失败"
|
||||
log " 已应用 $dropin(Port ${SSH_PORT}, root 仅密钥)"
|
||||
else
|
||||
warn " sshd 配置校验失败,已写入但未 reload,请人工检查 $dropin"
|
||||
fi
|
||||
}
|
||||
|
||||
# ── ③ 监控(Telegram 上报 + 阈值告警)─────────────────────────────────────────
|
||||
step_monitor() {
|
||||
log "[8/8] 监控(Telegram)"
|
||||
if [ -z "${TELEGRAM_BOT_TOKEN}" ] || [ -z "${TELEGRAM_CHAT_ID}" ]; then
|
||||
warn " 未配置 TELEGRAM_BOT_TOKEN/CHAT_ID,跳过监控安装"
|
||||
return
|
||||
fi
|
||||
install -m 755 "$SCRIPT_DIR/monitor/pangolin-monitor.sh" /usr/local/bin/pangolin-monitor
|
||||
umask 077
|
||||
cat > /etc/pangolin-monitor.env <<MENV
|
||||
TELEGRAM_BOT_TOKEN=${TELEGRAM_BOT_TOKEN}
|
||||
TELEGRAM_CHAT_ID=${TELEGRAM_CHAT_ID}
|
||||
NODE_NAME=${NODE_NAME}
|
||||
CPU_ALERT_PCT=${CPU_ALERT_PCT}
|
||||
MEM_ALERT_PCT=${MEM_ALERT_PCT}
|
||||
DISK_ALERT_PCT=${DISK_ALERT_PCT}
|
||||
MONITOR_UNITS=${MONITOR_UNITS}
|
||||
MONITOR_NET_PROBE=${MONITOR_NET_PROBE}
|
||||
MONITOR_SEND_OK_PULSE=${MONITOR_SEND_OK_PULSE}
|
||||
MENV
|
||||
umask 022
|
||||
sed -e "s/__INTERVAL__/${MONITOR_INTERVAL_MIN}/g" \
|
||||
"$SCRIPT_DIR/monitor/pangolin-monitor.timer" > /etc/systemd/system/pangolin-monitor.timer
|
||||
install -m 644 "$SCRIPT_DIR/monitor/pangolin-monitor.service" /etc/systemd/system/pangolin-monitor.service
|
||||
systemctl daemon-reload
|
||||
systemctl enable --now pangolin-monitor.timer >/dev/null
|
||||
log " 监控已装:每 ${MONITOR_INTERVAL_MIN} 分钟上报;发送一条启动通知"
|
||||
/usr/local/bin/pangolin-monitor --boot || true
|
||||
}
|
||||
|
||||
main() {
|
||||
step_base
|
||||
step_swap
|
||||
step_sysctl
|
||||
step_user
|
||||
step_firewall
|
||||
step_fail2ban
|
||||
step_ssh
|
||||
step_monitor
|
||||
log "✅ 初始化完成。摘要:"
|
||||
echo " OS : $(. /etc/os-release; echo "$PRETTY_NAME")"
|
||||
echo " swap : $(free -m | awk '/^Swap:/{print $2"MB"}')"
|
||||
echo " bbr : $(sysctl -n net.ipv4.tcp_congestion_control 2>/dev/null)"
|
||||
echo " ufw : $(ufw status | head -1)"
|
||||
echo " user : $(id pangolin 2>/dev/null || echo '未创建')"
|
||||
echo " 监控 : $(systemctl is-active pangolin-monitor.timer 2>/dev/null || echo '未装')"
|
||||
}
|
||||
|
||||
main "$@"
|
||||
Executable
+48
@@ -0,0 +1,48 @@
|
||||
#!/usr/bin/env bash
|
||||
# deadman-watch —— 死信守望。放在【常在线主机】(如 EC2)上,主动 TCP 探活各节点端口;
|
||||
# 连续失败到阈值 → 告警(节点宕机/不上报时,节点自己报不了,这层补上)。恢复也通知。
|
||||
#
|
||||
# 配置 /etc/pangolin-deadman.env:
|
||||
# TELEGRAM_BOT_TOKEN=... TELEGRAM_CHAT_ID=...
|
||||
# NODES="racknerd-la=107.172.55.251:443 tokyo=1.2.3.4:443" # name=host:port 空格分隔
|
||||
# FAIL_THRESHOLD=2 # 连续失败几次才告警(去抖)
|
||||
#
|
||||
# EC2 上每 3 分钟跑一次(cron,用户态即可):
|
||||
# */3 * * * * /usr/local/bin/deadman-watch >/dev/null 2>&1
|
||||
set -uo pipefail
|
||||
|
||||
ENV_FILE=/etc/pangolin-deadman.env
|
||||
[ -r "$ENV_FILE" ] || { echo "缺少 $ENV_FILE" >&2; exit 1; }
|
||||
# shellcheck disable=SC1090
|
||||
. "$ENV_FILE"
|
||||
: "${FAIL_THRESHOLD:=2}"
|
||||
STATE_DIR=/var/lib/pangolin-deadman
|
||||
mkdir -p "$STATE_DIR"
|
||||
|
||||
send_tg() {
|
||||
local text="$1"
|
||||
[ -n "${TELEGRAM_BOT_TOKEN:-}" ] && [ -n "${TELEGRAM_CHAT_ID:-}" ] || { echo "$text"; return; }
|
||||
curl -fsS --max-time 15 \
|
||||
"https://api.telegram.org/bot${TELEGRAM_BOT_TOKEN}/sendMessage" \
|
||||
--data-urlencode "chat_id=${TELEGRAM_CHAT_ID}" \
|
||||
--data-urlencode "text=${text}" >/dev/null 2>&1 || echo "Telegram 发送失败" >&2
|
||||
}
|
||||
|
||||
probe() { timeout 8 bash -c "exec 3<>/dev/tcp/$1/$2" 2>/dev/null; }
|
||||
|
||||
for entry in ${NODES:-}; do
|
||||
name="${entry%%=*}"; hp="${entry#*=}"; host="${hp%%:*}"; port="${hp##*:}"
|
||||
statef="${STATE_DIR}/${name}.fail"
|
||||
if probe "$host" "$port"; then
|
||||
if [ -f "$statef" ]; then
|
||||
send_tg "🟢 [${name}] 恢复可达(${host}:${port})"
|
||||
rm -f "$statef"
|
||||
fi
|
||||
else
|
||||
fails=$(( $(cat "$statef" 2>/dev/null || echo 0) + 1 ))
|
||||
echo "$fails" > "$statef"
|
||||
if [ "$fails" -eq "$FAIL_THRESHOLD" ]; then
|
||||
send_tg "🔴 [${name}] 不可达/疑似宕机(${host}:${port}),连续 ${fails} 次探测失败"
|
||||
fi
|
||||
fi
|
||||
done
|
||||
@@ -0,0 +1,12 @@
|
||||
[Unit]
|
||||
Description=Pangolin 节点监控(采集并经 Telegram 上报)
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
EnvironmentFile=/etc/pangolin-monitor.env
|
||||
ExecStart=/usr/local/bin/pangolin-monitor
|
||||
# 监控只读系统状态,降权运行
|
||||
User=root
|
||||
Nice=10
|
||||
Executable
+95
@@ -0,0 +1,95 @@
|
||||
#!/usr/bin/env bash
|
||||
# pangolin-monitor —— 节点自监控:采集 CPU/内存/磁盘/进程/网络,经 Telegram 上报。
|
||||
# 由 systemd timer 定期触发(见 pangolin-monitor.timer)。配置在 /etc/pangolin-monitor.env。
|
||||
#
|
||||
# 阈值越界 / 受监控单元掉线 / 出网不通 → 标 🔴 并必报;
|
||||
# 一切正常 → 🟢,是否推送由 MONITOR_SEND_OK_PULSE 决定(心跳)。
|
||||
# --boot:发送一条"初始化完成"启动通知。
|
||||
#
|
||||
# 注意:节点宕机后无法自报("不上报"类告警)需在常在线主机上跑 deadman-watch.sh(见 README)。
|
||||
set -uo pipefail
|
||||
|
||||
ENV_FILE=/etc/pangolin-monitor.env
|
||||
[ -r "$ENV_FILE" ] || { echo "缺少 $ENV_FILE" >&2; exit 1; }
|
||||
# shellcheck disable=SC1090
|
||||
. "$ENV_FILE"
|
||||
|
||||
: "${NODE_NAME:=$(hostname)}"
|
||||
: "${CPU_ALERT_PCT:=90}"; : "${MEM_ALERT_PCT:=90}"; : "${DISK_ALERT_PCT:=85}"
|
||||
: "${MONITOR_UNITS:=}"; : "${MONITOR_NET_PROBE:=https://www.cloudflare.com/cdn-cgi/trace}"
|
||||
: "${MONITOR_SEND_OK_PULSE:=true}"
|
||||
|
||||
send_tg() {
|
||||
local text="$1"
|
||||
[ -n "${TELEGRAM_BOT_TOKEN:-}" ] && [ -n "${TELEGRAM_CHAT_ID:-}" ] || { echo "$text"; return; }
|
||||
curl -fsS --max-time 15 \
|
||||
"https://api.telegram.org/bot${TELEGRAM_BOT_TOKEN}/sendMessage" \
|
||||
--data-urlencode "chat_id=${TELEGRAM_CHAT_ID}" \
|
||||
--data-urlencode "text=${text}" \
|
||||
--data "disable_web_page_preview=true" >/dev/null 2>&1 \
|
||||
|| echo "Telegram 发送失败" >&2
|
||||
}
|
||||
|
||||
# CPU% —— /proc/stat 两次采样差值
|
||||
cpu_pct() {
|
||||
local a_idle a_tot b_idle b_tot
|
||||
read -r a_idle a_tot < <(awk '/^cpu /{t=0;for(i=2;i<=NF;i++)t+=$i;print $5,t}' /proc/stat)
|
||||
sleep 1
|
||||
read -r b_idle b_tot < <(awk '/^cpu /{t=0;for(i=2;i<=NF;i++)t+=$i;print $5,t}' /proc/stat)
|
||||
awk -v ai="$a_idle" -v at="$a_tot" -v bi="$b_idle" -v bt="$b_tot" \
|
||||
'BEGIN{ d=bt-at; if(d<=0){print 0} else {printf "%.0f", 100*(1-(bi-ai)/d)} }'
|
||||
}
|
||||
|
||||
mem_pct() { free -m | awk '/^Mem:/{printf "%.0f", ($2-$7)/$2*100}'; } # 用 available 列
|
||||
disk_pct() { df -P / | awk 'NR==2{gsub("%","",$5); print $5}'; }
|
||||
|
||||
main() {
|
||||
local boot=false
|
||||
[ "${1:-}" = "--boot" ] && boot=true
|
||||
|
||||
local cpu mem disk net alert=0 lines=()
|
||||
cpu="$(cpu_pct)"; mem="$(mem_pct)"; disk="$(disk_pct)"
|
||||
|
||||
[ "${cpu:-0}" -ge "$CPU_ALERT_PCT" ] && alert=1
|
||||
[ "${mem:-0}" -ge "$MEM_ALERT_PCT" ] && alert=1
|
||||
[ "${disk:-0}" -ge "$DISK_ALERT_PCT" ] && alert=1
|
||||
|
||||
lines+=("CPU ${cpu}% 内存 ${mem}% 磁盘 ${disk}%")
|
||||
|
||||
# 进程/单元存活
|
||||
local u st ustr=""
|
||||
for u in $MONITOR_UNITS; do
|
||||
if ! systemctl list-unit-files "${u}.service" >/dev/null 2>&1 \
|
||||
&& ! systemctl cat "${u}.service" >/dev/null 2>&1; then
|
||||
continue # 单元不存在(此节点未部署该组件)→ 忽略
|
||||
fi
|
||||
st="$(systemctl is-active "${u}.service" 2>/dev/null || true)"
|
||||
[ "$st" = "active" ] || alert=1
|
||||
ustr+="${u}=${st} "
|
||||
done
|
||||
[ -n "$ustr" ] && lines+=("进程: ${ustr}")
|
||||
|
||||
# 出网连通
|
||||
if curl -fsS --max-time 8 "$MONITOR_NET_PROBE" >/dev/null 2>&1; then net="ok"; else net="FAIL"; alert=1; fi
|
||||
lines+=("出网: ${net}")
|
||||
|
||||
local head body ts
|
||||
ts="$(date -u '+%Y-%m-%d %H:%M:%SZ')"
|
||||
if $boot; then
|
||||
head="🟢 [${NODE_NAME}] 初始化完成 / 监控已上线"
|
||||
elif [ "$alert" -eq 1 ]; then
|
||||
head="🔴 [${NODE_NAME}] 异常告警"
|
||||
else
|
||||
head="🟢 [${NODE_NAME}] 正常"
|
||||
fi
|
||||
body="${head}\n${ts}\n"
|
||||
printf -v body '%s\n%s\n' "$head" "$ts"
|
||||
local l; for l in "${lines[@]}"; do body+="${l}"$'\n'; done
|
||||
|
||||
if $boot || [ "$alert" -eq 1 ] || [ "$MONITOR_SEND_OK_PULSE" = "true" ]; then
|
||||
send_tg "$body"
|
||||
fi
|
||||
return 0
|
||||
}
|
||||
|
||||
main "$@"
|
||||
@@ -0,0 +1,11 @@
|
||||
[Unit]
|
||||
Description=每 __INTERVAL__ 分钟运行 pangolin-monitor
|
||||
|
||||
[Timer]
|
||||
OnBootSec=2min
|
||||
OnUnitActiveSec=__INTERVAL__min
|
||||
AccuracySec=20s
|
||||
Persistent=true
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
Reference in New Issue
Block a user