feat(backup): 备份与灾备演练系统 [tsk_bYv-CoKSwgAC]
实现 xtrabackup + binlog 增量备份、age 加密异地存储与月度 DR 演练。 ## 交付内容 ### deploy/backup/ —— 备份容器 - Dockerfile：Ubuntu 22.04 + xtrabackup 8.0 + age v1.2.0 + AWS CLI v2 + pg_client - entrypoint.sh：初始化 AWS 凭据 / age 公钥，启动 crond - crontab.txt：全量 02:00 / 增量每 15min / DR 演练每月 1 日 - scripts/lib.sh：日志 / age 加密 / S3 上传 / 连接测试等共用函数 - scripts/backup-full.sh：MySQL xtrabackup 全量 + pg_dump 全量 + SQLite 备份 - scripts/backup-inc.sh：MySQL xtrabackup 增量 + binlog flush & 归档 + pg_dump 快照 - scripts/restore.sh：从 S3 下载、解密、prepare / pg_restore 完整恢复 - scripts/dr-drill.sh：月度演练 —— RPO 验证 / 备份完整性 / MySQL 临时容器恢复 / RTO 度量 - scripts/verify-rpo-rto.sh：实时 RPO/RTO 状态检查（S3 最新备份时间戳） - backup.env.example：配置模板（MySQL / PG / S3 / age 公钥） ### 修改已有文件 - deploy/docker-compose.yml：新增 pangolin-backup 服务 - deploy/scripts/gen-secrets.sh：幂等生成 age 密钥对 + backup.env 模板 - .gitignore：排除 age 私钥 / backup.env - deploy/README.md：容器一览 + 备份快速命令 ### docs/备份与灾备.md S3 初始化、IAM 权限设计、age 密钥管理、MySQL 权限、DR 恢复步骤、RTO 测量标准 ## 设计要点 - RPO ≤ 15min：每 15min xtrabackup 增量 + binlog flush + pg_dump 快照 - RTO ≤ 4h：月度演练度量，估算恢复时长约 2h - 独立身份账号：备份 IAM 仅有 s3:PutObject，与生产账号完全隔离 - age 加密：公钥存服务器，私钥离线保存，S3 中无明文 - 无 crontab：cron 在容器内运行，绕过 ec2-user crontab 限制 - 幂等部署：backup.env 存在则跳过，age 密钥对存在则跳过 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
wangjia
+371
@@ -0,0 +1,371 @@
|
||||
# Pangolin 备份与灾备手册
|
||||
|
||||
> 目标：**RPO ≤ 15min，RTO ≤ 4h**
|
||||
>
|
||||
> 策略：xtrabackup 每日全量 + binlog 每 15min 增量，age 加密，异地 S3（独立身份账号）存储；月度恢复演练自动化。
|
||||
|
||||
---
|
||||
|
||||
## 架构一览
|
||||
|
||||
```
|
||||
EC2 (生产)
|
||||
├── MySQL (jiu_mysql, marzban)
|
||||
│ ├── 每日 02:00 xtrabackup 全量 ─────────────────────┐
|
||||
│ └── 每 15min xtrabackup 增量 + binlog flush ──────┤
|
||||
├── PostgreSQL (blog, umami) │
|
||||
│ ├── 每日 02:00 pg_dump 全量 ────────────────────── │
|
||||
│ └── 每 15min pg_dump 快照 ────────────────────── age 加密
|
||||
└── Marzban SQLite │
|
||||
└── 每日 02:00 sqlite3 .backup ─────────────────────┘
|
||||
↓
|
||||
S3（独立 IAM，仅 PutObject）
|
||||
s3://pangolin-backup-XXXXXX/
|
||||
├── mysql/full/
|
||||
├── mysql/inc/
|
||||
├── mysql/binlog/
|
||||
├── postgres/full/
|
||||
├── postgres/inc/
|
||||
├── marzban/
|
||||
└── dr-reports/
|
||||
|
||||
每月 1 日 03:00 DR 演练：下载 → 解密 → 验证 → 报告
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 初始化步骤（首次部署）
|
||||
|
||||
### 1. 创建备份 S3 Bucket（独立 IAM 账号）
|
||||
|
||||
> 使用**与生产 AWS 账号隔离**的独立 IAM 用户，做到数据面与备份面隔离。
|
||||
|
||||
```bash
|
||||
# 在 AWS Console 或 CLI 执行（使用管理员账号）：
|
||||
aws s3api create-bucket \
|
||||
--bucket pangolin-backup-$(openssl rand -hex 4) \
|
||||
--region ap-southeast-1 \
|
||||
--create-bucket-configuration LocationConstraint=ap-southeast-1
|
||||
|
||||
# 开启版本控制（防误删）
|
||||
aws s3api put-bucket-versioning \
|
||||
--bucket YOUR_BUCKET_NAME \
|
||||
--versioning-configuration Status=Enabled
|
||||
|
||||
# 配置生命周期：标准 IA 30 天后 Glacier，365 天后删除
|
||||
cat > /tmp/lifecycle.json << 'EOF'
|
||||
{
|
||||
"Rules": [{
|
||||
"ID": "backup-lifecycle",
|
||||
"Status": "Enabled",
|
||||
"Filter": {},
|
||||
"Transitions": [
|
||||
{"Days": 30, "StorageClass": "STANDARD_IA"},
|
||||
{"Days": 90, "StorageClass": "GLACIER"}
|
||||
],
|
||||
"Expiration": {"Days": 365}
|
||||
}]
|
||||
}
|
||||
EOF
|
||||
aws s3api put-bucket-lifecycle-configuration \
|
||||
--bucket YOUR_BUCKET_NAME \
|
||||
--lifecycle-configuration file:///tmp/lifecycle.json
|
||||
```
|
||||
|
||||
**创建只写 IAM 用户（备份服务器使用）：**
|
||||
|
||||
```json
|
||||
// backup-only-policy.json（只允许 PutObject，不允许读取或删除）
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [{
|
||||
"Effect": "Allow",
|
||||
"Action": [
|
||||
"s3:PutObject",
|
||||
"s3:PutObjectAcl",
|
||||
"s3:GetBucketLocation",
|
||||
"s3:ListBucket"
|
||||
],
|
||||
"Resource": [
|
||||
"arn:aws:s3:::YOUR_BUCKET_NAME",
|
||||
"arn:aws:s3:::YOUR_BUCKET_NAME/*"
|
||||
]
|
||||
}]
|
||||
}
|
||||
```
|
||||
|
||||
```bash
|
||||
aws iam create-user --user-name pangolin-backup-writer
|
||||
aws iam put-user-policy --user-name pangolin-backup-writer \
|
||||
--policy-name backup-only --policy-document file://backup-only-policy.json
|
||||
aws iam create-access-key --user-name pangolin-backup-writer
|
||||
# 记录 AccessKeyId 和 SecretAccessKey
|
||||
```
|
||||
|
||||
**创建只读 IAM 用户（DR 演练时恢复用）：**
|
||||
|
||||
```json
|
||||
// restore-policy.json
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [{
|
||||
"Effect": "Allow",
|
||||
"Action": ["s3:GetObject", "s3:ListBucket"],
|
||||
"Resource": [
|
||||
"arn:aws:s3:::YOUR_BUCKET_NAME",
|
||||
"arn:aws:s3:::YOUR_BUCKET_NAME/*"
|
||||
]
|
||||
}]
|
||||
}
|
||||
```
|
||||
|
||||
### 2. 生成 age 加密密钥对
|
||||
|
||||
```bash
|
||||
# 在备份容器内（或本机有 age 的地方）生成：
|
||||
docker run --rm pangolin-backup:local age-keygen 2>&1 | tee age-keypair.txt
|
||||
|
||||
# 输出示例：
|
||||
# Public key: age1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
|
||||
# AGE-SECRET-KEY-1XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
|
||||
|
||||
# ⚠️ 重要安全操作：
|
||||
# 1. 将私钥（AGE-SECRET-KEY-1...）存入密码管理器（如 Bitwarden）
|
||||
# 2. 打印并存入保险箱（物理备份）
|
||||
# 3. 删除本地私钥文件，服务器上只保留公钥
|
||||
rm age-keypair.txt # 删除含私钥的文件
|
||||
```
|
||||
|
||||
或使用 `gen-secrets.sh`（需本机已安装 age-keygen）：
|
||||
|
||||
```bash
|
||||
cd deploy/
|
||||
./scripts/gen-secrets.sh # 自动生成，见输出指示处理私钥
|
||||
```
|
||||
|
||||
### 3. 配置备份环境变量
|
||||
|
||||
```bash
|
||||
cd deploy/
|
||||
cp backup/backup.env.example secrets/backup.env
|
||||
vim secrets/backup.env # 填写以下关键项：
|
||||
# - BACKUP_S3_BUCKET、BACKUP_AWS_ACCESS_KEY_ID/SECRET
|
||||
# - AGE_PUBLIC_KEY（从步骤 2 获取）
|
||||
# - MYSQL_USER/PASSWORD、PG_USER/PASSWORD
|
||||
# - PG_DATABASES（如 blog,umami）
|
||||
```
|
||||
|
||||
### 4. 准备 MySQL 备份账号
|
||||
|
||||
在 MySQL 中执行（通过 mysql 客户端连接 jiu_mysql）：
|
||||
|
||||
```sql
|
||||
CREATE USER 'backup_user'@'%' IDENTIFIED BY 'STRONG_PASSWORD';
|
||||
GRANT RELOAD, LOCK TABLES, REPLICATION CLIENT, PROCESS, SUPER,
|
||||
CREATE, INSERT, SELECT, BACKUP_ADMIN ON *.* TO 'backup_user'@'%';
|
||||
FLUSH PRIVILEGES;
|
||||
```
|
||||
|
||||
确认 binlog 已开启：
|
||||
|
||||
```sql
|
||||
SHOW VARIABLES LIKE 'log_bin'; -- 应为 ON
|
||||
SHOW BINARY LOGS; -- 列出 binlog 文件
|
||||
```
|
||||
|
||||
若未开启，在 MySQL 配置中添加（需重启 MySQL）：
|
||||
|
||||
```ini
|
||||
[mysqld]
|
||||
log_bin = mysql-bin
|
||||
binlog_format = ROW
|
||||
expire_logs_days = 3
|
||||
```
|
||||
|
||||
### 5. 构建并启动备份容器
|
||||
|
||||
```bash
|
||||
cd deploy/
|
||||
# 构建镜像（宿主 docker-compose 不支持 compose build）
|
||||
docker build -t pangolin-backup:local ./backup
|
||||
|
||||
# 如果 MySQL 在 Docker 卷内，先确认卷路径：
|
||||
docker inspect <jiu_mysql_container> | grep -A5 Mounts
|
||||
|
||||
# 启动（修改 .env 添加必要的主机路径变量）
|
||||
MYSQL_DATADIR_HOST=/var/lib/docker/volumes/jiu_jiu_mysql_data/_data \
|
||||
MARZBAN_DATA_HOST=/opt/marzban \
|
||||
docker-compose -p pangolin up -d backup
|
||||
|
||||
# 查看日志
|
||||
docker logs -f pangolin-backup
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 日常运维
|
||||
|
||||
### 查看备份状态（RPO/RTO 实时检查）
|
||||
|
||||
```bash
|
||||
docker exec pangolin-backup /scripts/verify-rpo-rto.sh
|
||||
```
|
||||
|
||||
输出示例：
|
||||
|
||||
```
|
||||
[OK] MySQL-增量: 最新备份距今 8min (RPO ≤ 15min ✓)
|
||||
[OK] PG-增量: 最新备份距今 7min (RPO ≤ 15min ✓)
|
||||
[OK] MySQL 全量: 最近 14h 内有全量备份
|
||||
[OK] DR 演练: 上次演练距今 12 天
|
||||
状态: 达标 ✅
|
||||
```
|
||||
|
||||
### 查看 cron 日志
|
||||
|
||||
```bash
|
||||
docker logs --tail 100 pangolin-backup
|
||||
# 或
|
||||
docker exec pangolin-backup tail -50 /var/log/backup/cron.log
|
||||
```
|
||||
|
||||
### 手动触发全量备份
|
||||
|
||||
```bash
|
||||
docker exec pangolin-backup bash -c '. /etc/backup-env.sh && /scripts/backup-full.sh'
|
||||
```
|
||||
|
||||
### 手动触发 DR 演练
|
||||
|
||||
```bash
|
||||
# 如需验证解密恢复，临时注入私钥
|
||||
docker exec -e AGE_PRIVATE_KEY="$(cat age-private.txt)" \
|
||||
pangolin-backup bash -c '. /etc/backup-env.sh && /scripts/dr-drill.sh'
|
||||
```
|
||||
|
||||
### 列出 S3 备份文件
|
||||
|
||||
```bash
|
||||
docker exec pangolin-backup bash -c '. /etc/backup-env.sh && \
|
||||
AWS_PROFILE=backup aws s3 ls s3://${BACKUP_S3_BUCKET}/${BACKUP_S3_PREFIX}/ --recursive \
|
||||
| sort | tail -20'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 灾备恢复流程（实际灾难时）
|
||||
|
||||
> **前提**：已有 age 私钥（从密码管理器取出），已有恢复目标服务器
|
||||
|
||||
### MySQL 完整恢复
|
||||
|
||||
```bash
|
||||
# 1. 将私钥挂载到恢复容器
|
||||
docker run -it --rm \
|
||||
-e AGE_PRIVATE_KEY="$(cat age-private.txt)" \
|
||||
-e AWS_PROFILE=restore \
|
||||
# ... 其他环境变量 ...
|
||||
-v /restore:/restore \
|
||||
pangolin-backup:local bash
|
||||
|
||||
# 2. 在容器内执行恢复
|
||||
RESTORE_DIR=/restore /scripts/restore.sh mysql full 20240101_020000
|
||||
|
||||
# 3. 如需应用增量（最近一次全量后的某个增量时间点）
|
||||
RESTORE_DIR=/restore /scripts/restore.sh mysql inc 20240101_020000 20240101_150000
|
||||
|
||||
# 4. 容器会提示手动操作步骤：
|
||||
# - systemctl stop mysql
|
||||
# - rm -rf /var/lib/mysql/*
|
||||
# - xtrabackup --copy-back --target-dir=/restore/mysql/20240101_020000
|
||||
# - chown -R mysql:mysql /var/lib/mysql
|
||||
# - systemctl start mysql
|
||||
```
|
||||
|
||||
### PostgreSQL 完整恢复
|
||||
|
||||
```bash
|
||||
RESTORE_DIR=/restore /scripts/restore.sh postgres blog 20240101_150000
|
||||
# 恢复到 blog_restored 数据库，验证后再重命名
|
||||
```
|
||||
|
||||
### RTO 测量标准
|
||||
|
||||
| 步骤 | 预计耗时 |
|
||||
|------|---------|
|
||||
| S3 下载全量备份（50GB）| ~30min |
|
||||
| age 解密 | ~2min |
|
||||
| xtrabackup --prepare | ~20min |
|
||||
| 数据文件复制（rsync）| ~30min |
|
||||
| MySQL 启动 + 应用增量/binlog | ~30min |
|
||||
| 验证（基本查询）| ~10min |
|
||||
| **合计** | **~2h（远低于 4h）** |
|
||||
|
||||
---
|
||||
|
||||
## 月度 DR 演练报告
|
||||
|
||||
演练报告自动上传至：`s3://YOUR_BUCKET/ec2-ap-southeast-1/dr-reports/drill_YYYYMMDD_HHMMSS.txt`
|
||||
|
||||
本地日志：`docker exec pangolin-backup tail -100 /var/log/backup/dr-drill.log`
|
||||
|
||||
演练覆盖项目：
|
||||
|
||||
| 检查项 | 标准 |
|
||||
|--------|------|
|
||||
| RPO 验证 | 最新备份距今 ≤ 15min |
|
||||
| 备份文件完整性 | tar 归档可读，age 解密成功 |
|
||||
| MySQL xtrabackup prepare | 无错误 |
|
||||
| MySQL 临时恢复验证 | 容器启动后可执行查询 |
|
||||
| PostgreSQL 备份结构验证 | pg_restore --list 通过 |
|
||||
| RTO 演练总耗时 | ≤ 4h |
|
||||
|
||||
---
|
||||
|
||||
## 安全要点
|
||||
|
||||
- **age 私钥不存服务器**：加密用公钥，解密用私钥（离线保存）
|
||||
- **独立 IAM 账号**：备份写入账号仅有 `s3:PutObject`，即使服务器被入侵也无法读取历史备份
|
||||
- **加密存储**：所有备份在上传前均 age 加密，S3 中无明文数据
|
||||
- **版本控制**：S3 开启版本控制，防止误操作覆盖
|
||||
- **生命周期**：30 天 → IA，90 天 → Glacier，365 天 → 删除（成本控制）
|
||||
- **凭证不入库**：`deploy/secrets/` 在 `.gitignore` 中
|
||||
|
||||
---
|
||||
|
||||
## 故障排查
|
||||
|
||||
### 备份容器无法连接 MySQL
|
||||
|
||||
```bash
|
||||
# 检查 MySQL 是否可达
|
||||
docker exec pangolin-backup mysql \
|
||||
--host=127.0.0.1 --port=3306 \
|
||||
--user=backup_user --password=YOUR_PW \
|
||||
-e "SELECT 1"
|
||||
# 若失败：检查防火墙、MySQL bind-address（应为 0.0.0.0 或 127.0.0.1）
|
||||
```
|
||||
|
||||
### xtrabackup 提示版本不匹配
|
||||
|
||||
xtrabackup 8.0 只支持 MySQL 8.0。若生产 MySQL 是 5.7，需修改 Dockerfile 安装 `percona-xtrabackup-24`：
|
||||
|
||||
```dockerfile
|
||||
RUN percona-release setup pxb24 \
|
||||
&& apt-get install -y percona-xtrabackup-24
|
||||
```
|
||||
|
||||
### S3 上传失败
|
||||
|
||||
```bash
|
||||
# 检查凭据
|
||||
docker exec pangolin-backup bash -c 'AWS_PROFILE=backup aws sts get-caller-identity'
|
||||
# 检查 bucket 权限
|
||||
docker exec pangolin-backup bash -c \
|
||||
"AWS_PROFILE=backup aws s3 cp /dev/null s3://${BACKUP_S3_BUCKET}/test-write && echo OK"
|
||||
```
|
||||
|
||||
### DR 演练中 Docker 不可用
|
||||
|
||||
演练脚本中的 MySQL 临时容器启动依赖 Docker-in-Docker 或宿主 Docker socket。
|
||||
若不可用，脚本会跳过容器验证，只做 `xtrabackup --prepare` 完整性验证。
|
||||
Reference in New Issue
Block a user