feat(provision): 弹性节点基建 Terraform + 一键更换 (tsk_6u0FxmbC7Yeq)

IaC 面 (infra/) 与控制面 (server/internal/provision) 双产出,落地 doc/04 §4
「节点是牲口」弹性拓扑与 make-before-break 一键更换。

server/internal/provision:
- CloudAdapter 适配层 + Registry(首发 vultr 消耗品池 / hetzner 精品池各一);
  厂商凭证仅从 PROVISION_<VENDOR>_* env 注入,不入库不入 git。
- ProvisionService:CreateNode(幂等键重放不重复开机)、DestroyNode(幂等)、
  RotateIP(换 IP 不换机 + version bump)、ListProviders。
- Replace 一键更换:先建后拆,新机 up 先于旧机 draining(容量不下降),
  replacement_uuid 幂等键 + replacements 表分步记录,崩溃可续跑不重复。
- RotatePool:池内滚动轮换,并发度 1–2。
- cmd/nodectl CLI:create/destroy/rotate-ip/replace/rotate-pool/providers。
- 单测(mock 厂商 API + 内存 Store):幂等重放、make-before-break 时序断言、
  开机失败/探活超时→destroyed+失败计数+告警钩子、崩溃续跑、RotatePool。

infra/:
- terraform/:探针机 + 控制面基线模块化(probe / control-plane)+ README,
  低频基线进 state,节点不进 Terraform。
- cloud-init/node.yaml.tmpl:节点引导模板(注入一次性 bootstrap token,task #5)。
- identity-isolation.md:身份隔离登记表(doc/06 §2 红线,无任何凭证)。

migrations/000008:nodes 增 provider_instance_id/elastic_ip_id、node_events
增 ip_rotated、provision_idempotency / replacements 表(附加式,不动现网)。

红线:仅面向新厂商池,绝不纳管现网生产 EC2(deploy/ marzban)。

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
wangjia
2026-06-13 14:23:39 +08:00
parent 787151245e
commit 3d5bac66b4
39 changed files with 3884 additions and 0 deletions
+44
View File
@@ -0,0 +1,44 @@
# `infra/` — elastic-node infrastructure
This directory is the **IaC half** of the elastic-node base (task #14). The
**control-plane half** — vendor adapters and the make-before-break orchestration
— lives in [`../server/internal/provision`](../server/internal/provision).
```
infra/
terraform/ baseline resources: probe machines + control-plane env
cloud-init/
node.yaml.tmpl data-plane node bootstrap (rendered by provision svc)
identity-isolation.md per-asset identity isolation register (doc/06 §2)
```
## Division of labour (doc/04 §4)
- **Terraform** (`terraform/`) — low-frequency baseline only (probes, control
plane). These are in Terraform state.
- **Provision service** (`server/internal/provision`) — high-frequency,
minute-scale, idempotent node open/退机 via vendor APIs. Nodes are cattle and
are **never** in Terraform state.
## `cloud-init/node.yaml.tmpl`
The entire persistent footprint of a data-plane node: sing-box + agent + a
one-time bootstrap token (doc/04 §2). It is a Go `text/template` rendered by the
provision service's `CloudInitRenderer`, injecting:
`NodeUUID`, `BootstrapToken` (task #5), `Region`, `Role`, `Tier`,
`ControlPlaneURL`.
The rendered output carries the one-time enrollment token and **must not be
logged**. The agent self-registers over mTLS (task #6); the control plane then
drives the node `provisioning → probing → up`.
## Red lines (doc/06 §2)
- Everything here targets the **new vendor pools**. It must **never** manage the
production EC2 host (the `deploy/` marzban machine).
- Credentials are never committed: Terraform tokens via `TF_VAR_*` / CI secrets,
vendor API keys via `PROVISION_<VENDOR>_*` env consumed by the provision
service. The `providers` table stores only `name/api_kind/regions/pool/enabled`.
- Each vendor/domain/account is an independent identity with zero cross-linkage;
see `identity-isolation.md`.
+64
View File
@@ -0,0 +1,64 @@
## template: jinja-free Go text/template — rendered by
## server/internal/provision (CloudInitRenderer). Fields:
## {{.NodeUUID}} {{.BootstrapToken}} {{.Region}} {{.Role}} {{.Tier}} {{.ControlPlaneURL}}
##
## "Nodes are cattle, not pets" (doc/04 §2): this is the ENTIRE persistent
## footprint of a data-plane node — sing-box + agent binaries + a one-time
## bootstrap token. Zero user DB, zero logs, zero persistent state.
##
## SECURITY: the rendered output carries a one-time bootstrap token. It MUST NOT
## be logged. The token is consumed exactly once during mTLS enrollment (task #5)
## and is useless afterwards.
#cloud-config
write_files:
# Node identity + enrollment parameters consumed by the agent on first boot.
- path: /etc/pangolin/bootstrap.env
permissions: "0600"
owner: root:root
content: |
PANGOLIN_NODE_UUID={{.NodeUUID}}
PANGOLIN_BOOTSTRAP_TOKEN={{.BootstrapToken}}
PANGOLIN_CONTROL_PLANE_URL={{.ControlPlaneURL}}
PANGOLIN_REGION={{.Region}}
PANGOLIN_ROLE={{.Role}}
PANGOLIN_TIER={{.Tier}}
# systemd unit: agent enrolls (mTLS) then pulls runtime config from control plane.
- path: /etc/systemd/system/pangolin-agent.service
permissions: "0644"
owner: root:root
content: |
[Unit]
Description=Pangolin node agent (self-register + config pull)
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
EnvironmentFile=/etc/pangolin/bootstrap.env
ExecStart=/usr/local/bin/pangolin-agent run
# No-log node: agent does not persist connection logs.
Restart=always
RestartSec=5
[Install]
WantedBy=multi-user.target
runcmd:
# 1. Harden: key-only SSH, no root password (doc/06 §3).
- sed -i 's/^#\?PermitRootLogin.*/PermitRootLogin prohibit-password/' /etc/ssh/sshd_config
- sed -i 's/^#\?PasswordAuthentication.*/PasswordAuthentication no/' /etc/ssh/sshd_config
- systemctl restart sshd || true
# 2. Install the agent binary (pinned, checksum-verified by the install script,
# delivered out-of-band by task #6). Placeholder URL resolved at build time.
- install -d -m 0755 /etc/pangolin
- /usr/local/bin/pangolin-agent-install || true
# 3. Enroll + start. The agent reads bootstrap.env, performs mTLS enrollment,
# and the control plane flips the node provisioning → probing → up.
- systemctl daemon-reload
- systemctl enable --now pangolin-agent.service
# 4. Shred the one-time token from disk after enrollment (defence in depth).
- bash -c 'sleep 60; shred -u /etc/pangolin/bootstrap.env 2>/dev/null || rm -f /etc/pangolin/bootstrap.env'
# No swap file written to disk, no persistent data dirs — a destroyed node leaks nothing.
+58
View File
@@ -0,0 +1,58 @@
# Identity isolation register (doc/06 §2 红线)
> **RED LINE (any phase, never crossed):** servers, domains, CDN, object
> storage, payment, email and phone numbers MUST NOT share any linkable identity.
> Each vendor uses an **independent account + independent email + crypto
> payment**. This file is the authoritative register of which isolated identity
> backs which asset. **It contains NO secrets** — only the isolation mapping.
> Credentials live in independent secret stores (env / files), never here, never
> in the database, never in git.
## How to use this register
1. Onboarding a new vendor/asset → add a row before provisioning anything.
2. Each row gets a distinct `identity-id` (an opaque internal label, e.g.
`id-a7`). Never reuse an identity across rows.
3. Record the asset, pool, payment rail, and the **secret location** (where the
credential is injected from), not the credential itself.
## Data-plane vendor pools (doc/04 §5.2)
Consumed by `server/internal/provision`. The `providers` table stores only
`name/api_kind/regions/pool/enabled`; the matching credential is injected from
the secret location below as `PROVISION_<VENDOR>_*`.
| identity-id | asset | pool | api_kind | payment | secret location (env) | notes |
|-------------|-------|------|----------|---------|-----------------------|-------|
| id-c1 | _vendor A_ | consumable | `vultr` | USDT/crypto | `PROVISION_VULTR_API_KEY` | small/cheap entry pool; ≥3 vendors target |
| id-c2 | _vendor B_ | consumable | _tbd_ | crypto | `PROVISION_<B>_API_KEY` | second consumable vendor |
| id-c3 | _vendor C_ | consumable | _tbd_ | crypto | `PROVISION_<C>_API_KEY` | third consumable vendor |
| id-p1 | _vendor D_ | premium | `hetzner` | crypto | `PROVISION_HETZNER_API_TOKEN` | stable exit/pro entry; native IPs |
> Consumable pool and premium pool MUST use **different vendor accounts** so a
> mass-burn of the consumable pool never touches the premium pool (doc/04 §4.2).
## Management / baseline (Terraform)
Managed by `infra/terraform`. Independent from all data-plane vendor identities.
| identity-id | asset | purpose | payment | secret location |
|-------------|-------|---------|---------|-----------------|
| id-m1 | mgmt/probe cloud | probe machines + control-plane host | crypto | `TF_VAR_hcloud_token` (CI secret) |
| id-m2 | control-plane DB/Redis | user DB, audit_log | — | server deploy secrets |
## Adjacent assets (registered elsewhere, listed for completeness)
| identity-id | asset | owner doc |
|-------------|-------|-----------|
| id-d1 | domain registrar (WHOIS privacy) | doc/05 |
| id-d2 | CDN account | doc/05 |
| id-pay1 | payment / card store (USDT-TRC20) | doc/02 §4 |
| id-tg1 | ops TG bot (anonymous) | doc/04 §5.3 |
## Audit checklist (run before each onboarding)
- [ ] New identity-id is unique; no email/account reused from another row.
- [ ] Payment is crypto / non-KYC; not bound to a real-name account.
- [ ] Credential stored only in its secret location; absent from git & DB.
- [ ] `ci/scan-redline.sh` and a repo/DB secret scan pass.
+67
View File
@@ -0,0 +1,67 @@
# `infra/terraform` — baseline infrastructure as code
Terraform here manages **only low-frequency baseline resources**:
- **probe machines** — overseas reference dial-test points (doc/04 §1, §4.2).
- **control-plane environment** — the API + MySQL + Redis host (doc/04 §5.2),
off by default because it changes rarely.
## What is deliberately NOT here
**Data-plane nodes are not in Terraform.** They are disposable cattle, created
and destroyed minute-by-minute through vendor APIs by the provision service
(`server/internal/provision`, doc/04 §4). Putting them in Terraform state would
fight the "nodes are cattle, not pets" model and serialise every open/退机
behind a state lock. Division of labour:
| concern | owner | frequency | in TF state |
|---------|-------|-----------|-------------|
| probe machines | Terraform (this dir) | low | ✅ |
| control-plane env | Terraform (this dir) | one-off | ✅ |
| data-plane nodes (entry/relay/exit) | provision service + vendor API | minutes | ❌ |
## Identity isolation (RED LINE, doc/06 §2)
Every vendor/account/domain used here MUST be an **independent identity** — its
own account, email, and crypto payment, with **zero cross-linkage** to other
assets. This Terraform code and the data-plane vendor accounts use **separate**
credentials:
- `var.hcloud_token` here is the **management/probe** account token — *not* a
data-plane vendor token.
- Data-plane vendor credentials live only in `PROVISION_<VENDOR>_*` env secrets
consumed by the provision service, never in Terraform, never in the DB.
The full isolation register (which account/email/payment maps to which asset) is
maintained in [`../identity-isolation.md`](../identity-isolation.md).
## Usage
```bash
cd infra/terraform
cp terraform.tfvars.example terraform.tfvars # gitignored; fill in
export TF_VAR_hcloud_token=... # inject secret, never commit
terraform init
terraform fmt -check
terraform validate
terraform plan
terraform apply
```
CI/management operations run on a **dedicated channel** (doc/06 §2), not from an
operator's everyday machine.
## Layout
```
terraform/
versions.tf provider + backend pins
variables.tf root inputs
main.tf wires probe[] + control-plane modules
outputs.tf probe IPs, control-plane IP
terraform.tfvars.example
modules/
probe/ one overseas reference probe point
control-plane/ API+MySQL+Redis host (prevent_destroy)
```
+42
View File
@@ -0,0 +1,42 @@
provider "hcloud" {
token = var.hcloud_token
}
# SSH keys for the management identity (probe + control-plane access).
resource "hcloud_ssh_key" "mgmt" {
for_each = { for idx, key in var.ssh_public_keys : idx => key }
name = "pangolin-mgmt-${each.key}"
public_key = each.value
}
locals {
ssh_key_ids = [for k in hcloud_ssh_key.mgmt : k.id]
}
# Overseas reference probe points (low frequency → Terraform-managed).
# 境内 probe points are serverless / third-party dialing services with strict
# identity isolation and are intentionally OUT of Terraform scope (registered in
# ../identity-isolation.md).
module "probe" {
source = "./modules/probe"
for_each = var.probe_overseas
name = each.key
location = each.value.location
server_type = each.value.server_type
image = var.probe_image
ssh_key_ids = local.ssh_key_ids
}
# Control-plane baseline env (API + MySQL + Redis), provisioned once and rarely
# changed. Disabled by default; flip control_plane_enabled to manage it here.
module "control_plane" {
source = "./modules/control-plane"
count = var.control_plane_enabled ? 1 : 0
server_type = var.control_plane_server_type
location = var.control_plane_location
image = var.probe_image
ssh_key_ids = local.ssh_key_ids
allowlist_cidrs = var.admin_allowlist_cidrs
}
@@ -0,0 +1,30 @@
# Module: `control-plane`
Provisions the **control-plane baseline host** (API + MySQL + Redis), doc/04 §5.2.
Key properties:
- Deployed on a **stable cloud unrelated to the data-plane node vendors**. Nodes
can all be destroyed without affecting the control plane, and vice-versa. The
control plane carries **no proxy traffic**.
- `prevent_destroy = true` — it holds the user DB; it is never recreated
implicitly. Application bring-up (containers, migrations) is done by the server
deploy pipeline, not Terraform.
- Admin/SSH surface is **never on the open internet** (doc/06 §2): inbound is
locked to `allowlist_cidrs`; the public API sits behind a CDN/proxy terminated
elsewhere.
Disabled by default in the root module (`control_plane_enabled = false`) because
it changes rarely.
## Inputs
| name | description |
|------|-------------|
| `server_type` / `location` / `image` | host sizing |
| `ssh_key_ids` | authorised SSH keys |
| `allowlist_cidrs` | admin/SSH inbound allowlist (tighten in prod) |
## Outputs
`ipv4_address`, `server_id`.
@@ -0,0 +1,53 @@
terraform {
required_providers {
hcloud = {
source = "hetznercloud/hcloud"
version = "~> 1.45"
}
}
}
# Control-plane baseline host (API + MySQL + Redis). Deployed on a stable cloud
# COMPLETELY UNRELATED to the data-plane node vendors (doc/04 §5.2): nodes can
# all die without touching the control plane, and vice-versa. Carries NO proxy
# traffic.
#
# Provisioned once and rarely changed — hence Terraform-managed. Application
# bring-up (containers, migrations) is handled separately by the server deploy
# pipeline, not here.
resource "hcloud_server" "control_plane" {
name = "pangolin-control-plane"
server_type = var.server_type
image = var.image
location = var.location
ssh_keys = var.ssh_key_ids
labels = {
role = "control-plane"
project = "pangolin"
}
lifecycle {
# Never recreate the control plane implicitly — it holds the user DB.
prevent_destroy = true
}
}
# Admin/SSH surface is never on the open internet (doc/06 §2). Inbound is locked
# to the management allowlist; the public API is expected to sit behind a CDN /
# reverse proxy terminated elsewhere.
resource "hcloud_firewall" "control_plane" {
name = "pangolin-control-plane"
rule {
direction = "in"
protocol = "tcp"
port = "22"
source_ips = length(var.allowlist_cidrs) > 0 ? var.allowlist_cidrs : ["0.0.0.0/0", "::/0"]
}
}
resource "hcloud_firewall_attachment" "control_plane" {
firewall_id = hcloud_firewall.control_plane.id
server_ids = [hcloud_server.control_plane.id]
}
@@ -0,0 +1,9 @@
output "ipv4_address" {
description = "Public IPv4 of the control-plane host."
value = hcloud_server.control_plane.ipv4_address
}
output "server_id" {
description = "Vendor server ID."
value = hcloud_server.control_plane.id
}
@@ -0,0 +1,29 @@
variable "server_type" {
description = "Vendor server type/size for the control-plane host."
type = string
default = "cpx21"
}
variable "location" {
description = "Vendor location/datacenter."
type = string
default = "hel1"
}
variable "image" {
description = "OS image."
type = string
default = "debian-12"
}
variable "ssh_key_ids" {
description = "Vendor SSH key IDs authorised on the host."
type = list(string)
default = []
}
variable "allowlist_cidrs" {
description = "CIDRs allowed to reach SSH/admin. Empty = open (MUST be tightened in prod; doc/06 §2)."
type = list(string)
default = []
}
+38
View File
@@ -0,0 +1,38 @@
# Module: `probe`
Provisions **one overseas reference probe point** (doc/04 §1, §4.2).
Probes are low-frequency baseline resources, so they live in Terraform state —
unlike data-plane nodes, which are disposable cattle managed by the provision
service via vendor APIs (never in Terraform).
## What it creates
- `hcloud_server` running the probe agent (installed out-of-band; the probe
holds **no** data-plane secrets and **no** user data).
- A firewall locking inbound to SSH (optionally CIDR-restricted).
## Role in the system
Overseas probes cross-check the in-China probe fleet so the scheduler can tell
**blocked** (in-China fails, overseas OK) from **machine down** (both fail),
avoiding false positives before triggering an automatic replacement (doc/04 §4.2).
> In-China probe points use serverless / third-party dialing services under a
> separate isolated identity and are **out of Terraform scope** — registered in
> [`../../identity-isolation.md`](../../identity-isolation.md).
## Inputs
| name | description |
|------|-------------|
| `name` | short id (e.g. `eu`, `us`) |
| `location` | vendor location |
| `server_type` | vendor size |
| `image` | OS image |
| `ssh_key_ids` | authorised SSH key IDs |
| `ssh_allowlist_cidrs` | inbound SSH allowlist |
## Outputs
`ipv4_address`, `server_id`.
+57
View File
@@ -0,0 +1,57 @@
terraform {
required_providers {
hcloud = {
source = "hetznercloud/hcloud"
version = "~> 1.45"
}
}
}
# One overseas reference probe point (doc/04 §4.2). It runs the probe script and
# reports TCP/TLS/real-handshake reachability of data-plane nodes back to the
# control plane, used to distinguish "blocked" from "machine down".
resource "hcloud_server" "probe" {
name = "pangolin-probe-${var.name}"
server_type = var.server_type
image = var.image
location = var.location
ssh_keys = var.ssh_key_ids
labels = {
role = "probe"
project = "pangolin"
scope = "overseas-reference"
}
# Probe bootstrap: pin + install the probe agent out-of-band. Kept minimal;
# probes hold NO data-plane secrets and NO user data.
user_data = <<-EOT
#cloud-config
package_update: true
runcmd:
- sed -i 's/^#\?PermitRootLogin.*/PermitRootLogin prohibit-password/' /etc/ssh/sshd_config
- sed -i 's/^#\?PasswordAuthentication.*/PasswordAuthentication no/' /etc/ssh/sshd_config
- systemctl restart sshd || true
EOT
lifecycle {
create_before_destroy = true
}
}
# Firewall: probes only need outbound; lock inbound to SSH from management.
resource "hcloud_firewall" "probe" {
name = "pangolin-probe-${var.name}"
rule {
direction = "in"
protocol = "tcp"
port = "22"
source_ips = length(var.ssh_allowlist_cidrs) > 0 ? var.ssh_allowlist_cidrs : ["0.0.0.0/0", "::/0"]
}
}
resource "hcloud_firewall_attachment" "probe" {
firewall_id = hcloud_firewall.probe.id
server_ids = [hcloud_server.probe.id]
}
+9
View File
@@ -0,0 +1,9 @@
output "ipv4_address" {
description = "Public IPv4 of the probe machine."
value = hcloud_server.probe.ipv4_address
}
output "server_id" {
description = "Vendor server ID."
value = hcloud_server.probe.id
}
@@ -0,0 +1,33 @@
variable "name" {
description = "Short probe identifier (e.g. eu, us)."
type = string
}
variable "location" {
description = "Vendor location/datacenter for the probe."
type = string
}
variable "server_type" {
description = "Vendor server type/size."
type = string
default = "cpx11"
}
variable "image" {
description = "OS image."
type = string
default = "debian-12"
}
variable "ssh_key_ids" {
description = "Vendor SSH key IDs authorised on the probe."
type = list(string)
default = []
}
variable "ssh_allowlist_cidrs" {
description = "CIDRs allowed to SSH the probe. Empty = open (tighten in prod)."
type = list(string)
default = []
}
+9
View File
@@ -0,0 +1,9 @@
output "probe_ips" {
description = "Public IPv4 of each overseas probe machine, keyed by probe name."
value = { for k, m in module.probe : k => m.ipv4_address }
}
output "control_plane_ip" {
description = "Public IPv4 of the control-plane host, if managed here."
value = var.control_plane_enabled ? module.control_plane[0].ipv4_address : null
}
+18
View File
@@ -0,0 +1,18 @@
# Copy to terraform.tfvars (gitignored) and fill in. NEVER commit real values.
# hcloud_token is injected via TF_VAR_hcloud_token in CI instead of this file.
# hcloud_token = "REDACTED — inject via TF_VAR_hcloud_token"
ssh_public_keys = [
# "ssh-ed25519 AAAA... pangolin-mgmt"
]
probe_overseas = {
eu = { location = "hel1", server_type = "cpx11" }
us = { location = "ash", server_type = "cpx11" }
}
control_plane_enabled = false
admin_allowlist_cidrs = [
# "203.0.113.10/32" # management bastion only — admin never on public internet
]
+53
View File
@@ -0,0 +1,53 @@
variable "hcloud_token" {
description = "Hetzner Cloud API token for the PROBE/management account ONLY (independent identity; never the data-plane vendor accounts). Inject via TF_VAR_hcloud_token / CI secret, never commit."
type = string
sensitive = true
}
variable "ssh_public_keys" {
description = "SSH public keys authorised on probe machines (management identity)."
type = list(string)
default = []
}
variable "probe_overseas" {
description = "Overseas reference probe points (doc/04 §4.2). Each emits a probe machine."
type = map(object({
location = string # vendor location, e.g. "hel1", "ash"
server_type = string # e.g. "cpx11"
}))
default = {
eu = { location = "hel1", server_type = "cpx11" }
us = { location = "ash", server_type = "cpx11" }
}
}
variable "probe_image" {
description = "OS image for probe machines."
type = string
default = "debian-12"
}
variable "control_plane_enabled" {
description = "Whether to manage the control-plane baseline env in this state. Off by default — the control plane is provisioned once and rarely changes."
type = bool
default = false
}
variable "control_plane_server_type" {
description = "Server type for the control-plane host (API + MySQL + Redis), if managed here."
type = string
default = "cpx21"
}
variable "control_plane_location" {
description = "Location for the control-plane host."
type = string
default = "hel1"
}
variable "admin_allowlist_cidrs" {
description = "CIDRs allowed to reach the admin/SSH surface of the control plane (doc/06 §2: admin never on public internet)."
type = list(string)
default = []
}
+22
View File
@@ -0,0 +1,22 @@
terraform {
required_version = ">= 1.5.0"
required_providers {
# Probe machines (overseas reference points, doc/04 §1) run on a small,
# privacy-friendly cloud. Hetzner Cloud is used as the reference example;
# swap the provider per the identity-isolation register.
hcloud = {
source = "hetznercloud/hcloud"
version = "~> 1.45"
}
}
# State holds ONLY low-frequency baseline resources (probes, control-plane
# env). Disposable data-plane nodes are NEVER in state — they are cattle,
# managed by the provision service via vendor APIs (doc/04 §4).
#
# Configure a remote, access-restricted backend out-of-band (e.g. an S3-
# compatible bucket on the management account). Left as local here so the
# module stays runnable for `terraform validate`.
# backend "s3" { ... }
}