Production fleet operator guide

To remove an optional node address, set its field to JSON null in the setup node object or the update-node configuration's node object. Omitting a field leaves its current value unchanged. Render and validate the replacement bundles, then use the normal rollout procedure; editing Fleet state alone does not change running listeners or external DNS. Retain the previous configuration for rollback.

Diagram source

flowchart LR
  subgraph Authority["Protected authority"]
    A["Topology + secrets + PKI"]
  end
  subgraph Hosts["Generated host bundles"]
    A --> C["Control + edge-control"]
    A --> D["DNS"]
    A --> E1["Edge A"]
    A --> E2["Edge B"]
  end

Each bundle is a security boundary. Never use a shared env_file: database, PowerDNS, bootstrap, identity, backup, and telemetry credentials are emitted only when that node's filtered services require them. Never copy a bundle, .env.prod, pki/, or secrets/ directory between nodes.

Work from an immutable checkout. For a fresh operator host:

git clone https://github.com/vaheed/CDNFoundry.git cdnfoundry
cd cdnfoundry
git checkout v1.2.1
git rev-parse --verify HEAD

Replace the example tag with the exact release or commit selected for the fleet. For deployment, copy the nine verified digest references from release-manifest.json into the corresponding CDNF_*_IMAGE keys in each node's extra_env; the renderer otherwise emits commit-tag references for offline validation and pre-publication testing. Never operate production from a moving branch or deploy unverified tag references.

This guide covers the full lifecycle of a CDNFoundry production fleet: first-time setup, control plus monitoring, additional DNS and edge nodes, validation, bundle transfer, operation, upgrades, recovery, and troubleshooting.

What the generator creates

The generator runs from a trusted checkout of the CDNFoundry repository and writes two protected outputs:

• Fleet state: topology, feature modes, immutable release identifiers, credentials, certificate authorities, node certificates, and state history.
• Node bundles: one minimal directory per host containing filtered Compose services, .env.prod, only the referenced runtime files, node-specific PKI, generated monitoring/logging files, validation/start scripts, and a node README.

Remote nodes do not need a repository clone. They receive only their own bundle.

Docker log retention on existing hosts

Every service in the production Compose source uses Docker json-file logging with 10 MB per file and three files. Fleet renders this policy into each node bundle. Docker applies changed logging options when it creates a container; an existing container keeps its old options until it is recreated. After rendering and validating a new bundle, use the normal canary and wave rollout so Compose recreates changed containers one host at a time. Check the effective options on each host with docker inspect CONTAINER and inspect its HostConfig.LogConfig field. Check free disk space before the rollout. Retain any required log evidence outside Docker first; recreation removes that container's old JSON log history. Do not run docker compose down --volumes or prune named volumes. If a wave fails, restore the previous bundle and repeat the normal rollback procedure.

Fastest supported setup

From the repository root, run:

./scripts/generate-production-env.sh

The wizard now performs the complete workflow instead of only initializing state. It asks for the global domains and release, offers a visible topology menu, adds hosts, validates the fleet, and renders bundles.

The default local paths are:

State:   ./.cdnfoundry-fleet
Bundles: ./build/fleet-bundles

Override them without editing the script:

CDNFOUNDRY_FLEET_STATE_DIR=/var/lib/cdnfoundry-fleet \
CDNFOUNDRY_FLEET_OUTPUT_DIR=/var/lib/cdnfoundry-fleet/bundles \
  sudo -E ./scripts/generate-production-env.sh

Control plus monitoring

Choose Control + monitoring on the same host in the wizard. This sets monitoring mode to colocated, selects both the control and telemetry profiles for the control node, and creates exporter targets for every enabled host.

Equivalent non-interactive command:

./scripts/cdnfoundry-fleet \
  --state-dir ./.cdnfoundry-fleet \
  --output-dir ./build/fleet-bundles \
  --repo-root "$PWD" \
  setup \
  --operator-domain ops.example.com \
  --platform-domain example.net \
  --release v1.2.1 \
  --preset control-monitoring \
  --control-ipv4 192.0.2.10 \
  --non-interactive

To enable it on an existing fleet:

./scripts/cdnfoundry-fleet --state-dir ./.cdnfoundry-fleet \
  configure-monitoring --mode colocated --non-interactive

./scripts/cdnfoundry-fleet \
  --state-dir ./.cdnfoundry-fleet \
  --output-dir ./build/fleet-bundles \
  --repo-root "$PWD" render

Verify the selected mode and services:

./scripts/cdnfoundry-fleet \
  --state-dir ./.cdnfoundry-fleet \
  --output-dir ./build/fleet-bundles status

grep -E '^(  )?(grafana|prometheus|clickhouse|loki|node-exporter):' \
  build/fleet-bundles/control-1/compose.yml

Config-driven setup

For repeatable automation, use a JSON file:

{
  "preset": "control-monitoring",
  "global": {
    "operator_domain": "ops.example.com",
    "platform_domain": "example.net",
    "release": "v1.2.1",
    "acme_email": "operations@example.com",
    "ipv6": false
  },
  "nodes": [
    {
      "name": "control-1",
      "role": "control",
      "region": "global",
      "location": "ashburn",
      "hostname": "control.ops.example.com",
      "public_ipv4": "192.0.2.10",
      "bind_ipv4": "0.0.0.0"
    },
    {
      "name": "dns-frankfurt",
      "role": "dns",
      "region": "europe",
      "location": "frankfurt",
      "public_ipv4": "192.0.2.20",
      "bind_ipv4": "0.0.0.0"
    },
    {
      "name": "edge-frankfurt",
      "role": "edge",
      "region": "europe",
      "location": "frankfurt",
      "public_ipv4": "192.0.2.30",
      "bind_ipv4": "0.0.0.0"
    }
  ],
  "features": {
    "monitoring": {"mode": "colocated", "host": null},
    "logs": {"mode": "disabled", "host": null, "endpoint": null},
    "backups": {"mode": "disabled", "repository": null, "region": "us-east-1"}
  }
}

Run:

./scripts/generate-production-env.sh \
  --config ./fleet-production.json \
  --non-interactive

The setup command is idempotent for named nodes in the config: existing nodes are updated and missing nodes are added. Secrets remain stable unless an explicit rotation command is used.

CLI map

Goal	Command
Full wizard/config setup	setup
Initialize state only	init
Add or change a host	add-node, update-node
Store manual edge UUID/token	configure-edge-registration
Remove one-time edge token	clear-edge-bootstrap-token
Replace a protected secret from file	set-secret
See fleet and feature modes	status
Machine-readable inventory	status --json, list-nodes
Check repository and tools	doctor
Set monitoring	configure-monitoring
Set centralized logs	configure-logs
Set backup metadata	configure-backups
Test rendering without replacing bundles	validate
Generate node bundles	render
Show deployment sequence	show-start-order
Import existing credentials	adopt-existing
Rotate a supported secret	rotate-secret

Every command has command-specific help:

./scripts/cdnfoundry-fleet setup --help
./scripts/cdnfoundry-fleet configure-monitoring --help
./scripts/cdnfoundry-fleet render --help

Preflight checks

Run before setup or after updating the repository:

./scripts/cdnfoundry-fleet --repo-root "$PWD" doctor

doctor verifies the production Compose file, deployment assets, Python, OpenSSL, and existing fleet state. Docker is reported separately because rendering can occur on a generator machine without starting containers, but Docker Compose is required on deployment hosts and for the final host-side validation.

Node roles

Control

Runs the application control plane, Valkey, migrations, and the edge-control TLS endpoint. By default it also runs embedded PostgreSQL. When node extra_env contains a non-control-db DB_HOST or a non-empty DB_URL, the generated bundle removes embedded control-db and points the application at the operator-managed database. In colocated monitoring mode it also runs the telemetry stack.

DNS

Runs node-local PostgreSQL, PowerDNS authoritative service, DNSdist, and required geo/MMDB support. Its database password and API key are unique to that node.

Edge

Runs the edge agent/runtime and gateway services. It receives the edge server CA, a node TLS certificate, and the generated edge-control URL.

DNS-edge

Combines the DNS and edge service sets on one host. It still uses its own local PowerDNS PostgreSQL database.

Monitoring

Runs a dedicated telemetry stack when monitoring mode is dedicated. It does not start a second control database. Configure the control node with an externally reachable DB_HOST, or set GRAFANA_POSTGRES_HOST explicitly on the monitoring node. Rendering fails closed without that endpoint. The generated bundle retains the bounded provisioning helper so the read-only Grafana role and sanitized views exist before Grafana starts.

PKI layout

The generator follows the production repository’s two-CA model:

• edge-identity-ca: used by the control plane for edge identity issuance and verification.
• edge-server-ca: signs edge-control, edge runtime, and DNS API TLS certificates.

CA private keys stay in the protected fleet state directory. Every node bundle receives the edge server CA certificate and its own public certificate. Control and DNS-only bundles receive their node private key; edge and combined DNS/edge bundles require that key from external secret storage on host tmpfs. Only the control bundle receives the edge identity CA private key because the control service requires it. The transferred control key begins root-only; the generated control start.sh must run as root and changes only this key to owner root, numeric group 82, mode 0640, allowing the immutable image's PHP-FPM worker to read it without making it public.

DNS activation also requires root: start.sh restricts the credential-bearing docker/pdns/pdns.conf to root:82, mode 0640, and the generated PowerDNS service receives that supplementary reader group. Keep the bundle directory mode 0700. Use the staged password rotation procedure for existing databases; rerendering alone does not rotate a stored credential.

Important generated environment paths include:

EDGE_IDENTITY_CA_CERTIFICATE=./pki/edge-identity-ca.crt
EDGE_IDENTITY_CA_PRIVATE_KEY=./pki/edge-identity-ca.key
PDNS_CA_CERTIFICATE=./pki/edge-server-ca.crt
EDGE_CONTROL_SERVER_CERTIFICATE=./pki/node.crt
EDGE_CONTROL_SERVER_PRIVATE_KEY=./pki/node.key
EDGE_CONTROL_CA_CERTIFICATE=./pki/edge-server-ca.crt
EDGE_RUNTIME_TLS_CERTIFICATE=/dev/shm/cdnfoundry/edge-runtime.crt
EDGE_RUNTIME_TLS_PRIVATE_KEY=/dev/shm/cdnfoundry/edge-runtime.key
EDGE_STATE_ENCRYPTION_KEY_FILE=/dev/shm/cdnfoundry/edge-state-encryption.key
DNS_API_SERVER_CERTIFICATE=./pki/node.crt
DNS_API_SERVER_PRIVATE_KEY=/dev/shm/cdnfoundry/node.key

Before starting an edge bundle, provision a dedicated bootstrap listener key and its matching certificate from external secret storage. Put the key and a stable, random 32-byte edge recovery key (64 hex characters) in the generated host tmpfs paths, owned by UID 10101 with mode 0400. Disable host swap and restore both keys after every host boot. On combined DNS/edge hosts, provision the node certificate's matching key at /dev/shm/cdnfoundry/node.key from external secret storage. The generated bundle excludes this key. Do not reuse it as the edge listener key. The edge agent stores encrypted recovery state in edge-agent-state; cells and agent reject disk-backed active key mounts.

Edge registration and mTLS enrollment

Fleet prepares the host bundle but is not part of control-plane enrollment. An edge display name may differ from its Fleet, Terraform, Ansible, or server name. Prepare and transfer each host bundle once. A combined dns-edge bundle starts DNS normally before an edge identity exists.

To enroll an edge:

1. In Infrastructure → Edges, create the edge and copy the two-line environment block from the one-time modal.

2. On the matching prepared host, replace the empty values in .env.prod:

   EDGE_ID=11111111-2222-3333-4444-555555555555
   EDGE_BOOTSTRAP_TOKEN=the-one-time-token

3. Keep .env.prod mode 0600, then start the edge profile explicitly:

   cd /opt/cdnfoundry
   sudo chmod 0600 .env.prod
   sudo docker compose --env-file .env.prod --profile edge up -d

For future whole-host starts, the operator may add --profile edge to the generated start.sh Compose up command. The script itself remains a fixed, transparent startup wrapper: it does not inspect enrollment state, wait for registration, edit .env.prod, or restart services. No Fleet command, bundle rerender, or second transfer is required.

After a fresh heartbeat, the bootstrap token is spent and cannot enroll again. Blank it in .env.prod during normal secret hygiene. An immediate second restart is not required for correct operation.

The agent creates its private key locally and persists the issued identity in edge-agent-state. Never copy or delete that volume. Directly assigned public service addresses need no EDGE_GATEWAY_ADDRESS_MAP; the gateway binds them directly. Configure the map once as advanced host state only for NAT or a load balancer, mapping every advertised address to its distinct private listener. Set EDGE_GATEWAY_REQUIRE_ADDRESS_MAP=true only where local policy requires explicit coverage.

Fleet automation may still use configure-edge-registration and clear-edge-bootstrap-token, but those commands are Fleet-specific and are not shown by the administrator panel.

Rotate an edge identity

Use rotation when an agent identity or its persistent state is lost or suspected compromised. This is an immediate revocation, not an overlapping certificate renewal. Open the edge under Infrastructure → Edges, choose Rotate identity, read the impact statement, and confirm only when the matching edge host is available.

The current certificate stops authenticating as soon as rotation is confirmed. The gateway and cells retain their last valid runtime, but agent heartbeat and configuration delivery pause until reenrollment. The replacement modal shows the same two environment values as initial enrollment. Replace them in the host's .env.prod and run the explicit --profile edge up -d command above. Compose recreates the agent when its environment changes. The agent keeps the previous local identity until the replacement certificate is issued, then atomically replaces it. Blank the spent replacement token afterward. DNS, edge-gateway, cells, and their last-valid runtime remain in place. If the modal is left before its token is saved, rotate again and use only the newest replacement token.

Embedded or remote control PostgreSQL

Embedded mode is the default. The control bundle contains control-db, and start.sh waits for control-db and redis before running the migration.

For remote mode, set the control node's optional Compose overrides and replace the generated database password with the real remote credential from a protected file:

sudo ./scripts/cdnfoundry-fleet \
  --state-dir /var/lib/cdnfoundry-fleet \
  update-node --node control-1 \
  --extra-env DB_HOST=postgres.internal.example \
  --extra-env DB_PORT=5432 \
  --extra-env DB_SSLMODE=verify-full \
  --non-interactive

sudo ./scripts/cdnfoundry-fleet \
  --state-dir /var/lib/cdnfoundry-fleet \
  set-secret --secret control-db-password \
  --from-file /root/cdnfoundry-postgres-password \
  --non-interactive

A non-empty DB_URL also selects remote mode. Do not place a password-bearing URL in version control or shell history.

In remote mode the renderer removes control-db, its volume, and every dependency on it. The generated start helper waits only for local Valkey before migration. When telemetry is colocated and DB_HOST is supplied, Grafana's control-database provisioning and datasource defaults inherit the same host, port, and SSL mode unless explicitly overridden.

Before rendering, ensure the external service has:

• database and application role expected by the project;
• TLS and certificate hostname verification where supported;
• exact-source network allowlists;
• capacity and connection limits for web, Horizon, scheduler, migrations, and Grafana provisioning;
• backups/PITR and an isolated restore test.

Validation and rendering

Use both steps in CI or before a production rollout:

./scripts/cdnfoundry-fleet \
  --state-dir ./.cdnfoundry-fleet \
  --output-dir ./build/fleet-bundles \
  --repo-root "$PWD" validate

./scripts/cdnfoundry-fleet \
  --state-dir ./.cdnfoundry-fleet \
  --output-dir ./build/fleet-bundles \
  --repo-root "$PWD" render

Validation renders into a temporary directory and leaves active bundles unchanged. Rendering builds each node in a temporary directory, then atomically replaces the active bundle while retaining .previous.

Fleet validates generated configuration; it does not verify public DNS/TLS, registrar delegation, account access or customer traffic. Follow the starter quick start for the ordered browser and API onboarding paths. Keep any tested host-local resolver/bridge Compose overrides in protected inventory; Fleet does not import them into its state. Merge them into the candidate bundle and validate before replacement. The quick start documents diagnosis, targeted activation and preservation of those overrides.

Bundle transfer and activation

Archive one node from the generator host:

tar --numeric-owner --owner=0 --group=0 \
  -C build/fleet-bundles \
  -czf /tmp/edge-frankfurt.tar.gz edge-frankfurt

On the target host:

install -d -m 0700 /opt/cdnfoundry.new
tar -xzf /tmp/edge-frankfurt.tar.gz --strip-components=1 -C /opt/cdnfoundry.new
cd /opt/cdnfoundry.new
# Apply retained host-local overrides before validation.
if [ -f secrets/metrics-token ]; then
  sudo chown 0:82 secrets/metrics-token
  sudo chmod 0640 secrets/metrics-token
fi
sudo ./validate.sh
cd /opt
mv cdnfoundry cdnfoundry.previous 2>/dev/null || true
mv cdnfoundry.new cdnfoundry
cd /opt/cdnfoundry
sudo ./start.sh

Never transfer the entire fleet state or another node’s bundle. Do not replace the generated control activation with a direct docker compose up: the activation applies the restricted PHP-worker access required for pki/edge-identity-ca.key. If core reports that the key is not readable, rerun sudo ./start.sh and verify stat -c '%u:%g %a %n' pki/edge-identity-ca.key reports 0:82 640.

validate.sh runs the pinned Caddy image's adapter against each Caddyfile present in the bundle. Treat any adapter error as a failed bundle and do not activate it. The validation container is short-lived and does not start dependencies; Docker may create the bundle's declared network or empty named-volume metadata while preparing the container.

If log-collector exits with code 78 and reports that LOG_AUTH_TOKEN is missing, do not put the token directly in the rendered Compose manifest. Verify that .env.prod contains a non-empty LOG_AUTH_TOKEN and that the rendered log-collector.environment maps LOG_AUTH_TOKEN from Compose interpolation, then rerender and transfer the corrected node bundle. Recreate only log-collector; its bounded operational-vector-data volume preserves buffered logs.

Remote metrics reachability

Monitoring bundles attach Prometheus to the existing private service networks and the outbound egress network. The private networks stay internal and no Prometheus port is published. Without outbound access the local datasource can be healthy while every remote PoP scrape fails. After activation inspect the Prometheus up query: all expected remote node, DNSdist, gateway and collector targets must be present and up. Datasource health alone is insufficient.

For an already deployed bundle generated before this correction, merge a prometheus service entry into the protected host-local Compose override, retaining its existing networks and adding egress. Validate Compose and recreate only Prometheus. Keep its volume and image digest. For the starter control bundle the complete network list is telemetry, control, dns-private, egress. Other role layouts must retain their own existing networks. Continue restricting metrics listeners to the monitoring source; do not open remote metrics publicly to work around an outbound routing failure. New Fleet renders include this attachment automatically.

Updating the fleet

Add a host:

./scripts/cdnfoundry-fleet --state-dir ./.cdnfoundry-fleet add-node \
  --node edge-tokyo --role edge --region asia --location tokyo \
  --public-ipv4 192.0.2.80 --non-interactive

Change a release or address:

./scripts/cdnfoundry-fleet --state-dir ./.cdnfoundry-fleet update-node \
  --node edge-tokyo --release v1.2.1 --public-ipv4 192.0.2.81 \
  --non-interactive

Render only that host:

./scripts/cdnfoundry-fleet \
  --state-dir ./.cdnfoundry-fleet \
  --output-dir ./build/fleet-bundles \
  --repo-root "$PWD" validate --node edge-tokyo

./scripts/cdnfoundry-fleet \
  --state-dir ./.cdnfoundry-fleet \
  --output-dir ./build/fleet-bundles \
  --repo-root "$PWD" render --node edge-tokyo

A changed hostname or public IP automatically causes that node certificate to be reissued with updated SANs.

Troubleshooting

The script prints only initialization output

Use the full wrapper or setup, not the low-level init command:

./scripts/generate-production-env.sh
# or
./scripts/cdnfoundry-fleet setup --help

init deliberately creates only protected state and secrets. It does not add nodes or render bundles.

No interactive questions appear

Interactive mode needs a real terminal. In CI, a pipe, or a non-TTY shell, provide all values with --non-interactive --config FILE. The CLI now returns a clear validation error instead of silently waiting on unavailable input.

I cannot find control plus monitoring

Run the setup wizard and choose Control + monitoring on the same host, or configure it explicitly:

./scripts/cdnfoundry-fleet --state-dir ./.cdnfoundry-fleet \
  configure-monitoring --mode colocated --non-interactive

Confirm with status.

State already exists

The full setup command reuses existing state. It does not rotate secrets. To inspect it:

./scripts/cdnfoundry-fleet --state-dir ./.cdnfoundry-fleet status

A required variable is missing

The renderer derives required variables from the final filtered Compose file. Add project-specific values to a node’s extra_env only when the repository introduces a new required variable that the generator does not yet know. Do not place secrets in version-controlled config files.

Compose input contains !reset or !override

The Fleet loader supports both Docker Compose tags. Older generator versions used yaml.safe_load directly and failed on Compose inputs containing them.

Docker is unavailable on the generator machine

You can still generate bundles and run Python-level validation. Run each bundle’s validate.sh on a host with Docker Compose v2 before activation.

Related documents

• Production quick start: three nodes
• Production quick start: multi-region fleet
• Production fleet reference
• Fleet configuration reference

Atomic render and recovery guarantees

Setup validates the complete candidate topology before persisting its nodes. Rendering prepares and fsyncs a complete fleet generation before activating it with Linux renameat2 exchange. A later-node render failure leaves every active bundle unchanged. The output directory remains addressable during replacement; <output-dir>.previous retains the complete previous generation, and per-node .previous directories remain available for existing operator workflows. This is local bundle publication; remote host activation remains a separate rollout operation and requires its own acknowledgements.

Use the supported Linux filesystem with same-filesystem rename and directory fsync. An unsupported atomic exchange fails with an actionable error instead of moving away the active bundle. If archival fails after exchange, retain the reported .candidate-* sibling directory: it contains the previous generation. Do not remove recovery material while diagnosing an interrupted render.

Fleet refuses to regenerate a CA when only its key or certificate survives. Restore the matching pair from backup. Reused CA and node certificates must verify, match their keys and remain valid for at least 24 hours. Expired or mismatched material requires an explicit, coordinated PKI recovery; a render must not silently replace trust. Keep the CA keys, application encryption key, artifact signing key and TLS material with the database backup.

Generated environment values preserve literal dollar signs, quotes and backslashes through Compose and Fleet adoption. JSON status redacts extra_env values and feature endpoints/repositories because they may contain credentials. Read protected state locally when an exact value is needed; do not copy it into logs or tickets. Dry runs do not create lock files or change state permissions.