Production fleet operator guide

Diagram source

flowchart LR
  subgraph Authority["Protected authority"]
    A["Topology + secrets + PKI"]
  end
  subgraph Hosts["Generated host bundles"]
    A --> C["Control + edge-control"]
    A --> D["DNS"]
    A --> E1["Edge A"]
    A --> E2["Edge B"]
  end

Each bundle is a security boundary. Never use a shared env_file: database, PowerDNS, bootstrap, identity, backup, and telemetry credentials are emitted only when that node's filtered services require them. Never copy a bundle, .env.prod, pki/, or secrets/ directory between nodes.

Work from an immutable checkout. For a fresh operator host:

bash

git clone https://github.com/vaheed/CDNFoundry.git cdnfoundry
cd cdnfoundry
git checkout v1.0.0
git rev-parse --verify HEAD

Replace the example tag with the exact release or commit selected for the fleet. For deployment, copy the nine verified digest references from release-manifest.json into the corresponding CDNF_*_IMAGE keys in each node's extra_env; the renderer otherwise emits commit-tag references for offline validation and pre-publication testing. Never operate production from a moving branch or deploy unverified tag references.

This guide covers the full lifecycle of a CDNFoundry production fleet: first-time setup, control plus monitoring, additional DNS and edge nodes, validation, bundle transfer, operation, upgrades, recovery, and troubleshooting.

What the generator creates

The generator runs from a trusted checkout of the CDNFoundry repository and writes two protected outputs:

Fleet state: topology, feature modes, immutable release identifiers, credentials, certificate authorities, node certificates, and state history.
Node bundles: one minimal directory per host containing filtered Compose services, .env.prod, only the referenced runtime files, node-specific PKI, generated monitoring/logging files, validation/start scripts, and a node README.

Remote nodes do not need a repository clone. They receive only their own bundle.

Fastest supported setup

From the repository root, run:

bash

./scripts/generate-production-env.sh

The wizard now performs the complete workflow instead of only initializing state. It asks for the global domains and release, offers a visible topology menu, adds hosts, validates the fleet, and renders bundles.

The default local paths are:

text

State:   ./.cdnfoundry-fleet
Bundles: ./build/fleet-bundles

Override them without editing the script:

bash

CDNFOUNDRY_FLEET_STATE_DIR=/var/lib/cdnfoundry-fleet \
CDNFOUNDRY_FLEET_OUTPUT_DIR=/var/lib/cdnfoundry-fleet/bundles \
  sudo -E ./scripts/generate-production-env.sh

Control plus monitoring

Choose Control + monitoring on the same host in the wizard. This sets monitoring mode to colocated, selects both the control and telemetry profiles for the control node, and creates exporter targets for every enabled host.

Equivalent non-interactive command:

bash

./scripts/cdnfoundry-fleet \
  --state-dir ./.cdnfoundry-fleet \
  --output-dir ./build/fleet-bundles \
  --repo-root "$PWD" \
  setup \
  --operator-domain ops.example.com \
  --platform-domain example.net \
  --release v1.0.0 \
  --preset control-monitoring \
  --control-ipv4 192.0.2.10 \
  --non-interactive

To enable it on an existing fleet:

bash

./scripts/cdnfoundry-fleet --state-dir ./.cdnfoundry-fleet \
  configure-monitoring --mode colocated --non-interactive

./scripts/cdnfoundry-fleet \
  --state-dir ./.cdnfoundry-fleet \
  --output-dir ./build/fleet-bundles \
  --repo-root "$PWD" render

Verify the selected mode and services:

bash

./scripts/cdnfoundry-fleet \
  --state-dir ./.cdnfoundry-fleet \
  --output-dir ./build/fleet-bundles status

grep -E '^(  )?(grafana|prometheus|clickhouse|loki|node-exporter):' \
  build/fleet-bundles/control-1/compose.yml

Config-driven setup

For repeatable automation, use a JSON file:

json

{
  "preset": "control-monitoring",
  "global": {
    "operator_domain": "ops.example.com",
    "platform_domain": "example.net",
    "release": "v1.0.0",
    "acme_email": "operations@example.com",
    "ipv6": false
  },
  "nodes": [
    {
      "name": "control-1",
      "role": "control",
      "region": "global",
      "location": "ashburn",
      "hostname": "control.ops.example.com",
      "public_ipv4": "192.0.2.10",
      "bind_ipv4": "0.0.0.0"
    },
    {
      "name": "dns-frankfurt",
      "role": "dns",
      "region": "europe",
      "location": "frankfurt",
      "public_ipv4": "192.0.2.20",
      "bind_ipv4": "0.0.0.0"
    },
    {
      "name": "edge-frankfurt",
      "role": "edge",
      "region": "europe",
      "location": "frankfurt",
      "public_ipv4": "192.0.2.30",
      "bind_ipv4": "0.0.0.0"
    }
  ],
  "features": {
    "monitoring": {"mode": "colocated", "host": null},
    "logs": {"mode": "disabled", "host": null, "endpoint": null},
    "backups": {"mode": "disabled", "repository": null, "region": "us-east-1"}
  }
}

Run:

bash

./scripts/generate-production-env.sh \
  --config ./fleet-production.json \
  --non-interactive

The setup command is idempotent for named nodes in the config: existing nodes are updated and missing nodes are added. Secrets remain stable unless an explicit rotation command is used.

CLI map

Goal	Command
Full wizard/config setup	`setup`
Initialize state only	`init`
Add or change a host	`add-node`, `update-node`
Store manual edge UUID/token	`configure-edge-registration`
Remove one-time edge token	`clear-edge-bootstrap-token`
Replace a protected secret from file	`set-secret`
See fleet and feature modes	`status`
Machine-readable inventory	`status --json`, `list-nodes`
Check repository and tools	`doctor`
Set monitoring	`configure-monitoring`
Set centralized logs	`configure-logs`
Set backup metadata	`configure-backups`
Test rendering without replacing bundles	`validate`
Generate node bundles	`render`
Show deployment sequence	`show-start-order`
Import existing credentials	`adopt-existing`
Rotate a supported secret	`rotate-secret`

Every command has command-specific help:

bash

./scripts/cdnfoundry-fleet setup --help
./scripts/cdnfoundry-fleet configure-monitoring --help
./scripts/cdnfoundry-fleet render --help

Preflight checks

Run before setup or after updating the repository:

bash

./scripts/cdnfoundry-fleet --repo-root "$PWD" doctor

doctor verifies the production Compose file, deployment assets, Python, OpenSSL, and existing fleet state. Docker is reported separately because rendering can occur on a generator machine without starting containers, but Docker Compose is required on deployment hosts and for the final host-side validation.

Node roles

Control

Runs the application control plane, Valkey, migrations, and the edge-control TLS endpoint. By default it also runs embedded PostgreSQL. When node extra_env contains a non-control-db DB_HOST or a non-empty DB_URL, the generated bundle removes embedded control-db and points the application at the operator-managed database. In colocated monitoring mode it also runs the telemetry stack.

DNS

Runs node-local PostgreSQL, PowerDNS authoritative service, DNSdist, and required geo/MMDB support. Its database password and API key are unique to that node.

Edge

Runs the edge agent/runtime and gateway services. It receives the edge server CA, a node TLS certificate, and the generated edge-control URL.

DNS-edge

Combines the DNS and edge service sets on one host. It still uses its own local PowerDNS PostgreSQL database.

Monitoring

Runs a dedicated telemetry stack when monitoring mode is dedicated. It does not start a second control database. Configure the control node with an externally reachable DB_HOST, or set GRAFANA_POSTGRES_HOST explicitly on the monitoring node. Rendering fails closed without that endpoint. The generated bundle retains the bounded provisioning helper so the read-only Grafana role and sanitized views exist before Grafana starts.

PKI layout

The generator follows the production repository’s two-CA model:

edge-identity-ca: used by the control plane for edge identity issuance and verification.
edge-server-ca: signs edge-control, edge runtime, and DNS API TLS certificates.

CA private keys stay in the protected fleet state directory. Every node bundle receives the edge server CA certificate plus its own certificate and private key. Only the control bundle receives the edge identity CA private key because the control service requires it. The transferred key begins root-only; the generated control start.sh must run as root and changes only this key to owner root, numeric group 82, mode 0640, allowing the immutable image's PHP-FPM worker to read it without making it public.

Important generated environment paths include:

text

EDGE_IDENTITY_CA_CERTIFICATE=./pki/edge-identity-ca.crt
EDGE_IDENTITY_CA_PRIVATE_KEY=./pki/edge-identity-ca.key
PDNS_CA_CERTIFICATE=./pki/edge-server-ca.crt
EDGE_CONTROL_SERVER_CERTIFICATE=./pki/node.crt
EDGE_CONTROL_SERVER_PRIVATE_KEY=./pki/node.key
EDGE_CONTROL_CA_CERTIFICATE=./pki/edge-server-ca.crt
EDGE_RUNTIME_TLS_CERTIFICATE=./pki/node.crt
EDGE_RUNTIME_TLS_PRIVATE_KEY=./pki/node.key
DNS_API_SERVER_CERTIFICATE=./pki/node.crt
DNS_API_SERVER_PRIVATE_KEY=./pki/node.key

Edge registration and mTLS enrollment

Fleet prepares the host bundle but is not part of control-plane enrollment. An edge display name may differ from its Fleet, Terraform, Ansible, or server name. Prepare and transfer each host bundle once. A combined dns-edge bundle starts DNS normally before an edge identity exists.

To enroll an edge:

In Infrastructure → Edges, create the edge and copy the two-line environment block from the one-time modal.

On the matching prepared host, replace the empty values in .env.prod:

dotenv

EDGE_ID=11111111-2222-3333-4444-555555555555
EDGE_BOOTSTRAP_TOKEN=the-one-time-token

Keep .env.prod mode 0600, then start the edge profile explicitly:

bash

cd /opt/cdnfoundry
sudo chmod 0600 .env.prod
sudo docker compose --env-file .env.prod --profile edge up -d

For future whole-host starts, the operator may add --profile edge to the generated start.sh Compose up command. The script itself remains a fixed, transparent startup wrapper: it does not inspect enrollment state, wait for registration, edit .env.prod, or restart services. No Fleet command, bundle rerender, or second transfer is required.

After a fresh heartbeat, the bootstrap token is spent and cannot enroll again. Blank it in .env.prod during normal secret hygiene. An immediate second restart is not required for correct operation.

The agent creates its private key locally and persists the issued identity in edge-agent-state. Never copy or delete that volume. Directly assigned public service addresses need no EDGE_GATEWAY_ADDRESS_MAP; the gateway binds them directly. Configure the map once as advanced host state only for NAT or a load balancer, mapping every advertised address to its distinct private listener. Set EDGE_GATEWAY_REQUIRE_ADDRESS_MAP=true only where local policy requires explicit coverage.

Fleet automation may still use configure-edge-registration and clear-edge-bootstrap-token, but those commands are Fleet-specific and are not shown by the administrator panel.

Rotate an edge identity

Use rotation when an agent identity or its persistent state is lost or suspected compromised. This is an immediate revocation, not an overlapping certificate renewal. Open the edge under Infrastructure → Edges, choose Rotate identity, read the impact statement, and confirm only when the matching edge host is available.

The current certificate stops authenticating as soon as rotation is confirmed. The gateway and cells retain their last valid runtime, but agent heartbeat and configuration delivery pause until reenrollment. The replacement modal shows the same two environment values as initial enrollment. Replace them in the host's .env.prod and run the explicit --profile edge up -d command above. Compose recreates the agent when its environment changes. The agent keeps the previous local identity until the replacement certificate is issued, then atomically replaces it. Blank the spent replacement token afterward. DNS, edge-gateway, cells, and their last-valid runtime remain in place. If the modal is left before its token is saved, rotate again and use only the newest replacement token.

Embedded or remote control PostgreSQL

Embedded mode is the default. The control bundle contains control-db, and start.sh waits for control-db and redis before running the migration.

For remote mode, set the control node's optional Compose overrides and replace the generated database password with the real remote credential from a protected file:

bash

sudo ./scripts/cdnfoundry-fleet \
  --state-dir /var/lib/cdnfoundry-fleet \
  update-node --node control-1 \
  --extra-env DB_HOST=postgres.internal.example \
  --extra-env DB_PORT=5432 \
  --extra-env DB_SSLMODE=verify-full \
  --non-interactive

sudo ./scripts/cdnfoundry-fleet \
  --state-dir /var/lib/cdnfoundry-fleet \
  set-secret --secret control-db-password \
  --from-file /root/cdnfoundry-postgres-password \
  --non-interactive

A non-empty DB_URL also selects remote mode. Do not place a password-bearing URL in version control or shell history.

In remote mode the renderer removes control-db, its volume, and every dependency on it. The generated start helper waits only for local Valkey before migration. When telemetry is colocated and DB_HOST is supplied, Grafana's control-database provisioning and datasource defaults inherit the same host, port, and SSL mode unless explicitly overridden.

Before rendering, ensure the external service has:

database and application role expected by the project;
TLS and certificate hostname verification where supported;
exact-source network allowlists;
capacity and connection limits for web, Horizon, scheduler, migrations, and Grafana provisioning;
backups/PITR and an isolated restore test.

Validation and rendering

Use both steps in CI or before a production rollout:

bash

./scripts/cdnfoundry-fleet \
  --state-dir ./.cdnfoundry-fleet \
  --output-dir ./build/fleet-bundles \
  --repo-root "$PWD" validate

./scripts/cdnfoundry-fleet \
  --state-dir ./.cdnfoundry-fleet \
  --output-dir ./build/fleet-bundles \
  --repo-root "$PWD" render

Validation renders into a temporary directory and leaves active bundles unchanged. Rendering builds each node in a temporary directory, then atomically replaces the active bundle while retaining .previous.

Bundle transfer and activation

Archive one node from the generator host:

bash

tar --numeric-owner --owner=0 --group=0 \
  -C build/fleet-bundles \
  -czf /tmp/edge-frankfurt.tar.gz edge-frankfurt

On the target host:

bash

install -d -m 0700 /opt/cdnfoundry.new
tar -xzf /tmp/edge-frankfurt.tar.gz --strip-components=1 -C /opt/cdnfoundry.new
cd /opt/cdnfoundry.new
./validate.sh
cd /opt
mv cdnfoundry cdnfoundry.previous 2>/dev/null || true
mv cdnfoundry.new cdnfoundry
cd /opt/cdnfoundry
sudo ./start.sh

Never transfer the entire fleet state or another node’s bundle. Do not replace the generated control activation with a direct docker compose up: the activation applies the restricted PHP-worker access required for pki/edge-identity-ca.key. If core reports that the key is not readable, rerun sudo ./start.sh and verify stat -c '%u:%g %a %n' pki/edge-identity-ca.key reports 0:82 640.

validate.sh runs the pinned Caddy image's adapter against each Caddyfile present in the bundle. Treat any adapter error as a failed bundle and do not activate it. The validation container is short-lived and does not start dependencies; Docker may create the bundle's declared network or empty named-volume metadata while preparing the container.

If log-collector exits with code 78 and reports that LOG_AUTH_TOKEN is missing, do not put the token directly in the rendered Compose manifest. Verify that .env.prod contains a non-empty LOG_AUTH_TOKEN and that the rendered log-collector.environment maps LOG_AUTH_TOKEN from Compose interpolation, then rerender and transfer the corrected node bundle. Recreate only log-collector; its bounded operational-vector-data volume preserves buffered logs.

Updating the fleet

Add a host:

bash

./scripts/cdnfoundry-fleet --state-dir ./.cdnfoundry-fleet add-node \
  --node edge-tokyo --role edge --region asia --location tokyo \
  --public-ipv4 192.0.2.80 --non-interactive

Change a release or address:

bash

./scripts/cdnfoundry-fleet --state-dir ./.cdnfoundry-fleet update-node \
  --node edge-tokyo --release v1.1.0 --public-ipv4 192.0.2.81 \
  --non-interactive

Render only that host:

bash

./scripts/cdnfoundry-fleet \
  --state-dir ./.cdnfoundry-fleet \
  --output-dir ./build/fleet-bundles \
  --repo-root "$PWD" validate --node edge-tokyo

./scripts/cdnfoundry-fleet \
  --state-dir ./.cdnfoundry-fleet \
  --output-dir ./build/fleet-bundles \
  --repo-root "$PWD" render --node edge-tokyo

A changed hostname or public IP automatically causes that node certificate to be reissued with updated SANs.

Troubleshooting

The script prints only initialization output

Use the full wrapper or setup, not the low-level init command:

bash

./scripts/generate-production-env.sh
# or
./scripts/cdnfoundry-fleet setup --help

init deliberately creates only protected state and secrets. It does not add nodes or render bundles.

No interactive questions appear

Interactive mode needs a real terminal. In CI, a pipe, or a non-TTY shell, provide all values with --non-interactive --config FILE. The CLI now returns a clear validation error instead of silently waiting on unavailable input.

I cannot find control plus monitoring

Run the setup wizard and choose Control + monitoring on the same host, or configure it explicitly:

bash

./scripts/cdnfoundry-fleet --state-dir ./.cdnfoundry-fleet \
  configure-monitoring --mode colocated --non-interactive

Confirm with status.

State already exists

The full setup command reuses existing state. It does not rotate secrets. To inspect it:

bash

./scripts/cdnfoundry-fleet --state-dir ./.cdnfoundry-fleet status

A required variable is missing

The renderer derives required variables from the final filtered Compose file. Add project-specific values to a node’s extra_env only when the repository introduces a new required variable that the generator does not yet know. Do not place secrets in version-controlled config files.

Compose input contains `!reset` or `!override`

The Fleet loader supports both Docker Compose tags. Older generator versions used yaml.safe_load directly and failed on Compose inputs containing them.

Docker is unavailable on the generator machine

You can still generate bundles and run Python-level validation. Run each bundle’s validate.sh on a host with Docker Compose v2 before activation.

Production fleet operator guide ​

What the generator creates ​

Fastest supported setup ​

Control plus monitoring ​

Config-driven setup ​

CLI map ​

Preflight checks ​

Node roles ​

Control ​

DNS ​

Edge ​

DNS-edge ​

Monitoring ​

PKI layout ​

Edge registration and mTLS enrollment ​

Rotate an edge identity ​

Embedded or remote control PostgreSQL ​

Validation and rendering ​

Bundle transfer and activation ​

Updating the fleet ​

Troubleshooting ​

The script prints only initialization output ​

No interactive questions appear ​

I cannot find control plus monitoring ​

State already exists ​

A required variable is missing ​

Compose input contains !reset or !override ​

Docker is unavailable on the generator machine ​

Related documents ​

Production fleet operator guide

What the generator creates

Fastest supported setup

Control plus monitoring

Config-driven setup

CLI map

Preflight checks

Node roles

Control

DNS

Edge

DNS-edge

Monitoring

PKI layout

Edge registration and mTLS enrollment

Rotate an edge identity

Embedded or remote control PostgreSQL

Validation and rendering

Bundle transfer and activation

Updating the fleet

Troubleshooting

The script prints only initialization output

No interactive questions appear

I cannot find control plus monitoring

State already exists

A required variable is missing

Compose input contains `!reset` or `!override`

Docker is unavailable on the generator machine

Related documents