← Back to Fleet Dashboard

🐳 Fleet Docker Compose Architecture & Hardware Lifecycle

Compose Path: /opt/docker-configs/fah-compose.yaml
Container Image: ghcr.io/linuxserver/foldingathome:latest
Lifecycle Control: /opt/bin/fah-start & /opt/bin/fah-stop
Applies To: itztli, foldimate, origami, smallpieces, scraps

1. Standardized Docker Compose Specification

Every Folding@Home node in the fleet runs an identical single-service Docker Compose deployment located at /opt/docker-configs/fah-compose.yaml (differing only in the host-specific MACHINE_NAME and local UID/GID):

services:
  foldingathome:
    image: ghcr.io/linuxserver/foldingathome:latest
    pull_policy: always
    container_name: foldingathome
    restart: no
    stop_grace_period: 90s
    ports:
      - "7396:7396"
    volumes:
      - /opt/docker-configs/foldingathome/config:/config
      - /etc/localtime:/etc/localtime:ro
    environment:
      - TZ=America/Los_Angeles
      - PUID=2005
      - PGID=2000
      - ACCOUNT_TOKEN=<redacted>
      - MACHINE_NAME=<hostname>
      - NVIDIA_VISIBLE_DEVICES=all
      - NVIDIA_DRIVER_CAPABILITIES=all
      - CUDA_DEVICE_ORDER=PCI_BUS_ID
      - OpenCL_DEVICE_ORDER=PCI_BUS_ID
      - OPENMM_CPU_THREADS=4
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              capabilities: [gpu, utility, compute]

2. Key Architectural Configurations & Rationale

  1. Device Pass-Through via Driver Reservations (NVIDIA_VISIBLE_DEVICES=all): Rather than hardcoding static device_ids: ['0', '1', '...'] arrays (which fail if a GPU drops or shifts indices), the compose spec delegates all GPU pass-through via modern NVIDIA Container Toolkit resource reservations (driver: nvidia, capabilities: [gpu, utility, compute]).
  2. PCIe Bus ID Alignment: The environment variables CUDA_DEVICE_ORDER=PCI_BUS_ID and OpenCL_DEVICE_ORDER=PCI_BUS_ID ensure that CUDA and OpenCL device numbering within the container maps strictly to physical PCIe bus hierarchy rather than arbitrary driver compute heuristics.
  3. Host RAM & Thread Pool Cap (OPENMM_CPU_THREADS=4) — Zero-Penalty Host Memory Relief:

    Even when a host is configured for pure GPU folding (cpus="0"), every FahCore_27 (OpenMM 8.2.0) GPU process scales its host resident memory (VmRSS) and process thread count linearly with the host CPU's logical thread count T (sysconf(_SC_NPROCESSORS_ONLN)) unless explicitly capped:

    • Binary Disassembly Root Cause (FahCore_27 0x433bbe vs. 0x433c64): Disassembling FahCore_27 reveals that each GPU worker constructs two OpenMM::Context objects at Work Unit startup:
      1. Primary GPU Simulation Context (0x433bbe, stored at this+0xb78): Invokes the 4-argument OpenMM::Context(System&, Integrator&, Platform&, map<string,string>&) constructor with explicit CUDA device properties. 100% of the molecular dynamics trajectory (1,250,000 steps per WU) executes inside this CUDA context in GPU VRAM.
      2. Secondary CPU Reference Context (0x433c64, stored at this+0xb80): Immediately after creating the CUDA context, FahCore_27 logs "Using CPU platform for reference calculations." (0x433bed) and invokes the 3-argument OpenMM::Context(System&, Integrator&, Platform&) constructor (Platform::getPlatformByName("CPU")) with no properties map—omitting the "Threads" property (OpenMM::CpuPlatform::CpuThreads).
    • OpenMM Unbounded Thread & Buffer Allocation (4T + 8 Threads & ~112–160 MiB/Thread per GPU): Because no "Threads" property is passed to the 3-argument constructor, libOpenMMCPU.so and libOpenMMPME.so fall back to getenv("OPENMM_CPU_THREADS") and, when unset, default to getNumProcessors() (sysconf(_SC_NPROCESSORS_ONLN) = all logical CPU threads T on the host). For every GPU in the system, OpenMM spawns 4 independent OpenMM::ThreadPool instances of T threads each (4T + 8 total threads per FahCore_27 process) and pre-allocates private per-thread force accumulation buffers (float[4 * N_atoms] in CpuPlatform::Data::threadForce) plus per-thread 3D FFT PME charge-spreading sub-grids (tempGrid via fftwf_malloc in CpuPmeReciprocalForce).
    • Why OPENMM_CPU_THREADS=4 Has Zero Impact on TPF or PPD (22 Frames Out of 1,250,000 Steps): Tracing all cross-references to 0x432bb0 ("Comparing forces and energies between GPU and CPU...") in FahCore_27 proves that the CPU reference context (this+0xb80) is never stepped during the MD simulation. It is called strictly twice at Step 0 (0x434564 before and 0x4345ea after energy minimization) and once at every 5% checkpoint (0x4115a9, every 62,500 GPU steps = 20 times per WU) to copy atomic coordinates from the GPU, compute a single static frame of forces/energies on the CPU, and verify that the GPU integrator has not diverged (NaN / hardware fault check). With OPENMM_CPU_THREADS=4, that single-frame check at each 5% checkpoint still completes in <50 ms every 35–60 minutes (<0.002% of WU runtime) while eliminating massive idle thread-buffer bloat across all 5 fleet folding hosts:
    Fleet Host CPU Model & Logical Threads (T) Active GPUs Uncapped Threads / GPU (4T + 8) Uncapped Host RSS / GPU Uncapped Total FahCore_27 RSS Capped Threads / GPU (OPENMM_CPU_THREADS=4) Capped Total FahCore_27 RSS
    origami v2 AMD EPYC 7543 (32C / 64T) 3 GPUs 264 threads (792 total) 9.04–12.19 GiB 33.40 GiB 24 threads (72 total) ~8.5–9.8 GiB (-24 GiB)
    smallpieces v3 AMD Ryzen 9 9900X (12C / 24T) 5 GPUs 104 threads (520 total) 4.45–6.29 GiB 25.95 GiB (on 32 GB host) 24 threads (120 total) ~13.5–15.0 GiB (-12 GiB)
    itztli AMD Ryzen 9 9950X3D (16C / 32T) 2 GPUs 136 threads (272 total) 5.40–7.20 GiB ~12.60 GiB 24 threads (48 total) ~5.6–6.4 GiB (-6.5 GiB)
    foldimate v1 AMD Ryzen 9 7950X (16C / 32T) 1 GPU 136 threads (136 total) 5.40–7.20 GiB ~6.30 GiB 24 threads (24 total) ~2.8–3.2 GiB (-3.3 GiB)
    scraps v2 Intel Core i7-7700 (4C / 8T) 4 GPUs 40 threads (160 total) 2.80–3.25 GiB 12.31 GiB 24 threads (96 total) ~10.2–11.0 GiB (-1.8 GiB)
  4. Stop Grace Period (stop_grace_period: 90s): Provides a generous 90-second shutdown buffer, allowing active OpenMM simulation steps and checkpoints to flush cleanly before SIGKILL.
  5. Service Lifecycle (restart: no), Utility Rate Pause Window & Host Reboot Trade-Off: Prevents Docker daemon auto-restart loops during hardware faults, permitting host systemd watchdog units (fah-runaway.service) to manage recovery and clean container restarts. On the dedicated folding servers (origami, smallpieces, scraps), fah-stop.timer and fah-start.timer pause folding daily from 15:00 to 00:00 local time (14:58 stop → 00:01 start) — widened from the former 16:00–21:00 window after changing the utility rate plan from TOU-C to EV2-A, which has "near-peak" and peak hours between 15:00 and midnight — while the desktop gaming rigs (itztli, foldimate) fold strictly overnight during off-peak hours (00:00/00:01–06:15). The trade-off of restart: no is that if a host workstation is rebooted during active folding hours, folding does not automatically start on boot and must be manually initiated. To make this convenient, the companion helper scripts /opt/bin/fah-start and /opt/bin/fah-stop were created. Because fleet scheduling relies on systemd oneshot services triggered by timers (fah-start.timer / fah-stop.timer), manually invoking systemctl start directly can leave a oneshot service stuck in an active state where subsequent timer alarms fail to trigger it; fah-start and fah-stop cleanly trigger the startup/shutdown sequence and immediately reset the service back to an inactive/clean state so future automated timer triggers fire reliably.

3. Graceful Hardware Swap Routine

To prevent orphaned slot bindings or phantom "unsupported" devices in F@H v8 client.db when replacing or relocating physical GPU cards on any fleet host:

  1. Finish WUs: In the F@H UI, set the container to Finish so active simulations complete cleanly without fetching new units.
  2. Unselect Devices: Once all units finish, unselect all active GPUs from the allowed devices list in the UI.
  3. Shutdown & Swap: Stop the compose service (/opt/bin/fah-stop or docker compose down) and execute the physical card swaps.
  4. Boot & Go: Start the container (/opt/bin/fah-start or docker compose up -d). F@H dynamically re-enumerates the fresh hardware complement without phantom entries. Select all live GPUs in the UI, save changes, and click the Play icon.

4. Fleet Host References