← Back to Fleet Summary

smallpieces Hardware, History, and Roadmap

Page Overview: Complete operational chronicle, physical slot mapping, Docker configuration, and upgrade roadmap for server smallpieces.


📜 1. Historical Journey (Feb 2026 – Present)

1.1 Executive Overview & Architectural Evolution

Server smallpieces originated as a budget multi-GPU prebuilt mining rig frame designed to process Folding@Home Work Units across cost-effective mid-tier graphics cards. Over time, it underwent a complete transformation from a memory-constrained Haswell system (smallpieces v1) to a high-density, 12-slot custom chassis built on Kaby Lake hardware (smallpieces v2).

smallpieces v1 Era (Feb 2026 – Jun 29, 2026) smallpieces v2 Era (Jun 30, 2026 – Present)
• GPNE Mining Rig Frame Case• Octominer X12 Ultra 12-Slot Chassis
• AX-B85M-ETH Motherboard• B250 Custom Board
• Intel Core i7-4790 (Upgraded from Celeron)• Intel Core i7-7700 (Kaby Lake 4C/8T)
• 8GB DDR3 SODIMM (Upgraded from 4GB)• 32GB DDR3L SODIMM (2x 16GB, Upgraded)
• Noctua NH-L9x65 Cooler• Thermalright AXP90-X36 Cooler
• 2400W Mining Power Supply• 4x Quad Octominer HPE 750W PSUs
• Shared 110V / 15A Circuit• Dedicated 240V / 30A Electrical Circuit
• Mixed GPUs (GTX 1070 / 2060 / 3060 Ti)• 1x RTX 5070 + 1x RTX 4070 + 4x RTX 5060 Ti

1.2 Phase 1: smallpieces v1 Era (Feb 2026 – June 2026)

Memory Constraints, "iowait Hell", and the zswap Hack

Early GPU Telemetry & The AMD RX 9060 XT Test


1.3 Phase 2: smallpieces v2 Migration & Octominer Discovery (June 27 – June 30, 2026)

Component Category Hardware Specification & Details
ChassisOctominer X12 Ultra Custom 12-Slot Server Rig
Electrical CircuitDedicated 240V / 30A AC Circuit
Processor (CPU)Intel Core i7-7700 (Kaby Lake 4C/8T, 3.6 GHz)
CPU CoolerNoctua NH-L9i Low-Profile Cooler
System Memory (RAM)32GB (2x 16GB) DDR3L-1600MHz PC3-12800 SODIMM
Power Delivery4x (Quad) Octominer HPE 750W 200V-277V Platinum Plus PSUs
Cables & PowerC13/C14 14AWG 15A/250V Power Extension Cords

Key Architectural Correctives

  1. Cooler Selection (Noctua NH-L9i): The planned Thermalright AXP90-X53 Full Copper cooler exceeded the lid clearance of the OX12U chassis. The system ran temporarily on a re-pasted stock cooler before installation of the low-profile Noctua NH-L9i (37mm height), which with the finstack parallel to the DRAM to prevent a heatwall, kept the CPU at 55–70°C with 4mm clearance between the fan and PSU cage.
  2. RAM Bottleneck Eliminated: Upgraded to 32GB DDR3L (2x 16GB SODIMM), completely removing the legacy 8GB Haswell swap constraint.
  3. Quad Power Supply Array: Powered by 4x (quad) Octominer HPE 750W Platinum Plus PSUs running on 240V input for optimal power distribution across 12 slots.

1.4 Phase 3: Modern Fleet Standard (July 2026 – Present)


🎛️ 2. Physical to Logical GPU Ordering Map (Octominer X12 Ultra)

The physical slot designations are silk-screened on the Octominer X12 Ultra motherboard.

Motherboard Slot GPU Model PCI Bus ID NVIDIA Index GPU UUID Target Container Port Notes
Slot 1 (x16)MSI VENTUS RTX 5070 OC Dual0000:01:00.00GPU-d87d48a0-49cf-c9ec-f567-9f46ad84851efoldingathome7396Active (12GB GDDR7, CPU-tied slot)
Slot 2 (x1)EmptyEmpty
Slot 3 (x1)EmptyEmpty
Slot 4 (x1)MSI RTX 5060 Ti 8G SHADOW #20000:07:00.05GPU-8854baa9-8a40-a186-cef9-38b66e867190foldingathome7396Active
Slot 5 (x1)EmptyEmpty
Slot 6 (x1)Zotac Blower RTX 40700000:04:00.04GPU-d65f3411-caf4-41a8-0248-0dcf200dee35foldingathome7396Active
Slot 7 (x1)EmptyEmpty
Slot 8 (x1)MSI RTX 5060 Ti 8G VENTUS0000:06:00.03GPU-56d10701-aced-b504-06b0-0d0fc856686dfoldingathome7396Active
Slot 9 (x1)EmptyEmpty
Slot 10 (x1)MSI RTX 5060 Ti 8G SHADOW #10000:03:00.02GPU-a83e55f9-8d3c-4770-8faf-cc15567610cdfoldingathome7396Active
Slot 11 (x1)EmptyEmpty
Slot 12 (x1)PNY RTX 5060 Ti 16GB VERTO0000:02:00.01GPU-2e4a1d96-f5b4-0b5a-c5e2-91c467224a00foldingathome7396Active (16GB Dual Fan)

🐳 3. Docker Compose Considerations

The smallpieces host runs a consolidated single-service Docker Compose deployment located at /opt/docker-configs/fah-compose.yaml:

services:
  foldingathome:
    image: lscr.io/linuxserver/foldingathome:latest
    pull_policy: always
    container_name: foldingathome
    restart: no
    stop_grace_period: 90s
    ports:
      - "7396:7396"
    volumes:
      - /opt/docker-configs/foldingathome/config:/config
      - /etc/localtime:/etc/localtime:ro
    environment:
      - TZ=America/Los_Angeles
      - PUID=2005
      - PGID=2000
      - ACCOUNT_TOKEN=
      - MACHINE_NAME=smallpieces
      - NVIDIA_VISIBLE_DEVICES=all
      - NVIDIA_DRIVER_CAPABILITIES=all
      - CUDA_DEVICE_ORDER=PCI_BUS_ID
      - OpenCL_DEVICE_ORDER=PCI_BUS_ID
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              capabilities: [gpu, utility, compute]

Key Architectural Configurations & Rationale

  1. Device Pass-Through via Driver Reservations (NVIDIA_VISIBLE_DEVICES=all): Rather than hardcoding static device_ids: ['0', '1', '...'] arrays (which fail if a GPU drops or shifts indices), the compose spec delegates all GPU pass-through via modern NVIDIA Container Toolkit resource reservations (driver: nvidia, capabilities: [gpu, utility, compute]).
  2. PCIe Bus ID Alignment: The environment variables CUDA_DEVICE_ORDER=PCI_BUS_ID and OpenCL_DEVICE_ORDER=PCI_BUS_ID ensure that CUDA and OpenCL device numbering within the container maps strictly to physical PCIe bus hierarchy rather than arbitrary driver compute heuristics.
  3. Stop Grace Period (stop_grace_period: 90s): Provides a generous 90-second shutdown buffer, allowing active OpenMM simulation steps and checkpoints to flush cleanly before SIGKILL.
  4. Service Lifecycle (restart: no) & Host Reboot Trade-Off: Prevents Docker daemon auto-restart loops during hardware faults, permitting host systemd watchdog units (fah-runaway.service) to manage recovery and clean container restarts. The trade-off is that if the host workstation is rebooted during active folding hours, folding does not automatically start on boot and must be manually initiated. To make this convenient, the companion helper scripts /opt/bin/fah-start and /opt/bin/fah-stop were created. Because fleet scheduling relies on systemd oneshot services triggered by timers (fah-start.timer / fah-stop.timer), manually invoking systemctl start directly can leave a oneshot service stuck in an active state where subsequent timer alarms fail to trigger it; fah-start and fah-stop cleanly trigger the startup/shutdown sequence and immediately reset the service back to an inactive/clean state so future automated timer triggers fire reliably.

Graceful Hardware Swap Routine

To prevent orphaned slot bindings or phantom "unsupported" devices in F@H v8 client.db when replacing physical GPU cards:

  1. Finish WUs: In the F@H UI, set the container to Finish so active simulations complete cleanly without fetching new units.
  2. Unselect Devices: Once all units finish, unselect all active GPUs from the allowed devices list in the UI.
  3. Shutdown & Swap: Stop the compose service (docker compose down), execute the physical card swaps on the Octominer board.
  4. Boot & Go: Start the container (docker compose up -d). F@H dynamically re-enumerates the fresh hardware complement without phantom entries. Select all live GPUs in the UI and, save changes, and click the Play icon.

🚀 4. Hardware Upgrade Roadmap (Completed August 8, 2026)

  1. Phase 1 (Slot Swap - Completed July 17, 2026): Relocated RTX 4070 from Slot 10 to physical Slot 1 (CPU-adjacent x16 slot) and moved MSI SHADOW RTX 5060 Ti to Slot 10 (x1 riser). Verified system stability and unthrottled x16 link throughput (+1.49M PPD net pair surge).
  2. Phase 2 (RTX 5070 Slot 1 Upgrade & 6-GPU Array Expansion - Completed July 18, 2026): Installed MSI GeForce RTX 5070 VENTUS 2X OC in physical Slot 1 (PCIe Gen3 x16), relocated ZOTAC RTX 4070 to Slot 6 (x1 riser), and expanded live smallpieces array to 6 GPUs.
    • Initial Observations:
      • Slot 1 (x16) Live WebSockets & Overnight Steady-State Truth (gpu:01:00:00): Overnight telemetry across 12 completed Work Units establishes the steady-state performance ceiling for the RTX 5070 VENTUS in Slot 1 x16 (gpu:01:00:00 | UUID d87d48a0..). On benchmark Project 18246, TPF converged down to 1m43.3s (103.3s), yielding a peak production rate of 9.01M PPD and a 95th percentile rate of 8.86M PPD (+9.3% net PPD surge and -6.2s/frame speedup over the RTX 4070 N-1 baseline in the same slot). High-frequency NVML profiling confirms peak PCIe bus throughput of 1.79 GB/s without link throttling.
      • Slot 6 (x1) 4070 Performance (gpu:04:00:00): RTX 4070 (gpu:04:00:00 in x1 riser | UUID d65f3411..) yields 4.66M PPD (@ 145W cap / 83.2W real load). High-frequency NVML sampler records peak burst transfers of 1.21 GB/s during CUDA frame step exchanges.
      • Array & Fleet Real-Time Yield: Live smallpieces host output currently streams at 27.60M PPD, bringing total live fleet production to ~66.5M PPD (projected ceiling ~69.2M PPD when itztli is not running).
      • Thermal & Power Stability: Total 6-GPU array power draw measured at 616.0W on 240V power with zero thermal throttling or watchdog faults.
  3. Phase 3 (Blackwell Scheduling & 7-GPU Array Feasibility - July 19, 2026):
    • Ada Spin-Locking vs. Blackwell Event Yielding: Overnight high CPU loads and container watchdog resets (fah-runaway) were traced to Ada Lovelace (RTX 4070) driver behavior (cuEventSynchronize busy-wait spin-locking), which pegged CPU cores at 93.2%. Blackwell cards (cudaDeviceScheduleYield) yield CPU threads cleanly to kernel idle (intel_idle), dropping host CPU utilization to 71.5% with 28.5% idle headroom under 5 Blackwell GPUs.
    • Slot 1 GPU CPU Workload Progression: Empirical baseline telemetry across regimes confirms that higher-tier cards in CPU-tied Slot 1 x16 demand progressively greater CPU overhead to service host-to-device event polling and Hilbert grid reordering:
      • Regime N-2 (RTX 5060 Ti in Slot 1 x16): ~11.5% host CPU demand (total 5-GPU host load 65.8%).
      • Regime N-1 (RTX 4070 in Slot 1 x16): ~13.2% host CPU demand (total 5-GPU host load 68.4%).
      • Regime N (RTX 5070 in Slot 1 x16 + 6th GPU): ~15.5% host CPU demand (total 6-GPU host load 93.2%).
    • 7-Blackwell GPU Array Feasibility Model: Because Blackwell GPUs yield host CPU threads cleanly, scaling smallpieces to 7 Blackwell GPUs (1x 5070 Dual + 6x 5060 Ti) is physically possible on the Core i7-7700 host, projecting 36.95M PPD at 41.60 WUs / day (~88-92% CPU load).
    • CRITICAL 4070 EXCLUSION RULE: Running 7 GPUs is ONLY possible if ALL installed cards are Blackwell (RTX 50-series). Running 7 GPUs with the RTX 4070 present would push host CPU load past 105%, triggering severe thread lockups and watchdog resets (fah-runaway).
    • Platform Upgrade Conclusion: The Intel B250 / LGA 1151v1 chipset natively maxes out at 4C/8T processors (i7-7700). While 6 Blackwell GPUs (or 7 Blackwell GPUs with all-50-series cards) represent the physical ceiling for this chassis, further compute expansion beyond 7 cards will require a next-generation server architecture.