A huge part of the fun of running a multi-rig Folding@Home fleet is seeing how much real science you can squeeze out of a fixed power budget, tight room thermals, and repurposed hardware. Anyone can plug a flagship GPU into a desktop at 100% factory power limit and let it turn a room into a sauna. The real game is doubling points-per-watt (PPD/W), keeping every card in the 50–66°C thermal sweet spot, and coaxing near-flagship efficiency out of older motherboards, mining frames, and power-capped GPUs.
This page collects the biggest engineering wins from our fleet history alongside a practical, copy-and-paste Field Guide of tips and tweaks for other folders.
10.11M ➔ 20.20M PPD) for Just +10 Watts (scraps v1 ➔ v2)Our salvage node scraps started life on a 2014 Haswell i7-4790 with 8GB DDR3 and PCIe Gen 2 x1 slots (AX-B85M-ETH), running 5 older GPUs (3x RTX 3060 Ti + RTX 2060 SUPER + RTX 2060). When smallpieces upgraded out of its Octominer X12 Ultra chassis (i7-7700 + 32GB DDR3L + PCIe Gen 3 x16/x1), we migrated scraps into the vacated Octominer, retired the two Turing 20-series cards, and paired the 3x RTX 3060 Ti (125W cap) with an RTX 3070 (145W cap) and a blower RTX 4070 (145W cap).
After unlocking the mining board's CPU multiplier, dialing in a 3.00 GHz undervolt sweet spot, and isolating each GPU's main CUDA polling thread onto its own CPU core, scraps v2 delivers literally 2.0x the PPD at the exact same wall power—while running 10–14°C cooler:
| Metric | scraps v1 (5 GPUs: 3x 3060 Ti + 2060S + 2060) |
scraps v2 Now (5 GPUs: 4070 + 3070 + 3x 3060 Ti) |
Net Improvement |
|---|---|---|---|
| Total Node Production | 10.11M PPD |
20.20M PPD (20,197,060 PPD) |
+99.8% (2.0x PPD) |
| GPU + CPU Silicon Power | 636.9 W (586.1W GPU + 50.8W i7-4790) |
647.3 W (628.8W GPU + 18.5W i7-7700) |
+1.6% (+10.4 W) |
| Whole-Node Silicon Efficiency | 15.87k PPD/Watt |
31.20k PPD/Watt |
+96.6% (~2.0x PPD/W) |
| GPU Steady-State Temps | 66–77°C |
55–63°C (RTX 4070 at 62°C, RTX 3070 at 63°C) |
-10°C to -14°C cooler |
| CPU Temp, Vcore & Power | 74–82°C (50.8W Haswell) |
66.0°C (0.98V, 18.5W @ 3.00 GHz) |
-63.6% CPU power |
"Unicorn" Tier (smallpieces v2 ➔ v3)For three months, our five Blackwell GPUs (1x RTX 5070 at 165W + 4x RTX 5060 Ti at 130W) ran inside the Octominer X12 Ultra (i7-7700 + DDR3L-1600 + PCIe Gen 3.0 x1 PCH slots). Even though the cards themselves were brand-new, the 1x PCIe bus and 4-core CPU held those 5 cards to ~23–25M PPD (~27.6M PPD with a 6th card).
When we rebuilt smallpieces v3 around an AMD Ryzen 9 9900X (65W Eco Mode), 32GB DDR5-6000, and an ADT-Link R34A x4/x4/x4/x4 PCIe Gen 5.0 bifurcation adapter (giving four GPUs CPU-direct PCIe Gen 5.0 x4 and the fifth GPU PCIe Gen 4.0 x4), those exact same 5 GPUs at the exact same 165W/130W power caps jumped to 45.33M PPD (+78%) and 59.4k PPD/Watt—putting power-capped GPUs squarely into the "Unicorn" performance band on LAR.systems.
OPENMM_CPU_THREADS=4)Early on, adding GPUs to a Linux host with a high thread count caused mysterious kernel page-table lockups, zswap thrashing, and multi-gigabyte memory footprints per container.
FahCore_27 / FahCore_28) spawn host CPU helper threads for step-boundary tasks. By default, OpenMM inspects the host's total logical CPU count (24 threads on a 9900X, 32 on a 9950X3D, 64 on an EPYC 7543) and spawns a matching pool of threads per GPU—allocating per-thread working buffers that never get used while the GPU does 99% of the math.OPENMM_CPU_THREADS=4 in our standardized Docker Compose environment cut host RAM usage by 60–75% across every node (0.00 KB/s swap across the entire fleet) with zero loss in PPD.MSR 0x1a0 Bit 38)When we first audited the Octominer X12 Ultra (B250 mining motherboard + i7-7700), we discovered why its CPU always ran at 50°C and 4.3W: the mining BIOS intentionally sets bit 38 of IA32_MISC_ENABLE (MSR 0x1a0 = 0x4000850089) at boot, which traps the Linux intel_pstate driver in its 8x (800 MHz) Lowest Frequency Mode 93% of the time.
Simply clearing bit 38 of MSR 0x1a0 (wrmsr -a 0x1a0 0x850089) unlocked hardware P-states up to 3.60 GHz and instantly shaved 4.0–5.5 seconds off every single frame (+11% to +15% PPD across the whole chassis) before touching a single GPU setting.
FahCore Spin-Wait Collisions & Cross-CCD Bouncing (nvpowerctl v1.3.4 Thread Affinity)When we installed a 5th GPU (RTX 4070) onto scraps's 4-core / 8-thread i7-7700, nvtop showed the RTX 4070's SM utilization chopping wildly between 42% and 100% every few seconds.
FahCore process has one main CUDA dispatch thread (TID == PID) that busy-polls cuEventSynchronize at 100% CPU, plus ~27 low-duty helper threads. Left to default Linux scheduling, two 100%-spinning main CUDA loops frequently landed on the same logical CPU core, time-slicing each other 50%/50% while other logical cores sat idle.nvpowerctl v1.3.4 (fah_thread_affinity = enabled, fah_main_cpus, and optional semicolon-separated per-GPU fah_helper_cpus). Using os.sched_setaffinity(tid, mask), nvpowerctl pins only the main thread (tid == pid) of each FahCore to its own dedicated logical core (0, 3, 1, 2, 7 on scraps; 0, 16, 24 on origami) while pinning OpenMM helper threads (tid != pid) either across all cores ({0..7} on scraps) or to dedicated CCD-local physical cores (1-9; 17-23,10-11; 25-31,12-13 on origami's 32-core EPYC 7543). Result: nvtop SM% locked flat at 94–100% and scraps output jumped +1.5M PPD for zero extra watts.968-WU Controlled A/B Experiment)Recent PC hardware analyses note that when you measure whole-system energy (Perf / (P_gpu + P_host)), aggressive 60% power caps can actually hurt efficiency due to fixed VRAM/board draw (35–55W), silicon V_min voltage floors, and desktop idle overhead (~120–150W)—especially in Folding@Home where Quick Return Bonus scales super-linearly (PPD ~ Speed^1.5). We put that hypothesis to a 48-hour, zero-straddler, project-normalized A/B test (801 PRE vs 167 POST WUs + 16,094 neumon snapshots) across smallpieces v3 and scraps v2, using 7 simultaneous untouched GPUs (±0.4% drift) as live controls:
| Host & GPU Tested | Change Applied | Real Power (dW) |
Like-for-Like PPD Delta | Marginal dPPD / dW |
Node Wall PPD/W Impact |
|---|---|---|---|---|---|
origami RTX PRO 4000 (6f99ccc4) |
135W ➔ 145W (100% max) |
+9.6 W |
+10.2% (11.63M ➔ 12.82M on P18264) |
123,900 PPD/W |
+1.9% GPU PPD/W (145W wins!) |
smallpieces RTX 5070 (d87d48a0) |
175W ➔ 185W |
+9.8 W (+5.4%) |
+1.90% (+269k PPD; 14.73M on P18258) |
26,619 PPD/W (> a 3060 Ti!) |
-0.75% combined node wall PPD/W |
smallpieces 5060 Ti 16GB (2e4a1d96) |
150W ➔ 160W |
+3.5 W (+2.3%) |
+0.79% (+63k PPD; +1.17% vs 8GB controls) |
18,125 PPD/W |
(included in -0.75% above) |
scraps 3x RTX 3060 Ti |
125W ➔ 140W each |
+39.7 W total (+10.5%) |
+2.57%..+4.96% (+345k PPD; +6.1% on P15506) |
8,689 PPD/W |
-4.20% combined node wall PPD/W |
scraps RTX 4070 (d65f3411) |
145W ➔ 160W |
+12.2 W (+8.2%) |
+1.66% (+118k PPD; +2.6% on P18264) |
9,688 PPD/W |
(included in -4.20% above) |
scraps RTX 3070 (1522578d) |
145W cap kept; fan auto ➔ 65% |
-0.9 W leakage |
+0.22% (+8.5k PPD; 60°C ➔ 52°C) |
Free PPD + less W! | +1.03% GPU PPD/W, -60% blips |
100% Limit Wins on Enterprise Silicon (RTX PRO 4000): Because the single-slot RTX PRO 4000 (GB203) is factory-capped at 145W (~55% of native desktop die TGP), throttling it to 135W pushes it onto its V_min floor while 24GB GDDR7 keeps burning fixed watts—so 145W wins on both speed and PPD/W.scraps's caps jumped core clocks +105..+270 MHz, two physical laws muted wall-efficiency gains:
Speed ~ SM_Clock^0.25, not Clock^1.0): Power limits raise only SM core clock while VRAM clock (6,800 / 10,251 / 13,801 MHz), PCIe Gen3 x1 transfers, and DDR3L 5%-checkpoint stalls stay fixed—so 166k-atom P15506 gained up to +6.1% PPD while memory-heavy 919k-atom P18258 gained only +0.8%..+3.4%.P_host Down to ~12W/GPU: A 1-GPU gaming PC carries ~150W of fixed host overhead on one card (flattening % dWall), whereas scraps shares 18.4W CPU + ~42W board across 5 GPUs. Thus 62.5%–72.5% consumer TGP (125W 3060 Ti, 145W 4070, 175W 5070) is the true multi-GPU wall-efficiency peak, while 70%–80% TGP (140W/160W/185W) is the max-throughput gear (+804k PPD combined, kept active for our 16.07B+ leaderboard climb)."Unicorn" on Some Projects and "Normal" / "OK" on OthersA common question when tuning for efficiency is: "If I power-cap my GPUs by 25–35%, how does LAR.systems' webclient decide whether a slot is "Unicorn", "Normal", "OK", or "OOF"—and why can the exact same card jump from "OK" on one project to "Unicorn (+88%)" on the next?"
(GPU Model, OS, Project ID))RTX 4070 against an RTX 5070 or RTX PRO 4000 on the same project. Instead, the webclient fetches a model-specific JSON profile (https://folding.lar.systems/api/gpu_ppd/<device_name_api>.json, e.g., ad104_geforce_rtx_4070.json), looks up the (os_type, project_id) entry (findProjectByOsAndId), and compares your live slot PPD strictly against other samples submitted for that exact GPU model, OS (Linux vs Windows), and Project ID:
gpuPerfRatio = ((slot_ppd_raw / avg_ppd) * 100) - 100UNICORN: >= +30% | GREAT: +20%..+30% | GOOD: +10%..+20% | NORMAL: -10%..+10% | OK: -20%..-10% | POOR: -30%..-20% | OOF: < -30%RTX 4070 on scraps v2 Scores "OK (-14%)" on P18258 and "Unicorn (+88%)" on P18264When our ZOTAC RTX 4070 Blower (d65f3411) ran its first two Alzheimer's work units (919,221 atoms) on scraps v2, LAR.systems rated the first WU "OK" (-14%) and the second WU "UNICORN" (+88%) at 6.21M PPD (and across its next 39 WUs it climbed to 6.94M–7.94M median PPD across P18264/P18258/P15243):
Work Unit (scraps v2 RTX 4070) |
Project (919,221 atoms) |
LAR (RTX 4070, Linux) avg_ppd |
LAR Linux samples |
Our scraps v2 RTX 4070 PPD |
LAR Ratio & Badge | What Populated LAR's Baseline |
|---|---|---|---|---|---|---|
1st WU (18258-122-9-45) |
P18258 |
7,997,554 PPD |
15 |
~6.88M PPD |
-14.0% (OK) |
Small sample (n=15) of un-capped 200W desktop RTX 4070 cards on PCIe x16 + modern DDR4/DDR5 CPUs. |
2nd WU (18264-202-3-327) |
P18264 |
3,305,581 PPD |
1,112 |
6.21M–6.94M PPD |
+88%..+110% (UNICORN) |
Self-referential historical telemetry from smallpieces v2! Whenever the LAR webclient tab is open, send_device_ppd_project_push_delay() pushes live slot samples to LAR. Nearly all 1,112 Linux P18264 samples for the RTX 4070 were submitted by our own RTX 4070 when it was choked at ~3.3M PPD on a PCIe 3.0 x1 riser (Slot 6) + 800 MHz BIOS-locked i7-7700 inside smallpieces v2! |
In short, the +88% "Unicorn" badge on P18264 is a live, self-referential measurement of how much faster that exact RTX 4070 runs on scraps v2 (Slot 1 PCIe Gen 3.0 x16 + 3.00 GHz unlocked i7-7700 + nvpowerctl TID == PID core isolation = 6.21M PPD) compared to its old smallpieces v2 self (3.31M PPD).
When comparing against un-skewed desktop baselines (like P18258 on the RTX 4070 or P15506 on the RTX 3070), moving our Blackwell cards onto smallpieces v3 (Ryzen 9 9900X + Gen 5/4 x4) and our Ada/Ampere cards onto the Octominer (scraps v2) isolates the five hardware factors that determine whether a power-capped GPU scores "Unicorn" or "Normal" / "OK":
| Bottleneck Dimension | smallpieces v3 ("Unicorn" Tier) |
scraps v2 ("Normal" / "OK" on Un-Skewed Projects; "Unicorn" vs sp v2 on P18264) |
Real-World Impact on *070 GPUs |
|---|---|---|---|
| 1. PCIe Bandwidth per GPU | 4x PCIe Gen 5.0 x4 (15.8 GB/s) + 1x Gen 4.0 x4 (7.9 GB/s), all wired directly to the CPU (0 chipset contention). |
RTX 4070 on Slot 1 Gen 3.0 x16 (15.8 GB/s), but RTX 3070 on Slot 2 Gen 3.0 x1 (~0.98 GB/s) shared through the B250 chipset with 3x 3060 Ti. |
Moving the RTX 4070 from sp v2 Slot 6 (Gen3 x1 + 800 MHz CPU) to scraps Slot 1 (Gen3 x16 + 3.0 GHz CPU) lifted its P18264 output +88% (3.31M ➔ 6.21M PPD); moving the RTX 3070 from Slot 1 (x16) to Slot 2 (x1) slowed P15506 frame time from 43.5s (4.69M) ➔ 46.5s (4.10M, -12.5%). |
| 2. Physical CPU Cores & Single-Thread IPC | 12 Zen 5 cores / 24 threads (~5.2 GHz boost) for 5 GPUs (2.4 physical cores per GPU). OpenMM's 4 barrier threads run on idle physical cores without ever touching a CUDA polling core. |
4 Kaby Lake cores / 8 threads (3.00 GHz) for 5 GPUs (0.8 physical cores per GPU). With 5 GPUs on 4 physical cores, every core's execution pipeline is shared across CUDA polling + 20 OpenMM threads. |
Zen 5 IPC on smallpieces v3 even lets d87d48a0 (RTX 5070) beat origami's EPYC 7543 (3.7 GHz Milan) RTX 5070 (dd40d876) across 4/4 shared projects (+4.7% to +8.9% PPD, 14.73M vs 13.53M on P18258). |
| 3. CPU L3 Cache & RAM Speed | 64 MB L3 Cache + DDR5-6000 (~75 GB/s). OpenMM host force arrays stay inside L3 cache; 5% checkpoints compress in <1.2s. |
8 MB L3 Cache (1/8th) + DDR3L-1600 (~21 GB/s, 1/3.5th). Host atom arrays spill to DDR3L RAM; 5% checkpoints stall the RTX 4070 for 5.0–6.0s. |
On Project 18249, the RTX 4070 runs 1% steps in 37s, but loses 5–6s every 5% checkpoint (39.0s effective TPF = ~6.05M PPD). |
| 4. Driver Spin-Wait vs Yield | Blackwell (RTX 50-series) uses event-driven yielding (~71% host CPU load across 5 GPUs), leaving breathing room for host tasks. |
Ada (RTX 40-series) uses aggressive cuEventSynchronize spin-polling (100% core lock), saturating a full logical thread nonstop. |
Requires TID == PID thread affinity isolation on 4-core CPUs so an Ada card's spin loop doesn't starve neighboring GPUs. |
| 5. Silicon Node V/F Curve & Memory Amdahl's Law | Blackwell (TSMC 4NP) at 175–185W (5070) / 150–160W (5060 Ti) retains 95–98% of stock boost clock (2,660–2,820 MHz), easily beating stock-TGP LAR averages on older CPUs. |
Ampere/Ada gain +105..+270 MHz core clock raising 3060 Ti (125➔140W) and 4070 (145➔160W), yielding +3.8% / +1.7% PPD (capped by fixed GDDR6/GDDR6X memory clocks + Gen3 x1). |
62.5%–72.5% TGP maximizes multi-GPU wall PPD/W; 70%–80% TGP trades -0.8%..-4.2% wall PPD/W for +804k extra fleet PPD. |
If you are building or tuning a Linux Folding@Home rig—whether it's a single gaming PC folding overnight or a dense multi-GPU frame—these seven tweaks deliver the biggest bang for the effort.
PPD/W Cap (62%–72% TGP) and High-Throughput Cap (70%–85% TGP)Modern consumer GPUs ship 15–25% past the flat part of their voltage/frequency curve to win gaming benchmarks by 3%. Our controlled 968-WU power-cap experiment showed two distinct operating sweet spots depending on your goal:
145W cards (RTX PRO 4000 Blackwell): Keep at 145W (100%)—135W loses on both PPD and PPD/W.PPD/Watt (62.5%–72.5% consumer TGP): OpenMM molecular dynamics is ~65–75% VRAM-bandwidth bound (Speed ~ SM_Clock^0.25), so once host power is amortized across 3–5 GPUs, 125W (3060 Ti), 145W (4070), and 175W (5070) hit peak kWh efficiency.P_host ~ 120W) or max leaderboard PPD (70%–85% consumer TGP): Stepping up +10–15W (140W 3060 Ti, 160W 4070/16GB 5060 Ti, 185W 5070) buys +1.7% to +5.0% extra QRB PPD (with RTX 5070 delivering 26.6k marginal PPD/dW):| GPU Model | Stock TGP | Peak PPD/W Cap |
Active Throughput Cap | Typical Folding Clock & Temp | Measured True GPU PPD/W |
|---|---|---|---|---|---|
| NVIDIA RTX 4090 | 450 W |
315 W |
315 W (70% TGP) |
~2,550 MHz @ 60–63°C |
104.7k – 115.2k PPD/W |
| NVIDIA RTX PRO 4000 Blackwell | 145 W |
145 W (100%) |
145 W (beats 135W!) |
~2,010–2,150 MHz @ 68–78°C |
88.1k – 92.8k PPD/W |
| NVIDIA GeForce RTX 5070 | 250 W |
175 W (80.1k/W) |
180 W – 185 W (+1.9% PPD) |
~2,745–2,752 MHz @ 60–65°C |
74.3k – 82.9k PPD/W |
| NVIDIA GeForce RTX 5060 Ti | 180 W |
150 W (56.5k/W) |
150 W (8G) / 160 W (16G) |
~2,660–2,820 MHz @ 62–65°C |
54.3k – 59.4k PPD/W |
| NVIDIA GeForce RTX 4070 Ti | 285 W |
195 W |
195 W (Overnight) |
~2,520 MHz @ 72–75°C |
61.1k – 67.9k PPD/W |
| NVIDIA GeForce RTX 4070 | 200 W |
145 W (53.7k/W) |
160 W (+1.7% PPD) |
~2,535–2,550 MHz @ 64–66°C |
49.1k – 53.7k PPD/W |
| NVIDIA GeForce RTX 3070 | 220 W |
145 W (65% fan) |
145 W (52°C @ 65% fan) |
~1,605–1,695 MHz @ 52°C |
28.0k – 30.5k PPD/W |
| NVIDIA GeForce RTX 3060 Ti | 200 W |
125 W (25.0k/W) |
140 W (+3.8% PPD avg) |
~1,635–1,830 MHz @ 56–60°C |
23.1k – 25.7k PPD/W |
OPENMM_CPU_THREADS=4 on Multi-GPU & High-Core HostsIf you run Folding@Home on a 12-core, 16-core, or 32-core CPU—or run multiple GPUs on a machine with limited RAM—pass OPENMM_CPU_THREADS=4 to the F@H client environment (in Docker Compose environment: or your systemd drop-in /etc/systemd/system/fah-client.service.d/override.conf). It stops OpenMM from spawning dozens of redundant host threads per GPU, slashes RAM usage by up to 75%, and reduces CPU cache pollution during step barriers.
FahCore Main Threads (TID == PID) If You Have More GPUs Than Physical CPU CoresIf you are running 4, 5, or 6 GPUs on a quad-core CPU (i7-4790, i7-7700, Ryzen 3/5), check whether two FahCore main threads are fighting over the same logical CPU core:
ps -eLo pid,tid,pcpu,psr,stat,comm | awk '$1==$2 && $6~/FahCore/'Look at column 4 (PSR, the logical CPU number). If two FahCore processes show the same PSR number, they are time-slicing each other at 50% speed!
Crucial Rule: Do not use taskset -cp <cpu> <pid> with -a (all threads), because forcing all 28 threads of a FahCore process onto a single CPU core serializes OpenMM's 4 barrier threads and slows down the GPU. Instead, pin only the main thread (TID == PID) to its own dedicated logical CPU (taskset -cp <cpu> <pid> without -a), leaving the child helper threads free to run across the whole CPU.
PCIe x1 Mining Splitters for Anything Faster Than an RTX 3060 TiOur controlled slot swaps across scraps and smallpieces showed a clear cutoff for PCIe bus bandwidth in Folding@Home v8 (FahCore_27 / FahCore_28):
PCIe Gen 2.0 x1 (~500 MB/s): Bottlenecks every modern GPU (costs -10% to -18% even on an RTX 3060 Ti).PCIe Gen 3.0 x1 (~985 MB/s): Adequate for an RTX 3060 Ti (~94–96% of full speed), noticeably taxes an RTX 3070 (-12.5% PPD vs x16), and heavily starves RTX 4070 / RTX 5060 Ti / RTX 5070 cards (-25% to -40% PPD).PCIe Gen 4.0 x4 (7.9 GB/s) & Gen 5.0 x4 (15.8 GB/s): Zero bottleneck even on RTX 5070 and RTX PRO 4000 Blackwell cards. If your motherboard supports x4/x4/x4/x4 PCIe bifurcation on its primary x16 slot, a passive Gen 4/5 bifurcation riser card (ADT-Link R34A) is vastly superior to x1 mining risers.800 MHz (MSR 0x1a0)Many B250/H110 mining motherboards (including Octominer boards) boot with Intel SpeedStep / Turbo disabled via Model-Specific Register 0x1a0 bit 38, trapping the CPU at 800 MHz (8x multiplier). Check your live core clocks under load:
grep "cpu MHz" /proc/cpuinfoIf your cores are stuck at 800.000 MHz, install msr-tools, load modprobe msr, and inspect sudo rdmsr 0x1a0. If bit 38 (0x4000000000) is set, clear it with sudo wrmsr -a 0x1a0 0x850089 and set a sensible ceiling in /sys/devices/system/cpu/cpu*/cpufreq/scaling_max_freq (such as 2400000 for 4 GPUs or 3000000 for 5 GPUs) paired with the performance governor so cores never dip back to 800 MHz between OpenMM barriers.
libnvidia-ml.so.1 C-Library Calls (and Check PMU Throttle Counters on Ampere)Consumer GPU VBIOS fan curves are tuned for gaming acoustics: they often let a card climb to 75–78°C while barely spinning the fans at 40–49%. In a 24/7 multi-GPU folding rig, those extra 12–15°C cost you 1–2 GPU Boost clock bins, cook neighboring cards, and bake VRAM pads.
On headless Linux servers, operators often assume GPU fan control is impossible without running a dummy Xorg server (nvidia-settings + Coolbits=4), because the nvidia-smi CLI utility only supports reading fan.speed and exposes no CLI flag to set fan PWM. However, on NVIDIA drivers R525+, root-level fan control is exposed directly inside the NVML C shared library (libnvidia-ml.so.1) over /dev/nvidiactl via three C functions—requiring zero X11, display server, or third-party packages:
nvmlDeviceGetNumFans(handle, &num_fans) — enumerates independent physical fan controllers on the GPU (1 on blowers, 2–3 on multi-fan axial cards).nvmlDeviceSetFanSpeed_v2(handle, fan_idx, speed_pct) — switches that fan controller to manual mode and pins PWM to speed_pct (0–100%).nvmlDeviceSetDefaultFanSpeed_v2(handle, fan_idx) — releases manual override back to the card's hardware VBIOS automatic curve when folding pauses.# Minimal headless fan pin / release via Python stdlib ctypes (run as root)
import ctypes
nvml = ctypes.CDLL("libnvidia-ml.so.1")
nvml.nvmlInit_v2()
h = ctypes.c_void_p()
nvml.nvmlDeviceGetHandleByIndex_v2(0, ctypes.byref(h)) # GPU index 0 (or match by UUID)
n = ctypes.c_uint()
nvml.nvmlDeviceGetNumFans(h, ctypes.byref(n))
for f in range(n.value):
nvml.nvmlDeviceSetFanSpeed_v2(h, f, 65) # Pin all fans on GPU 0 to 65%
# nvml.nvmlDeviceSetDefaultFanSpeed_v2(h, f) # Restore VBIOS auto fan curveDiagnostic Bonus (SW Thermal Slowdown Blips vs True PMU Residency): On Ampere GA104 cards (RTX 3070 / 3060 Ti), instantaneous pollers (clocks_throttle_reasons.sw_thermal_slowdown) can fire single-poll (~2s) transient alerts during 150W boost spikes at 58–60°C edge temp even when cumulative hardware PMU residency (nvidia-smi -q ➔ Clocks Event Reasons Counters ➔ SW Thermal Slowdown: 0 us) proves zero microseconds of real throttling. Pinning 65%–90% positional fan floors via gpufanpin v1.4.0 (65% on scraps's RTX 4070 & RTX 3070; 75% on smallpieces's RTX 5070 & bottom 16GB 5060 Ti; 90% on origami's 2x RTX PRO 4000 blowers) keeps core and hotspot deltas cool while releasing silently to VBIOS auto during off-shift pauses.
Taming High-CFM 4-Pin Powered Chassis Hubs & Asymmetric Bank Offsets (gpufanpin v1.4.0 + Alphacool ES Guardian): Desktop BIOS Q-Fan curves assume consumer fans whose floor sits at 25–35% PWM (pwm=89/255), which makes Molex-powered server fan hubs (such as 2x Alphacool ES Guardian 4-Port Powerfan Hubs driving 8x 120mm wind-tunnel fans on smallpieces v3) scream at 2,432 / 2,903 RPM even at idle—and BIOS curves can only track CPU temperature, ignoring GPU load! Even more subtly, intake and exhaust banks facing different static-pressure impedance spin ~550–650 RPM apart at the exact same PWM duty cycle (0% PWM = 847 RPM Intake vs 1,518 RPM Exhaust). By driving /sys/class/hwmon/hwmon*/pwm* (nct6799) in raw 0–255 integer steps with positional [Intake, Exhaust] offsets (idle = [4, 0], load_min = [26, 6], load_max = [102, 71] across 48°C–74°C max GPU temp) and declaratively syncing /etc/sensors.d/gpufanpin-sensors.conf (CHA2_Intake, CHA1_Exhaust, CPU_FAN), gpufanpin v1.4.0 balances steady-state 5-GPU (754W) airflow to 2,061 RPM Intake / 2,173 RPM Exhaust (59–67°C across all 5 Blackwell GPUs) while dropping off-shift idle noise by ~1,400 RPM.
NVIDIA R620+ Driver Note (620.70.06+): Starting in NVIDIA R620, nvidia-smi --query-gpu=temperature.gpu --format=csv,noheader,nounits emits decimal floats (48.36 instead of 48). Never guard GPU temperature strings with .isdigit() in custom scripts—always parse with float()!
65W Eco Mode / PPT 88W)On a dedicated GPU folder with a modern 12-core or 16-core AMD Ryzen CPU (7950X, 9900X, 9950X3D), the CPU only dispatches CUDA kernels and runs brief 4-thread OpenMM step reductions. Enabling AMD 65W Eco Mode (PPT = 88W, TDC = 75A, EDC = 150A) keeps Zen 4/5 cores boosting to 4.7–5.2 GHz on light threads while capping worst-case CPU heat so a compact low-profile copper cooler (Thermalright AXP90-X53 Full Copper) can cool a 5-GPU 45.3M+ PPD node at 65–67°C.