How to Control Dell PowerEdge Fan Speed from Linux with GPU Temperatures (iDRAC IPMI Fan Control Guide)

I built this in my homelab and run it on a PowerEdge with a passively cooled datacenter GPU. Where something comes from Dell’s or NVIDIA’s documentation rather than my own testing, I say so.

What it is

A Dell PowerEdge server cools itself. Its iDRAC, the baseboard management controller (BMC) built into the motherboard, reads the CPU, inlet, exhaust and other sensors and sets the fan speed on its own. It does that well for the parts Dell sells and validates.

It does it badly for a GPU Dell didn’t ship. The iDRAC can’t read the card’s temperature, so it falls back to a third-party PCIe card cooling response: it assumes the worst and runs the fans hard, often near 100%, all the time. Turn that response off and you get the opposite problem. Datacenter GPUs such as NVIDIA’s Tesla cards have no fans of their own and rely entirely on the server’s airflow, and now nothing is watching them.

The fix in this guide is a small Linux service that closes the loop itself:

1. It reads the GPU’s temperature with nvidia-smi, and the iDRAC’s CPU, inlet and exhaust sensors over IPMI.
2. It runs each reading through a fan curve and takes the highest answer, so whichever part is hottest sets the speed.
3. It puts the iDRAC in manual fan mode and sets that speed with an IPMI raw command.
4. When it can’t read a sensor, it treats that as hot, not cold, and goes to a fixed fail-safe speed.

The result is a server that’s quiet at idle and still ramps hard when the GPU works.

When to use it, and when not to

A good fit:

• A PowerEdge whose iDRAC still accepts manual fan commands over IPMI (see Firmware and version notes). That’s 13th generation and older, plus early 14th-generation firmware.
• A GPU, or another PCIe card, the iDRAC doesn’t recognize, especially a passively cooled one.
• A lab, a home office, or anywhere the noise of a server at full fan speed is the real problem.

Not a good fit:

• iDRAC9 on current firmware. Dell removed manual fan control over IPMI, deliberately (see Firmware and version notes). Use the iDRAC’s own thermal settings instead.
• A production server under a support contract. The fan commands below are Dell OEM IPMI commands that Dell doesn’t document for this use, apart from the PCIe cooling response. Dell’s thermal settings are the supported route.
• A Dell-validated GPU. If the iDRAC already recognizes your card and reads its temperature, its own fan control is the better choice.
• When one of Dell’s documented settings is enough. Sometimes simply turning off the third-party PCIe cooling response leaves the fans quiet and the card cool enough. Measure that first (step 1).

What you need

Requirement	Why
A PowerEdge with an iDRAC that accepts raw fan commands	The whole method depends on it. Test it by hand in step 3 before building anything.
IPMI over LAN enabled on the iDRAC, or an OS with in-band IPMI access (/dev/ipmi0)	ipmitool reaches the iDRAC either over the network or through the local IPMI driver.
A dedicated iDRAC user for the controller	So its credential can be scoped, rotated and audited on its own.
A Linux machine that can run nvidia-smi against the GPU	Wherever the GPU driver lives: the server’s own OS, or a VM the GPU is passed through to.
ipmitool and Python 3 on that machine	The controller below uses only Python’s standard library.
systemd	For the service, its watchdog, and the stop hook that leaves the fans safe.
A second machine, or the server’s host OS, for a watchdog (recommended)	Something has to hand the fans back to Dell if the controller’s machine dies.

Firmware and version notes

Whether this works depends on the server’s generation and its iDRAC firmware. Find both on the iDRAC’s overview page, or with ipmitool … mc info.

PowerEdge generation	iDRAC	Manual fan control over IPMI	Third-party PCIe cooling response
11th (R710 era)	iDRAC6	Works	Not covered by Dell’s article; test it with the query command
12th (R720 era)	iDRAC7	Works	Not covered by Dell’s article; test it with the query command
13th (R730 era)	iDRAC8	Works	Dell-documented raw commands (step 4)
14th (R740 era)	iDRAC9	Works only up to 3.30.30.30; removed from 3.34.34.34; 3.31 and 3.32 unreported	iDRAC9 thermal and PCIe airflow settings
15th and 16th	iDRAC9	No. These servers shipped with firmware past the cut-off	iDRAC9 thermal and PCIe airflow settings
17th	iDRAC10	No. It never had the commands	iDRAC thermal settings

“Works” means owners report it, so still test it by hand (step 3) before building anything. Tower and rack models of a generation share its iDRAC. If your server is in a “No” row, skip to Dell’s thermal settings: a thermal profile, a fan speed offset, a minimum fan speed, and the PCIe airflow settings for the GPU’s slot. They can’t follow the GPU’s temperature, but setting a minimum airflow for that slot is the supported way to keep a passive card cool.

• iDRAC9 removed manual fan control over IPMI. Owners report that the raw 0x30 0x30 fan commands worked up to iDRAC9 firmware 3.30.30.30 and are refused from 3.34.34.34 on, with an Invalid command or Insufficient privilege level error. A Dell support answer quoted on Dell’s own community forum says access is “not going to be allowed” going forward, and points to thermal profiles and fan offsets instead. Newer iDRAC9 firmware also blocks downgrades past certain releases, so treat this as one-way.
• The third-party PCIe cooling response has a documented command for 13th-generation servers. Dell’s knowledge base article lists the raw commands to disable, re-enable and query it on the R230, R330, R430, R530, R630, R730, R730xd, R830 and R930. On iDRAC9, the equivalent lives in the iDRAC’s thermal and PCIe airflow settings.
• IPMI over LAN is off by default on recent iDRAC firmware, and cipher suite 0 (no authentication) has been off by default since iDRAC7 1.37.35, according to Dell. You’ll turn on the first and leave the second off.
• ipmitool’s upstream GitHub repository is archived. Your distribution’s package still works; nothing here needs a newer build.

Limitations

• One speed for every fan. The command sets all fans to the same duty. The iDRAC’s own control can drive zones separately; this can’t.
• The iDRAC can take control back. An iDRAC reset, a firmware update, or another tool can return the fans to automatic mode. The controller below re-asserts manual mode on every cycle, so it heals within one interval.
• Only one thing may write the fans. Two controllers fight, last writer wins, and the fan speed bounces between their answers. That includes “fan control” features in hypervisor plugins and old scripts you forgot about.
• The iDRAC is slow to answer. Each sdr read walks the iDRAC’s sensor list over the network, and other monitoring reading it at the same moment can push a read past its timeout. See Troubleshooting.
• Undocumented commands can change. A future firmware update can remove them, as iDRAC9’s did. Your fail-safe and watchdog are what keep that from becoming a thermal problem.

Decisions to make before you build

Where the controller runs

Run it on the machine that can read the GPU, because nvidia-smi needs the driver. If the GPU is passed through to a VM, that’s the VM, and the controller reaches the iDRAC over the network (IPMI over LAN). If the GPU driver runs on the server’s own OS, the controller can use in-band IPMI instead and you can leave IPMI over LAN off.

Read the GPU on the host side of any container boundary. My first controller ran in a container. After the host reloaded its service manager, the container quietly lost access to the GPU, read it as 0 °C, and held the fans at their lowest setting while the GPU ran hot under load. Its health check only proved the loop was running, so it stayed green for weeks. A native service that calls nvidia-smi directly can’t lose the GPU that way.

Unreadable means hot

Decide now what the fans do when a reading fails, and make it a fixed, generous speed. A failed read must never look like a cold one. The controller below checks every reading is a plausible temperature (above 0 and at most 120 °C) and goes to the fail-safe speed (70% by default) when any check fails. 70% is loud but keeps a loaded server inside its limits; set it higher if your inlet air is warm.

Your curves

Every curve is a list of (temperature, fan %) points with straight lines between them. Get your GPU’s limits from the card itself rather than from a forum post:

nvidia-smi -q -d TEMPERATURE

GPU Slowdown Temp is where the card starts throttling; GPU Shutdown Temp is where it turns itself off. Put your curve’s 100% point a few degrees below the slowdown temperature. The defaults in the script are what I settled on for a passive datacenter card; treat them as a starting point.

Ramp up fast, come down slowly

Raising the fan speed should be immediate. Lowering it should wait until the target has been lower for several cycles in a row, and ignore changes of a few percent. Otherwise the fans hunt up and down with every GPU burst, which is more noticeable than a steady speed.

Who catches the controller’s machine dying

systemd restarts a crashed controller, and its stop hook leaves the fans at the fail-safe speed. Neither helps if the whole machine (or VM) dies, because the iDRAC holds the last speed it was given, which might be 20%. A watchdog on a different machine closes that gap: if the controller’s heartbeat goes stale, it hands the fans back to Dell’s automatic control, which is loud but safe. In my lab the watchdog runs on the server’s host OS and uses in-band IPMI, so it needs no network and no iDRAC password.

Step-by-step build

1. Measure first

Note the fan speed and noise with Dell’s control, then read what the controller will read:

ipmitool -I lanplus -H <idrac> -U <user> -E sdr type temperature
ipmitool -I lanplus -H <idrac> -U <user> -E sdr type fan
nvidia-smi --query-gpu=temperature.gpu --format=csv,noheader,nounits

-E makes ipmitool read the password from the IPMI_PASSWORD environment variable, so it never appears in the process list or your shell history: read -rs IPMI_PASSWORD; export IPMI_PASSWORD.

The temperature output has one line per sensor. The second column is the sensor ID:

Inlet Temp       | 04h | ok  |  7.1 | 24 degrees C
Exhaust Temp     | 01h | ok  |  7.1 | 38 degrees C
Temp             | 0Eh | ok  |  3.1 | 55 degrees C
Temp             | 0Fh | ok  |  3.2 | 52 degrees C

Those four IDs are what my server reports for inlet, exhaust, CPU 1 and CPU 2. The two CPUs share the name Temp, so the script matches on ID, not name. Check yours and edit SENSOR_IDS in the script if they differ.

2. Create a dedicated iDRAC user and enable IPMI over LAN

In the iDRAC web interface, add a user for the controller with a long, unique password, and enable IPMI over LAN (skip that if the controller uses in-band IPMI). Raw OEM commands generally need an Administrator-privilege IPMI user; try a lower privilege first and keep it if step 3 works. Security best practices covers the trade-offs.

3. Test the fan commands by hand

Keep the “return to automatic” command ready before you send anything else:

# Manual mode
ipmitool -I lanplus -H <idrac> -U <user> -E raw 0x30 0x30 0x01 0x00
# All fans to 30% (the last byte is the percentage in hex: 0x1e = 30)
ipmitool -I lanplus -H <idrac> -U <user> -E raw 0x30 0x30 0x02 0xff 0x1e
# Back to Dell automatic control
ipmitool -I lanplus -H <idrac> -U <user> -E raw 0x30 0x30 0x01 0x01

If the fans audibly drop after the second command and come back after the third, your iDRAC supports this. If the first command fails with Invalid command or Insufficient privilege level, check the user’s privilege first, then the firmware notes above. Don’t build the rest until this works.

4. Turn off the third-party PCIe cooling response

Without this, the iDRAC keeps ramping the fans for the card it doesn’t recognize, even in manual mode. On 13th-generation servers, Dell documents these commands:

# Query: ...00 01 00 00 means disabled, ...00 00 00 00 enabled
ipmitool -I lanplus -H <idrac> -U <user> -E raw 0x30 0xce 0x01 0x16 0x05 0x00 0x00 0x00
# Disable the response
ipmitool -I lanplus -H <idrac> -U <user> -E raw 0x30 0xce 0x00 0x16 0x05 0x00 0x00 0x00 0x05 0x00 0x01 0x00 0x00

On iDRAC9, use the iDRAC’s thermal and PCIe airflow settings instead. This setting persists in the iDRAC, so it’s a one-time step. Now is a good moment to re-check step 1: if Dell’s own control is quiet enough with only this change, and the GPU stays well below its slowdown temperature under a long load, you can stop here.

5. Install the controller

Save this as /opt/fan-control/fan_control.py. The same code, with the systemd units, the watchdog and unit tests, is also in the poweredge-gpu-fan-control repository on GitHub.

#!/usr/bin/env python3
"""GPU-aware fan controller for a Dell PowerEdge iDRAC (stdlib only).

Sets one fan duty for the whole chassis from the highest of several
temperature curves. Any sensor it cannot read counts as hot, never cold.
"""
import json, logging, os, re, socket, subprocess, sys, time

log = logging.getLogger("fan-control")
E = os.environ

HOST, USER = E["IDRAC_HOST"], E["IDRAC_USER"]   # password: IPMI_PASSWORD, read by ipmitool -E
INTERVAL = int(E.get("CHECK_INTERVAL", "15"))
FLOOR = int(E.get("FLOOR_PCT", "20"))
FAILSAFE = int(E.get("FAILSAFE_PCT", "70"))
HYST = int(E.get("HYSTERESIS_PP", "3"))
DECAY = int(E.get("DECAY_CYCLES", "4"))
STATUS = E.get("STATUS_PATH", "/run/fan-control/status.json")
NVIDIA_SMI = E.get("NVIDIA_SMI", "nvidia-smi")

# (temperature C, fan %) points, ascending. Tune these to your hardware.
CURVES = {
    "gpu":     [(45, 20), (60, 35), (75, 60), (82, 90), (85, 100)],
    "cpu":     [(50, 20), (65, 40), (75, 70), (80, 100)],
    "inlet":   [(28, 20), (33, 45), (38, 70)],
    "exhaust": [(69, 0), (70, 80), (90, 100)],
}
# Sensor IDs from `ipmitool sdr type temperature` (column 2). Check yours.
SENSOR_IDS = {"04h": "inlet", "01h": "exhaust", "0eh": "cpu1", "0fh": "cpu2"}


def run(*argv, timeout=10):
    """stdout of argv, or None (logged) on any failure."""
    try:
        p = subprocess.run(argv, capture_output=True, text=True, timeout=timeout)
    except Exception as e:
        log.warning("%s: %s", argv[0], e)
        return None
    if p.returncode:
        log.warning("%s: exit %d: %s", argv[0], p.returncode, p.stderr.strip()[-200:])
        return None
    return p.stdout


def ipmi(*args):
    return run("ipmitool", "-I", "lanplus", "-H", HOST, "-U", USER, "-E", *args)


def raw(*hexbytes):
    return ipmi("raw", *hexbytes)


def plausible(t):
    return t if t is not None and 0 < t <= 120 else None


def read_gpu():
    out = run(NVIDIA_SMI, "--query-gpu=temperature.gpu", "--format=csv,noheader,nounits")
    try:
        t = plausible(max(int(x) for x in out.split()))      # hottest GPU
    except (AttributeError, ValueError):
        t = None
    if t is None:
        log.warning("bad GPU temperature read: %r", out)
    return t


def read_ipmi():
    out = ipmi("sdr", "type", "temperature")
    temps = {}
    for line in (out or "").splitlines():
        cols = [c.strip() for c in line.split("|")]
        key = SENSOR_IDS.get(cols[1].lower()) if len(cols) >= 5 else None
        m = re.search(r"(-?\d+) degrees", cols[4]) if key else None
        if key:
            temps[key] = plausible(int(m.group(1))) if m else None
    if len(temps) != len(SENSOR_IDS) or None in temps.values():
        log.warning("bad IPMI temperature read: %s", temps)
        return None
    return temps


def interp(table, x):
    if x <= table[0][0]:
        return table[0][1]
    for (x0, y0), (x1, y1) in zip(table, table[1:]):
        if x <= x1:
            return y0 + (x - x0) / (x1 - x0) * (y1 - y0)
    return table[-1][1]


def target(t):
    want = max(interp(CURVES["gpu"], t["gpu"]),
               interp(CURVES["cpu"], max(t["cpu1"], t["cpu2"])),
               interp(CURVES["inlet"], t["inlet"]),
               interp(CURVES["exhaust"], t["exhaust"]))
    return int(round(max(FLOOR, min(100, want))))


class Smoother:
    """Up at once, down only after DECAY cycles in a row, ignoring small moves."""
    def __init__(self):
        self.cur, self.below = None, 0

    def next(self, want):
        if self.cur is None or want - self.cur >= HYST:
            self.cur, self.below = want, 0
        elif want < self.cur:
            self.below += 1
            if self.below >= DECAY and self.cur - want >= HYST:
                self.cur, self.below = want, 0
        else:
            self.below = 0
        return self.cur


def set_fans(pct):
    raw("0x30", "0x30", "0x01", "0x00")                  # manual mode, every time
    raw("0x30", "0x30", "0x02", "0xff", "0x%02x" % pct)  # all fans to pct


def notify(msg):
    addr = E.get("NOTIFY_SOCKET")
    if addr:
        s = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
        s.sendto(msg.encode(), "\0" + addr[1:] if addr[0] == "@" else addr)
        s.close()


def write_status(payload):
    os.makedirs(os.path.dirname(STATUS), exist_ok=True)
    with open(STATUS + ".tmp", "w") as f:
        json.dump(payload, f)
    os.replace(STATUS + ".tmp", STATUS)


def main():
    logging.basicConfig(level=logging.INFO, format="%(levelname)s %(message)s")
    if "--failsafe" in sys.argv:            # ExecStopPost: leave the fans safe
        set_fans(FAILSAFE)
        log.warning("stopped: fans held at fail-safe %d%%", FAILSAFE)
        return
    smooth = Smoother()
    notify("READY=1")
    while True:
        try:
            gpu, ipmi_t = read_gpu(), read_ipmi()
            if gpu is None or ipmi_t is None:
                pct, mode, temps = FAILSAFE, "failsafe", {"gpu": gpu, **(ipmi_t or {})}
            else:
                temps = {"gpu": gpu, **ipmi_t}
                pct, mode = smooth.next(target(temps)), "curve"
            set_fans(pct)
            log.info("mode=%s pct=%d temps=%s", mode, pct, temps)
        except Exception as e:              # never let the loop die quietly
            log.error("loop error: %s", e)
            pct, mode, temps = FAILSAFE, "error", {}
            set_fans(pct)
        write_status({"ts": int(time.time()), "mode": mode, "pct": pct, "temps": temps})
        notify("WATCHDOG=1")
        time.sleep(INTERVAL)


if __name__ == "__main__":
    main()

A few things in it are deliberate:

• Manual mode is re-sent every cycle, right before the speed. A set-speed command is ignored unless manual mode is active, and an iDRAC reset silently reverts it.
• Every failure is logged with its reason, so a fail-safe line in the journal always has a cause next to it.
• ts in the status file is a heartbeat for the watchdog in step 8.

6. Write the config

Create /etc/fan-control/config.env, readable by root only, because it holds the iDRAC password:

IDRAC_HOST=192.0.2.10
IDRAC_USER=fanctl
IPMI_PASSWORD=change-me
# Optional (defaults shown)
CHECK_INTERVAL=15
FLOOR_PCT=20
FAILSAFE_PCT=70
HYSTERESIS_PP=3
DECAY_CYCLES=4

sudo install -d -m 0700 /etc/fan-control
sudo chmod 0600 /etc/fan-control/config.env

7. Run it as a systemd service

/etc/systemd/system/fan-control.service:

[Unit]
Description=GPU-aware PowerEdge fan controller
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
NotifyAccess=main
EnvironmentFile=/etc/fan-control/config.env
ExecStart=/usr/bin/python3 /opt/fan-control/fan_control.py
# On any stop or crash, leave the fans at the fail-safe speed
ExecStopPost=/usr/bin/python3 /opt/fan-control/fan_control.py --failsafe
Restart=always
RestartSec=10
# Restart it if a cycle hangs for a minute
WatchdogSec=60
RuntimeDirectory=fan-control

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable --now fan-control
journalctl -u fan-control -f

You should see a mode=curve pct=… temps={…} line every 15 seconds.

8. Add the watchdog

The watchdog needs to read the heartbeat from another machine. This tiny server publishes the status file and nothing else. Save it as /opt/fan-control/status_http.py:

"""Serve the fan controller's status file read-only, and nothing else."""
import http.server

STATUS = "/run/fan-control/status.json"


class Handler(http.server.BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path not in ("/", "/status"):
            return self.send_error(404)
        try:
            with open(STATUS, "rb") as f:
                body = f.read()
        except OSError:
            return self.send_error(503)
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)


http.server.ThreadingHTTPServer(("0.0.0.0", 8910), Handler).serve_forever()

Run it as an unprivileged user, sandboxed, and started before the controller so that it stops after it at shutdown. I learned the ordering the hard way: with it the other way round, a slow shutdown left the controller running with no visible heartbeat, and the watchdog and controller took turns setting the fan mode until the machine finally stopped.

/etc/systemd/system/fan-status-http.service:

[Unit]
Description=Read-only HTTP mirror of the fan controller status
After=network-online.target
Before=fan-control.service

[Service]
ExecStart=/usr/bin/python3 /opt/fan-control/status_http.py
Restart=always
DynamicUser=yes
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
PrivateDevices=yes
RestrictAddressFamilies=AF_INET AF_INET6
CapabilityBoundingSet=
MemoryDenyWriteExecute=yes

[Install]
WantedBy=multi-user.target

Then, on the machine that outlives the controller (the server’s host OS in my case), run this from cron every three minutes:

#!/bin/bash
# If the fan controller's heartbeat is missing or stale, hand the fans back
# to Dell's automatic control. Runs from cron on a machine that outlives the
# controller. Here: the server's own host OS, in-band through /dev/ipmi0.
set -u
STATUS_URL=${STATUS_URL:-http://controller.example:8910/status}
STALE_SECS=${STALE_SECS:-120}          # 8 missed 15-second cycles

now=$(date +%s)
ts=$(curl -fsS --max-time 10 "$STATUS_URL" 2>/dev/null \
     | grep -o '"ts": *[0-9]*' | grep -o '[0-9]*$')

if [ -z "$ts" ]; then
  reason="no heartbeat"
elif [ $(( now - ts )) -gt "$STALE_SECS" ]; then
  reason="heartbeat $(( now - ts ))s old"
else
  exit 0
fi

logger -t fan-watchdog "$reason: returning fans to Dell automatic control"
ipmitool raw 0x30 0x30 0x01 0x01 >/dev/null \
  || logger -t fan-watchdog "ERROR: could not return fans to automatic"

Plain ipmitool raw with no -H talks to the local BMC through /dev/ipmi0, which needs the ipmi_devintf kernel module. Some hypervisor distributions ship FreeIPMI rather than ipmitool; the same command there is ipmi-raw 0x00 0x30 0x30 0x01 0x01. If the watchdog runs on a separate machine instead, use -I lanplus -H <idrac> -U <user> -E as before.

The watchdog and the controller don’t fight in normal operation: the watchdog acts only when the heartbeat is gone, and if the controller comes back, its next cycle takes manual control again.

Verify it works

Do these in order. Each one gates the next.

1. The controller follows the curves. cat /run/fan-control/status.json shows "mode": "curve", and the pct matches what your curves give for the temperatures shown. ipmitool … sdr type fan shows the RPMs moving with it.
2. It ramps under load. Run a long GPU job. The fan speed should rise as the GPU warms, and the GPU should level off well below its slowdown temperature. Stop the job: the speed should hold for a few cycles, then step down.
3. A failed read fails safe. Add NVIDIA_SMI=/bin/false to the config and restart. The journal should show bad GPU temperature read and mode=failsafe pct=70. Remove the line and restart.
4. Stopping it fails safe. systemctl stop fan-control. The journal shows stopped: fans held at fail-safe 70%, and the fans settle there. Start it again.
5. It heals a reset. Send the “back to automatic” command from step 3 by hand. Within one cycle the controller should take manual control again.
6. The watchdog fires. Stop both services, wait past the stale threshold plus one cron run, and check the watchdog machine’s log for returning fans to Dell automatic control. The fans should go to Dell’s automatic speed. Start the services again.

Troubleshooting

• Fans stay near 100% whatever you set. The third-party PCIe cooling response is still on. Query it (step 4).
• Invalid command or Insufficient privilege level on raw 0x30 0x30. Check the iDRAC user’s IPMI privilege, then the firmware. On iDRAC9 from 3.34.34.34, this is the removal described above, not a bug.
• Fans ramp while the GPU is idle. Read the journal: the temps show which sensor is driving the speed. In my lab this has been a CPU, more than once: a GPU workload that lost its GPU handle spun on the CPU instead, and the CPU curve was doing exactly its job.
• Occasional fail-safe cycles with a timeout in the log. The iDRAC answers slowly when several tools read it at once. Make the sensor reads cheaper with a local copy of the sensor list: ipmitool … sdr dump /var/cache/fan-control/sdr.cache once, then add -S /var/cache/fan-control/sdr.cache to the sdr reads. Rebuild the cache at start, daily, and after any iDRAC firmware update. In my lab this cut a cycle from about 8.5 seconds to under one.
• A single CPU reads No Reading for one cycle. Some iDRACs do this now and then, and the controller fails safe for that cycle, which is the intended behavior. If it’s frequent enough to annoy you, holding that one CPU’s last good value for a cycle or two is a reasonable refinement; never hold more than one sensor, or for long.
• The fan speed bounces between two values. Two writers. Look for another controller, a hypervisor plugin’s fan feature, or an old script still on a timer.

Rollback

1. sudo systemctl disable --now fan-control fan-status-http, and remove the watchdog’s cron entry.
2. Return the fans to Dell: ipmitool -I lanplus -H <idrac> -U <user> -E raw 0x30 0x30 0x01 0x01. Do this after step 1, or the controller’s stop hook leaves the fans at the fail-safe speed in manual mode.
3. Re-enable the third-party PCIe cooling response if the card is staying: raw 0x30 0xce 0x00 0x16 0x05 0x00 0x00 0x00 0x05 0x00 0x00 0x00 0x00.
4. Delete the controller’s iDRAC user, and turn IPMI over LAN back off if nothing else needs it.
5. Remove /opt/fan-control, /etc/fan-control and both unit files.

Security best practices

Security decisions for any server belong to its administrator, who knows its risks, requirements and constraints. This section lays out the recommended practices for this build and names the guidance behind each one, so you can weigh them against your own environment.

The guidance referenced is the NSA and CISA joint information sheet on hardening baseboard management controllers, NIST SP 800-123 (Guide to General Server Security), NIST SP 800-193 (Platform Firmware Resiliency Guidelines), and Dell’s iDRAC documentation.

A BMC is worth this attention because of what it can do. It runs below the operating system, can power the server off, mount media and reflash firmware, and keeps working when the OS is down. This build asks you to turn on a network protocol to it and store a credential for it.

Limit who can reach the iDRAC

Recommended practice	What it does	Trade-off	Basis
Use in-band IPMI and leave IPMI over LAN off, where the controller runs on the server’s own OS	No new network service on the BMC at all	Not possible when the GPU is in a VM; the controller then needs the network	NSA/CISA BMC hardening: reduce exposed services
Put the iDRAC on a dedicated management network that only management hosts can reach	An attacker on the general network can’t talk to the BMC	Needs a separate VLAN or port, and firewall rules for the controller’s machine	NSA/CISA BMC hardening: segment BMC traffic. NIST SP 800-123: network protection for servers
Allow only the controller’s address with the iDRAC’s IP blocking/filtering, or a firewall in front of it	Narrows who can even attempt a login	Another rule to update if the controller moves	NSA/CISA BMC hardening
Never expose the iDRAC to the internet	Removes the largest exposure	None worth taking	NSA/CISA BMC hardening

Treat the IPMI credential as a privileged secret

IPMI 2.0’s login design (RAKP) lets anyone who can reach the service request a salted hash of a valid user’s password and crack it offline (CVE-2013-4786). It’s a protocol issue, not a bug a firmware update fixes, so the defenses are the network limits above and a password that can’t be cracked.

Recommended practice	What it does	Trade-off	Basis
A dedicated user with a long, random password, not the default root account	A cracked or leaked credential exposes one account you can disable, not the server’s admin login	One more account to manage	NSA/CISA BMC hardening: unique credentials. CVE-2013-4786
The lowest privilege that works	Limits what a stolen credential can do	Raw fan commands may require Administrator; test before settling	NIST SP 800-123: least privilege
Keep cipher suite 0 disabled	Suite 0 accepts logins with no authentication at all	None	Dell knowledge base: IPMI cipher 0
Keep the password off the command line (-E, root-only config.env)	It never appears in ps, shell history or logs	None	NIST SP 800-123: protect authentication data
Rotate it, and on any suspicion of exposure	Limits the life of a cracked hash	Edit the config and restart the service	NSA/CISA BMC hardening

Keep the BMC’s firmware current, and know the trade-off

Recommended practice	What it does	Trade-off	Basis
Update iDRAC firmware, and subscribe to Dell’s security advisories	Fixes BMC vulnerabilities, which are real and periodically severe	On iDRAC9, an update past the cut-off removes manual fan control, and downgrades can be blocked. Decide which matters more before you update, and plan to fall back to Dell’s thermal settings	NIST SP 800-193: firmware protection and recovery. NSA/CISA BMC hardening
Prefer current, in-support hardware for new builds	Keeps getting firmware fixes	Newer iDRACs don’t allow this method, so use their thermal settings	NIST SP 800-193

That last tension is the honest one. On hardware where this method still works, the iDRAC’s network exposure is what you control, so lean hard on the network limits above.

Keep the controller itself small

Recommended practice	What it does	Trade-off	Basis
The status endpoint serves one file, read-only, as a sandboxed user with no capabilities	A bug in it can’t become more than a read of a JSON file	None	NIST SP 800-123: least functionality
Bind or firewall the status port to the watchdog’s address	Nothing else learns your temperatures or when the controller is down	A rule to keep in sync	NIST SP 800-123
Watchdog in-band, with no stored credential, where it runs on the server’s own OS	The watchdog holds no secret to steal	Needs local IPMI access on that OS	Least privilege
Alert on fail-safe and on a stale heartbeat	You learn about a failing sensor or controller before the noise tells you	Needs a monitoring system to send it to	NIST SP 800-123: logging and monitoring

References

This guide’s code

• poweredge-gpu-fan-control on GitHub: the controller, systemd units, status server, watchdog and unit tests from this guide, MIT licensed

Dell documentation and knowledge base

• How to disable the third-party PCIe card default cooling response on PowerEdge 13G servers: the disable, enable and query commands, and the supported models
• How to check if IPMI cipher 0 is off: cipher 0 and the IPMI over LAN default
• iDRAC9 User’s Guide: see Modifying thermal settings and Modifying PCIe airflow settings, the supported route on iDRAC9

Firmware and compatibility

• Dell community: fan speed control removed in iDRAC9 3.34.34.34 and later: owner reports, a quoted Dell support answer, and Dell moderators pointing to thermal profiles and fan offsets

NVIDIA

• nvidia-smi documentation: the temperature fields, including GPU Slowdown Temp and GPU Shutdown Temp. nvidia-smi --help-query-gpu lists the query fields, including temperature.gpu.

Linux and systemd

• systemd.service: Type=notify, WatchdogSec=, ExecStopPost= and Restart=

Security guidance and advisories

• CISA and NSA: Hardening Baseboard Management Controllers: credentials, firmware and network segmentation for BMCs
• CVE-2013-4786: the IPMI 2.0 RAKP password hash disclosure
• NIST SP 800-123, Guide to General Server Security
• NIST SP 800-193, Platform Firmware Resiliency Guidelines