Why my NFS shares went stale every night

For a while, every NFS client in my homelab went stale on a schedule. Media apps would suddenly see empty libraries, and df still reported the full, correct capacity for mounts where every read returned ESTALE. It looked like data loss, not an outage, which is the worst kind of failure.

The symptom

dmesg on the clients said:

NFS: server <nas> error: fileid changed

It happened right after midnight, on every client at once.

The cause

The storage server runs Unraid, which has a nightly mover: files land on a fast SSD cache first, and the mover later relocates them to the main array. Through Unraid’s user-share layer the path stays the same, but a relocated file gets a new inode.

NFSv3 file handles are derived from the inode. Change the inode and every client’s handle for that file is dead, all at the same moment the mover runs.

What didn’t fix it

The obvious guess was “use NFSv4”. I tested it: a vers=4.2 mount went stale at the same minute as the v3 mounts. fileid changed means the server broke file identity, and no NFS version tolerates that. The protocol version was never the problem.

What did

I moved all shared storage to SMB/CIFS, which resolves files by path rather than by an inode-derived handle. A deliberately triggered mover run changed a file’s inode while the CIFS mount kept working and an NFS control mount went stale beside it.

There were two follow-on lessons:

1. CIFS caches server inode numbers too, by default (serverino). That brings back the same identity problem one layer up. The fix is noserverino on every mount. The usual objection is that it breaks hardlinks. I tested it: links still get created and counted correctly. What you lose is inode equality (find -samefile and dedup tools), which nothing in my stack relied on.
2. Permissions moved from the server to the client. NFS with all_squash had quietly been answering “yes” to everyone. CIFS checks the process’s uid/gid against the mount’s uid=/gid= options on the client. Every consumer needed the right ids, and one unprivileged container needed noperm, because its uid map can’t represent the share owner at all.

The takeaway

Test a migration by doing a real write as the real service user on every consumer, not by checking that the mount shows up. The three permission problems above each went unnoticed for days because the mounts looked fine.