Slow kernel memory leak (6.14 -> 6.18 regression)

[hoshinolina]

I'm observing some strange kernel behavior since updating to F43. Memory usage steadily increases, until the system starts OOMing and other badness and I need to reboot. One time I caught this, I tried shutting down essentially ~all services and daemons, and yet a significant fraction of memory was still not being released despite no visible process (or tmpfs mount that I could see) using significant RAM. Unless I missed something hiding in plain sight, this suggests there's a kernel side memory leak.

This is a j474 server, so no UI or significant GPU usage (the GPU device is not open normally, and I don't think anything in particular would try to open it transiently though I'm not entirely sure).

Note that the machine had been stable for a long time, there is a clear change in behavior on 3/12 when I updated to F43. On 3/27 I added swap (zswap, replacing zram) but that didn't really help.

Kernel versions:

• Prior to the regression: 6.14.8-400.asahi.fc42
• After the regression: 6.18.15-400.asahi.fc43
• As of last night: 6.19.13-400.asahi.fc43

I will report back whether the new upgrade fixed anything, though it'll take a couple weeks to become evident in the graphs.

Poking around Node Exporter stats, these are the most relevant ones:

So Active + Inactive memory (which is basically userspace memory) remains relatively steady throughout. However, there is a very clear change in behavior after the regression date. Prior, MemAvailable closely tracked Cached. After, the kernel claims the leaked RAM as MemAvailable, but it is not cached (and it is definitely not available, I OOMed the system while testing this with ~all userspace shut down and a high number in MemAvailable). The leaked RAM is also reflected in Committed_AS.

For reference, the math used for "memory free" in the first screenshot is MemTotal - MemFree - (Cached + Buffers + SReclaimable). This is essentially a manual computation similar to MemAvailable, but it differs from what the kernel reports.

This machine is running a bunch of stuff, perhaps notably on the kernel side a couple CephFS mounts, as well as a number of daemons, services, and periodic timers, so it's hard to pin down what could be the trigger here.

[hoshinolina]

I'm not seeing the same issue on two other machines with the same kernel version. Those two have a similar hardware/driver workload, so I think that rules out things like USB mass storage or ethernet. That leaves CephFS on the kernel side, or something triggered by a specific userspace app as the likely root causes.

[hoshinolina]

This one very clearly shows the problem:

That's MemAvailable - (Cached + Buffers + SReclaimable + MemFree). No other machine I'm monitoring has this upwards trend. It's also interesting that sometimes it has a discrete jump up...

[axboe]

I ran into this, and I believe it was CONFIG_PERSISTENT_HUGE_ZERO_FOLIO which was the culprit. Turn that off in the kernel config.

[hoshinolina]

Alas,

# grep CONFIG_PERSISTENT_HUGE_ZERO_FOLIO /boot/config-6.1*
/boot/config-6.18.15-400.asahi.fc42.aarch64+16k:# CONFIG_PERSISTENT_HUGE_ZERO_FOLIO is not set
/boot/config-6.18.15-400.asahi.fc43.aarch64+16k:# CONFIG_PERSISTENT_HUGE_ZERO_FOLIO is not set
/boot/config-6.19.13-400.asahi.fc43.aarch64+16k:# CONFIG_PERSISTENT_HUGE_ZERO_FOLIO is not set

Looking at the discrete 8GB jump yesterday (which happened within 15 seconds, from one sample to the next), the only thing in the logs lining up with it is a bunch of complaining from sudo and selinux about something that was misconfigured, but that happens daily and on other machines so it's unlikely to be the culprit. A similar 8GB jump on 04/01 doesn't line up with anything in the logs, really. 06:05:00 to 06:05:15 8GB of memory was leaked, and all I've got is:

Apr 01 06:05:00 flamingo ceph-mgr[1638]: ::ffff:192.168.3.41 - - [01/Apr/2026:06:05:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/3.10.0"
Apr 01 06:05:10 flamingo pushgateway[1499]: ts=2026-03-31T21:05:10.765Z caller=diskmetricstore.go:221 level=info msg="metrics persisted" file=/var/lib/pushgateway/persist
Apr 01 06:05:15 flamingo ceph-mgr[1638]: ::ffff:192.168.3.41 - - [01/Apr/2026:06:05:15] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/3.10.0"

The most recent jump looks like this:

Edit: Simultaneously with the above, the ~same amount of memory moved from Inactive(file) to Active(file) (however, memory always seems to be bouncing between active and inactive anyway).

Now it's kind of interesting that both of these happened on the first of the month. I don't really have anything running on that schedule on the machine itself, but I do have daily backups from a bunch of sources involved, and some of those would do extra things on the 1st (delete old backups, make a full backup, etc).

The 4/1 06:05 leak happened after a Windows machine had been writing a ~terabyte of data from 3:30 to 5:40 (samba to kcephfs) and some deletions between 5:57 and 6:02. The same thing happened yesterday on 5/1, with a large backup happening from 3:30 to 6:20 and then some deletions later. I'm wondering if this is a case of a bunch of kcephfs I/O putting things into the page cache, and then somehow that cache is leaked once it ages out/the caps are released, instead of being properly freed?

The only thing unique about those Windows backups vs. everything else, is that they write one massive file (1.2TB single file).

[hoshinolina]

I think it's this: https://lore.kernel.org/all/20251110182008.71e0858b@xps15mal/

[hoshinolina]

Yeah, I can easily repro with

while true; do head -c 50000000000 /dev/zero | pv > test.file; echo 3 > /proc/sys/vm/drop_caches; done

on the CephFS mount. The leak happens on the drop_caches:

Looks like there's a patch, folio leak (cephfs was recently converted to folios it seems): https://lore.kernel.org/all/b4ae204ba4d5191cc9641180ad1193b9463a6beb.camel@ibm.com/

See also: https://tracker.ceph.com/issues/74156

[hoshinolina]

I managed to get just ceph.ko rebuilt with the patch (lazy) and it seems happy now. Feel free to close, I guess upstream will deal with this one in due time.