Writing terabytes to disk in Go: Stopping the OS Page Cache from eating all your RAM (FADV_DONTNEED)

Hello everyone! This is the second article about the development of RUSEON-core, a Zero-Copy video streaming server for AI platforms and Edge video infrastructure. In the first article, I talked about the fundamental reason why we decided to create our own server in the first place. I also covered the main problem with most similar solutions — the "thundering herd" — and how we managed to squeeze out 8 Gbps on a single CPU core. By the way, I forgot to mention in that article that besides simple streaming, we also record the streams in fMP4 format. It’s stored locally for N amount of time, and it can fly off to an S3 bucket (depending on how long the clients want to keep the recordings). This article is precisely about a non-obvious (well, at least to me, maybe for someone else it's an everyday thing) problem related to data storage and its specifics across all Operating Systems. So, let's dive in. We rolled out our first release to production (100 cameras), made the clients happy, and started working. About an hour passed, and the alerts started flying. I SSH into the server, open htop, and see there's only 100 MB of free RAM. Uh-oh. I should clarify that the production server had 32 gigs of RAM. The expected behavior was that the CPU is chilling, the network card is chewing through the traffic, RAM usage is around 250-300 MB, and the disks are not heavily loaded. So, when you see numbers like that in htop, you start blaming yourself and your crooked hands that wrote this piece of "garbage". But still, we decided to go to Google, ChatGPT, and the like. Fortunately, the answer was found quickly, and we stopped beating ourselves up. The code was absolutely not the culprit; Linux itself ate the memory. If you've ever written tons of data to a disk, I think you already know what’s going on. There is an "invisible enemy" known as the Page Cache. That was exactly the root of this problem. How does the Page Cache work and what to do with it? The typical solution to the problem is writing a bash script that runs echo 3 > /proc/sys/vm/drop_caches once an hour. And some people just ignore it and let the system kill random processes via the OOM Killer. But we are building a fault-tolerant thing. That doesn't work for us. The task is to explain to the OS kernel that our fMP4 video archive segments are write-only trash for a short amount of time (because if the client doesn't want to keep records for long, the archive gets cleaned up, and if they do, we send the archive to S3 after N time, and still clean up the local copy). So everything should work on the principle of "write and forget". How to tame the Linux kernel via Go (the right way) But there is one cruel nuance here, which I stumbled upon myself and racked my brain over for a long time (and all it took was reading the docs, but here, just like with assembling IKEA furniture: "Why do I need a manual, I know how to do it myself"). The Linux kernel does not remove pages from the cache if they are so-called "dirty" — meaning they haven't been physically written to the disk platter yet. If you just call Fadvise, nothing will happen. First, you need to do a hard Sync(). // File: pkg/storage/localfs/file_linux.go package localfs import ( "os" "golang.org/x/sys/unix" ) type FileWrapper struct { *os.File } func (fw *FileWrapper) DropCache() error { // First, flush dirty pages to disk! if err := fw.File.Sync(); err != nil { return err } // And only then order the kernel to forget them return unix.Fadvise(int(fw.File.Fd()), 0, 0, unix.FADV_DONTNEED) } Architecture: How not to break the build on Windows Therefore, a rather elegant (I invite you to argue this statement in the comments) interface was implemented. In the core of the recorder, there is now a check to see if the file descriptor knows how to drop the cache: // OPTIMIZATION: Saving RAM from Page Cache if dropper, ok := file.(registry.CacheDropper); ok { _ = dropper.DropCache() } And then comes the magic of Go build tags (thank you for those). In the file_linux.go file (with the //go:build linux tag), we call unix.Fadvise. And right next to it lies the file_others.go file (with the //go:build !linux tag), where the DropCache() method just does a regular Sync() and returns. The code is crystal clear, the linters are happy, and the build works everywhere. So, what was the result? By the way, when you are backing up databases, parsing giant logs, or simply writing heavy files — this feature will save you a mountain of headaches. I also don't understand why this flag is barely talked about or written about anywhere. Usually, people only write about how to properly allocate slices, but there is complete silence regarding the fact that at the level of file operations, your OS can reduce all your efforts to zero. In short, in the open-source part of ruseon-core, this logic is now wired deep into the recording engine. The conclusion and advice I want to give is — don't blindly trust the kernel with memory, hoping that the OS is hypothetically "perfect and maximally thought out." Sometimes you have to slap the kernel on the wrist. Otherwise, you will run into similar problems as I did. The source code, as always, is available on GitHub: https://github.com/RUSEGAL/ruseon-core