QuillNEW

The ghost in the write

by Oren Eini

Every now and then, you run into the sort of bugs that are flat-out impossible to figure out. Recently one of our RavenDB servers started to throw impossible errors. Now, I know what you are going to say, you’ve all seen “impossible” bugs that were very clear in retrospect.

In this case, this is a piece of code that worked tirelessly for over a decade, so it has both tenure and length of service to show that there is no obvious bad stuff there. The relevant failure was in writing RavenDB’s journal files, to give you some context.

That is the sort of thing that has gone through the wringer. That is something that we went over, tested, verified, etc. And suddenly, within the span of a few weeks, we had several separate incidents where journal writing & recovery failed.

We investigated, because that is a Bad Thing. The Emergency Council for Horrible Bugs was convened. The council has over 50 years of cumulative experience in RavenDB. It issued a ruling: that Was Not Supposed To Happen.

Behind the scenes, there were three guys scratching their heads about this. Because it really shouldn’t be possible. The only lead we had was that the errors all happened on the same instance, and that was an ARM machine. RavenDB has been running on ARM for a long time, that isn’t something new.

We scanned the code, we consulted the auguries (Claude), but nothing made sense. Then we tried to reduce the problem to its essential components. Here is what we came up with. A short Python script that can reproduce the problem:


import ctypes, mmap, os, signal


MB = 1024 * 1024


def region(i):
    fd = os.memfd_create(f"region{i}")
    os.ftruncate(fd, 4 * MB)
    m = mmap.mmap(fd, 4 * MB)  # MAP_SHARED
    # 2 MB aligned
    off = -ctypes.addressof(ctypes.c_char.from_buffer(m)) % (2*MB) 
    return fd, m, off




for _ in range(os.cpu_count() - 1):  # fork process per core
    if os.fork() == 0:
        break


regions = [region(i) for i in range(16)]
it = 0
while True:
    it += 1
    for i, (fd, m, off) in enumerate(regions):
        m[off:off + 16] = b"%2d %13d" % (i, it)  # write via mmap
    for i, (fd, m, off) in enumerate(regions):
        found = os.pread(fd, 16, off)  # read via file I/O
        if found != b"%2d %13d" % (i, it):
            print(
    f"pid {os.getpid()}: "
    f"wrote {b'%2d %13d' % (i, it)}, "
    f"but the file holds {found}",
    flush=True,
            )


            os.killpg(0, signal.SIGTERM)  # stop all


    for fd, m, off in regions: # done with the buffer, release the memory
        m.madvise(mmap.MADV_DONTNEED, off, 2*MB)

If you run this code on a 2-core ARM instance, such as AWS’ c7g.large, you’ll eventually hit a failure.

The code here isn’t complex. We spawn a (separate) process per core, and in each one we create 16 (separate) files and write to them using mmap and then read them via file I/O. Linux explicitly ensures that this is valid, and that both the file I/O and mmap writes will use the same physical memory in the page cache.

Note that after each loop, we discard the memory to free the buffers. Linux doesn’t guarantee what we’ll see after we call this method, but since we read only our own writes, that is fine.

There is no concurrent access to the same data here. We use multiple processes, but they all run on their own data, there are no threads, etc.

You can look at the code above for as long as you like. There is nothing there to see. The problem lies a lot lower than this, in the kernel.

Our original problem, with messed-up journal writes, started when the ARM instance we were running was upgraded to kernel version 7.0.0-1014-aws. That version is affected by CVE-2026-74674, which we encountered on our ARM instance.

The underlying issue is that the kernel messes up page mapping, leading you to write to someone else’s memory. Yes, that includes some other random process, in this case. That is why this is a CVE, and not “just” a bug.

In our case, we can close the chapter on this impossible issue with a pat on the back. Yes, that was Not Our Fault. We’ll reconvene tomorrow for the next bug (which will probably be our fault).

I do want to point out something really interesting here. It took us a very short amount of time to figure out what the problem was when we started to actually look at this. That is solely because of the use of AI. We have experience with handling similar bugs in the past, and the typical time to resolve them was in the order of many weeks. This one was solved within a much shorter time frame, and most of the time was just figuring out where the problem was, so we could look into it.

Woah, already finished? 🤯

If you found the article interesting, don’t miss a chance to try our database solution – totally for free!

Try nowtry now arrow icon