RandomAccess.FlushToDisk

[ericmutta]

Following the handy guidance by @adamsitnik on how to do fast file I/O on .NET 6, I decided to take advantage of the RandomAccess class in .NET 6 and so far it has all the methods I need except one: the equivalent for the FileStream.Flush(bool) function.

Basically, I am working on a database-like project where multiple threads can write to random 4KB pages in the same file (i.e. exactly the kind of thing RandomAccess was designed to handle). At some point I need to make a call to fsync() (on Linux) or FlushFileBuffers() (on Windows). If I had a FileStream on hand I could just call myFileStream.Flush(true) and that would flush changes to disk using fsync in an OS-independent way. However, I do NOT have a FileStream object, only a SafeFileHandle because I use the new File.OpenHandle() function to open my files.

This leads to two questions:

1. Is there a technical reason that the RandomAccess class does not have a Flush(bool) function that does an fsync under the covers?
2. What can I do now if all I have is a SafeFileHandle and I want to fsync any writes made via that handle?

I know there are several FileStream constructors that take a SafeFileHandle parameter, but creating a FileStream object only so I can call Flush(bool) doesn't feel quite right and I suspect I am missing something!

[ghost]

Tagging subscribers to this area: @dotnet/area-system-io
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Following the handy guidance by @adamsitnik on how to do fast file I/O on .NET 6, I decided to take advantage of the RandomAccess class in .NET 6 and so far it has all the methods I need except one: the equivalent for the FileStream.Flush(bool) function.

Basically, I am working on a database-like project where multiple threads can write to random 4KB pages in the same file (i.e. exactly the kind of thing RandomAccess was designed to handle). At some point I need to make a call to fsync() (on Linux) or FlushFileBuffers() (on Windows). If I had a FileStream on hand I could just call myFileStream.Flush(true) and that would flush changes to disk using fsync in an OS-independent way. However, I do NOT have a FileStream object, only a SafeFileHandle because I use the new File.OpenHandle() function to open my files.

This leads to two questions:

1. Is there a technical reason that the RandomAccess class does not have a Flush(bool) function that does an fsync under the covers?
2. What can I do now if all I have is a SafeFileHandle and I want to fsync any writes made via that handle?

I know there are several FileStream constructors that take a SafeFileHandle parameter, but creating a FileStream object only so I can call Flush(bool) doesn't feel quite right and I suspect I am missing something!

Author:	ericmutta
Assignees:	-
Labels:	area-System.IO, untriaged
Milestone:	-

[danmoseley]

I think this might be an oversight. @adamsitnik ?

[ericmutta]

Thanks @danmoseley for following up! 👍

While waiting for @adamsitnik to provide some more insight, I was checking out the docs.

For Windows, I found this sentence about the flags we can pass to the CreateFile function:

...the FILE_FLAG_WRITE_THROUGH flag is often used with the FILE_FLAG_NO_BUFFERING flag as a replacement for calling the FlushFileBuffers function after each write,

For Linux, I found this sentence about the O_SYNC flag by running man "open(2)" :

By the time write(2) (or similar) returns, the output data and associated file metadata have been transferred to the underlying hardware (i.e., as though each write(2) was followed by a call to fsync(2)).

The Windows FILE_FLAG_WRITE_THROUGH flag and the Linux O_SYNC flag is set when you specify the FileOptions.WriteThrough option (e.g. see this code for Linux) when calling File.OpenHandle(). Effectively, this means one can avoid the need for doing the fsync operation entirely if the file is opened with the FileOptions.WriteThrough option. Perhaps this is the reason there's no RandomAccess.Flush(bool) function?

In any case, I think adding RandomAccess.Flush(bool) may be useful to "round out" the RandomAccess API surface, especially for users who don't want to implement their own caching/buffering (which is required to avoid hitting the disk with each write when you specify the FileOptions.WriteThrough option). In the scenario that prompted me to create this issue, I am actually very glad I can avoid the need to do fsync entirely by having all my writes go directly to disk so I don't have to deal with the complexities of handling fsync errors.

[adamsitnik]

Effectively, this means one can avoid the need for doing the fsync operation entirely if the file is opened with the FileOptions.WriteThrough option

FileOptions.WriteThrough performs flush to disk after every write and can cause major perf loss for those who perform multiple writes, but want to flush to disk just once.

As @danmoseley wrote, it's an oversight ;)

Based on your feedback I believe that we should add following method:

namespace System.IO;

public static class RandomAccess
{
+    public static void FlushToDisk(SafeFileHandle handle);
}

For now you can use this code as a workaround, but please keep in mind that it won't work on WASM and for non-regular files like Pipes & Sockets.

[adamsitnik]

Background and motivation

FileStream exposes a way to flush the data to disk by calling Flush(flushToDisk: true), RandomAccess does not.

We don't need Flush(flushToDisk: false), as there is nothing to flush because RandomAccess does not perform any buffering like FileStream does.

API Proposal

namespace System.IO;

public static class RandomAccess
{
+    public static void FlushToDisk(SafeFileHandle handle);
}

API Usage

using SafeFileHandle sfh = File.OpenHandle("test.txt", FileMode.CreateNew, FileAccess.Write, FileShare.Read);

RandomAccess.Write(sfh, new byte[1024], fileOffset: 0);
RandomAccess.FlushToDisk(sfh);

Alternative Designs

None of the supported OSes exposes a way to perform an async flush to disk, but anyway we could include an async overload too.

Risks

I can't see any.

[danmoseley]

I wonder whether Flush() would be a better name. It's consistent with FileStream (albeit with true). FlushToDisk() makes me stop and think what other kind of flush there is here (is there a buffer?).

[ericmutta]

Thanks @adamsitnik for following up and for the workaround code 👍

As far as naming goes, using Flush() would keep things consistent with FileStream as @danmoseley mentions, though I like the longer FlushToDisk() name.

When someone wants to flush, they type RandomAccess, then a dot, then Flu... at that point the editor will show the function we want regardless of whether it is called Flush or FlushToDisk because all names have the same Flush prefix. I like FlushToDisk because it is unambiguous: you are going to hit the disk when you call this function. With just Flush, I would have to read the docs to understand what this means (especially since in FileStream it has two meanings: flushing internal buffers into the kernel when you call the overload without a bool parameter, or flushing kernel buffers onto the disk when you call the overload with the bool parameter set to true).

So I vote FlushToDisk but if Flush wins, nobody will die, so that's fine too :-)

Incidentally, when writing the docs for this function it may be worth advising users to check for errors (especially if they are following the "do many writes then issue one fsync at the end" approach). On Linux, according to this issue encountered by the Postgres guys, if fsync fails due to write-back errors, the kernel marks the dirty pages clean allowing them to be evicted and causing (potentially silent) data loss 😨

PS: thanks to both of you and the rest of the team on the amazing work you are doing with .NET! The RandomAccess class is a key enabling technology for the work I am doing and has made my life so much easier (oh, and MemoryMarshal.Cast is just pure awesomeness too if you are implementing a database-like project in C#) 🙏

[danmoseley]

Incidentally, when writing the docs for this function it may be worth advising users to check for errors

presumably it would throw IOException in such a case, just like FileStream.Flush() does.

[adamsitnik]

The RandomAccess class is a key enabling technology for the work I am doing and has made my life so much easier

Thank you for your kind words!

in FileStream it has two meanings: flushing internal buffers into the kernel when you call the overload without a bool parameter

My motivation was to avoid this confusion. I am sure that the API Review Board is going to choose the best name.

presumably it would throw IOException in such a case, just like FileStream.Flush() does.

We are definitely going to throw on error

[MichalPetryka]

(oh, and MemoryMarshal.Cast is just pure awesomeness too if you are implementing a database-like project in C#)

Remember that you need to ensure sufficient buffer alignment (and proper endianness and padding) when using Cast. When reading from a file, I'd recommend using a correctly typed array and AsBytes instead.

[ericmutta]

@MichalPetryka Remember that you need to ensure sufficient buffer alignment (and proper endianness and padding) when using Cast. When reading from a file, I'd recommend using a correctly typed array and AsBytes instead.

I am doing something like this:

    // allocate 4KB page buffer.
    var buffer = new byte[4096];

    // read page from disk into buffer using RandomAccess.ReadAsync().
    ReadPageFromDisk(buffer);

    // access the buffer as if it was an array of 64-bit elements.
    var elements = MemoryMarshal.Cast<byte, ulong>(buffer);
    elements[0] = ulong.MinValue;
    elements[1] = ulong.MaxValue;

Presumably the CLR takes care of aligning the buffer allocated via new byte[4096] so that when I later pass it to MemoryMarshal.Cast, the Span<ulong> I get back will find the ulong values properly aligned too...correct?

Speaking of buffer alignment issues while we are on the topic of enhancing the RandomAccess class, I just discovered NativeMemory.AlignedAlloc yesterday, which looks like it could be handy if .NET provides platform independent support for direct/unbuffered I/O ( which uses the Windows FILE_FLAG_NO_BUFFERING flag and the Linux O_DIRECT flag). This would have been handy in the work I am doing right now, but I have opted to stay away because thinking about memory alignment issues for a database-like library that can be used on everything from a Raspberry Pi to an IBM S390x mainframe makes my head hurt 😄

[adamsitnik]

Linux O_DIRECT flag

I took a stab at it in the past: #55463

You can leave your feedback here: #27408

[MichalPetryka]

Presumably the CLR takes care of aligning the buffer allocated via new byte[4096] so that when I later pass it to MemoryMarshal.Cast, the Span<ulong> I get back will find the ulong values properly aligned too...correct?

No, if you want to have your code be portable and work on platforms like S390X you'd need to do:

    // allocate 4KB page buffer.
    var elements = new ulong[4096 / sizeof(ulong)];

    // read page from disk into buffer using RandomAccess.ReadAsync().
    ReadPageFromDisk(MemoryMarshal.AsBytes(buffer));

    if (!BitConverter.IsLittleEndian)
        for (int i = 0; i < elements.Length; i++)
            elements[i] = BinaryPrimitives.ReverseEndianness(elements[i]);

    // access the buffer as if it was an array of 64-bit elements.
    elements[0] = ulong.MinValue;
    elements[1] = ulong.MaxValue;

With custom structs you'd also probably need to use Explicit layouts to maintain same layouts for all platforms.

[ericmutta]

@MichalPetryka: No, if you want to have your code be portable and work on platforms like S390X you'd need to do ...

Many thanks for the tip, I will research this some more to make sure I am considering all the issues 👍

@adamsitnik: You can leave your feedback here ...

Awesome, I have been toying around with some ideas today, will share them there 👍

[danmoseley]

@adamsitnik Did you intend to add a label to send this to API review?

[adamsitnik]

Did you intend to add a label to send this to API review?

I thought I already did ;) Thanks!

[terrajobst]

Video

• Looks good as proposed

namespace System.IO;

public static class RandomAccess
{
    public static void FlushToDisk(SafeFileHandle handle);
}

[danmoseley]

@ericmutta now this is approved, any interest in adding it ?

[ericmutta]

@danmoseley yes, I am interested in this one too because it relates directly to that database-like project I mentioned earlier.

But before I ask for the issue to be assigned to me, I just need to sort out a couple of other tasks on my plate over the next week or two, then I will be back 👍

PS: I believe the plan is to ship this with .NET 8 in November this year. Is there a cut-off date for implementation of features planned for .NET 8? I would like to plan accordingly so that if I take this on, I can do it well and wrap up everything in time for .NET 8.

[danmoseley]

Not announced (@jeffschwMSFT @jeffhandley do you plan to put up the usual sticky issue with the shutdown schedule?). My guess is for community contributions like this, merged no later than mid August - possibly earlier. Allow plenty of time for the PR process. But it's not a days away thing yet.

[ericmutta]

@danmoseley I am beginning to get the hang of this (e.g. I have finally managed to open the relevant code in Visual Studio and have the full editing experience). Please assign the task to me since I have already started working on it (I know I said I would be back in a week or two but I couldn't resist 😁)

[adamsitnik]

@ericmutta I've assigned you!

Here are some tips that you may find useful:

• here you can find our workflow instructions, the first step is to install everything required to build the product
• to build the clr, libs and tests you need to build following subsets:

.\build.cmd -c Release -subset clr+libs+libs.tests

• the RandomAccess type is included in corelib, not in System.IO.dll: https://github.com/dotnet/runtime/blob/main/src/libraries/System.Private.CoreLib/src/System/IO/RandomAccess.cs
• its tests are in System.IO.FileSystem: https://github.com/dotnet/runtime/tree/main/src/libraries/System.IO.FileSystem/tests/RandomAccess
• the test projects can access only the public classes and methods that were added to the ref assembly: https://github.com/dotnet/runtime/blob/c88b3776d0fb3bb33fcd4192b476147c0895bd6f/src/libraries/System.Runtime/ref/System.Runtime.cs#L10014
• To rebuild only the corelib (where RandomAccess is) and run the tests you can run following command:

.\build.cmd -c Release -subset clr.corelib+clr.nativecorelib+libs.PreTest && .\dotnet.cmd build -c Release .\src\libraries\System.IO.FileSystem\System.IO.FileSystem.sln && .\dotnet.cmd build -c Release .\src\libraries\System.IO.FileSystem\tests\System.IO.FileSystem.Tests.csproj /t:Test

Good luck!

[ericmutta]

Many thanks @adamsitnik, the tips you gave were a huuuge time-saver 🚀

For the implementation I am thinking of simply calling FileStreamHelpers.FlushToDisk(handle) which automatically does the right thing for Windows and for Unix. This is the same helper function used by FileStream.Flush(bool) and I think it makes sense to share the logic...please let me know if there's a better or more preferred way 👍

My prototype looks like this:

public static void FlushToDisk(SafeFileHandle handle)
{
   ValidateInput(handle, fileOffset: 0);

   FileStreamHelpers.FlushToDisk(handle);
}

I have also managed to setup a dedicated test class with a failing test, which looks like this:

using Microsoft.Win32.SafeHandles;
using Xunit;

namespace System.IO.Tests
{
    public class RandomAccess_FlushToDisk : RandomAccess_Base<long>
    {
        protected override long MethodUnderTest(SafeFileHandle handle, byte[] bytes, long fileOffset)
        {
            RandomAccess.FlushToDisk(handle);
            return 0;
        }

        protected override bool UsesOffsets => false;

        [Fact]
        public void FailingTest()
        {
            Assert.True(false, "This test is failing because it is not implemented yet.");
        }
    }
}

Everything builds and runs as expected so far, so assuming I am heading in the right direction, I am ready to start the official implementation (tests, XML docs, etc) 👍

[adamsitnik]

I am thinking of simply calling FileStreamHelpers.FlushToDisk(handle)

Sounds great to me!

Regarding the tests and the implementation the only open question I have is what should be the behavior for SafeFileHandle instances that represent non-seekable files like pipes or sockets. IIRC we don't have tests for that for FileStream.Flush(true), but I can see in the fsync man page that it reports an error for pipes and sockets:

EROFS, EINVAL
fd is bound to a special file (e.g., a pipe, FIFO, or
socket) which does not support synchronization.

On Windows, FlushFileBuffers does the following:

If hFile is a handle to the server end of a named pipe, the function does not return until the client has read all buffered data from the pipe.

The simplest solution would be to unify the behavior and just throw for non-seekable files before calling the sys-call. But by doing that, we would loose the extra capability provided by Windows.

@stephentoub what is your opinion on this?

[ericmutta]

The simplest solution would be to unify the behavior and just throw for non-seekable files before calling the sys-call. But by doing that, we would loose the extra capability provided by Windows.

It looks like the current Unix implementation already does this by ignoring errors returned by fsync() for non-seekable files. I believe this means that one can flush non-seekable files on both platforms and Windows would provide its extra capability while for Unix, the operation is effectively a no-op.

[adamsitnik]

@ericmutta let's keep the current behavior then 👍

[ericmutta]

Thanks @adamsitnik 👍

One more thing...how can we really test that FlushToDisk() is doing its thing? The typical flow for a test would be:

1. Open a file and write some data.
2. Call FlushToDisk() to flush the contents of the file system cache.
3. Read the file contents back to confirm that you get what was written in step 1.

The challenge is in step 3: when reading the data back, that read is most likely going to be served from the file system cache. For the test to really prove that the data was written to disk, we need the read to be performed directly from disk, bypassing the file cache entirely. This implies the need for the FILE_FLAG_NO_BUFFERING and O_DIRECT flags that are not supported yet...which makes testing this function rather tough. I welcome any ideas here 👍

PS: this being the age of AI and all, I asked ChatGPT how to test fsync and it had several suggestions. One easy one was to check the file's timestamp after calling FlushToDisk() so I am going to research that to see if it makes sense.

[adamsitnik]

This implies the need for the FILE_FLAG_NO_BUFFERING and O_DIRECT flags that #27408

This is tricky indeed. We actually support FILE_FLAG_NO_BUFFERING on Windows in a very weird way: we accept one magic int value that is not part of the enum:

runtime/src/libraries/System.Private.CoreLib/src/System/IO/Strategies/FileStreamHelpers.cs

Line 81 in f25bc7b

if (options != FileOptions.None && (options & ~(FileOptions.WriteThrough | FileOptions.Asynchronous | FileOptions.RandomAccess | FileOptions.DeleteOnClose | FileOptions.SequentialScan | FileOptions.Encrypted | (FileOptions)0x20000000 /* NoBuffering */)) != 0)

We have some tests that use it:

runtime/src/libraries/System.IO.FileSystem/tests/RandomAccess/NoBuffering.Windows.cs

Line 13 in f25bc7b

public class RandomAccess_NoBuffering : FileSystemTest

You can extend these tests and test it properly on Windows.

When it comes to Unix my best idea is to simply ensure that it does not throw.