.NET 6 FileStream fails with System.IO.IOException: 'The parameter is incorrect' when FILE_FLAG_NO_BUFFERING used

[HighPerfDotNet]

Description

New .NET 6 FileStream has got unexpected behavior that was not present in all previous .NET frameworks starting from at least .NETFX 2. There appears to be an exception System.IO.IOException: 'The parameter is incorrect' thrown after whole file is read if the file is not a multiple of read buffer size

Reproduction Steps

// .NET 6 console code to repro
// needs to be multilple of sector size: 4k covers 512 and 4k sectors
int iBufferSize = 512 * 8;

// test 1: multiple of buffer reads: works in all .NET's
TestReads(iBufferSize * 2);

// test 2: slightly larger size of file: fails in .NET 6 new FileStream object, works if config uses: "System.IO.UseNet5CompatFileStream": true
TestReads(1+ (iBufferSize * 2));

void TestReads(int iDataSize)
{
    Console.WriteLine("Testing file with {0} bytes",iDataSize);

    string sFile= @"data.dat";

    // Set FILE_FLAG_NO_BUFFERING (more in this here: https://docs.microsoft.com/en-us/windows/win32/fileio/file-buffering )
    FileOptions FileFlagNoBuffering = (FileOptions)0x20000000;

    byte[] buffer = new byte[iBufferSize];

    File.WriteAllBytes(sFile, new byte[iDataSize]);

    using (FileStream fs = new FileStream(sFile, FileMode.Open, FileAccess.Read, FileShare.Read, buffer.Length, FileFlagNoBuffering))
    {
        int iTotalBytes = 0;

        while (true)
        {
            int iBytes = fs.Read(buffer, 0, buffer.Length);
            
            // stopping reads if nothing left
            if (iBytes == 0)
                break;

            iTotalBytes += iBytes;

            Console.WriteLine("{0} bytes read, total: {1}", iBytes, iTotalBytes);
        }
    }

}

Console.WriteLine("done");

Expected behavior

Should work by returning 0 read bytes on last call, just like in all previous frameworks (including .NET 5).

Actual behavior

On second test Throws System.IO.IOException: 'The parameter is incorrect'

Regression?

Yes - worked in all previous Frameworks including FX since 2.0

Known Workarounds

Turning off new .NET 6 FileStream object in app config HOWEVER that option is said to be removed in .NET 7 and it means that benefits of new implementation are not used.

    "configProperties": {
      "System.IO.UseNet5CompatFileStream": true
    }

It's also possible to check file size before reading and use that to avoid making last call that should have just read 0 bytes, BUT that is ugly and unnecessary, there just should be no exception thrown.

Configuration

.NET 6, current Windows 10 on x64, using Console app build.

Other information

Detailed exception thrown - 

System.IO.IOException
  HResult=0x80070057
  Message=The parameter is incorrect. : 'data.dat'
  Source=System.Private.CoreLib
  StackTrace:
   at System.IO.RandomAccess.ReadAtOffset(SafeFileHandle handle, Span`1 buffer, Int64 fileOffset)
   at System.IO.Strategies.OSFileStreamStrategy.Read(Span`1 buffer)
   at System.IO.Strategies.OSFileStreamStrategy.Read(Byte[] buffer, Int32 offset, Int32 count)
   at System.IO.Strategies.BufferedFileStreamStrategy.ReadSpan(Span`1 destination, ArraySegment`1 arraySegment)
   at System.IO.Strategies.BufferedFileStreamStrategy.Read(Byte[] buffer, Int32 offset, Int32 count)
   at System.IO.FileStream.Read(Byte[] buffer, Int32 offset, Int32 count)
   at Program.<<Main>$>g__TestWrites|0_0(Int32 iDataSize, <>c__DisplayClass0_0& ) in FileStreamBug\Program.cs:line 29
   at Program.<Main>$(String[] args) in FileStreamBug\Program.cs:line 8

[ghost]

Tagging subscribers to this area: @dotnet/area-system-io
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

New .NET 6 FileStream has got unexpected behavior that was not present in all previous .NET frameworks starting from at least .NETFX 2. There appears to be an exception System.IO.IOException: 'The parameter is incorrect' thrown after whole file is read if the file is not a multiple of read buffer size

Reproduction Steps

// .NET 6 console code to repro
// needs to be multilple of sector size: 4k covers 512 and 4k sectors
int iBufferSize = 512 * 8;

// test 1: multiple of buffer reads: works in all .NET's
TestReads(iBufferSize * 2);

// test 2: slightly larger size of file: fails in .NET 6 new FileStream object, works if config uses: "System.IO.UseNet5CompatFileStream": true
TestReads(1+ (iBufferSize * 2));

void TestReads(int iDataSize)
{
Console.WriteLine("Testing file with {0} bytes",iDataSize);

string sFile= @"data.dat";

// Set FILE_FLAG_NO_BUFFERING (more in this here: https://docs.microsoft.com/en-us/windows/win32/fileio/file-buffering )
FileOptions FileFlagNoBuffering = (FileOptions)0x20000000;

byte[] buffer = new byte[iBufferSize];

File.WriteAllBytes(sFile, new byte[iDataSize]);

using (FileStream fs = new FileStream(sFile, FileMode.Open, FileAccess.Read, FileShare.Read, buffer.Length, FileFlagNoBuffering))
{
    int iTotalBytes = 0;

    while (true)
    {
        int iBytes = fs.Read(buffer, 0, buffer.Length);
        
        // stopping reads if nothing left
        if (iBytes == 0)
            break;

        iTotalBytes += iBytes;

        Console.WriteLine("{0} bytes read, total: {1}", iBytes, iTotalBytes);
    }
}

}

Console.WriteLine("done");

Expected behavior

Should work by returning 0 read bytes on last call, just like in all previous frameworks (including .NET 5).

Actual behavior

On second test Throws System.IO.IOException: 'The parameter is incorrect'

Regression?

Yes - worked in all previous Frameworks including FX since 2.0

Known Workarounds

Turning off new .NET 6 FileStream object in app config HOWEVER that option is said to be removed in .NET 7 and it means that benefits of new implementation are not used.

"configProperties": {
  "System.IO.UseNet5CompatFileStream": true
}

It's also possible to check file size before reading and use that to avoid making last call that should have just read 0 bytes, BUT that is ugly and unnecessary, there just should be no exception thrown.

Configuration

.NET 6, current Windows 10 on x64, using Console app build.

Other information

Detailed exception thrown -

System.IO.IOException
HResult=0x80070057
Message=The parameter is incorrect. : 'data.dat'
Source=System.Private.CoreLib
StackTrace:
at System.IO.RandomAccess.ReadAtOffset(SafeFileHandle handle, Span1 buffer, Int64 fileOffset) at System.IO.Strategies.OSFileStreamStrategy.Read(Span1 buffer)
at System.IO.Strategies.OSFileStreamStrategy.Read(Byte[] buffer, Int32 offset, Int32 count)
at System.IO.Strategies.BufferedFileStreamStrategy.ReadSpan(Span1 destination, ArraySegment1 arraySegment)
at System.IO.Strategies.BufferedFileStreamStrategy.Read(Byte[] buffer, Int32 offset, Int32 count)
at System.IO.FileStream.Read(Byte[] buffer, Int32 offset, Int32 count)
at Program.<

$>g__TestWrites|0_0(Int32 iDataSize, <>c__DisplayClass0_0& ) in C:\Users\Alex\source\repos\FileStreamBug\Program.cs:line 29
at Program.$(String[] args) in C:\Users\Alex\source\repos\FileStreamBug\Program.cs:line 8

Author:	HighPerfDotNet
Assignees:	-
Labels:	area-System.IO, untriaged
Milestone:	-

[adamsitnik]

Hi @HighPerfDotNet

Big thanks for a very detailed bug report. I've sent a fix in #63625 and we are going to solve this problem in .NET 7.

Is there any chance you could provide your feedback in #27408 ?

[HighPerfDotNet]

@adamsitnik - this sounds good to me, thank you for fixing this very important to us bug. We'll use .NET 5 Stream fallback option for the time being, that seems to work well, closing this issue now. I'll check out that other link for feedback.

[adamsitnik]

@HighPerfDotNet I am going to close the issue once the PR gets merged. Thank you for your feedback!

[adamsitnik]

#63625 got merged, the fix will be shipped as part of .NET 7 Preview 1