Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add symbolic link APIs #54253

Merged
merged 51 commits into from
Jul 9, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
b63c7b7
ref declarations and src empty definitions
carlossanlop Jun 1, 2021
b223304
Implementation for Unix symlinks APIs and cross-platform unit tests.
carlossanlop Jun 1, 2021
2e822fd
Add windows implementation of [sym]link APIs
jozkee Jun 11, 2021
32e196c
Use ValueStringBuilder in ResolveLinkTarget
carlossanlop Jun 15, 2021
8ed2772
Build failure in FileSystemWatcher csproj due to missing new interop …
carlossanlop Jun 17, 2021
abbecf0
Fix build failure. Address documentation suggestions.
Jun 17, 2021
fc49855
Add missing csproj references to System.Net.Ping.Functional.Tests (pr…
carlossanlop Jun 18, 2021
86cbd19
Move CanCreateSymbolicLinks from FileSystemWatcher one class above, s…
Jun 18, 2021
480d3df
Fix some Microsoft.IO.Redist CI failures.
carlossanlop Jun 18, 2021
a6cd9da
Fix CI build failures from Net5Compat.Tests
carlossanlop Jun 18, 2021
6952eb7
Clean CanCreateSymbolicLinks for readability. Remove incomplete code.
carlossanlop Jun 21, 2021
797ace5
Support long paths in GetFinalPathNameByHandle
jozkee Jun 18, 2021
8639e3b
Use MS.IO.Redist friendly APIs for marshalling
jozkee Jun 19, 2021
6bb1aa9
Add IsBrowser check to CanCreateSymbolicLinks
carlossanlop Jun 21, 2021
88bcadb
Add more tests and fix some issues in windows
jozkee Jun 22, 2021
8701118
Add path to IO_InconsistentLinkType exception msg
carlossanlop Jun 22, 2021
d10e00c
Fix failure in test creating inconsistent file/dir type.
carlossanlop Jun 22, 2021
fa5ff40
Fix net48 error
jozkee Jun 22, 2021
6788db7
Address CreateSymbolicLink initial check on Unix to verify pathToTarg…
Jun 22, 2021
45392b4
Use RemoteExecutor for a test related to change the current working dir
jozkee Jun 23, 2021
38b6e88
Add missing argument to exception message in MS.IO.Redist
Jun 23, 2021
a70cdef
nit: Clean comment on CreateSymbolicLink p/invoke.
Jun 23, 2021
cbcf7af
Fix CI bug in Windows Nano where %TEMP% points to "C:\Windows\TEMP" b…
carlossanlop Jun 26, 2021
c43664a
Use ALLOW_UNPRIVILEGED_CREATE only on windows versions >= 10.0.14972
jozkee Jun 26, 2021
9460264
Do not use IsWindowsVersionAtLeast since isn't avail in ns2.0
jozkee Jun 28, 2021
bbaa35a
Use PrintName (Dos) instead of SubstituteName (NT)
jozkee Jun 28, 2021
7e2eaa9
Fix issues related to server share paths
jozkee Jun 29, 2021
0d4250b
Fix CI issues
jozkee Jun 29, 2021
1a05a56
Address suggestions
jozkee Jun 29, 2021
d656aed
Address suggestions
jozkee Jun 29, 2021
adcca16
Address suggestions about using PathInternal.IsExtended
jozkee Jun 29, 2021
35165b0
Add more scenarios for "file system entry type is inconsistent with t…
jozkee Jul 1, 2021
9d2694b
Remove duplicated validation checks
jozkee Jul 1, 2021
c79df26
Address suggestions for Unix
jozkee Jul 1, 2021
fc4241d
ifdef the list of paths used for theories
jozkee Jul 1, 2021
4881f2b
Add tests to verify the limit of followed links
jozkee Jul 2, 2021
b377cb0
Fix bug related to follow links limit in Unix
jozkee Jul 2, 2021
496487f
Trim extended prefix when the passed-in path is not extended
jozkee Jul 2, 2021
beaf701
Remove helper AssertFullNameEquals since is no longer needed and fix …
jozkee Jul 2, 2021
a5986fb
Add a check for versions < win10 build 14972 to fix silent error
jozkee Jul 2, 2021
b7ec269
Add Windows limit for ResolveLinkTarget to remarks
jozkee Jul 2, 2021
7f37a86
Remove pipe test
jozkee Jul 2, 2021
3e36f72
Remove lstat from ResolveLinkTarget
jozkee Jul 6, 2021
6f5b99a
Avoid testing the reparse point/soft link limit very precisely
jozkee Jul 8, 2021
2d09f7e
Fix LinkTarget invalidation logic
jozkee Jul 8, 2021
4b5e133
Make returnFinalTarget no longer optional
jozkee Jul 8, 2021
82dc88a
Merge branch 'main' of https://github.com/dotnet/runtime into symlinks
jozkee Jul 8, 2021
48e1af1
Try to fix failing tests found in CI
jozkee Jul 8, 2021
0495f59
Try smaller chains of links
jozkee Jul 8, 2021
40cb737
Avoid using GetTestFilePath in order to correctly compare against the…
jozkee Jul 9, 2021
995ead5
Use chain length of 20 instead of 30, which fails in Win 1809
jozkee Jul 9, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.

internal static partial class Interop
{
// Unix max paths are typically 1K or 4K UTF-8 bytes, 256 should handle the majority of paths
// without putting too much pressure on the stack.
internal const int DefaultPathBufferSize = 256;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if the team has taken any measurements on this? What is typical path size?
I would choose half of the maximum i.e. 512 so as to always have no more than one fallback (bufferSize *= 2).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question, I don't know if we have data around this. Since this number is used in some other APIs (and this const is now consumed in those places) I would rather not make that change now. Feel free to open an issue though.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: This will give us a new const for Unix, while we already have Interop.Kernel32.MAX_PATH for Windows. It would be nice to have one const for both platforms.

}
Original file line number Diff line number Diff line change
Expand Up @@ -14,18 +14,16 @@ internal static partial class Sys

internal static unsafe string GetCwd()
{
const int StackLimit = 256;

// First try to get the path into a buffer on the stack
byte* stackBuf = stackalloc byte[StackLimit];
string? result = GetCwdHelper(stackBuf, StackLimit);
byte* stackBuf = stackalloc byte[DefaultPathBufferSize];
string? result = GetCwdHelper(stackBuf, DefaultPathBufferSize);
if (result != null)
{
return result;
}

// If that was too small, try increasing large buffer sizes
int bufferSize = StackLimit;
int bufferSize = DefaultPathBufferSize;
while (true)
{
checked { bufferSize *= 2; }
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
using System.Runtime.InteropServices;
using System.Buffers;
using System.Text;
using System;

internal static partial class Interop
{
Expand All @@ -20,24 +21,31 @@ internal static partial class Sys
/// Returns the number of bytes placed into the buffer on success; bufferSize if the buffer is too small; and -1 on error.
/// </returns>
[DllImport(Libraries.SystemNative, EntryPoint = "SystemNative_ReadLink", SetLastError = true)]
private static extern int ReadLink(string path, byte[] buffer, int bufferSize);
private static extern int ReadLink(ref byte path, byte[] buffer, int bufferSize);

/// <summary>
/// Takes a path to a symbolic link and returns the link target path.
/// </summary>
/// <param name="path">The path to the symlink</param>
/// <returns>
/// Returns the link to the target path on success; and null otherwise.
/// </returns>
public static string? ReadLink(string path)
/// <param name="path">The path to the symlink.</param>
/// <returns>Returns the link to the target path on success; and null otherwise.</returns>
internal static string? ReadLink(ReadOnlySpan<char> path)
{
int bufferSize = 256;
int outputBufferSize = 1024;

// Use an initial buffer size that prevents disposing and renting
// a second time when calling ConvertAndTerminateString.
using var converter = new ValueUtf8Converter(stackalloc byte[1024]);

while (true)
{
byte[] buffer = ArrayPool<byte>.Shared.Rent(bufferSize);
byte[] buffer = ArrayPool<byte>.Shared.Rent(outputBufferSize);
try
{
int resultLength = Interop.Sys.ReadLink(path, buffer, buffer.Length);
int resultLength = Interop.Sys.ReadLink(
ref MemoryMarshal.GetReference(converter.ConvertAndTerminateString(path)),
buffer,
buffer.Length);

if (resultLength < 0)
{
// error
Expand All @@ -54,8 +62,8 @@ internal static partial class Sys
ArrayPool<byte>.Shared.Return(buffer);
}

// buffer was too small, loop around again and try with a larger buffer.
bufferSize *= 2;
// Output buffer was too small, loop around again and try with a larger buffer.
outputBufferSize = buffer.Length * 2;
}
}
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,16 +9,12 @@ internal static partial class Interop
{
internal static partial class Sys
{
// Unix max paths are typically 1K or 4K UTF-8 bytes, 256 should handle the majority of paths
// without putting too much pressure on the stack.
private const int StackBufferSize = 256;

[DllImport(Libraries.SystemNative, EntryPoint = "SystemNative_Stat", SetLastError = true)]
internal static extern int Stat(ref byte path, out FileStatus output);

internal static int Stat(ReadOnlySpan<char> path, out FileStatus output)
{
var converter = new ValueUtf8Converter(stackalloc byte[StackBufferSize]);
var converter = new ValueUtf8Converter(stackalloc byte[DefaultPathBufferSize]);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: can this be using var instead of calling Dispose?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, ValueUtf8Converter is not disposable.

@GrabYourPitchforks Do you know why it was decided not to make this type disposable? Is there anything special about its Dispose method that would make it unsafe to call inside a using?

internal ref struct ValueUtf8Converter

Copy link
Member

@hoyosjs hoyosjs Jun 23, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you do IDisposable you'd need boxing if anyone ever tries to store it or pass it as IDisposable; also, ref structs can't implement interfaces (precisely because they can't be boxed). You can still do using var even if it doesn't implement IDisposable: SharpLab

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting, thanks for showing me @hoyosjs . Didn't know using could be used with non-IDisposables.

We have a list of to-dos for changes that are unrelated to this PR. We'll add this suggestion to the list.

int result = Stat(ref MemoryMarshal.GetReference(converter.ConvertAndTerminateString(path)), out output);
converter.Dispose();
return result;
Expand All @@ -29,7 +25,7 @@ internal static int Stat(ReadOnlySpan<char> path, out FileStatus output)

internal static int LStat(ReadOnlySpan<char> path, out FileStatus output)
{
var converter = new ValueUtf8Converter(stackalloc byte[StackBufferSize]);
var converter = new ValueUtf8Converter(stackalloc byte[DefaultPathBufferSize]);
int result = LStat(ref MemoryMarshal.GetReference(converter.ConvertAndTerminateString(path)), out output);
converter.Dispose();
return result;
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.

using System;
using System.Runtime.InteropServices;

internal static partial class Interop
{
internal static partial class Sys
{
[DllImport(Libraries.SystemNative, EntryPoint = "SystemNative_SymLink", SetLastError = true)]
internal static extern int SymLink(string target, string linkPath);
}
}
1 change: 1 addition & 0 deletions src/libraries/Common/src/Interop/Windows/Interop.Errors.cs
Original file line number Diff line number Diff line change
Expand Up @@ -91,5 +91,6 @@ internal static partial class Errors
internal const int ERROR_EVENTLOG_FILE_CHANGED = 0x5DF;
internal const int ERROR_TRUSTED_RELATIONSHIP_FAILURE = 0x6FD;
internal const int ERROR_RESOURCE_LANG_NOT_FOUND = 0x717;
internal const int ERROR_NOT_A_REPARSE_POINT = 0x1126;
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.

using System;
using System.IO;
using System.Runtime.InteropServices;

internal static partial class Interop
{
internal static partial class Kernel32
{
/// <summary>
/// The link target is a directory.
/// </summary>
internal const int SYMBOLIC_LINK_FLAG_DIRECTORY = 0x1;

/// <summary>
/// Allows creation of symbolic links from a process that is not elevated. Requires Windows 10 Insiders build 14972 or later.
/// Developer Mode must first be enabled on the machine before this option will function.
adamsitnik marked this conversation as resolved.
Show resolved Hide resolved
/// </summary>
internal const int SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE = 0x2;

[DllImport(Libraries.Kernel32, EntryPoint = "CreateSymbolicLinkW", SetLastError = true, CharSet = CharSet.Unicode, BestFitMapping = false, ExactSpelling = true)]
private static extern bool CreateSymbolicLinkPrivate(string lpSymlinkFileName, string lpTargetFileName, int dwFlags);

/// <summary>
/// Creates a symbolic link.
/// </summary>
/// <param name="symlinkFileName">The symbolic link to be created.</param>
/// <param name="targetFileName">The name of the target for the symbolic link to be created.
/// If it has a device name associated with it, the link is treated as an absolute link; otherwise, the link is treated as a relative link.</param>
/// <param name="isDirectory"><see langword="true" /> if the link target is a directory; <see langword="false" /> otherwise.</param>
internal static void CreateSymbolicLink(string symlinkFileName, string targetFileName, bool isDirectory)
{
string originalPath = symlinkFileName;
symlinkFileName = PathInternal.EnsureExtendedPrefixIfNeeded(symlinkFileName);
targetFileName = PathInternal.EnsureExtendedPrefixIfNeeded(targetFileName);

int flags = 0;

bool isAtLeastWin10Build14972 =
Environment.OSVersion.Version.Major == 10 && Environment.OSVersion.Version.Build >= 14972 ||
Environment.OSVersion.Version.Major >= 11;

if (isAtLeastWin10Build14972)
{
flags = SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE;
}

if (isDirectory)
{
flags |= SYMBOLIC_LINK_FLAG_DIRECTORY;
}

bool success = CreateSymbolicLinkPrivate(symlinkFileName, targetFileName, flags);

int error;
if (!success)
{
throw Win32Marshal.GetExceptionForLastWin32Error(originalPath);
}
// In older versions we need to check GetLastWin32Error regardless of the return value of CreateSymbolicLink,
// e.g: if the user doesn't have enough privileges to create a symlink the method returns success which we can consider as a silent failure.
else if (!isAtLeastWin10Build14972 && (error = Marshal.GetLastWin32Error()) != 0)
{
throw Win32Marshal.GetExceptionForWin32Error(error, originalPath);
}
}
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.

using System;
using System.IO;
using System.Runtime.InteropServices;

internal static partial class Interop
{
internal static partial class Kernel32
{
// https://docs.microsoft.com/windows/win32/api/winioctl/ni-winioctl-fsctl_get_reparse_point
internal const int FSCTL_GET_REPARSE_POINT = 0x000900a8;
jozkee marked this conversation as resolved.
Show resolved Hide resolved

[DllImport(Libraries.Kernel32, EntryPoint = "DeviceIoControl", SetLastError = true, CharSet = CharSet.Unicode, ExactSpelling = true)]
adamsitnik marked this conversation as resolved.
Show resolved Hide resolved
internal static extern bool DeviceIoControl(
SafeHandle hDevice,
uint dwIoControlCode,
IntPtr lpInBuffer,
uint nInBufferSize,
byte[] lpOutBuffer,
uint nOutBufferSize,
out uint lpBytesReturned,
IntPtr lpOverlapped);
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ internal static partial class IOReparseOptions
{
internal const uint IO_REPARSE_TAG_FILE_PLACEHOLDER = 0x80000015;
internal const uint IO_REPARSE_TAG_MOUNT_POINT = 0xA0000003;
internal const uint IO_REPARSE_TAG_SYMLINK = 0xA000000C;
}

internal static partial class FileOperations
Expand All @@ -18,6 +19,7 @@ internal static partial class FileOperations

internal const int FILE_FLAG_BACKUP_SEMANTICS = 0x02000000;
internal const int FILE_FLAG_FIRST_PIPE_INSTANCE = 0x00080000;
internal const int FILE_FLAG_OPEN_REPARSE_POINT = 0x00200000;
internal const int FILE_FLAG_OVERLAPPED = 0x40000000;

internal const int FILE_LIST_DIRECTORY = 0x0001;
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.

using System;
using System.IO;
using System.Runtime.InteropServices;
using Microsoft.Win32.SafeHandles;

internal static partial class Interop
{
internal static partial class Kernel32
{
internal const uint FILE_NAME_NORMALIZED = 0x0;

// https://docs.microsoft.com/windows/desktop/api/fileapi/nf-fileapi-getfinalpathnamebyhandlew (kernel32)
[DllImport(Libraries.Kernel32, EntryPoint = "GetFinalPathNameByHandleW", CharSet = CharSet.Unicode, SetLastError = true, ExactSpelling = true)]
internal static unsafe extern uint GetFinalPathNameByHandle(
SafeFileHandle hFile,
char* lpszFilePath,
uint cchFilePath,
uint dwFlags);
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.

using System;
using System.Runtime.InteropServices;

internal static partial class Interop
{
internal static partial class Kernel32
{
// https://docs.microsoft.com/windows-hardware/drivers/ifs/fsctl-get-reparse-point
internal const int MAXIMUM_REPARSE_DATA_BUFFER_SIZE = 16 * 1024;

internal const uint SYMLINK_FLAG_RELATIVE = 1;

// https://msdn.microsoft.com/library/windows/hardware/ff552012.aspx
// We don't need all the struct fields; omitting the rest.
[StructLayout(LayoutKind.Sequential)]
internal unsafe struct REPARSE_DATA_BUFFER
{
internal uint ReparseTag;
internal ushort ReparseDataLength;
internal ushort Reserved;
internal SymbolicLinkReparseBuffer ReparseBufferSymbolicLink;

[StructLayout(LayoutKind.Sequential)]
internal struct SymbolicLinkReparseBuffer
{
internal ushort SubstituteNameOffset;
internal ushort SubstituteNameLength;
internal ushort PrintNameOffset;
internal ushort PrintNameLength;
internal uint Flags;
}
}
}
}
36 changes: 23 additions & 13 deletions src/libraries/Common/src/System/IO/FileSystem.Attributes.Windows.cs
Original file line number Diff line number Diff line change
Expand Up @@ -64,19 +64,7 @@ internal static int FillAttributeInfo(string? path, ref Interop.Kernel32.WIN32_F
{
errorCode = Marshal.GetLastWin32Error();

if (errorCode != Interop.Errors.ERROR_FILE_NOT_FOUND
&& errorCode != Interop.Errors.ERROR_PATH_NOT_FOUND
&& errorCode != Interop.Errors.ERROR_NOT_READY
&& errorCode != Interop.Errors.ERROR_INVALID_NAME
&& errorCode != Interop.Errors.ERROR_BAD_PATHNAME
&& errorCode != Interop.Errors.ERROR_BAD_NETPATH
&& errorCode != Interop.Errors.ERROR_BAD_NET_NAME
&& errorCode != Interop.Errors.ERROR_INVALID_PARAMETER
&& errorCode != Interop.Errors.ERROR_NETWORK_UNREACHABLE
&& errorCode != Interop.Errors.ERROR_NETWORK_ACCESS_DENIED
&& errorCode != Interop.Errors.ERROR_INVALID_HANDLE // eg from \\.\CON
&& errorCode != Interop.Errors.ERROR_FILENAME_EXCED_RANGE // Path is too long
)
if (!IsPathUnreachableError(errorCode))
{
// Assert so we can track down other cases (if any) to add to our test suite
Debug.Assert(errorCode == Interop.Errors.ERROR_ACCESS_DENIED || errorCode == Interop.Errors.ERROR_SHARING_VIOLATION || errorCode == Interop.Errors.ERROR_SEM_TIMEOUT,
Expand Down Expand Up @@ -127,5 +115,27 @@ internal static int FillAttributeInfo(string? path, ref Interop.Kernel32.WIN32_F

return errorCode;
}

internal static bool IsPathUnreachableError(int errorCode)
{
switch (errorCode)
{
case Interop.Errors.ERROR_FILE_NOT_FOUND:
case Interop.Errors.ERROR_PATH_NOT_FOUND:
case Interop.Errors.ERROR_NOT_READY:
case Interop.Errors.ERROR_INVALID_NAME:
case Interop.Errors.ERROR_BAD_PATHNAME:
case Interop.Errors.ERROR_BAD_NETPATH:
case Interop.Errors.ERROR_BAD_NET_NAME:
case Interop.Errors.ERROR_INVALID_PARAMETER:
case Interop.Errors.ERROR_NETWORK_UNREACHABLE:
case Interop.Errors.ERROR_NETWORK_ACCESS_DENIED:
case Interop.Errors.ERROR_INVALID_HANDLE: // eg from \\.\CON
case Interop.Errors.ERROR_FILENAME_EXCED_RANGE: // Path is too long
return true;
default:
return false;
}
}
}
}
Loading