Replace harvested package assets with live built configurations #47530

ViktorHofer · 2021-01-27T16:23:06Z

The runtime repository currently produces 98 packages from libraries and 25 of those contain prebuilts (assets which are not built in the master/main branch anymore). Such prebuilt assets are redistributed into packages via a mechanism called “harvesting”. Without going into detail, the harvesting infrastructure downloads the latest available assets of the “current band” and places them into a specified subfolder inside the package.

In contrast to simplification of the build graph, harvesting has numerous downsides:

Increased cost of servicing as code fixes need to be applied in the repository’s branch in which the code lives, which is not necessarily the same place where the package is produced and shipped from.
Dependency between servicing releases, especially during MSRCs which forces schedules to be aligned.
Source link and newer compiler features (IL size reduction / optimization) are not applicable to harvested assets which causes NuGet Package Explorer to list the whole package as unhealthy.
Package validation costs
Fat packages
Harvesting build validation causes builds to fail in PRs every time new assets are shipped to indicate that harvesting defines should be updated.
Maintenance of code bases which are not tested anymore.

Based on these disadvantages, we would like to remove harvesting in dotnet/runtime entirely and only ship what is being built live.

Implementation

For the following libraries, build configurations need to be added and/or harvesting configurations in the pkgproj need to be removed.

Add a build warning wherever functionality would regress.

cc @danmosemsft @ericstj @terrajobst @tommcdon

ghost · 2021-01-27T16:23:09Z

Tagging subscribers to this area: @safern, @ViktorHofer
See info in area-owners.md if you want to be subscribed.

Issue Details

The runtime repository currently produces 98 packages from libraries and 25 of those contain prebuilts (assets which are not built in the master/main branch anymore). Such prebuilt assets are redistributed into packages via a mechanism called “harvesting”. Without going into detail, the harvesting infrastructure downloads the latest available assets of the “current band” and places them into a specified subfolder inside the package.

In contrast to simplification of the build graph, harvesting has numerous downsides:

Increased cost of servicing as code fixes need to be applied in the repository’s branch in which the code lives, which is not necessarily the same place where the package is produced and shipped from.
Dependency between servicing releases, especially during MSRCs which forces schedules to be aligned.
Source link and newer compiler features (IL size reduction / optimization) are not applicable to harvested assets which causes NuGet Package Explorer to list the whole package as unhealthy.
Package validation costs
Fat packages
Harvesting build validation causes builds to fail in PRs every time new assets are shipped to indicate that harvesting defines should be updated.
Maintenance of code bases which are not tested anymore.

Based on these disadvantages, we would like to remove harvesting in dotnet/runtime entirely and only ship what is being built live.

Implementation

For the following libraries, build configurations need to be added and/or harvesting configurations in the pkgproj need to be removed.

cc @danmosemsft @ericstj @terrajobst @tommcdon

Author:	ViktorHofer
Assignees:	ViktorHofer
Labels:	`area-Infrastructure-libraries`
Milestone:	6.0.0

The netstandard1.6 configuration of Microsoft.Extensions.DependencyModel isn't built anymore. Instead the already built matching binary from the latest available package version is redistributed when packaging the DependencyModel library. Also dropping the netstandard1.3 asset and the net451 one as the minimum supported set of platforms are ones that support netstandard2.0. In addition to the harvesting removal, cleaning up the src project which had an unnecessary condition and property set. Contributes to dotnet#47530

The net46 and netstandard1.3 configuration of Microsoft.Win32.Registry and Microsoft.Win32.Registry.AccessControl isn't built anymore. Instead the already built matching binary from the latest available package version is redistributed when packaging these libraries. Dropping the netstandard1.3 asset and the net46 one as the minimum supported set of platforms are ones that support netstandard2.0. For .NET Framework, there's still a net461 configuration to avoid binding redirect issues. Contributes to dotnet#47530

* Remove harvesting of M.E.DependencyModel The netstandard1.6 configuration of Microsoft.Extensions.DependencyModel isn't built anymore. Instead the already built matching binary from the latest available package version is redistributed when packaging the DependencyModel library. Also dropping the netstandard1.3 asset and the net451 one as the minimum supported set of platforms are ones that support netstandard2.0. In addition to the harvesting removal, cleaning up the src project which had an unnecessary condition and property set. Contributes to #47530

The netstandard1.0 and portable* configurations of the System.Composition.* packages aren't built anymore. Instead the already built matching binary from the latest available package version is redistributed when packaging these libraries. Dropping the netstandard1.0 asset and the portable* one as the minimum supported set of platforms are ones that support netstandard2.0. For .NET Framework, there's still a net461 configuration to avoid binding redirect issues. Contributes to dotnet#47530

The removed configurations (few netstandard1.x, portable*, netcore50) of the touched packages aren't built anymore. Instead the already built matching binaries from the latest available packages are redistributed when packaging these libraries. Dropping these assets as the minimum supported set of platforms are ones that support netstandard2.0. For .NET Framework, there's still a net461 configuration present to avoid binding redirect issues. Contributes to dotnet#47530

The netstandard1.0 and portable* configurations of the System.Composition.* packages aren't built anymore. Instead the already built matching binary from the latest available package version is redistributed when packaging these libraries. Dropping the netstandard1.0 asset and the portable* one as the minimum supported set of platforms are ones that support netstandard2.0. For .NET Framework, there's still a net461 configuration to avoid binding redirect issues. Contributes to #47530

danmoseley · 2021-04-22T13:54:33Z

Are there any updates to the plan (since this was opened) that we need to share here?

ViktorHofer · 2021-04-22T14:01:02Z

Only minor adjustments, two configurations which are dropped instead of brought back. I'm in regular sync with @terrajobst about this. I do have an excel spreadsheet which has more details which I will share with you again offline.

danmoseley · 2021-04-22T14:05:01Z

@ViktorHofer I don't have concerns that I'm not up to date. I just wanted to make sure that community was up to date, presumably here or possibly in some linked design.

* Stop harvesting old frameworks in some libraries The removed configurations (few netstandard1.x, portable*, netcore50) of the touched packages aren't built anymore. Instead the already built matching binaries from the latest available packages are redistributed when packaging these libraries. Dropping these assets as the minimum supported set of platforms are ones that support netstandard2.0. For .NET Framework, there's still a net461 configuration present to avoid binding redirect issues. Contributes to #47530

To reduce complexity of the cryptography cng library, dead ending its package as most of the configurations are partial facades anyway and adding Cng to the targeting pack. It's already part of the runtime and exposed in aspnetcore's targeting pack. .NET Standard libraries can continue to use the latest available package which harvests for .NET Standard configurations. Contributes to dotnet#47530

The netstandard1.3 and net46 configurations of the System.Security.Cryptography.ProtectedData packages aren't built anymore. Instead the already built matching binary from the latest available package version is redistributed when packaging these libraries. Dropping the netstandard1.3 asset and the net46 one as the minimum supported set of platforms are ones that support netstandard2.0. For .NET Framework, there's still a net461 configuration to avoid binding redirect issues. Contributes to dotnet#47530

ViktorHofer · 2021-04-26T16:08:44Z

Replied offline.

The netstandard1.3 and net46 configurations of the System.Security.Cryptography.ProtectedData packages aren't built anymore. Instead the already built matching binary from the latest available package version is redistributed when packaging these libraries. Dropping the netstandard1.3 asset and the net46 one as the minimum supported set of platforms are ones that support netstandard2.0. For .NET Framework, there's still a net461 configuration to avoid binding redirect issues. Contributes to #47530

* Dead end S.Security.Cryptography.OpenSsl package To reduce complexity of the cryptography OpenSsl library, dead ending its package as most of the configurations are partial facades anyway and adding OpenSsl to the targeting pack. It's already part of the runtime and exposed in aspnetcore's targeting pack. .NET Standard libraries can continue to use the latest available package which harvests for .NET Standard configurations. Contributes to #47530

Nearly all configurations in the System.ComponentModel.Annotations package weren't built live anymore and just redistributed from older packages. Except for the netstandard2.0 configuration, the package didn't receive any changes. As the library is exposed as part of the shared framework, a reference to the package on .NETCoreApp isn't necessary and will be ignored by the SDK. Based on these reasons, dead-ending the package to remove the burden of maintaining the harvesting of old configurations. Contributes to dotnet#47530

* Dead end S.Security.Cryptography.Cng package To reduce complexity of the cryptography cng library, dead ending its package as most of the configurations are partial facades anyway and adding Cng to the targeting pack. It's already part of the runtime and exposed in aspnetcore's targeting pack. .NET Standard libraries can continue to use the latest available package which harvests for .NET Standard configurations. Contributes to #47530

The removed configurations (few netstandard1.x, portable*, netcore50, net46) of the touched packages aren't built anymore. Instead the already built matching binaries from the latest available packages are redistributed when packaging these libraries. Dropping these assets as the minimum supported set of platforms are ones that support netstandard2.0. For .NET Framework, there's still a net461 configuration present to avoid binding redirect issues. Contributes to dotnet#47530

Removing the netcoreapp2.0, netstandard2.0 and net461 configurations of System.Drawing.Common as these aren't built live anymore (harvested). Replace the netstandard2.0 and net461 harvested asset with live builds and stop supporting netcoreapp2.x. Bringing these assets back as the minimum supported set of platforms are ones that support netstandard2.0 and are still in support. Contributes to dotnet#47530

* Stop harvesting old frameworks in more libraries The removed configurations (few netstandard1.x, portable*, netcore50, net46) of the touched packages aren't built anymore. Instead the already built matching binaries from the latest available packages are redistributed when packaging these libraries. Dropping these assets as the minimum supported set of platforms are ones that support netstandard2.0. For .NET Framework, there's still a net461 configuration present to avoid binding redirect issues. Contributes to #47530

Nearly all configurations in the System.ComponentModel.Annotations package weren't built live anymore and just redistributed from older packages. Except for the netstandard2.0 configuration, the package didn't receive any changes. As the library is exposed as part of the shared framework, a reference to the package on .NETCoreApp isn't necessary and will be ignored by the SDK. Based on these reasons, dead-ending the package to remove the burden of maintaining the harvesting of old configurations. Contributes to #47530

* Drawing: Replace harvested assets with live config Removing the netcoreapp2.0, netstandard2.0 and net461 configurations of System.Drawing.Common as these aren't built live anymore (harvested). Replace the netstandard2.0 and net461 harvested asset with live builds and stop supporting netcoreapp2.x. Bringing these assets back as the minimum supported set of platforms are ones that support netstandard2.0 and are still in support. Contributes to #47530

The removed configurations (netstandard1.3, netcoreapp2.1, net46) of the touched packages aren't built anymore. Instead the already built matching binaries from the latest available packages are redistributed when packaging these libraries. Dropping these assets as the minimum supported set of platforms are ones that support netstandard2.0 and are still in-support. As an example, also dropping the netcoreapp2.1 asset which isn't supported anymore and adding a target to prevent the package being used for netcoreapp2.x. For .NET Framework, there's still a net461 configuration present to avoid binding redirect issues. Contributes to dotnet#47530

* Stop harvesting S.Sec.Cryptography.Pkcs The removed configurations (netstandard1.3, netcoreapp2.1, net46) of the touched packages aren't built anymore. Instead the already built matching binaries from the latest available packages are redistributed when packaging these libraries. Dropping these assets as the minimum supported set of platforms are ones that support netstandard2.0 and are still in-support. As an example, also dropping the netcoreapp2.1 asset which isn't supported anymore and adding a target to prevent the package being used for netcoreapp2.x. For .NET Framework, there's still a net461 configuration present to avoid binding redirect issues. Contributes to #47530 * Fix the build * Remove workaround and mark Xamarin supported

The removed configurations (netstandard1.3, netcoreapp2.1, net46) aren't built anymore. Instead the already built matching binaries from the latest available packages are redistributed when packaging these libraries. Dropping these assets as the minimum supported set of platforms are ones that support netstandard2.0 and are still in-support. As an example, also dropping the netcoreapp2.1 asset which isn't supported anymore and upgrading it to netcoreapp3.1.. For .NET Framework, there's still a net461 configuration present to avoid binding redirect issues. The ugly part of this change is that System.IO.Pipes.AccessControl depends on an internal API of System.IO.Pipes for all .NETCoreApp configurations. Because of that, for configurations that don't apply to the current band (netcoreapp3.1, net5.0), the runtime packs need to be downloaded so that runtime assembly can be referenced. Ideally we will come up with a better solution to not require runtime packs to be fetched as that adds 60MB of download size to the repo and breaks source build. Contributes to dotnet#47530

* Stop harvesting S.IO.Pipes.AccessControl The removed configurations (netstandard1.3, netcoreapp2.1, net46) aren't built anymore. Instead the already built matching binaries from the latest available packages are redistributed when packaging these libraries. Dropping these assets as the minimum supported set of platforms are ones that support netstandard2.0 and are still in-support. As an example, also dropping the netcoreapp2.1 asset which isn't supported anymore and upgrading it to netcoreapp3.1.. For .NET Framework, there's still a net461 configuration present to avoid binding redirect issues. The ugly part of this change is that System.IO.Pipes.AccessControl depends on an internal API of System.IO.Pipes for all .NETCoreApp configurations. Because of that, for configurations that don't apply to the current band (netcoreapp3.1, net5.0), the runtime packs need to be downloaded so that runtime assembly can be referenced. Ideally we will come up with a better solution to not require runtime packs to be fetched as that adds 60MB of download size to the repo and breaks source build. Contributes to #47530 * Use non shipping refs for Pipes typeforwards

ViktorHofer · 2021-05-11T15:41:12Z

Closing as the tracked work is completed.

ViktorHofer added the area-Infrastructure-libraries label Jan 27, 2021

ViktorHofer added this to the 6.0.0 milestone Jan 27, 2021

ViktorHofer self-assigned this Jan 27, 2021

dotnet-issue-labeler bot added the untriaged New issue has not been triaged by the area owner label Jan 27, 2021

ViktorHofer removed the untriaged New issue has not been triaged by the area owner label Jan 27, 2021

safern mentioned this issue Feb 1, 2021

Future of System.IO.Pipes.AccessControl Package. #44600

Closed

ViktorHofer mentioned this issue Apr 20, 2021

Remove harvesting of M.E.DependencyModel #51582

Merged

ViktorHofer mentioned this issue Apr 21, 2021

Remove harvesting of Microsoft.Win32.Registry* #51616

Closed

ViktorHofer mentioned this issue Apr 21, 2021

Remove harvesting of System.Composition* #51617

Merged

ViktorHofer mentioned this issue Apr 22, 2021

Stop harvesting old frameworks in some libraries #51689

Merged

ViktorHofer mentioned this issue Apr 26, 2021

Dead end S.Security.Cryptography.Cng package #51853

Merged

ViktorHofer mentioned this issue Apr 26, 2021

Dead end S.Security.Cryptography.OpenSsl package #51854

Merged

ViktorHofer mentioned this issue Apr 26, 2021

Stop harvesting S.Security.Crypto.ProtectedData #51872

Merged

ViktorHofer mentioned this issue Apr 26, 2021

Dead end System.ComponentModel.Annotations package #51891

Merged

ViktorHofer mentioned this issue Apr 26, 2021

Stop harvesting old frameworks in more libraries #51894

Merged

ViktorHofer mentioned this issue Apr 27, 2021

Drawing: Replace harvested assets with live config #51934

Merged

ViktorHofer mentioned this issue Apr 29, 2021

Stop harvesting S.Security.Cryptography.Pkcs #52045

Merged

ViktorHofer mentioned this issue May 5, 2021

Stop harvesting S.IO.Pipes.AccessControl #52294

Merged

ViktorHofer closed this as completed May 11, 2021

ViktorHofer mentioned this issue May 26, 2021

Adhere to the package support policy by building and packaging supported frameworks only #53282

Closed

ghost locked as resolved and limited conversation to collaborators Jul 9, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replace harvested package assets with live built configurations #47530

Replace harvested package assets with live built configurations #47530

ViktorHofer commented Jan 27, 2021 •

edited

Loading

ghost commented Jan 27, 2021

Implementation

danmoseley commented Apr 22, 2021

ViktorHofer commented Apr 22, 2021 •

edited

Loading

danmoseley commented Apr 22, 2021

ViktorHofer commented Apr 26, 2021

ViktorHofer commented May 11, 2021

Replace harvested package assets with live built configurations #47530

Replace harvested package assets with live built configurations #47530

Comments

ViktorHofer commented Jan 27, 2021 • edited Loading

Implementation

ghost commented Jan 27, 2021

Implementation

danmoseley commented Apr 22, 2021

ViktorHofer commented Apr 22, 2021 • edited Loading

danmoseley commented Apr 22, 2021

ViktorHofer commented Apr 26, 2021

ViktorHofer commented May 11, 2021

ViktorHofer commented Jan 27, 2021 •

edited

Loading

ViktorHofer commented Apr 22, 2021 •

edited

Loading