Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Release PR for .NET 8.0 doesn't update prebuilts/artifact versions correctly #3344

Closed
premun opened this issue Mar 27, 2023 · 13 comments
Closed
Assignees
Labels
area-release-infra Release infrastructure owned by .NET Product Construction

Comments

@premun
Copy link
Member

premun commented Mar 27, 2023

Context

The release pipeline opens a PR against the installer repo and updates VMR's eng/Versions.props file, specifically the part with prebuilt package versions.

These properties changed between 7.0 and 8.0 and need to be changed for 8.0 too.

Goal

We need to have a look at the official VMR build associated with the release and find out the names of the artifacts from there and update the file using these names.

@premun premun self-assigned this Mar 27, 2023
@dotnet-issue-labeler dotnet-issue-labeler bot added area-prebuilts Reducing the number of prebuilt packages in the tarball untriaged labels Mar 27, 2023
@MichaelSimons MichaelSimons added area-release Release tasks or related issues and removed area-prebuilts Reducing the number of prebuilt packages in the tarball untriaged labels Mar 30, 2023
@premun premun removed their assignment Apr 19, 2023
@tkapin
Copy link
Member

tkapin commented May 11, 2023

@ali-turan7 - assigned this to you as it's close to the issues you've already implemented.
@mthalman - could you please support @ali-turan7 in this if needed? Thanks

@MichaelSimons MichaelSimons added area-release-infra Release infrastructure owned by .NET Product Construction and removed area-release Release tasks or related issues labels May 11, 2023
@mthalman
Copy link
Member

@ali-turan7 - The two relevant URLs referenced in the Versions.props files are uploaded by these release pipeline steps:

- template: ../steps/upload-to-blob-storage.yml
parameters:
file: $(PIPELINE.WORKSPACE)/$(sourceBuiltArtifactsFileName)
accountName: $(storageAccountName)
containerName: $(blobContainerName)
uploadPath: $(artifactsUploadBaseFilePath)
azureStorageKey: $(dotnetcli-storage-key)
- template: ../steps/upload-to-blob-storage.yml
parameters:
file: $(PIPELINE.WORKSPACE)/$(sdkArtifactFileName)
accountName: $(storageAccountName)
containerName: $(blobContainerName)
uploadPath: $(sdkUploadBaseFilePath)
azureStorageKey: $(dotnetcli-storage-key)

So basically, you just need to resolve what the names of those two files are. The file names are defined here:

- name: sourceBuiltArtifactsFileName
value: Private.SourceBuilt.Artifacts.8.0.100.centos.8-x64.tar.gz
- name: sdkArtifactFileName
value: dotnet-sdk-*.tar.gz

You can see that they use wildcards for the versions. (Actually, the artifacts file name doesn't use a wildcard yet. That change is coming in #3451).

It's also important to note that the $(sdkVersion) variable can't be used here like it is for 6.0/7.0. Since, 8.0 is currently in preview, it doesn't use a stable version pattern and thus doesn't match what source-build produces. This is expected.

So an easy way to determine the version number used in the file is to resolve the wildcards locally on the build agent with a bash script using basename:

- script: |
    set -euo pipefail
    resolvedSdkName=$(basename $(PIPELINE.WORKSPACE)/$(sdkArtifactFileName))
    echo "##vso[task.setvariable variable=ResolvedSdkName]$resolvedSdkName"

That script would set ResolvedSdkName to the actual file name with no wildcards in it. Then it's just a matter of using that value to replace what's in the Versions.props file. Since the pipeline already has the wildcard pattern for these two files, you could actually use those strings and dynamically convert them into a regular expression pattern that could then be passed to a sed command to do the replacement.

@mthalman
Copy link
Member

In addition to the updates that are needed to PrivateSourceBuiltArtifactsUrl and PrivateSourceBuiltSdkUrl_CentOS8Stream for 8.0, there also needs to be updates made for 7.0.

Currently, for both 6.0 and 7.0, only the PrivateSourceBuiltArtifactsPackageVersion property is being updated. But 7.0 also requires that the PrivateSourceBuiltSDKVersion property be updated as well. These two should have the same value. It's not possible for PrivateSourceBuiltArtifactsPackageVersion to simply reference the PrivateSourceBuiltSDKVersion property because the logic which reads these values is doing XML parsing outside the context of MSBuild and can't resolve property references.

To summarize:

@mthalman
Copy link
Member

I just realized that the updates needed for 7.0 that I mentioned above are captured in this issue: #3385. I think it probably makes sense to fix both of these together since they really are the same scenario, just impacting different versions in a different way, and they're in the same part of the code.

@mthalman
Copy link
Member

Here's a sed command that can convert the wildcard pattern for the filename into a regular expression:

sed 's/\./\\./g; s/\*/.*/g'

This simply replaces all instances of . with \. and all instances of * with .*. For example, this would transform Private.SourceBuilt.Artifacts.*.centos.8-x64.tar.gz to Private\.SourceBuilt\.Artifacts\..*\.centos\.8-x64\.tar\.gz, the latter of which is a regular expression that is equivalent to the wildcard pattern of the former.

This regular expression can then be used to match the URL in the Versions.props file.

Here's something I typed up which demonstrates what I'm thinking (completely untested):

filenameRegex=$(echo "$(sourceBuiltArtifactsFileName)" | sed 's/\./\\./g; s/\*/.*/g')
actualFilename=$(basename <whatever>)

sed -i "s#\(<PrivateSourceBuiltArtifactsUrl>\).*$filenameRegex\(</PrivateSourceBuiltArtifactsUrl>\)#\1$actualFilename\2#" $versions_props_path

@ali-turan7
Copy link
Contributor

@ali-turan7
Copy link
Contributor

In DryRun mode, the submit-sorce-build-release-pr script has been called.
In the final phase, right before raising the PR, the creation of the pull request has been disabled.
The contents of the VersionsProps file were compared and printed to ensure that the desired changes were implemented correctly.
The results for each version are shown below.

Test for 8.0
https://dev.azure.com/dnceng/internal/_build/results?buildId=2184104&view=logs&j=19992227-62fb-5b50-4e29-3b72bd33eea1&t=9281c869-17dc-5ef7-6bc6-1c1da071debd

Test for 7.0
https://dev.azure.com/dnceng/internal/_build/results?buildId=2185462&view=logs&j=19992227-62fb-5b50-4e29-3b72bd33eea1&t=9281c869-17dc-5ef7-6bc6-1c1da071debd

Test for 6.0
https://dev.azure.com/dnceng/internal/_build/results?buildId=2185416&view=logs&j=19992227-62fb-5b50-4e29-3b72bd33eea1&t=9281c869-17dc-5ef7-6bc6-1c1da071debd

@ali-turan7
Copy link
Contributor

It is because the diff comes after the commit.

@mthalman
Copy link
Member

mthalman commented May 23, 2023

So how do you verify the file is actually updated with the expected content?

@mthalman
Copy link
Member

Ok, looks good.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area-release-infra Release infrastructure owned by .NET Product Construction
Projects
Archived in project
Development

No branches or pull requests

5 participants