Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG]: #4268

Closed
1 of 4 tasks
svbrburgman opened this issue May 9, 2023 · 19 comments
Closed
1 of 4 tasks

[BUG]: #4268

svbrburgman opened this issue May 9, 2023 · 19 comments
Assignees

Comments

@svbrburgman
Copy link

svbrburgman commented May 9, 2023

What happened?

warning resulting in an error later in the pipeline with agent 3.220.1
2023-05-09T10:09:43.4362072Z ##[warning]Artifact does not contain any files to download.
2023-05-09T10:09:43.4381870Z 0 placed file(s): 0 downloaded, 0 empty

The artifact does exist and job works with agent 3.220.0

downgraded the agent from 3.220.1 (pre-release) to 3.220.0 (latest release).

The job is using the xldeploy plugin and should upload the build artifacts to xldeploy.

Versions

Azure DevOps Server 2020 Update 1.2
agent 3.220.1

Environment type (Please select at least one enviroment where you face this issue)

  • Self-Hosted
  • Microsoft Hosted
  • VMSS Pool
  • Container

Azure DevOps Server type

Azure DevOps Server (Please specify exact version in the textbox below)

Azure DevOps Server Version (if applicable)

Azure DevOps Server 2020 Update 1.2

Operation system

Windows 2016 (10.0.14393)

Version controll system

No response

Relevant log output

2023-05-09T10:09:43.4362072Z ##[warning]Artifact does not contain any files to download.
2023-05-09T10:09:43.4381870Z 0 placed file(s): 0 downloaded, 0 empty


##[error]Task ended with an exception: Error: No package was found using search pattern 'c:\Agent\ws\r16\a\xxxxx\xxx-dar *\dar-xxxxx *.dar'.
@ismayilov-ismayil
Copy link
Contributor

Hi @svbrburgman, thanks for reporting. Could you please share agent logs to take a look? You can retrieve them by following docs . You would also need to set up 'Agent.Diagnostic' pipeline variable as 'true' before pipeline run.

@Ondrashx
Copy link

Hi, we run into similar issue yesterday. What helped was to rename the artifactName of the published task, so it does not contain any dots. So, e.g. from ENV.Release -> ENVRelease. The download task than was able to download all the files in artifact.

@kauemendes
Copy link

I believe I open a repeat bug: #4270
Can´t say for sure, most definetely we are facing issues with download artifact. I am not sure if is related to the timeout into the MetadataProvider added in the last 3.220.1 version. But I don´t have enough knowloadge into C# and into this project to full undestand what impacts this can produce in such change.

@kraduk
Copy link

kraduk commented May 11, 2023

we are seeing this issue downloading artifacts with the DownloadPipelineArtifact@2. Nothing changed in the multiple pipelines it just failed on the agent update

@MATTHEWTAYLOR1995
Copy link

We've had the same thing. I've just downgraded the agents to 3.220.0 until MS have a fix as it was breaking pipelines left, right and center.

@epmenard
Copy link

Same issue here. Download build artifact creates an empty folder and cannot retrieve the files when the artifact contains a dot '.' e.g. project_alpha.build

The problem appeared anywhere between Tuesday, May 9th 16:25 EDT and Wednesday, May 10th 11:18 EDT. Removing dot '.' from our published artifact name does bypass the introduced flaw.

@epmenard
Copy link

epmenard commented May 11, 2023

@ismayilov-ismayil : here are two logs. The first with a dot-named artifact showing only a single match (the folder) and the second targeting the same build pipeline with a dotless artifact name producing 2 matches (the folder and it's zip file).

artifact name: lob.core

2023-05-11T15:20:33.4363886Z ##[debug]Evaluating condition for step: 'Download Assembly'
2023-05-11T15:20:33.4364611Z ##[debug]Evaluating: SucceededNode()
2023-05-11T15:20:33.4364829Z ##[debug]Evaluating SucceededNode:
2023-05-11T15:20:33.4365250Z ##[debug]=> True
2023-05-11T15:20:33.4365505Z ##[debug]Result: True
2023-05-11T15:20:33.4365751Z ##[section]Starting: Download Assembly
2023-05-11T15:20:33.4370542Z ==============================================================================
2023-05-11T15:20:33.4370662Z Task         : Download build artifacts
2023-05-11T15:20:33.4370749Z Description  : Download files that were saved as artifacts of a completed build
2023-05-11T15:20:33.4370848Z Version      : 1.220.0
2023-05-11T15:20:33.4370909Z Author       : Microsoft Corporation
2023-05-11T15:20:33.4370994Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/download-build-artifacts
2023-05-11T15:20:33.4371105Z ==============================================================================
2023-05-11T15:20:33.7903862Z Download from the specified build: #876524
2023-05-11T15:20:33.7916969Z ##[debug]Processed: ##vso[task.setvariable variable=BuildNumber;issecret=False;]876524
2023-05-11T15:20:33.7917731Z Download artifact to: /agent/_work/2/assembly
2023-05-11T15:20:34.3766656Z Pattern: **
2023-05-11T15:20:34.3776204Z Minimatch patterns: [**]
2023-05-11T15:20:34.3776439Z Applying include pattern against original list.
2023-05-11T15:20:34.3984529Z 1 matches
2023-05-11T15:20:34.4230625Z 1 final results
2023-05-11T15:20:34.4263123Z Start downloading FCS artifact- lob.core
2023-05-11T15:20:34.4386893Z Downloading artifact finished.
2023-05-11T15:20:34.4435275Z ##[section]Finishing: Download Assembly

artifact name: build

2023-05-11T15:31:37.7956161Z ##[section]Starting: Download Assembly
2023-05-11T15:31:37.7961367Z ==============================================================================
2023-05-11T15:31:37.7961504Z Task         : Download build artifacts
2023-05-11T15:31:37.7961590Z Description  : Download files that were saved as artifacts of a completed build
2023-05-11T15:31:37.7961689Z Version      : 1.220.0
2023-05-11T15:31:37.7961750Z Author       : Microsoft Corporation
2023-05-11T15:31:37.7961832Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/download-build-artifacts
2023-05-11T15:31:37.7961942Z ==============================================================================
2023-05-11T15:31:38.1509955Z Download from the specified build: #878144
2023-05-11T15:31:38.1528557Z Download artifact to: /agent/_work/1/assembly
2023-05-11T15:31:38.7185967Z Pattern: **
2023-05-11T15:31:38.7194139Z Minimatch patterns: [**]
2023-05-11T15:31:38.7210838Z Applying include pattern against original list.
2023-05-11T15:31:38.7418332Z 2 matches
2023-05-11T15:31:38.7639181Z 2 final results
2023-05-11T15:31:38.7687456Z Start downloading FCS artifact- build
2023-05-11T15:31:38.7735380Z Using default max parallelism.
2023-05-11T15:31:38.7748578Z Max dedup parallelism: 192
2023-05-11T15:31:38.8732363Z Downloading: /agent/_work/1/assembly/build/878144.zip
2023-05-11T15:31:39.1024193Z Downloading artifact finished.
2023-05-11T15:31:39.1031854Z ##[section]Finishing: Download Assembly

@LiliaSabitova
Copy link
Contributor

@svbrburgman @epmenard @Ondrashx @kauemendes @kraduk @MATTHEWTAYLOR1995
Could you please provide example pipeline YAML-file?
Because we cannot reproduce the issue.

@Ondrashx
Copy link

@LiliaSabitova
Sorry can not provide the whole yaml file :( , but here is the change which fixed the issue:

image

@kauemendes
Copy link

kauemendes commented May 12, 2023

Sure @LiliaSabitova, I can share two versions, a versions that stopped working and a version that receives a workaround:

A version that before the update was perfectly working as is:

stages:
- stage: ci
  jobs:
    - job: build
      displayName: "buildxpto"
      workspace:
        clean: all
      pool:
        name: "ubuntu-20.04"
      continueOnError: false
      steps:
        - checkout: self
          persistCredentials: true

        - powershell: |
            $length = "$(Build.BuildId)".length
            $MajorVersion="$(Build.BuildId)".substring(0, 1)
            $MinorVersion="$(Build.BuildId)".substring(1, 1)
            $PatchVersion="$(Build.BuildId)".substring($length -4)
            Write-Host "##vso[build.updatebuildnumber]$MajorVersion.$MinorVersion.$PatchVersion"
          displayName: 'update buildnumber: version Major.Minor.Patch alike'
          enabled: true
        
        - task: ArchiveFiles@2
          displayName: 'commpress: something'
          inputs:
            rootFolderOrFile:       $(build.sourcesdirectory)/xxxx #Something to Compress
            includeRootFolder:      false
            archiveType:            'zip'
            archiveFile:            $(build.artifactstagingdirectory)/$(Build.BuildNumber).zip
            replaceExistingArchive: true
          enabled: true

        - task: PublishBuildArtifacts@1
          displayName: 'publish: $(Build.BuildNumber)'
          inputs:
            ArtifactName:  '$(Build.BuildNumber)'
            PathtoPublish: '$(Build.ArtifactStagingDirectory)'
          enabled: true
          
- stage: cd
  dependsOn: ci
  jobs:
  - deployment: cd_deploy
    displayName: cd_deploy
    pool:
      name: "ubuntu-20.04"
    continueOnError: 'false'
    environment: some-env
    strategy:
      runOnce:
        deploy:
          steps:
          - checkout: self
            persistCredentials: true
          
          - bash: |
              echo "Check if File Exists"
              ls -la $(Pipeline.Workspace) 
              ls -la $(Pipeline.Workspace)/$(Build.BuildNumber) 
              unzip $(Pipeline.Workspace)/$(Build.BuildNumber)/$(Build.BuildNumber).zip
            displayName: "verify: files"

And the version with workaround proposed by @epmenard and demonstrated by @Ondrashx (I believe)

stages:
- stage: ci
  jobs:
    - job: build
      displayName: "buildxpto"
      workspace:
        clean: all
      pool:
        name: "ubuntu-20.04"
      continueOnError: false
      steps:
        - checkout: self
          persistCredentials: true

        - powershell: |
            $length = "$(Build.BuildId)".length
            $MajorVersion="$(Build.BuildId)".substring(0, 1)
            $MinorVersion="$(Build.BuildId)".substring(1, 1)
            $PatchVersion="$(Build.BuildId)".substring($length -4)
            Write-Host "##vso[build.updatebuildnumber]$MajorVersion.$MinorVersion.$PatchVersion"
          displayName: 'update buildnumber: version Major.Minor.Patch alike'
          enabled: true
        
        - task: ArchiveFiles@2
          displayName: 'commpress: something'
          inputs:
            rootFolderOrFile:       $(build.sourcesdirectory)/xxxx #Something to Compress
            includeRootFolder:      false
            archiveType:            'zip'
            archiveFile:            $(build.artifactstagingdirectory)/$(Build.BuildNumber).zip
            replaceExistingArchive: true
          enabled: true

        - task: PublishBuildArtifacts@1
          displayName: 'publish: file' 
          inputs:
            ArtifactName:  'file' #Changed to file instead of name dot-separated
            PathtoPublish: '$(Build.ArtifactStagingDirectory)'
          enabled: true
          
- stage: cd
  dependsOn: ci
  jobs:
  - deployment: cd_deploy
    displayName: cd_deploy
    pool:
      name: "ubuntu-20.04"
    continueOnError: 'false'
    environment: some-env
    strategy:
      runOnce:
        deploy:
          steps:
          - checkout: self
            persistCredentials: true
          
          - bash: |
              echo "Check if File Exists"
              ls -la $(Pipeline.Workspace) 
              ls -la $(Pipeline.Workspace)/file #Changed to file
              unzip $(Pipeline.Workspace)/file/$(Build.BuildNumber).zip
            displayName: "verify: files"

The logic of this pipeline is irrelevant, it serves only to demonstrate, please be aware of the fact that a file is being compressed and published as a build artifact and then the pipeline itself does the Download Artifact (automatic), but with dots in the publish bucket the the folder comes empty, and without dots we have the artifact.

Another question is, how a pre-release version such as 3.220.1 is automatically rollout? Isn´t latest official version yet.

@svbrburgman
Copy link
Author

We are using an onpremise environment. And 3.220.0 was scheduled to be deployed.
From 3.220.0 to 3.220..1 suggested just a minor update and didn't show any issues on the staging environment. So choosing the pre-release was a bit opportune.

As many reported, our artifact also has serveral dot's in it's name. (version x.y.z)

@LiliaSabitova
Copy link
Contributor

Thanks all for providing the example pipelines and details!
We were able to reproduce the issue and define the root cause and we're working on fixing it.

@markatky46
Copy link

We are also seeing this issue. We are unable to downgrade the agent as we use VMSS instances that are ephemeral. We identified a quick fix without downgrading the agent:
ISSUE:-
task: DownloadBuildArtifacts@1
This task is downloading the container but its not downloading the contents in the container ie., zip file. Hence facing the path error.

FIX:- download task version to 0: DownloadBuildArtifacts@0

May i ask also why a pre-release version has been rolled out and being consumed like a general release by my cloud service? Is there any way for us to select only general release versions?

@LiliaSabitova
Copy link
Contributor

Hi @markatky46 could you please advise what is your organization name?
And I will check why you got pre-release version.
For security reasons you can e-mail me your organization name: v-lisabitova@microsoft.com

@markatky46
Copy link

@LiliaSabitova i emailed you. Were you able to look into it?

@LiliaSabitova
Copy link
Contributor

Hi @Ondrashx @svbrburgman @kauemendes @kraduk @MATTHEWTAYLOR1995 @epmenard
Could you please also share the names of your ADO organizations.
For security reasons you can e-mail your organization names here: v-lisabitova@microsoft.com

Need them to define which changes in the vss api package caused the issue.
Thank you.

@MATTHEWTAYLOR1995
Copy link

MATTHEWTAYLOR1995 commented May 18, 2023

There's been a new agent version pushed 3.220.2

We've tested it and it's resolved our problem with downloading artifacts.

@kraduk
Copy link

kraduk commented May 18, 2023

same here

@LiliaSabitova
Copy link
Contributor

We've rolled out the related changes for this issue.
Feel free to reopen the ticket in case you face any problem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

9 participants