[BUG]: Self-hosted agent fails to mount whole work directory for second run in container with checkout path

[asaril]

What happened?

Hi,
I am trying to build a project using the west build system in a container-based pipeline on self-hosted agents.

Due to the way west is operating, the sources need to be cloned one level deeper, as west will create additional files next to the repository.
For this, I use a

      - checkout: self
        path: s/west_stage/sources

step. In this case, only one repository is checked out for this job, no multi-repo checkout. There are other repo resources defined in the pipeline, but this is the only checkout step in this job.

The resulting directory structure is correct (sources located at /__w/20/s/west_stage/sources in the container, and in /home/azure_cloud/adc_agent_03/_work/20/s/west_stage on the host).

However, this only works for the first run. If the work directory exists already on the agent host, it will detect the west_stage/sources as default working directory, and try to mount west_stage only for the container.

The paths are still correct, but the level that is mounted seems to be tied to $(Build.SourcesDirectory)/.. instead of the $(Pipeline.Workspace).

This leads to the binary and artifact staging directories being unavailable in the container, breaking the build.

[edit: this was one level too deep still, as a one-repo clone will use /s directly]
As a final step, I retested with a checkout path of just s/sources (I would like to avoid that, as west will pollute the work dir /s with this).
This setup still results in the same issue.
For the second run, only the s/ directory will be mounted, instead of the whole work directory.
[/edit]

Versions

Agent Version: 3.227.2
Agent OS: Linux

Environment type (Please select at least one enviroment where you face this issue)

• Self-Hosted
• Microsoft Hosted
• VMSS Pool
• Container

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

No response

Version controll system

git

Relevant log output

First run extracts:
2023-10-19T08:49:53.1679168Z Prepare build directory.
2023-10-19T08:49:53.1896870Z ##[debug]Creating build directory: '/home/azure_cloud/adc_agent_03/_work/20'
2023-10-19T08:49:53.1899581Z ##[debug]Delete existing artifacts directory: '/home/azure_cloud/adc_agent_03/_work/20/a'
2023-10-19T08:49:53.1902430Z ##[debug]Creating artifacts directory: '/home/azure_cloud/adc_agent_03/_work/20/a'
2023-10-19T08:49:53.1905002Z ##[debug]Delete existing test results directory: '/home/azure_cloud/adc_agent_03/_work/20/TestResults'
2023-10-19T08:49:53.1906811Z ##[debug]Creating test results directory: '/home/azure_cloud/adc_agent_03/_work/20/TestResults'
2023-10-19T08:49:53.1909270Z ##[debug]Creating binaries directory: '/home/azure_cloud/adc_agent_03/_work/20/b'
2023-10-19T08:49:53.1911581Z ##[debug]Creating source directory: '/home/azure_cloud/adc_agent_03/_work/20/s'
2023-10-19T08:49:53.1968456Z Set build variables.
[...]
2023-10-19T08:49:57.2533544Z ##[debug]Default Working Directory /__w/20/s
2023-10-19T08:49:57.2533813Z ##[debug]Working Directory /__w/20
2023-10-19T08:49:57.2534172Z ##[debug]Mount Working Directory /home/azure_cloud/adc_agent_03/_work/20
[...]
2023-10-19T08:49:57.2971487Z ##[command]/usr/bin/docker create --name [redacted] --label e6ad1e --network vsts_network_[redacted]  -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/home/azure_cloud/adc_agent_03/_work/20":"/__w/20" -v "/home/azure_cloud/adc_agent_03/_work/_temp":"/__w/_temp" -v "/home/azure_cloud/adc_agent_03/_work/_tasks":"/__w/_tasks" -v "/home/azure_cloud/adc_agent_03/_work/_tool":"/__t" -v "/home/azure_cloud/adc_agent_03/externals":"/__a/externals":ro -v "/home/azure_cloud/adc_agent_03/_work/.taskkey":"/__w/.taskkey" [redacted container image] "/__a/externals/node/bin/node" -e "setInterval(function(){}, 24 * 60 * 60 * 1000);"


-----------------------------------------------------------------------------------------


Second run extracts:
2023-10-19T08:51:36.1483553Z Prepare build directory.
2023-10-19T08:51:36.1743556Z ##[debug]Delete existing artifacts directory: '/home/azure_cloud/adc_agent_03/_work/20/a'
2023-10-19T08:51:36.1746566Z ##[debug]Deleting artifacts directory: '/home/azure_cloud/adc_agent_03/_work/20/a'
2023-10-19T08:51:36.1775724Z ##[debug]Creating artifacts directory: '/home/azure_cloud/adc_agent_03/_work/20/a'
2023-10-19T08:51:36.1778818Z ##[debug]Delete existing test results directory: '/home/azure_cloud/adc_agent_03/_work/20/TestResults'
2023-10-19T08:51:36.1780774Z ##[debug]Deleting test results directory: '/home/azure_cloud/adc_agent_03/_work/20/TestResults'
2023-10-19T08:51:36.1810315Z ##[debug]Creating test results directory: '/home/azure_cloud/adc_agent_03/_work/20/TestResults'
2023-10-19T08:51:36.1870870Z Set build variables.
[...]
2023-10-19T08:51:40.2474587Z ##[debug]Default Working Directory /__w/20/s/west_stage/sources
2023-10-19T08:51:40.2474959Z ##[debug]Working Directory /__w/20/s/west_stage
2023-10-19T08:51:40.2475457Z ##[debug]Mount Working Directory /home/azure_cloud/adc_agent_03/_work/20/s/west_stage
[...]
2023-10-19T08:51:40.2934404Z ##[command]/usr/bin/docker create --name [redacted] --label e6ad1e --network vsts_network_[redacted]  -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/home/azure_cloud/adc_agent_03/_work/20/s/west_stage":"/__w/20/s/west_stage" -v "/home/azure_cloud/adc_agent_03/_work/_temp":"/__w/_temp" -v "/home/azure_cloud/adc_agent_03/_work/_tasks":"/__w/_tasks" -v "/home/azure_cloud/adc_agent_03/_work/_tool":"/__t" -v "/home/azure_cloud/adc_agent_03/externals":"/__a/externals":ro -v "/home/azure_cloud/adc_agent_03/_work/.taskkey":"/__w/.taskkey" [redacted container id] "/__a/externals/node/bin/node" -e "setInterval(function(){}, 24 * 60 * 60 * 1000);"


--------------------------------------------------------------------------

Variables in container:
they are the same for both runs, and also similar on the host (adapted in the real path)

echo "Pipeline.Workspace             = /__w/20"
echo "Build.StagingDirectory         = /__w/20/a"
echo "Build.SourcesDirectory         = /__w/20/s/west_stage/sources"
echo "Build.Repository.LocalPath     = /__w/20/s/west_stage/sources"
echo "Build.BinariesDirectory        = /__w/20/b"
echo "Build.ArtifactStagingDirectory = /__w/20/a"
echo "Agent.WorkFolder               = /__w"
echo "Agent.ToolsDirectory           = /__t"
echo "Agent.TempDirectory            = /__w/_temp"
echo "Agent.HomeDirectory            = /__a"
echo "Agent.BuildDirectory           = /__w/20"
echo "System.DefaultWorkingDirectory = /__w/20/s/west_stage/sources"

[DmitriiBobreshev]

Hi @asaril, thanks for the feedback, we'll take a look

[asaril]

I have tried three more cases:

• Using no path: -> works as intended, code gets checked out to __w/20/s, and second run uses the whole mount dir. (This does not confirm nor disprove the bug, as __w/20/s/.. is the same as the Pipeline.Workspace. It also will lead to west placing files directly into the workspace, which technically works, but seems a bit "unclean".
• Using a path: s2 -> also works. So it seems the issue is not directly related to the fact of stating a path: or not, but rather to the path being deeper than one level under the work dir
• Using a multi-repo checkout with one path: s/west_stage/sources and one path: s/source2 -> works. Default Working Directory and Mount Working Directory get set to __w/20 for this. -> So a possible workaround would be, to clone any dummy repo to have the job use multi-repo checkout instead.

[DenisRumyantsev]

@asaril can you confirm that this issue is fixed?

[DenisRumyantsev]

@asaril I have used this pipeline to verify that this issue is fixed —

trigger: none

pool: <Self-Hosted Agent Pool>

resources:
  containers:
  - container: box
    image: <image:tag>

steps:
- checkout: self
  path: sources/project/repo

- bash: |
    cd ../..
    touch ./$(build.buildid).label
  target: box

This issue is fixed. I am closing it.