[WIP] [BUG]: An error occurred trying to start process '/agent/externals/node20_1/bin/node'

[Copilot]

Thanks for assigning this issue to me. I'm starting to work on it and will keep this PR's description up to date as I form a plan and make progress.

Original issue description:

What happened?

Hello,

We are experiencing inconsistent, occasional crashes in our pipelines. The error always says:

An error occurred trying to start process '/agent/externals/node20_1/bin/node' with working directory '/agent/_work/1/s'. No such file or directory

We have yaml pipelines with a few stages. These pipelines are mostly shared (through git) and run hundreds of times daily over multiple projects and repos. Most of the time they work correctly, but sometimes they crash. Maybe once in 10 to 100 runs, it is hard to tell an exact number.

I am not sure if it is a problem of a task or an agent, but the same issue occurs in different tasks, and on top of that it reports missing nodejs (which we are not using in that task) and a version of nodejs we are not using anywhere, I am leaning toward it being a problem with an agent or some other part of the system.

When it happens, it appears to always happen in the first task in a stage (not counting standard checkout tasks). We see in happening in:

  - script: |
      echo $PATH
      node --version
    displayName: 'Node.js info'
    continueOnError: true

jobs:
  - job: CheckPullRequest
    displayName: 'Check pull request'
    pool:
      name: pool-name-foo
      vmImage: ubuntu-latest
      workspace:
        clean: all
    steps:
      - download: none

      - checkout: self
        path: sources
        fetchDepth: 0
        clean: true
        persistCredentials: true

      - checkout: SharedPipelinesRepository
        path: shared-pipelines
        fetchDepth: 1
        
      - script: |
          echo $PATH
          node --version
        displayName: 'Node.js info'
        continueOnError: true

      - task: UseNode@1
        displayName: 'Install Node.js'
        inputs:
          version: '20.18.0'

      - task: AzureKeyVault@2
        inputs:
          connectedServiceName: 'foo'
          keyVaultName: 'bar'
          runAsPreJob: true
          secretsFilter: 'secret'

We even tried to create a separate agent pool and move a single pipeline to it (so we could rule out some different pipeline breaking the environments) but it happened anyway. That pipeline run 3 parallel stages, first one had node, second didn't, and the third had it available again. That is wild.

Those stages all had task: UseNode@1 as a first task (except checkout tasks). Here are timestamps from those UseNode tasks:

Stage A, Job A
2025-03-05T06:28:21.9568752Z Found tool in cache: node 20.18.0 x64
 
Stage B, Job B
2025-03-05T06:28:42.8821219Z ##[error]An error occurred trying to start process '/agent/externals/node20_1/bin/node' with working directory '/agent/_work/1/s'. No such file or directory
 
Stage C, Job AC
2025-03-05T06:29:06.4968152Z Found tool in cache: node 20.18.0 x64

We are trying to create a repro, but so far no luck. This is not happening on MS hosted agents.

Could you give us some hint what could be the root cause? Why is nodejs 20.1 needed to run - script: echo $PATH? Is it used by the system itself? What could cause it sometimes not being present?

I can add more info if needed.

Thank for any help!

Versions

Current agent version: '4.251.0'
Agent running as: 'AzDevOps'
Image: ubuntu-latest

Environment type (Please select at least one enviroment where you face this issue)

• Self-Hosted
• Microsoft Hosted
• VMSS Pool
• Container

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

No response

Version controll system

Git

Relevant log output

2025-03-11T00:16:33.8605603Z ##[section]Starting: Node.js info
2025-03-11T00:16:33.8621208Z ==============================================================================
2025-03-11T00:16:33.8621756Z Task         : Command line
2025-03-11T00:16:33.8622069Z Description  : Run a command line script using Bash on Linux and macOS and cmd.exe on Windows
2025-03-11T00:16:33.8622609Z Version      : 2.250.1
2025-03-11T00:16:33.8622928Z Author       : Microsoft Corporation
2025-03-11T00:16:33.8623299Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/command-line
2025-03-11T00:16:33.8623801Z ==============================================================================
2025-03-11T00:16:34.0871007Z ##[error]An error occurred trying to start process '/agent/externals/node20_1/bin/node' with working directory '/agent/_work/1/s'. No such file or directory
2025-03-11T00:16:34.0890327Z ##[section]Finishing: Node.js info


2025-03-11T00:16:34.0936432Z ##[section]Starting: Install Node.js
2025-03-11T00:16:34.0945645Z ==============================================================================
2025-03-11T00:16:34.0945964Z Task         : Use Node.js ecosystem
2025-03-11T00:16:34.0946154Z Description  : Set up a Node.js environment and add it to the PATH, additionally providing proxy support
2025-03-11T00:16:34.0946482Z Version      : 1.248.1
2025-03-11T00:16:34.0946659Z Author       : Microsoft Corporation
2025-03-11T00:16:34.0946875Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks
2025-03-11T00:16:34.0947120Z ==============================================================================
2025-03-11T00:16:34.2777399Z ##[error]An error occurred trying to start process '/agent/externals/node20_1/bin/node' with working directory '/agent/_work/1/s'. No such file or directory
2025-03-11T00:16:34.2787536Z ##[section]Finishing: Install Node.js

Fixes #5151.

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.