[LanguageWorker][Debugging]Function Host fails to restart language worker process on file edit if debugger is attached

[pavanagrawal123]

I'm launching the debugger from VSCode. The run time is v2, and language is Node.js.

Every time I make a change to the file, the debug process crashes and I have to start the function again, as well as the debug session.

• Function App version (1.0 or 2.0): 2.0
• Function App name: runnning locally

I'm on Windows!

Repro steps

1. Download VSCode
2. Install Node.js
3. Install the Azure Functions Runtime
4. Create a function
5. Launch a debug session
6. Make a change to function, save file, see crash.

Expected behavior

I don't expect the function runtime to crash.

Here's a sample log from when it happens:

info: Host.General[0]
     Restarting host.
info: Host.General[0]
     Building host: startup suppressed:False, configuration suppressed: False
info: Host.Startup[0]
     Reading host configuration file 'C:\volunteerManagement\host.json'
info: Host.Startup[0]
     Host configuration file read:
     {
       "version": "2.0"
     }
[10/1/2018 7:00:19 PM] Initializing Host.
[10/1/2018 7:00:19 PM] Host initialization: ConsecutiveErrors=0, StartupCount=2
[10/1/2018 7:00:19 PM] Starting JobHost
[10/1/2018 7:00:19 PM] Starting Host (HostId=desktoplglnm4m-1873661633, InstanceId=aada58ee-e906-47b5-bde3-669548211425, Version=2.0.12115.0, ProcessId=4536, AppDomainId=1, Debug=False, FunctionsExtensionVersion=)
[10/1/2018 7:00:19 PM] Starting language worker process:node  --inspect=5858 "C:\Users\Admin\AppData\Roaming\npm\node_modules\azure-functions-core-tools\bin\workers\node\dist/src/nodejsWorker.js" --host 127.0.0.1 --port 49634 --workerId 40137e5e-1db2-4007-9067-57bcd26ecfb2 --requestId 2fdb954b-ba34-4ac7-862c-d785b047d56b --grpcMaxMessageLength 134217728
[10/1/2018 7:00:19 PM] node process with Id=7328 started
[10/1/2018 7:00:19 PM] Generating 4 job function(s)
[10/1/2018 7:00:19 PM] Found the following functions:
[10/1/2018 7:00:19 PM] Host.Functions.getEventsForRegistration
[10/1/2018 7:00:19 PM] Host.Functions.getIfCurrent
[10/1/2018 7:00:19 PM] Host.Functions.registerMember[10/1/2018 7:00:19 PM] Starting inspector on 127.0.0.1:5858 failed: address already in use

[10/1/2018 7:00:19 PM] Host.Functions.signupEvent
[10/1/2018 7:00:19 PM]
[10/1/2018 7:00:19 PM] Language Worker Process exited.
[10/1/2018 7:00:19 PM] Host initialized (278ms)
[10/1/2018 7:00:19 PM] node exited with code 12
.
[10/1/2018 7:00:19 PM] Host started (288ms)
[10/1/2018 7:00:19 PM] Job host started
[10/1/2018 7:00:19 PM] Starting language worker process:node  --inspect=5858 "C:\Users\Admin\AppData\Roaming\npm\node_modules\azure-functions-core-tools\bin\workers\node\dist/src/nodejsWorker.js" --host 127.0.0.1 --port 49634 --workerId cc3e30e5-2c72-40f7-baa3-e685c0c2adfc --requestId ad1b93ad-e6aa-4021-a66c-1bd7cd8498e3 --grpcMaxMessageLength 134217728
[10/1/2018 7:00:19 PM] node process with Id=22756 started
[10/1/2018 7:00:19 PM] Stopping JobHost
[10/1/2018 7:00:19 PM] Job host stopped
info: Host.General[0]
     Host restarted.
[10/1/2018 7:00:19 PM] Language Worker Process exited.[10/1/2018 7:00:19 PM] Debugger listening on ws://127.0.0.1:5858/ac861383-518f-451b-affe-65d33b3c5d73

[10/1/2018 7:00:20 PM] node exited with code -1
.
[10/1/2018 7:00:20 PM] For help, see: https://nodejs.org/en/docs/inspector
[10/1/2018 7:00:20 PM] Worker cc3e30e5-2c72-40f7-baa3-e685c0c2adfc connecting on 127.0.0.1:49634
[10/1/2018 7:00:24 PM] Host lock lease acquired by instance ID '0000000000000000000000004C66A269'.

[mhoeger]

The root cause is issue #3415

[pragnagopa]

Are you seeing this when you are changing file and saving it while the debugger is attached?

[pavanagrawal123]

Yes, that's exactly where I'm seeing the issue

[graysonwebster]

I'm also seeing this issue.

[rhummelmose]

Any update on when this'll be fixed? Makes it hard to follow a real flow of work

[rhummelmose]

For anybody who may have an interest in how to get a decent workflow up and running in VSCode until this issue is resolved I've put the following together to allow me to automatically start/stop the functions host when I launch the debugger.

launch.json

{
    // Use IntelliSense to learn about possible attributes.
    // Hover to view descriptions of existing attributes.
    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
    "version": "0.2.0",
    "configurations": [
        {
            "type": "node",
            "request": "attach",
            "name": "Run Functions host and attach",
            "protocol": "inspector",
            "preLaunchTask": "Compile TS, Run Functions (Shell)",
            "postDebugTask": "Stop Functions",
            "outFiles": [ "{workspaceRoot}/TypeScriptFunction/index.js" ],
            "port": 5858
        },
        {
            "type": "node",
            "request": "attach",
            "name": "Attach",
            "protocol": "inspector",
            "outFiles": [ "{workspaceRoot}/TypeScriptFunction/index.js" ],
            "port": 5858
        }
    ]
}

tasks.json

{
  "version": "2.0.0",
  "tasks": [
    {
      "label": "Run Functions",
      "type": "shell",
      "command": "sh scripts/functions_host_background.sh",
      "problemMatcher": []
    },
    {
      "label": "Stop Functions",
      "type": "shell",
      "command": "tmux kill-session -t azure-functions-playground",
      "problemMatcher": []
    },
    {
      "label": "Compile TypeScript",
      "type": "shell",
      "command": "tsc --outDir TypeScriptFunction --sourceMap --lib es2015 TypeScriptFunction/index.ts",
      "problemMatcher": []
    },
    {
      "label": "Compile TS, Run Functions (Compound)",
      "type": "shell",
      "dependsOn": [
        "Compile TypeScript",
        "Run Functions"
      ],
      "problemMatcher": []
    },
    {
      "label": "Compile TS, Run Functions (Shell)",
      "type": "shell",
      "command": "tsc --outDir TypeScriptFunction --sourceMap --lib es2015 TypeScriptFunction/index.ts;sh scripts/functions_host_background.sh",
      "problemMatcher": []
    }
  ]
}

functions_host_background.sh

tmux kill-session -t azure-functions-playground
tmux start-server
tmux new-session -d -s azure-functions-playground -n host
tmux send-keys -t azure-functions-playground:host "func host start --language-worker -- \"--inspect=5858\"" C-m

You need tmux, but it makes life bearable..

Add the presentation options to your tasks to keep them out of sight.

I couldn't use the compound key because of this bug: microsoft/vscode#52737

[tsuga]

I have the same issue, and am hoping this will be fixed soon.

[IMAMBAKS]

I have the same issue... Please fix this asap...

[jeffhollan]

Adding a P1 as in addition to 'expectation' from JavaScript develop experience, believe it's needed for placeholders

[Wankishh]

Guys do we have any solution yet, I have the same error and on prod side, working with I/O operations including creating new multiline js files in a different folder from the functions keeps restarting the host.

[mhoeger]

@Wankishh - what's the scenario you are creating js files for? From our perspective this is an issue that affects people debugging their local code and not a production issue. Remember that function invocations should be regarded as stateless. If you are trying to preserve state between function invocations, you should explore Durable Functions.

[pavanagrawal123]

hi! I'm also experiencing this on MacOS with Python functions, any info available on Python files?

[fabiocav]

@jeffhollan the work to address this will be tracked by #3415. I'm assigning a P1 to that issue now so it will be properly triaged, but we should consider closing this as a duplicate as well.

[meds]

I may be way off but I found this works with tsc -w so that changing the code does not kill the process and the debugger remains attached (just not with workspaces where the debugger does not attach):

//tasks.json
    {
      "type": "func",
      "command": "npm start",
      "problemMatcher": "$func-watch",
      "isBackground": true,
      "dependsOn": "npm build",
    },
    },
    {
      "type": "shell",
      "label": "npm build",
      "command": "npm run build",
      "dependsOn": [
        "func: extensions install",
      ],
      "problemMatcher": "$tsc"
    },
//launch.json
    {
      "name": "Attach to Node Functions",
      "type": "node",
      "request": "attach",
      "port": 9228,
      "preLaunchTask": "func: host start"
    },

I've been using this config for a while, not sure how I stumbled on it but it works quite well, F5 in vscode then run tsc -w in a terminal to ensure the code live updates as I change it (shouldn't have to do that with javascript).

As I mentioend here: microsoft/vscode-azurefunctions#1517 the only issue is the vscode debugger will not attach to it if the proejct is running in a workspace so I'm forced to run the azure functions in a seperate vscode instance.

[osbornm]

Is there any follow up?

[KarFan-FS]

I can confirm that this is still happening.
OS: Windows10, AFunc Lang: Python (3.7.4)

ETA on fix, devs?

[macrozone]

this is a surprisingly bad developer experience, which is weird, because the vs extensions seems polished otherwise.

But having to kill the process all the time is just annoying.

[jawa-the-hutt]

I wanted to drop this info here as I have arrived at a mostly workable solution that allows me to use a Workspace with multiple Typescript function apps. The only downside is that when a Function app recompiles it does drop the debugger, but it's only a matter of a couple of seconds to reattach it based on all the configs below. I am using an npm package named tsc-watch to control the recompile. This allows me to only have one console open per Function app versus having to have two (one for the tsc -w and one for the actual function app).

Here are the configs I'm using and I would be interested to see if anyone can improve on them.

workspace config file:

{
  "folders": [
    {
      "name": "project-one",
      "path": "project-one"
    },
    {
      "name": "project-two",
      "path": "project-two"
    }
  ],
  "settings": {
    "debug.internalConsoleOptions": "neverOpen"
  },
  "launch": {
      "configurations": [],
      "compounds": [
        {
          "name": "Workspace - Attach all functions",
          "configurations": [
            "Attach Project-One Functions",
            "Attach Project-Two Functions"
          ]
        },
        {
          "name": "Workspace - Attach Project-One Functions",
          "configurations": [
            "Attach Project-One Functions"
          ]
        },
        {
          "name": "Workspace - Attach Project-Two Functions",
          "configurations": [
            "Attach Project-Two Functions"
          ]
        }
     ]
  }
}

package.json - this is essentially the same for all the Function app projects minus any actual other dependencies specific to your needs

{
  "name": "project-one",
  "version": "1.0.0",
  "description": "",
  "scripts": {
    "build": "tsc",
    "watch": "tsc-watch --onSuccess \"func start\"",
    "prestart": "npm run build && func extensions install",
    "start": "func host start",
    "build:production": "npm run prestart && npm prune --production",
    "test": "echo \"No tests yet...\""
  },
  "dependencies": {
  },
  "devDependencies": {
    "@azure/functions": "^1.2.2",
    "tsc-watch": "^4.2.9",
    "typescript": "^3.9.7"
  }
}

launch.json - notice I am setting a port here. this is important as you need to set a corresponding port in your local.settings.json file that i'll include below

{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Attach Project-One Functions",
      "type": "node",
      "protocol": "inspector",
      "request": "attach",
      "stopOnEntry": false,
      "port": 9001,
      "preLaunchTask": "npm start",
      "sourceMaps": true,
      "cwd": "${workspaceFolder}",
      "outFiles": ["${workspaceRoot}/dist/**/*.js"],
    }
  ]
}

local.settings.json -- Notice the NODE_OPTIONS value's port should match what is set in the launch.json file. Also, I've left the host option in as a way to config the function to start on a specific port since you would probably want multiple Function apps starting at the same time and they can't use the same port.

{
  "IsEncrypted": false,
  "Values": {
    "NODE_OPTIONS": "--inspect=9001",
    "FUNCTIONS_WORKER_RUNTIME": "node",
  },
  "Host": {
    "LocalHttpPort": 5001,
    "CORS": "*",
    "CORSCredentials": false
  }
}

tasks.json -- this is by far the biggest departure to norms, but it works. Notice the endPattern match on the npm start section. It looks for that pattern to exist in the console so that it will attach the debugger. Also notice the presentation options for each section. By grouping them this way, I can leave the watcher in the same console as the Function app and not have to move between the two. If you setup another function, then you need to change the group item to a different name so that all the work for that Function app is done in its own console. All you have to do then is hit the drop down in the top right of the console area of vscode to switch between your different Function apps to see their console.

{
  "version": "2.0.0",
  "tasks": [
    // {
    //   "type": "func",
    //   "command": "host start",
    //   "problemMatcher": "$func-watch",
    //   "isBackground": true,
    //   "dependsOn": "npm build",
    // },
    {
      "type": "shell",
      "label": "npm start",
      "dependsOn": "npm build",
      "command": "npm run watch",
      "isBackground": true,
      "problemMatcher": [
        {
          "pattern": [
            {
              "regexp": ".",
              "file": 1,
              "location": 2,
              "message": 3
            }
          ],
          "background": {
            "activeOnStart": true,
            "beginsPattern": ".",
            "endsPattern": "Host lock lease acquired by instance ID"
          }
        }
      ],
      "presentation": {
        "echo": true,
        "reveal": "silent",
        "focus": false,
        "panel": "shared",
        "showReuseMessage": true,
        "clear": false,
        "group": "projectone"
      },
    },
    {
      "type": "shell",
      "label": "npm build",
      "command": "npm run build",
      "dependsOn": "npm install",
      "problemMatcher": "$tsc",
      "presentation": {
        "echo": true,
        "reveal": "always",
        "focus": true,
        "panel": "shared",
        "showReuseMessage": true,
        "clear": false,
        "group": "projectone"
      }
    },
    {
      "type": "shell",
      "label": "npm install",
      "command": "npm install",
      "presentation": {
        "echo": true,
        "reveal": "always",
        "focus": true,
        "panel": "shared",
        "showReuseMessage": true,
        "clear": false,
        "group": "projectone"
      }
    },
    {
      "type": "shell",
      "label": "npm prune",
      "command": "npm prune --production",
      "dependsOn": "npm build",
      "problemMatcher": []
    }
  ]
}

Finally and again: If you change a ts file and the watcher recompiles the Function app, then it will detach the debugger. You then just need to reattach it via the usual methods in vscode. Hitting F5 may not do this correctly on it's own so you may have to navigate to Debug area in vscode and hit the drop down to manually reattach to just that Function app. If you have multiple Function apps running in the same workspace, then the other Function apps won't detach, so you would only need to reattach to the single Function app.

The caveat to this is if you have a shared folder that contains code that multiple Function apps are looking at, then if you save a ts file in the shared folder and your normal config causes all Function apps to recompile, then it will detach the debugger on all Function apps so you would have to reattach all them again. This is why I have included the option in the workspace file to attach to all of them at the same time so that you only have to select that option and all Function apps will reattach. This is really handy once you have more than a few Function apps you are working with in the workspace.

[fabiocav]

Assigning to sprint 85 for initial investigation.

[fabiocav]

@stefanushinardi were you planning on scheduling some time to discuss this?

[stefanushinardi]

@fabiocav I was under the impression that one of the devs would take a look first, but I will schedule a time slot next to discuss the issue. Thanks!

[stefanushinardi]

After a discussion with @fabiocav @anthonychu @pragnagopa and @mhoeger, we believe that the issue occurs when there is an overlap while recycling processes, that results in the new process fails to open the debugging port as it is still being held by the process that is being brought down.

The current approach that we are thinking is to remove the overlapping condition when running the host in development environments.

[mhoeger]

Didn't get much time to work on this larger item with being on-call, pushing to next sprint. Apologies all, still definitely a priority :)

[fabiocav]

Work for this is currently in progress, but we expect completion to happen next sprint. Updated the target to reflect that.

[mhoeger]

This issue is resolved on the azure-functions-host side with host versions > 3.0.15149 (currently, this means our next release).

However, this will not automatically enable a continuous debugging experience on file change. Changes also need to be made to the VS Code Extension, and those can be tracked here: microsoft/vscode-azurefunctions#781

In addition to a resolution for microsoft/vscode-azurefunctions#781, for Python functions, this change needs to be made:
microsoft/vscode-azurefunctions#2576

Closing as issue described in title, "Function Host fails to restart language worker process on file edit if debugger is attached," is resolved.