[BUG] npx failed to install package and exec, when the user HOME dir is a symlink #3431

zthxxx · 2021-06-17T07:10:16Z

Is there an existing issue for this?

I have searched the existing issues

Current Behavior

I have't globally or locally install serve, and the npx serve command is first command I executed after npm i -g npm@7, so I don't have any npm/npx cache.

$ npx serve
sh: 1: serve: not found

$ npx create-react-app
sh: 1: create-react-app: not found

$ npx pm2
sh: 1: pm2: not found

Expected Behavior

$ npx serve

   ┌───────────────────────────────────┐
   │                                   │
   │   Serving!                        │
   │                                   │
   │   Local:  http://localhost:5000   │
   │                                   │
   └───────────────────────────────────┘




$ npx create-react-app
Please specify the project directory:
  create-react-app <project-directory>

For example:
  create-react-app my-react-app

Run create-react-app --help to see all options.



$ npx pm2

                        -------------

__/\\\\\\\\\\\\\____/\\\\____________/\\\\____/\\\\\\\\\_____
 _\/\\\/////////\\\_\/\\\\\\________/\\\\\\__/\\\///////\\\___
  _\/\\\_______\/\\\_\/\\\//\\\____/\\\//\\\_\///______\//\\\__
   _\/\\\\\\\\\\\\\/__\/\\\\///\\\/\\\/_\/\\\___________/\\\/___
    _\/\\\/////////____\/\\\__\///\\\/___\/\\\________/\\\//_____
     _\/\\\_____________\/\\\____\///_____\/\\\_____/\\\//________
      _\/\\\_____________\/\\\_____________\/\\\___/\\\/___________
       _\/\\\_____________\/\\\_____________\/\\\__/\\\\\\\\\\\\\\\_
        _\///______________\///______________\///__\///////////////__


                          Runtime Edition

        PM2 is a Production Process Manager for Node.js applications
                     with a built-in Load Balancer.
       ......

Steps To Reproduce

make sure user HOME (~) dir is a symlink (like mount on some other disk)
clear npx install cache with remove ~/.npm/_npx/
use npm7 to exec npx serve

# in this demo case, HOME dir at /home/username is link to /data00/home/username
# I use those command to show you
$ echo ~ && echo $HOME
/home/username
/home/username

$ ls -l ~
lrwxrwxrwx 1 root root 26 Dec 10  2020 /home/username -> /data00/home/username

$ realpath ~
/data00/home/username


# clear npx install cache
$ rm -rf ~/.npm/_npx/


# direct execution npx will be failed !
$ npx serve
sh: 1: serve: not found


$ npx create-react-app
sh: 1: create-react-app: not found


# -------------------------------------------
#
# show more info of npx install cache below

$ ls -l ~/.npm/_npx/
drwxr-xr-x 2 username username 4096 Jun 17 14:52 aab42732f01924e5


# "aab42732f01924e5" is sha512 prefix of "serve@12.0.0" (sourcecode in  https://github.com/npm/cli/blob/v7.17.0/node_modules/libnpmexec/lib/cache-install-dir.js)
#
# and you can see there is no `node_modules` dir, serve not installed !

$ ls -l ~/.npm/_npx/aab42732f01924e5
-rw-r--r-- 1 username username    50 Jun 17 14:52 package.json
-rw-r--r-- 1 username username 52211 Jun 17 14:52 package-lock.json


$ cat ~/.npm/_npx/aab42732f01924e5/package.json
{
  "dependencies": {
    "serve": "12.0.0"
  }
}



# -------------------------------------------
# in this case, I identified the problem as  `user HOME home is symlink` by make `HOME` var to it realpath, and npx will run successful.

$ HOME=`realpath ~` npx serve

   ┌───────────────────────────────────┐
   │                                   │
   │   Serving!                        │
   │                                   │
   │   Local:  http://localhost:5000   │
   │                                   │
   └───────────────────────────────────┘

^C
INFO: Gracefully shutting down. Please wait...


$ export HOME=`realpath ~`
$ npx create-react-app
Please specify the project directory:
  create-react-app <project-directory>

For example:
  create-react-app my-react-app

Run create-react-app --help to see all options.



# then you can see `node_modules` in  "aab42732f01924e5" dir
$ ls -l ~/.npm/_npx/aab42732f01924e5
drwxr-xr-x 85 username username  4096 Jun 17 15:05 node_modules
-rw-r--r--  1 username username    50 Jun 17 14:52 package.json
-rw-r--r--  1 username username 76823 Jun 17 15:05 package-lock.json

Environment

OS: Debian 9.11
Node: v14.15.1 (by debug in npm, i think this bug is not about node version)
npm: 7.17.0

The text was updated successfully, but these errors were encountered:

zthxxx · 2021-06-17T07:25:34Z

self troubleshooting:

I found the different logic that handles whether the HOME dir is a symlink is in arborist/build-ideal-tree.js _rootNodeFromPackage method

but it is not exclusively belong to npx, just common npm install algorithm.

https://github.com/npm/cli/blob/v7.17.0/node_modules/@npmcli/arborist/lib/arborist/build-ideal-tree.js#L357-L364
in Step.1 at line 363 and 364, this.path maybe /home/username/.npm/_npx/xxx but const real is /data00/home/username/.npm/_npx/xxx

it will make this.idealTree and this.actualTree to ArboristLink rather than correct ArboristNode in reify step,
then arborist Diff.calculate() will get incorrect diff result without missing diff children property.

https://github.com/npm/cli/blob/v7.17.0/node_modules/%40npmcli/arborist/lib/arborist/reify.js#L317-L324

in Step.2, the incorrect Arborist this.path is from this.option.path which is from cacheInstallDir({ cache: npm.flatOptions.cache })

https://github.com/npm/cli/blob/v7.17.0/node_modules/libnpmexec/lib/index.js#L127-L132

and npm.flatOptions.cache default value is from definitions.js

https://github.com/npm/cli/blob/v7.17.0/lib/utils/config/definitions.js#L313-L314

https://github.com/npm/cli/blob/v7.17.0/lib/utils/config/definitions.js#L81-L83

there is the source of not realpath ~

if npm run scripts in package.json which has npx, (e.g. npm run server) needs to set unset npm_config_cache in script like this:

  ...
  "scripts": {
    "server": "unset npm_config_cache && HOME=`realpath ~` npx serve",
    ...
  },
  ...

due to npm.flatOptions.cache will load from npm.data.get('env').data.cache, which is load from env var npm_config_cache, but it set by npm with not realpath before npx.

https://github.com/npm/cli/blob/v7.17.0/node_modules/@npmcli/config/lib/index.js#L346-L358

…symlink, see npm/cli#3431 detail issue: npm/cli#3431

zthxxx · 2021-06-17T10:06:16Z

npm/libnpmexec#6

try fix by make cacheInstallDir always be a realpath

…symlink, see npm/cli#3431 detail issue: npm/cli#3431

isaacs · 2021-06-17T17:48:26Z

Here's the root cause fix: https://gist.github.com/isaacs/c02d6c2f72cbbbb0c7c5be856d8bac8a

Repro cases: https://gist.github.com/isaacs/3e61667df31f750f88030eb16ed9191a

zthxxx · 2021-06-17T20:11:23Z

Basically, we should be doing this always, not just for global installs. Not sure why we had that restriction

I have read the diff.js, but also not sure why need judge only global install, maybe npx default to install in global node_modules when npm@v6?

This expands the behavior that was previously only done for global roots, but should really be done for _any_ root dir that is not accessed via its realpath. Fixes: npm/cli#3431

isaacs · 2021-06-18T23:46:07Z

Will be fixed when npm/arborist#294 lands.

isaacs · 2021-06-18T23:46:57Z

I have read the diff.js, but also not sure why need judge only global install, maybe npx default to install in global node_modules when npm@v6?

npx in npm v6 didn't use any of this code. npm v7 is a very different animal.

zthxxx · 2021-06-28T06:59:16Z

thanks, I have confirmed it fixed in npm@7.19.0

zthxxx added Bug thing that needs fixing Needs Triage needs review for next steps Release 7.x work is associated with a specific npm 7 release labels Jun 17, 2021

zthxxx changed the title ~~[BUG] npx failed to install package~~ [BUG] npx failed to install package and exec, when the user HOME dir is a symlink Jun 17, 2021

zthxxx added a commit to zthxxx/libnpmexec that referenced this issue Jun 17, 2021

fix: fix npx failed to install package and exec, when cache dir is a …

cd78006

…symlink, see npm/cli#3431 detail issue: npm/cli#3431

zthxxx mentioned this issue Jun 17, 2021

fix: fix npx failed to install package and exec, when cache dir is a symlink npm/libnpmexec#6

Open

zthxxx added a commit to zthxxx/libnpmexec that referenced this issue Jun 17, 2021

fix: fix npx failed to install package and exec, when cache dir is a …

4c124c8

…symlink, see npm/cli#3431 detail issue: npm/cli#3431

isaacs mentioned this issue Jun 17, 2021

[BUG] Diff skips deps of non-global link root npm/arborist#292

Closed

isaacs removed the Needs Triage needs review for next steps label Jun 18, 2021

isaacs mentioned this issue Jun 18, 2021

fix(diff): walk target children if root is a link npm/arborist#294

Closed

isaacs closed this as completed in npm/arborist@42b952e Jun 24, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] npx failed to install package and exec, when the user HOME dir is a symlink #3431

[BUG] npx failed to install package and exec, when the user HOME dir is a symlink #3431

zthxxx commented Jun 17, 2021 •

edited

Loading

zthxxx commented Jun 17, 2021 •

edited

Loading

zthxxx commented Jun 17, 2021

isaacs commented Jun 17, 2021

zthxxx commented Jun 17, 2021

isaacs commented Jun 18, 2021

isaacs commented Jun 18, 2021

zthxxx commented Jun 28, 2021

[BUG] npx failed to install package and exec, when the user HOME dir is a symlink #3431

[BUG] npx failed to install package and exec, when the user HOME dir is a symlink #3431

Comments

zthxxx commented Jun 17, 2021 • edited Loading

Is there an existing issue for this?

Current Behavior

Expected Behavior

Steps To Reproduce

Environment

zthxxx commented Jun 17, 2021 • edited Loading

zthxxx commented Jun 17, 2021

isaacs commented Jun 17, 2021

zthxxx commented Jun 17, 2021

isaacs commented Jun 18, 2021

isaacs commented Jun 18, 2021

zthxxx commented Jun 28, 2021

zthxxx commented Jun 17, 2021 •

edited

Loading

zthxxx commented Jun 17, 2021 •

edited

Loading