[ci] ask pip to always install local artifact but not download package from PyPI

[StrikerRUS]

Initially I was trying to use the following pip flag:

--no-index

Ignore package index (only looking at --find-links URLs instead).

https://pip.pypa.io/en/stable/cli/pip_install/#cmdoption-no-index

to prohibit pip to install LightGBM from PyPI like here:

Successfully built lightgbm-4.5.0.tar.gz
--- installing lightgbm ---
Looking in links: .
Collecting lightgbm
  Downloading lightgbm-4.5.0-py3-none-manylinux_2_28_x86_64.whl.metadata (17 kB)
Downloading lightgbm-4.5.0-py3-none-manylinux_2_28_x86_64.whl (3.6 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.6/3.6 MB 97.6 MB/s eta 0:00:00
Installing collected packages: lightgbm
Successfully installed lightgbm-4.5.0

But unfortunately that flag cannot be used without --no-build-isolation flag because pip needs setuptools during installation:

--- installing lightgbm ---
Looking in links: .
Processing ./lightgbm-4.5.0.tar.gz
  Installing build dependencies: started
  Installing build dependencies: finished with status 'error'
  error: subprocess-exited-with-error
  
  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [3 lines of output]
      Looking in links: .
      ERROR: Could not find a version that satisfies the requirement setuptools (from versions: none)
      ERROR: No matching distribution found for setuptools
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

So I found that being more precise with *.tar.gz/*.whl extension is enough to prevent pip from downloading wheel file from PyPI.

[StrikerRUS]

--ignore-installed was added in #6526 for the following reason:

It also proposes remove the pip uninstall lightgbm in that script with passing --ignore-installed to pip install... that way the locally-built version overwrites the already-installed one every time, without generating a log message like "WARNING: Skipping lightgbm as it is not installed."

But it seems that the right flag is --force-reinstall for that.
--ignore-installed may mix old and new lightgbm installations which can result in corrupted installation. Refer to the detailed explanation here: https://stackoverflow.com/a/51916623.

[jameslamb]

You're right, --force-reinstall is preferable now that we are also using --no-deps here (#6256).

I'd previously not wanted to use it because it applies to all dependencies, not just to the package being installed, and that led to an existing numpy or scipy being force-reinstalled too.

[jameslamb]

Thanks for this. I tested locally to be sure sh build-python.sh install --precompile works as well, and it did.

I agree, this is an improvement! Not having the ability to silently fall back to installing from PyPI is really important for ensuring that CI tests what we think it's testing.

Funny enough, I recently worked through similar issues over in RAPIDS:

• Remove uses of 'pip --find-links' in CI rapidsai/build-planning#69
• explore 'pip --no-index' to more tightly control wheel builds rapidsai/build-planning#79

Never thought about how build-python.sh here could be in need of similar changes.

Thanks very much!!