ENH: Don't add system library directories to rpath #1168

Job	Run time
test_cygwin (39, windows-latest)	5m 12s
ci_setuptools (3.10, ubuntu-latest)	3m 47s
test (3.8, ubuntu-latest)	45s
collateral (diffcov)	47s
test (3.8, macos-latest)	59s
collateral (docs)	27s
test_msvc_python_mingw	2m 28s
test_msys2_mingw (mingw64, x86_64, gcc, g++)	1m 35s
test (3.8, windows-latest)	42s
test_msys2_mingw (mingw32, i686, gcc, g++)	2m 16s
test_msys2_mingw (ucrt64, ucrt-x86_64, gcc, g++)	1m 34s
test_msys2_mingw (clang64, clang-x86_64, clang, clang++)	1m 52s
test (3.12, ubuntu-latest)	41s
test (3.12, macos-latest)	46s
test (3.12, windows-latest)	44s
test (3.9, ubuntu-latest)	46s
test (3.10, ubuntu-latest)	51s
test (3.11, ubuntu-latest)	50s
test (pypy3.10, ubuntu-latest)	58s
check	1s
release	0s
	28m 1s

Provide feedback