Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Relocate image and video_reader extensions' dependencies on conda #2818

Open
wants to merge 52 commits into
base: main
Choose a base branch
from
Open

Relocate image and video_reader extensions' dependencies on conda #2818

wants to merge 52 commits into from

Conversation

andfoy
Copy link
Contributor

@andfoy andfoy commented Oct 14, 2020

Follow-up to #2777 (comment)

andfoy and others added 30 commits October 14, 2020 16:38
@andfoy
Include FFmpeg as a dependency on Windows
5bd6fdc
@fmassa
Merge branch 'master' into ffmpeg_win
b2b58fe
@andfoy
Merge branch 'master' into ffmpeg_win
da3ff1e
@andfoy
Try to redefine build_ext
96701c7
@andfoy
Try to relocate Linux so
f27538d
@andfoy
Fix command cwd
3b2b7f6
@andfoy
Do not overwrite output library dir
5652f85
@andfoy
Do not delete torchvision
cfb5d56
@andfoy
Minor error correction
6e9a62a
@andfoy
Install auditwheel on Linux
78fad76
@andfoy
Minor error correction
71b2a44
@andfoy
Try to build on conda
889baf1
@andfoy
Try to install auditwheel in build.sh
b98cddc
@andfoy
Try to install auditwheel manually
aac5598
@andfoy
Add pbr as dependency
3c2df7f
@andfoy
Install patchelf dependencies only on linux
3a564b1
@andfoy
Debug if build.sh causes some errors
fff9902
@andfoy
Debug if build.sh causes some errors
6edfadc
@andfoy
Restore meta.yaml
6ca59e8
@andfoy
Try to install again the extra dependencies
f0c1ea5
@andfoy
Check if wheel introduces conflicts
d1a33bc
@andfoy
Debug if patchelf causes errors
418ad50
@andfoy
See if wheel causes errors
7d2701f
@andfoy
Try to install pyelftools solely
60aa67a
@andfoy
Call lddtree locally
6839724
@andfoy
Set valid true
8781317
@andfoy
Install pyelftools on type checking CI
3a08cd1
@andfoy
Update LD_LIBRARY_PATH
0931e1b
@andfoy
Do not require libpng, libjpeg nor ffmpeg at runtime
5756273
@andfoy
Regenerate circleci file
88f2956
@andfoy
Copy link
Contributor Author

andfoy commented Oct 30, 2020 •
edited
Loading

@fmassa, I managed PyAV and torchvision to use their own versions of FFmpeg, however, it would seem that the test errors are legitimate.

@andfoy
Copy link
Contributor Author

andfoy commented Oct 30, 2020

@peterjc123, do you know how to fix the cl not found error on the conda (GPU) build?

@peterjc123
Copy link
Contributor

@andfoy There's a bug in cpp_extensions.cpp. I'll try to fix it in PyTorch.

@peterjc123 peterjc123 mentioned this pull request Oct 31, 2020
Closed
@facebook-github-bot
Copy link

This pull request has been merged in d73a8db.

@pmeier
Copy link
Collaborator

pmeier commented Nov 11, 2020

Merely, #2818 (comment) was fixed in pytorch/pytorch#47150, but this does not include the rest of this PR.

@pmeier pmeier reopened this Nov 11, 2020
@pmeier pmeier removed the Merged label Nov 11, 2020
@andfoy
Copy link
Contributor Author

andfoy commented Nov 11, 2020

Thanks @pmeier!

seemethere pushed a commit to seemethere/pytorch that referenced this pull request Dec 7, 2020
@peterjc123 @seemethere
Use local env for building CUDA extensions on Windows (pytorch#47150)
429743d 
Summary:
Fixes pytorch/vision#2818 (comment)
After activating the VC env multiple times, the following error will be raised when building a CUDA extension.
```
FAILED: C:/tools/MINICO~1/CONDA-~2/TORCHV~1/work/build/temp.win-amd64-3.8/Release/tools/MINICO~1/CONDA-~2/TORCHV~1/work/torchvision/csrc/cuda/PSROIAlign_cuda.obj
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\bin\nvcc -Xcompiler /MD -Xcompiler /wd4819 -Xcompiler /wd4251 -Xcompiler /wd4244 -Xcompiler /wd4267 -Xcompiler /wd4275 -Xcompiler /wd4018 -Xcompiler /wd4190 -Xcompiler /EHsc -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -DWITH_CUDA -Dtorchvision_EXPORTS -IC:\tools\MINICO~1\CONDA-~2\TORCHV~1\work\torchvision\csrc -I%PREFIX%\lib\site-packages\torch\include -I%PREFIX%\lib\site-packages\torch\include\torch\csrc\api\include -I%PREFIX%\lib\site-packages\torch\include\TH -I%PREFIX%\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\include" -I%PREFIX%\include -I%PREFIX%\include "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.27.29110\ATLMFC\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.27.29110\include" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\cppwinrt" "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.27.29110\ATLMFC\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.27.29110\include" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\cppwinrt" -I%PREFIX%\Library\include -c C:\tools\MINICO~1\CONDA-~2\TORCHV~1\work\torchvision\csrc\cuda\PSROIAlign_cuda.cu -o C:\tools\MINICO~1\CONDA-~2\TORCHV~1\work\build\temp.win-amd64-3.8\Release\tools\MINICO~1\CONDA-~2\TORCHV~1\work\torchvision\csrc\cuda\PSROIAlign_cuda.obj -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_35,code=sm_35 -gencode=arch=compute_50,code=sm_50 -gencode=arch=compute_60,code=sm_60 -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_75,code=sm_75 -gencode=arch=compute_80,code=sm_80 -gencode=arch=compute_50,code=compute_50 -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=_C -D_GLIBCXX_USE_CXX11_ABI=0
'cl.exe' is not recognized as an internal or external command,
operable program or batch file.
```

Pull Request resolved: pytorch#47150

Reviewed By: agolynski

Differential Revision: D24706019

Pulled By: ezyang

fbshipit-source-id: c13dc29f62d2d12d6a56f33dd450b467a1bf193b
(cherry picked from commit d73a8db)
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>
@seemethere seemethere mentioned this pull request Dec 7, 2020
Merged
seemethere added a commit to pytorch/pytorch that referenced this pull request Dec 7, 2020
@seemethere @peterjc123
[v.1.7.x] Use local env for building CUDA extensions on Windows (#47150…
661d1a0 
…) (#48937)

Summary:
Fixes pytorch/vision#2818 (comment)
After activating the VC env multiple times, the following error will be raised when building a CUDA extension.
```
FAILED: C:/tools/MINICO~1/CONDA-~2/TORCHV~1/work/build/temp.win-amd64-3.8/Release/tools/MINICO~1/CONDA-~2/TORCHV~1/work/torchvision/csrc/cuda/PSROIAlign_cuda.obj
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\bin\nvcc -Xcompiler /MD -Xcompiler /wd4819 -Xcompiler /wd4251 -Xcompiler /wd4244 -Xcompiler /wd4267 -Xcompiler /wd4275 -Xcompiler /wd4018 -Xcompiler /wd4190 -Xcompiler /EHsc -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -DWITH_CUDA -Dtorchvision_EXPORTS -IC:\tools\MINICO~1\CONDA-~2\TORCHV~1\work\torchvision\csrc -I%PREFIX%\lib\site-packages\torch\include -I%PREFIX%\lib\site-packages\torch\include\torch\csrc\api\include -I%PREFIX%\lib\site-packages\torch\include\TH -I%PREFIX%\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\include" -I%PREFIX%\include -I%PREFIX%\include "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.27.29110\ATLMFC\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.27.29110\include" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\cppwinrt" "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.27.29110\ATLMFC\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.27.29110\include" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\cppwinrt" -I%PREFIX%\Library\include -c C:\tools\MINICO~1\CONDA-~2\TORCHV~1\work\torchvision\csrc\cuda\PSROIAlign_cuda.cu -o C:\tools\MINICO~1\CONDA-~2\TORCHV~1\work\build\temp.win-amd64-3.8\Release\tools\MINICO~1\CONDA-~2\TORCHV~1\work\torchvision\csrc\cuda\PSROIAlign_cuda.obj -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_35,code=sm_35 -gencode=arch=compute_50,code=sm_50 -gencode=arch=compute_60,code=sm_60 -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_75,code=sm_75 -gencode=arch=compute_80,code=sm_80 -gencode=arch=compute_50,code=compute_50 -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=_C -D_GLIBCXX_USE_CXX11_ABI=0
'cl.exe' is not recognized as an internal or external command,
operable program or batch file.
```

Pull Request resolved: #47150

Reviewed By: agolynski

Differential Revision: D24706019

Pulled By: ezyang

fbshipit-source-id: c13dc29f62d2d12d6a56f33dd450b467a1bf193b
(cherry picked from commit d73a8db)
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>

Co-authored-by: peter <peterghost86@gmail.com>
@fmassa fmassa mentioned this pull request Dec 8, 2020
Merged
@andfoy andfoy mentioned this pull request Dec 9, 2020
Closed
@andfoy andfoy mentioned this pull request Feb 11, 2021
Merged
@andfoy andfoy mentioned this pull request Apr 22, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
cla signed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@andfoy @peterjc123 @facebook-github-bot @pmeier @fmassa