Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

torch.Tensor.index_select Trigger heap-buffer-overflow with AddressSanitizer #88940

Closed
Kristoff-starling opened this issue Nov 12, 2022 · 1 comment
Closed

torch.Tensor.index_select Trigger heap-buffer-overflow with AddressSanitizer #88940

Kristoff-starling opened this issue Nov 12, 2022 · 1 comment
Labels
module: sanitizers triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module

Comments

@Kristoff-starling
Copy link
Contributor

Kristoff-starling commented Nov 12, 2022 •
edited
Loading

🐛 Describe the bug

A test case for torch.Tensor.index_select triggers heap-buffer-overflow error with address sanitizer. Without sanitizers, the test terminates normally.
Test:

import torch

def test():
    arg_1 = torch.rand([], dtype=torch.float32).clone()
    arg_3 = torch.zeros([2], dtype=torch.int64).clone()
    res = torch.Tensor.index_select(arg_1,0,arg_3,)

test()

Error log:

==100125==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x609000000284 at pc 0x7fb6f1b2e155 bp 0x7fff89752850 sp 0x7fff89752848
WRITE of size 4 at 0x609000000284 thread T0
    #0 0x7fb6f1b2e154 in at::native::index_select_out_cpu_(at::Tensor const&, long, at::Tensor const&, at::Tensor&)::$_10::operator()() const TensorAdvancedIndexing.cpp
    #1 0x7fb6f1b23bec in at::native::index_select_out_cpu_(at::Tensor const&, long, at::Tensor const&, at::Tensor&) (/home/yuyao/anaconda3/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so+0x9d18bec) (BuildId: c7876a8b2ea547346cadf9022e3da8df32f4bd9b)
    #2 0x7fb6f1b33666 in at::native::index_select_cpu_(at::Tensor const&, long, at::Tensor const&) (/home/yuyao/anaconda3/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so+0x9d28666) (BuildId: c7876a8b2ea547346cadf9022e3da8df32f4bd9b)
    #3 0x7fb6f388a01a in c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoFunctor_<c10::CompileTimeFunctionPointer<at::Tensor (at::Tensor const&, long, at::Tensor const&), &at::(anonymous namespace)::(anonymous namespace)::wrapper__index_select(at::Tensor const&, long, at::Tensor const&)>, at::Tensor, c10::guts::typelist::typelist<at::Tensor const&, long, at::Tensor const&>>, at::Tensor (at::Tensor const&, long, at::Tensor const&)>::call(c10::OperatorKernel*, c10::DispatchKeySet, at::Tensor const&, long, at::Tensor const&) RegisterCPU.cpp
    #4 0x7fb6f27a3466 in at::Tensor c10::Dispatcher::redispatch<at::Tensor, at::Tensor const&, long, at::Tensor const&>(c10::TypedOperatorHandle<at::Tensor (at::Tensor const&, long, at::Tensor const&)> const&, c10::DispatchKeySet, at::Tensor const&, long, at::Tensor const&) const (/home/yuyao/anaconda3/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so+0xa998466) (BuildId: c7876a8b2ea547346cadf9022e3da8df32f4bd9b)
    #5 0x7fb6f2546bd8 in at::_ops::index_select::redispatch(c10::DispatchKeySet, at::Tensor const&, long, at::Tensor const&) (/home/yuyao/anaconda3/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so+0xa73bbd8) (BuildId: c7876a8b2ea547346cadf9022e3da8df32f4bd9b)
    #6 0x7fb6f71eaca1 in torch::autograd::VariableType::(anonymous namespace)::index_select(c10::DispatchKeySet, at::Tensor const&, long, at::Tensor const&) VariableType_0.cpp
    #7 0x7fb6f71e9ff3 in c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoFunctor_<c10::CompileTimeFunctionPointer<at::Tensor (c10::DispatchKeySet, at::Tensor const&, long, at::Tensor const&), &torch::autograd::VariableType::(anonymous namespace)::index_select(c10::DispatchKeySet, at::Tensor const&, long, at::Tensor const&)>, at::Tensor, c10::guts::typelist::typelist<c10::DispatchKeySet, at::Tensor const&, long, at::Tensor const&>>, at::Tensor (c10::DispatchKeySet, at::Tensor const&, long, at::Tensor const&)>::call(c10::OperatorKernel*, c10::DispatchKeySet, at::Tensor const&, long, at::Tensor const&) VariableType_0.cpp
    #8 0x7fb6f2546625 in at::_ops::index_select::call(at::Tensor const&, long, at::Tensor const&) (/home/yuyao/anaconda3/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so+0xa73b625) (BuildId: c7876a8b2ea547346cadf9022e3da8df32f4bd9b)
    #9 0x7fb7131f6487 in torch::autograd::THPVariable_index_select(_object*, _object*, _object*) python_variable_methods.cpp
    #10 0x55bcca48db9b in method_vectorcall_VARARGS_KEYWORDS /opt/conda/conda-bld/python-split_1649141344976/work/Objects/descrobject.c:348
    #11 0x55bcca3cd754 in _PyObject_VectorcallTstate /opt/conda/conda-bld/python-split_1649141344976/work/Include/cpython/abstract.h:118
    #12 0x55bcca3cd754 in PyObject_Vectorcall /opt/conda/conda-bld/python-split_1649141344976/work/Include/cpython/abstract.h:127
    #13 0x55bcca3cd754 in call_function /opt/conda/conda-bld/python-split_1649141344976/work/Python/ceval.c:5077
    #14 0x55bcca3cd754 in _PyEval_EvalFrameDefault.cold.2984 /opt/conda/conda-bld/python-split_1649141344976/work/Python/ceval.c:3489
    #15 0x55bcca465283 in _PyEval_EvalFrame /opt/conda/conda-bld/python-split_1649141344976/work/Include/internal/pycore_ceval.h:40
    #16 0x55bcca465283 in function_code_fastcall /opt/conda/conda-bld/python-split_1649141344976/work/Objects/call.c:330
    #17 0x55bcca465283 in _PyFunction_Vectorcall /opt/conda/conda-bld/python-split_1649141344976/work/Objects/call.c:367
    #18 0x55bcca3cbae5 in _PyObject_VectorcallTstate /opt/conda/conda-bld/python-split_1649141344976/work/Include/cpython/abstract.h:118
    #19 0x55bcca3cbae5 in PyObject_Vectorcall /opt/conda/conda-bld/python-split_1649141344976/work/Include/cpython/abstract.h:127
    #20 0x55bcca3cbae5 in call_function /opt/conda/conda-bld/python-split_1649141344976/work/Python/ceval.c:5077
    #21 0x55bcca3cbae5 in _PyEval_EvalFrameDefault.cold.2984 /opt/conda/conda-bld/python-split_1649141344976/work/Python/ceval.c:3520
    #22 0x55bcca464662 in _PyEval_EvalFrame /opt/conda/conda-bld/python-split_1649141344976/work/Include/internal/pycore_ceval.h:40
    #23 0x55bcca464662 in _PyEval_EvalCode /opt/conda/conda-bld/python-split_1649141344976/work/Python/ceval.c:4329
    #24 0x55bcca51145b in _PyEval_EvalCodeWithName /opt/conda/conda-bld/python-split_1649141344976/work/Python/ceval.c:4361
    #25 0x55bcca51145b in PyEval_EvalCodeEx /opt/conda/conda-bld/python-split_1649141344976/work/Python/ceval.c:4377
    #26 0x55bcca46545a in PyEval_EvalCode /opt/conda/conda-bld/python-split_1649141344976/work/Python/ceval.c:828
    #27 0x55bcca51150a in run_eval_code_obj /opt/conda/conda-bld/python-split_1649141344976/work/Python/pythonrun.c:1221
    #28 0x55bcca541f74 in run_mod /opt/conda/conda-bld/python-split_1649141344976/work/Python/pythonrun.c:1242
    #29 0x55bcca3e2986 in pyrun_file.cold.3080 /opt/conda/conda-bld/python-split_1649141344976/work/Python/pythonrun.c:1140
    #30 0x55bcca547a2e in pyrun_simple_file /opt/conda/conda-bld/python-split_1649141344976/work/Python/pythonrun.c:450
    #31 0x55bcca547a2e in PyRun_SimpleFileExFlags /opt/conda/conda-bld/python-split_1649141344976/work/Python/pythonrun.c:483
    #32 0x55bcca54810a in pymain_run_file /opt/conda/conda-bld/python-split_1649141344976/work/Modules/main.c:379
    #33 0x55bcca54810a in pymain_run_python /opt/conda/conda-bld/python-split_1649141344976/work/Modules/main.c:604
    #34 0x55bcca54810a in Py_RunMain /opt/conda/conda-bld/python-split_1649141344976/work/Modules/main.c:683
    #35 0x55bcca548308 in Py_BytesMain /opt/conda/conda-bld/python-split_1649141344976/work/Modules/main.c:1129
    #36 0x7fb71a321d8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
    #37 0x7fb71a321e3f in __libc_start_main csu/../csu/libc-start.c:392:3
    #38 0x55bcca4cf09f in _start (/home/yuyao/anaconda3/bin/python3.9+0x20109f)

0x609000000284 is located 0 bytes to the right of 4-byte region [0x609000000280,0x609000000284)
allocated by thread T0 here:
    #0 0x7fb71a706817 in posix_memalign (/usr/lib/llvm-15/lib/clang/15.0.4/lib/linux/libclang_rt.asan-x86_64.so+0xd0817) (BuildId: c3e83b91f2d38fca7fefc78874591b233340d131)
    #1 0x7fb6e7cb88e9 in c10::alloc_cpu(unsigned long) (/home/yuyao/anaconda3/lib/python3.9/site-packages/torch/lib/libc10.so+0x12d8e9) (BuildId: 04808d6a15270bc9548529381a08b11ae3ad24c7)
    #2 0x7fb6e7c4b325 in c10::DefaultCPUAllocator::allocate(unsigned long) const (/home/yuyao/anaconda3/lib/python3.9/site-packages/torch/lib/libc10.so+0xc0325) (BuildId: 04808d6a15270bc9548529381a08b11ae3ad24c7)
    #3 0x7fb6f1a0e5e6 in at::native::resize_impl_cpu_(c10::TensorImpl*, c10::ArrayRef<long>, c10::OptionalArrayRef<long>, bool) (/home/yuyao/anaconda3/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so+0x9c035e6) (BuildId: c7876a8b2ea547346cadf9022e3da8df32f4bd9b)
    #4 0x7fb6f1a0bd5c in at::native::resize_(at::Tensor const&, c10::ArrayRef<long>, c10::optional<c10::MemoryFormat>) (/home/yuyao/anaconda3/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so+0x9c00d5c) (BuildId: c7876a8b2ea547346cadf9022e3da8df32f4bd9b)
    #5 0x7fb6f1a0bbf5 in at::native::resize_output(at::Tensor const&, c10::ArrayRef<long>) (/home/yuyao/anaconda3/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so+0x9c00bf5) (BuildId: c7876a8b2ea547346cadf9022e3da8df32f4bd9b)

SUMMARY: AddressSanitizer: heap-buffer-overflow TensorAdvancedIndexing.cpp in at::native::index_select_out_cpu_(at::Tensor const&, long, at::Tensor const&, at::Tensor&)::$_10::operator()() const
Shadow bytes around the buggy address:
  0x0c127fff8000: fa fa fa fa fa fa fa fa fd fa fa fa fa fa fa fa
  0x0c127fff8010: fa fa fa fa fa fa fa fa 04 fa fa fa fa fa fa fa
  0x0c127fff8020: fa fa fa fa fa fa fa fa fd fd fa fa fa fa fa fa
  0x0c127fff8030: fa fa fa fa fa fa fa fa 00 00 fa fa fa fa fa fa
  0x0c127fff8040: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c127fff8050:[04]fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c127fff8060: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c127fff8070: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c127fff8080: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c127fff8090: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c127fff80a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==100125==ABORTING

Versions

PyTorch version: 1.14.0a0+git6e5f736
Is debug build: False
CUDA used to build PyTorch: Could not collect
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.1 LTS (x86_64)
GCC version: (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
Clang version: 11.1.0-6
CMake version: version 3.22.1
Libc version: glibc-2.35

Python version: 3.9.12 (main, Apr  5 2022, 06:56:58)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.15.0-52-generic-x86_64-with-glibc2.35
Is CUDA available: False
CUDA runtime version: 11.8.89
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: 
GPU 0: NVIDIA GeForce RTX 3090
GPU 1: NVIDIA GeForce RTX 3090
GPU 2: NVIDIA GeForce RTX 3090

Nvidia driver version: 515.65.01
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.4.1
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.21.5
[pip3] numpydoc==1.2
[pip3] torch==1.14.0a0+git6e5f736
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               11.3.1               h2bc3f7f_2  
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-service               2.4.0            py39h7f8727e_0  
[conda] mkl_fft                   1.3.1            py39hd3c417c_0  
[conda] mkl_random                1.2.2            py39h51133e4_0  
[conda] numpy                     1.21.5           py39he7a7128_1  
[conda] numpy-base                1.21.5           py39hf524024_1  
[conda] numpydoc                  1.2                pyhd3eb1b0_0  
[conda] torch                     1.14.0a0+git6e5f736          pypi_0    pypi
@malfet malfet mentioned this issue Nov 14, 2022
Open
@Kristoff-starling
Copy link
Contributor Author

Here is a gist for reproduction. We upload a recent version of PyTorch (built with sanitizers) on pypi and reproduce the error on Colab.
cc @malfet

@bdhirsh bdhirsh added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Nov 15, 2022
@ganler ganler mentioned this issue Feb 7, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
module: sanitizers triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: forbid multi-index for index_select over scalar ganler/pytorch
3 participants
@malfet @bdhirsh @Kristoff-starling