Debugger Does Not Enumerate ctypes Arrays

[TheRealQuantam]

Describe the bug
The debugger in Visual Studio 2022 does not treat ctypes arrays as containers, and hovering over them or looking at them in the variables window does not show length or allow the items to be enumerated like a list or dict. This is a new problem in VS2022, and did not occur in VS2017 or VS2019, which is why I've only recently started using VS2022 and this problem is making it very unpleasant as I use ctypes extensively in my work.

Ordinarily I would post in a forum to ask if there is some setting that's set incorrectly (the settings are default for Python), but as there's no forum here I'm making a bug report.

Steps to Reproduce
For me this occurs any and all times I use ctypes arrays in any context, so I don't really have any specific instructions to reproduce. The only thing worth noting is that this occurs regardless of whether "Enable debugging of the Python standard library" is enabled (I mostly work with it enabled). Maybe read a random binary file into a bytearray and do something like

x = (ctypes.c_uint8 * len(file_bytes)).from_buffer(file_bytes)

Then hover over x (or look at it in the variables window). It will only show "special variables" and "protected variables".

Expected behavior
In past VS versions the debugger would display as if it was a list, showing the length and list of all elements.

Configuration information
VS Version: Community 2022 (64-bit) 17.3.0
PTVS version: 17.0.22179.3
Python version: 3.9.7 64-bit (x64)
Debugger Type: I don't think this is relevant for VS2022?

[AdamYoblick]

I wasn't even able to get past the import ctypes statement in VS.

Here's the code I ran:

import ctypes

prime_numbers = [2, 3, 5, 7]
byte_array = bytearray(prime_numbers)
x = (ctypes.c_uint8 * len(byte_array)).from_buffer(byte_array)

print(x)

Here's vscode running this just fine:

But when I try the same code in VS, I get an exception from the debugger process:

Traceback (most recent call last):
  File "C:\Users\advolker\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\advolker\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "c:\users\advolker\appdata\local\microsoft\visualstudio\17.0_f467929dexp\extensions\microsoft corporation\python\17.0.0\debugpy\__main__.py", line 37, in <module>
    from debugpy.server import cli
  File "c:\users\advolker\appdata\local\microsoft\visualstudio\17.0_f467929dexp\extensions\microsoft corporation\python\17.0.0\debugpy/..\debugpy\server\__init__.py", line 7, in <module>
    import debugpy._vendored.force_pydevd  # noqa
  File "c:\users\advolker\appdata\local\microsoft\visualstudio\17.0_f467929dexp\extensions\microsoft corporation\python\17.0.0\debugpy/..\debugpy\_vendored\force_pydevd.py", line 36, in <module>
    preimport('pydevd', [
  File "c:\users\advolker\appdata\local\microsoft\visualstudio\17.0_f467929dexp\extensions\microsoft corporation\python\17.0.0\debugpy/..\debugpy\_vendored\__init__.py", line 126, in preimport
    import_module(name)
  File "C:\Users\advolker\AppData\Local\Programs\Python\Python39\lib\importlib\__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "c:\users\advolker\appdata\local\microsoft\visualstudio\17.0_f467929dexp\extensions\microsoft corporation\python\17.0.0\debugpy\_vendored\pydevd\pydevd.py", line 45, in <module>
    from _pydevd_bundle import pydevd_utils
  File "c:\users\advolker\appdata\local\microsoft\visualstudio\17.0_f467929dexp\extensions\microsoft corporation\python\17.0.0\debugpy\_vendored\pydevd\_pydevd_bundle\pydevd_utils.py", line 9, in <module>
    import ctypes
  File "C:\Users\advolker\source\repos\ctypes\ctypes\ctypes.py", line 6, in <module>
    x = (ctypes.c_uint8 * len(byte_array)).from_buffer(byte_array)
AttributeError: partially initialized module 'ctypes' has no attribute 'c_uint8' (most likely due to a circular import)
Press any key to continue . . .

And if I try to remove the import statement, I get an error that ctypes is not defined.

[AdamYoblick]

@int19h Are you able to look into this whenever you can? I'm not really sure what's going on here.

[TheRealQuantam]

Very strange that you can't import ctypes in VS. At least it looks like you're able to reproduce it on VSCode. In VS _length_ is hidden away under "protected variables" (as with all variables that start with _), but your image doesn't show enumeration of entries.

[int19h]

@AdamYoblick The error is because your sample is named ctypes.py, so when you import ctypes it imports itself.

Aside from that I'm pretty sure the behavior is identical between VS and VSCode - we don't do anything special in debugpy that'd make a difference.

ctypes arrays have len() and are iterable though, so not showing items is a bug for sure.

[AdamYoblick]

@AdamYoblick Adam Yoblick FTE The error is because your sample is named ctypes.py, so when you import ctypes it imports itself.

aaaand I feel like a fool 😆

@TheRealQuantam I see what you're saying about the variable data being presented in a slightly different way.

Here's VSCode:

And here's VS:

This has to do with Visual Studio variable presentation options found in Tools -> Options -> Python -> Debugging.

If you don't want protected variables to be shown in a group, select Inline for that option, then restart debugging:

Then the view will look like VSCode:

Please let me know if this fixes your issue and I can close this out :)

[AdamYoblick]

@int19h I was under the impression that VSCode used group for everything, but it sounds like it uses inline for some things. I'd like to change the default behavior of VS to match VSCode. Can you please point me to variable presentation options passed to debugpy from VSCode if the user doesn't change anything?

[int19h]

VSCode doesn't actually wire it up in any special way - whatever is in launch.json is flown through as is, and if it's not there, debugpy defaults are used, which are to group everything except for "protected":

debugpy/src/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_api.py

Lines 72 to 74 in db43846

	class VariablePresentation(object):
	def __init__(self, special='group', function='group', class_='group', protected='inline'):

Note though that showing _length_ as an attribute directly under the variable is separate from showing the actual items and length for a collection. If you expand prime_numbers in the repro, you'll see all the items, and length represented as a synthetic member len(). This is what the ctypes array should be doing as well.

[AdamYoblick]

Ok so I'll create a new issue to have protected variables be presented as Inline, to match the vs code defaults. And I'll move this issue to debugpy per Pavel's instructions.

[int19h]

It looks like we don't actually handle collections in a sufficiently generic way for it to "just work":

debugpy/src/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_xml.py

Lines 36 to 114 in ea423ae

	def _create_default_type_map():
	default_type_map = [
	# None means that it should not be treated as a compound variable
	# isintance does not accept a tuple on some versions of python, so, we must declare it expanded
	(type(None), None,),
	(int, None),
	(float, None),
	(complex, None),
	(str, None),
	(tuple, pydevd_resolver.tupleResolver),
	(list, pydevd_resolver.tupleResolver),
	(dict, pydevd_resolver.dictResolver),
	]
	try:
	from collections import OrderedDict
	default_type_map.insert(0, (OrderedDict, pydevd_resolver.orderedDictResolver))
	# we should put it before dict
	except:
	pass
	try:
	default_type_map.append((long, None)) # @UndefinedVariable
	except:
	pass # not available on all python versions
	default_type_map.append((DAPGrouper, pydevd_resolver.dapGrouperResolver))
	try:
	default_type_map.append((set, pydevd_resolver.setResolver))
	except:
	pass # not available on all python versions
	try:
	default_type_map.append((frozenset, pydevd_resolver.setResolver))
	except:
	pass # not available on all python versions
	try:
	from django.utils.datastructures import MultiValueDict
	default_type_map.insert(0, (MultiValueDict, pydevd_resolver.multiValueDictResolver))
	# we should put it before dict
	except:
	pass # django may not be installed
	try:
	from django.forms import BaseForm
	default_type_map.insert(0, (BaseForm, pydevd_resolver.djangoFormResolver))
	# we should put it before instance resolver
	except:
	pass # django may not be installed
	try:
	from collections import deque
	default_type_map.append((deque, pydevd_resolver.dequeResolver))
	except:
	pass
	if frame_type is not None:
	default_type_map.append((frame_type, pydevd_resolver.frameResolver))
	if _IS_JYTHON:
	from org.python import core # @UnresolvedImport
	default_type_map.append((core.PyNone, None))
	default_type_map.append((core.PyInteger, None))
	default_type_map.append((core.PyLong, None))
	default_type_map.append((core.PyFloat, None))
	default_type_map.append((core.PyComplex, None))
	default_type_map.append((core.PyString, None))
	default_type_map.append((core.PyTuple, pydevd_resolver.tupleResolver))
	default_type_map.append((core.PyList, pydevd_resolver.tupleResolver))
	default_type_map.append((core.PyDictionary, pydevd_resolver.dictResolver))
	default_type_map.append((core.PyStringMap, pydevd_resolver.dictResolver))
	if hasattr(core, 'PyJavaInstance'):
	# Jython 2.5b3 removed it.
	default_type_map.append((core.PyJavaInstance, pydevd_resolver.instanceResolver))
	return default_type_map

So far as I can see, TupleResolver should work as is if it's registered as a resolver for ctypes arrays. However, because different array sizes produce different types, this will have to be implemented as a TypeResolveProvider.

[TheRealQuantam]

When and how will this fix be deployed?

[int19h]

Whenever a new version of debugpy is published on PyPI, the following PTVS release will normally pick it up. We do not have a regular release schedule for debugpy, so I can't specifically say when this will happen.

Note that you can always deploy fixes locally by placing a more up-to-date version of debugpy under "c:\users\advolker\appdata\local\microsoft\visualstudio\17.0_f467929dexp\extensions\microsoft corporation\python". It should work even if you just get it directly from git and copy it over, although you'll need to build it first (via setup.py bdist) to get native code optimizations that improve performance.