Inconsistent raising of IndentationError on for and if blocks after mismatching indented comment

[JonathanGrant92]

Description

Similar to issue #171, whitespace of single-line comments at the top scope causes inconsistent raising of exceptions in code blocks like for and if blocks defined at the top scope anywhere after the indented comment in the same cell. Not reproducible with the same CPython version in IDLE Shell.

Reproduce

1. Add an indented single-line comment at the top level, for example, not matching expected indentation of a scope by 4 whitespaces.
2. Add an if or for code block in the same cell anywhere after indented single-line comment
3. Copy code to CPython's IDLE for a control trial
4. Compare test and control trials for discrepancies and inconsistency in output.

Expected behavior

Consistent raising and handling of IndentationError

Context

• Operating System and version: Windows 11 Home 10.0.22621
• Python: 3.11.4 (tags/v3.11.4:d2340ef, Jun 7 2023, 05:45:37) [MSC v.1934 64 bit (AMD64)]
• VSCode and version:
  Version: 1.84.0 (user setup)
  Commit: d037ac076cee195194f93ce6fe2bdfe2969cc82d
  Date: 2023-11-01T11:29:04.398Z
  Electron: 25.9.2
  ElectronBuildId: 24603566
  Chromium: 114.0.5735.289
  Node.js: 18.15.0
  V8: 11.4.183.29-electron.0
  OS: Windows_NT x64 10.0.22621

• Jupyter: v2023.10.1003070148

Troubleshoot Output

indented_comments_break_jupyter.pdf

Command Line Output

18:31:21.496 [debug] notebook communications already initialized for editor ~\Documents\python\datasci\helloworld\indented_comments_break_jupyter.ipynb
18:31:27.370 [info] Handle Execution of Cells 0 for ~\Documents\python\datasci\helloworld\indented_comments_break_jupyter.ipynb
18:31:27.370 [debug] start the kernel, options.disableUI=false for ~\Documents\python\datasci\helloworld\indented_comments_break_jupyter.ipynb
18:31:27.371 [debug] Execution Request Sent to Kernel for cell 0
18:31:27.375 [debug] Start cell 0 execution @ 1699057887375 (clear output)
18:31:27.375 [info] Kernel acknowledged execution of cell 0 @ 1699057887375
18:31:28.594 [info] End cell 0 execution @ 1699057888594, started @ 1699057887375, elapsed time = 1.219s
18:31:28.594 [debug] Cell 0 executed with state Success
18:31:29.156 [info] Handle Execution of Cells 1 for ~\Documents\python\datasci\helloworld\indented_comments_break_jupyter.ipynb
18:31:29.156 [debug] start the kernel, options.disableUI=false for ~\Documents\python\datasci\helloworld\indented_comments_break_jupyter.ipynb
18:31:29.157 [debug] Execution Request Sent to Kernel for cell 1
18:31:29.161 [debug] Start cell 1 execution @ 1699057889161 (clear output)
18:31:29.161 [info] Kernel acknowledged execution of cell 1 @ 1699057889161
18:31:29.165 [info] End cell 1 execution @ 1699057889165, started @ 1699057889161, elapsed time = 0.004s
18:31:29.165 [warn] Cancel all remaining cells due to cancellation or failure in execution
18:31:29.165 [debug] Cancel pending cells
18:31:29.166 [debug] Cell 1 executed with state Error
18:31:31.026 [info] Handle Execution of Cells 2 for ~\Documents\python\datasci\helloworld\indented_comments_break_jupyter.ipynb
18:31:31.026 [debug] start the kernel, options.disableUI=false for ~\Documents\python\datasci\helloworld\indented_comments_break_jupyter.ipynb
18:31:31.027 [debug] Execution Request Sent to Kernel for cell 2
18:31:31.031 [debug] Start cell 2 execution @ 1699057891031 (clear output)
18:31:31.031 [info] Kernel acknowledged execution of cell 2 @ 1699057891031
18:31:31.035 [info] End cell 2 execution @ 1699057891035, started @ 1699057891031, elapsed time = 0.004s
18:31:31.035 [warn] Cancel all remaining cells due to cancellation or failure in execution
18:31:31.035 [debug] Cancel pending cells
18:31:31.035 [debug] Cell 2 executed with state Error
18:31:32.607 [info] Handle Execution of Cells 3 for ~\Documents\python\datasci\helloworld\indented_comments_break_jupyter.ipynb
18:31:32.607 [debug] start the kernel, options.disableUI=false for ~\Documents\python\datasci\helloworld\indented_comments_break_jupyter.ipynb
18:31:32.608 [debug] Execution Request Sent to Kernel for cell 3
18:31:32.613 [debug] Start cell 3 execution @ 1699057892613 (clear output)
18:31:32.613 [info] Kernel acknowledged execution of cell 3 @ 1699057892613
18:31:32.616 [info] End cell 3 execution @ 1699057892616, started @ 1699057892613, elapsed time = 0.003s
18:31:32.616 [debug] Cell 3 executed with state Success
18:31:34.053 [info] Handle Execution of Cells 4 for ~\Documents\python\datasci\helloworld\indented_comments_break_jupyter.ipynb
18:31:34.054 [debug] start the kernel, options.disableUI=false for ~\Documents\python\datasci\helloworld\indented_comments_break_jupyter.ipynb
18:31:34.054 [debug] Execution Request Sent to Kernel for cell 4
18:31:34.059 [debug] Start cell 4 execution @ 1699057894058 (clear output)
18:31:34.059 [info] Kernel acknowledged execution of cell 4 @ 1699057894058
18:31:34.061 [info] End cell 4 execution @ 1699057894061, started @ 1699057894058, elapsed time = 0.003s
18:31:34.062 [debug] Cell 4 executed with state Success
18:31:35.183 [info] Handle Execution of Cells 5 for ~\Documents\python\datasci\helloworld\indented_comments_break_jupyter.ipynb
18:31:35.183 [debug] start the kernel, options.disableUI=false for ~\Documents\python\datasci\helloworld\indented_comments_break_jupyter.ipynb
18:31:35.184 [debug] Execution Request Sent to Kernel for cell 5
18:31:35.188 [debug] Start cell 5 execution @ 1699057895188 (clear output)
18:31:35.188 [info] Kernel acknowledged execution of cell 5 @ 1699057895188
18:31:35.191 [info] End cell 5 execution @ 1699057895191, started @ 1699057895188, elapsed time = 0.003s
18:31:35.192 [warn] Cancel all remaining cells due to cancellation or failure in execution
18:31:35.192 [debug] Cancel pending cells
18:31:35.192 [debug] Cell 5 executed with state Error
18:31:36.340 [info] Handle Execution of Cells 6 for ~\Documents\python\datasci\helloworld\indented_comments_break_jupyter.ipynb
18:31:36.341 [debug] start the kernel, options.disableUI=false for ~\Documents\python\datasci\helloworld\indented_comments_break_jupyter.ipynb
18:31:36.341 [debug] Execution Request Sent to Kernel for cell 6
18:31:36.345 [debug] Start cell 6 execution @ 1699057896345 (clear output)
18:31:36.345 [info] Kernel acknowledged execution of cell 6 @ 1699057896345
18:31:36.349 [info] End cell 6 execution @ 1699057896349, started @ 1699057896345, elapsed time = 0.004s
18:31:36.349 [warn] Cancel all remaining cells due to cancellation or failure in execution
18:31:36.349 [debug] Cancel pending cells
18:31:36.349 [debug] Cell 6 executed with state Error
18:31:37.474 [info] Handle Execution of Cells 7 for ~\Documents\python\datasci\helloworld\indented_comments_break_jupyter.ipynb
18:31:37.475 [debug] start the kernel, options.disableUI=false for ~\Documents\python\datasci\helloworld\indented_comments_break_jupyter.ipynb
18:31:37.476 [debug] Execution Request Sent to Kernel for cell 7
18:31:37.481 [debug] Start cell 7 execution @ 1699057897480 (clear output)
18:31:37.481 [info] Kernel acknowledged execution of cell 7 @ 1699057897480
18:31:37.484 [info] End cell 7 execution @ 1699057897484, started @ 1699057897480, elapsed time = 0.004s
18:31:37.484 [debug] Cell 7 executed with state Success

ipynb cell samples exported to python

# %%
!jupyter --version

# %%
    # 4 whitespace indented comment throws IndentationError
print('This doesn\'t affect me!')
for x in range(0, 3):
    print("We're on time %d" % (x))

# %%
    # 4 whitespace indented comment throws IndentationError
if 1:
    print("We're here!")

# %%
 # 1 whitespace indented comment throws no error
  # 2 whitespace indented comment throws no error
   # 3 whitespace indented comment throws no error
     # 5 whitespace indented comment throws no error
      # 6 whitespace indented comment throws no error
       # 7 whitespace indented comment throws no error
        # 8 whitespace indented comment throws no error
print('This doesn\'t affect me!')
for x in range(0, 3):
    print("We're on time %d" % (x))

# %%
# 4 whitespace indent on next line throws no error; related to issue jupyter/notebook#171 reported on Jun 25, 2015
     
if 1:
    print(0)

    for b in range(2):
        print(b)

# %%
     # 5 whitespace indented comment throws IndentationError
if 1:
    print(0)

    for b in range(2):
        print(b)

# %%
       # 7 whitespace indented comment throws IndentationError
if 1:
    print(0)

    for b in range(2):
        print(b)

# %%
   # 3 whitespace indented comment throws no error
if 1:
    print(0)

    for b in range(2):
        print(b)

# %%
import sys
sys.version

[JonathanGrant92]

Sample where I first encountered this issue:

    # Create DataFrame for cat plot using `pd.melt` using just the values from 
    # 'cholesterol', 'gluc', 'smoke', 'alco', 'active', and 'overweight'.
# https://pandas.pydata.org/docs/reference/api/pandas.melt.html
df_cat = df.loc[:,'cholesterol':'overweight'].melt(id_vars=['cardio'])

# Group and reformat the data to split it by 'cardio'. Show the counts of each feature. 
# You will have to rename one of the columns for the catplot to work correctly.
df_cat = df.groupby('cardio')
for i in range(0, 10):
    print(i)

Output is unexpected:

{
	"name": "IndentationError",
	"message": "expected an indented block after 'for' statement on line 9 (638309045.py, line 10)",
	"stack": "  Cell In[7], line 10
    print(i)
    ^
IndentationError: expected an indented block after 'for' statement on line 9
"
}

In IDLE, the code was minimally modified to not require pandas import and output is expected; indented comments are ignored.

[JonathanGrant92]

This isn't a critical issue, so even if we agree this is a bug, we can also agree to triage this as a maintenance issue, because we can work around the issue by:

• using multi-line comment syntax to preserve desired indentation
• changing indentation of single line comments that cause the exception

Ideally, Jupyter's implementation should ignore indentation of single line comments like the CPython implementation, albeit, afaik Jupyter is using the same implementation as IDLE, yet I can't reproduce this unexpected behavior in IDLE Shell.

[RRosio]

Hi @JonathanGrant92 thank you for writing this issue! As noted on the older issue you linked this issue seems to be better suited for the IPython repository as that is where changes would be applied. I will close this issue but I would encourage you to open this issue in the IPython repo.