Prevent catastrophic failures from resulting in no results

[chrispyles]

It's possible for a catastrophic failure to occur when Otter runs the student notebook s.t. the executed notebook produces no results pickle file, e.g.

Traceback (most recent call last):
  File "/autograder/source/run_otter.py", line 6, in <module>
    run_autograder("/autograder")
  File "/root/mambaforge/envs/otter-env/lib/python3.9/site-packages/otter/run/run_autograder/__init__.py", line 68, in main
    scores = runner.run()
  File "/root/mambaforge/envs/otter-env/lib/python3.9/site-packages/otter/run/run_autograder/runners/python_runner.py", line 125, in run
    scores = grade_notebook(
  File "/root/mambaforge/envs/otter-env/lib/python3.9/site-packages/otter/execute/__init__.py", line 103, in grade_notebook
    results = pickle.load(ntf)
EOFError: Ran out of input

Otter should handle this case gracefully by returning a score of 0 and alerting the student/instructor to the failure instead of completely erroring out.

[ctnelson1997]

@chrispyles This is causing us a lot of issues, appears to stem from this change 461ec0f. Anyway to do a hotfix/5.1.4?

[ctnelson1997]

Relevant pr #728

[chrispyles]

I'm currently working on a fix for this but not sure how long it will take. If it's a super important issue I can put this into a bugfix release (i.e. v5.1.4) ahead of v5.2.0. As noted in my comment in #728, that PR does not actually fix the underlying issue.

[ctnelson1997]

@chrispyles We have around 1000 students using this and running into this issue (example error below). It still emits the ZIP file that we need, so it's not blocking, but there is confusion over it.

Running your submission against local test cases...


C:\Users\ColeNelson\anaconda3\Lib\site-packages\zmq\_future.py:679: RuntimeWarning: Proactor event loop does not implement add_reader family of methods required for zmq. Registering an additional selector thread for add_reader support via tornado. Use `asyncio.set_event_loop_policy(WindowsSelectorEventLoopPolicy())` to avoid this warning.
  self._get_loop()
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\ColeNelson\anaconda3\Lib\site-packages\otter\check\validate_export\__main__.py", line 46, in <module>
    main()
  File "C:\Users\ColeNelson\anaconda3\Lib\site-packages\otter\check\validate_export\__main__.py", line 31, in main
    results = grade_notebook(
              ^^^^^^^^^^^^^^^
  File "C:\Users\ColeNelson\anaconda3\Lib\site-packages\otter\execute\__init__.py", line 103, in grade_notebook
    results = pickle.load(ntf)
              ^^^^^^^^^^^^^^^^
EOFError: Ran out of input

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[16], line 2
      1 # Save your notebook first, then run this cell to export your submission.
----> 2 grader.export(pdf=False, run_tests=True)

File ~\anaconda3\Lib\site-packages\otter\check\utils.py:166, in grading_mode_disabled(wrapped, self, args, kwargs)
    164 if type(self)._grading_mode:
    165     return
--> 166 return wrapped(*args, **kwargs)

File ~\anaconda3\Lib\site-packages\otter\check\utils.py:153, in incompatible_with.<locals>.incompatible(wrapped, self, args, kwargs)
    151     else:
    152         return
--> 153 return wrapped(*args, **kwargs)

File ~\anaconda3\Lib\site-packages\otter\check\utils.py:203, in logs_event.<locals>.event_logger(wrapped, self, args, kwargs)
    201 except Exception as e:
    202     self._log_event(event_type, success=False, error=e)
--> 203     raise e
    205 else:
    206     self._log_event(event_type, results=results, question=question, shelve_env=shelve_env)

File ~\anaconda3\Lib\site-packages\otter\check\utils.py:197, in logs_event.<locals>.event_logger(wrapped, self, args, kwargs)
    194     question, results, shelve_env = wrapped(*args, **kwargs)
    196 else:
--> 197     results = wrapped(*args, **kwargs)
    198     shelve_env = {}
    199     question = None

File ~\anaconda3\Lib\site-packages\otter\check\notebook.py:447, in Notebook.export(self, nb_path, export_path, pdf, filtering, pagebreaks, files, display_link, force_save, run_tests)
    444         display(HTML(out_html))
    446 if pdf_created or not otter_nb_config.get(REQUIRE_CONFIRMATION_NO_PDF_EXPORT_KEY, False):
--> 447     continue_export()
    448 else:
    449     display_pdf_confirmation_widget(
    450         otter_nb_config.get(NO_PDF_EXPORT_MESSAGE_KEY), continue_export)

File ~\anaconda3\Lib\site-packages\otter\check\notebook.py:429, in Notebook.export.<locals>.continue_export()
    427 if run_tests:
    428     print("Running your submission against local test cases...\n")
--> 429     results = grade_zip_file(zip_path, nb_path, self._path)
    430     print(
    431         "Your submission received the following results when run against " + \
    432         "available test cases:\n\n" + indent(results.summary(), "    "))
    434 if display_link:
    435     # create and display output HTML

File ~\anaconda3\Lib\site-packages\otter\check\utils.py:91, in grade_zip_file(zip_path, nb_arcname, tests_dir)
     88 print(results.stderr.decode("utf-8"))
     90 if results.stderr:
---> 91     raise RuntimeError(results.stderr)
     93 with open(results_path, "rb") as f:
     94     results = dill.load(f)

RuntimeError: b'C:\\Users\\ColeNelson\\anaconda3\\Lib\\site-packages\\zmq\\_future.py:679: RuntimeWarning: Proactor event loop does not implement add_reader family of methods required for zmq. Registering an additional selector thread for add_reader support via tornado. Use `asyncio.set_event_loop_policy(WindowsSelectorEventLoopPolicy())` to avoid this warning.\r\n  self._get_loop()\r\nTraceback (most recent call last):\r\n  File "<frozen runpy>", line 198, in _run_module_as_main\r\n  File "<frozen runpy>", line 88, in _run_code\r\n  File "C:\\Users\\ColeNelson\\anaconda3\\Lib\\site-packages\\otter\\check\\validate_export\\__main__.py", line 46, in <module>\r\n    main()\r\n  File "C:\\Users\\ColeNelson\\anaconda3\\Lib\\site-packages\\otter\\check\\validate_export\\__main__.py", line 31, in main\r\n    results = grade_notebook(\r\n              ^^^^^^^^^^^^^^^\r\n  File "C:\\Users\\ColeNelson\\anaconda3\\Lib\\site-packages\\otter\\execute\\__init__.py", line 103, in grade_notebook\r\n    results = pickle.load(ntf)\r\n              ^^^^^^^^^^^^^^^^\r\nEOFError: Ran out of input\r\n'

[chrispyles]

@ctnelson1997 I'm not sure how much a fix for this will help for your issue. The fix I'm proposing here doesn't fix the fact that an EOFError occurs, it just handles it more gracefully.

I'm not sure why the error you provided is getting thrown -- it looks like something unrelated to this issue and would require more debugging. If you just want to silence the error, you could disable the submission zip validation in grader.export by setting export_cell: run_tests: false in your assignment config (or by manually removing run_tests=True from the grader.export call).

[ctnelson1997]

Hi @chrispyles this appears to be a platform compatibility issue with the use of NamedTemporaryFile. For anyone not on a Unix machine (e.g. Windows, like mine) the NamedTemporaryFile cannot be opened again. Thus, results cannot be written, which I believe is what's causing the file to be empty, leading to the EOFError: Ran out of input