Pwntools doesn't in Jupyter because of Jupyter-Curses interactions

[cr019283]

Although the github demo works well for IPython, it fails for jupyter notebook.

Here is the received output for "from pwn import *" and "from pwnlib import *":

In 1: from pwn import *
Out 1:
---------------------------------------------------------------------------
error                                     Traceback (most recent call last)
<ipython-input-1-d810d1e70089> in <module>()
----> 1 from pwn import *

/usr/local/lib/python2.7/dist-packages/pwn/__init__.py in <module>()
      1 # Promote useful stuff to toplevel
----> 2 from .toplevel import *
      3 
      4 pwnlib.args.initialize()
      5 pwnlib.log.install_default_handler()

/usr/local/lib/python2.7/dist-packages/pwn/toplevel.py in <module>()
     18 from pprint import pprint
     19 
---> 20 import pwnlib
     21 from pwnlib import *
     22 from pwnlib.asm import *

/usr/local/lib/python2.7/dist-packages/pwnlib/__init__.py in <module>()
     38 
     39 for module in __all__:
---> 40     importlib.import_module('.%s' % module, 'pwnlib')

/usr/lib/python2.7/importlib/__init__.pyc in import_module(name, package)
     35             level += 1
     36         name = _resolve_name(name[level:], package, level)
---> 37     __import__(name)
     38     return sys.modules[name]

/usr/local/lib/python2.7/dist-packages/pwnlib/args.py in <module>()
     46 import sys
     47 
---> 48 from . import term
     49 from .context import context
     50 

/usr/local/lib/python2.7/dist-packages/pwnlib/term/__init__.py in <module>()
      1 import sys
      2 
----> 3 from . import completer
      4 from . import key
      5 from . import keymap

/usr/local/lib/python2.7/dist-packages/pwnlib/term/completer.py in <module>()
      2 import re
      3 
----> 4 from . import readline
      5 
      6 

/usr/local/lib/python2.7/dist-packages/pwnlib/term/readline.py in <module>()
      2 from . import keymap as km
      3 from . import term
----> 4 from . import text
      5 
      6 cursor = text.reverse

/usr/local/lib/python2.7/dist-packages/pwnlib/term/text.pyc in <module>()
    113 
    114 tether = sys.modules[__name__]
--> 115 sys.modules[__name__] = Module()

/usr/local/lib/python2.7/dist-packages/pwnlib/term/text.pyc in __init__(self)
     24         self.__file__ = __file__
     25         self.__name__ = __name__
---> 26         self.num_colors = termcap.get('colors', default = 8)
     27         self.has_bright = self.num_colors >= 16
     28         self.has_gray = self.has_bright

/usr/local/lib/python2.7/dist-packages/pwnlib/term/termcap.pyc in get(cap, *args, **kwargs)
     23     s = cache.get(cap)
     24     if not s:
---> 25         s = curses.tigetstr(cap)
     26         if s == None:
     27             s = curses.tigetnum(cap)

error: must call (at least) setupterm() first

​

In 2: import curses

In 3: curses.setupterm()
Out 2:
---------------------------------------------------------------------------
UnsupportedOperation                      Traceback (most recent call last)
<ipython-input-5-82805162dd9f> in <module>()
----> 1 curses.setupterm()

/usr/local/lib/python2.7/dist-packages/ipykernel/iostream.pyc in fileno(self)
    304 
    305     def fileno(self):
--> 306         raise UnsupportedOperation("IOStream has no fileno.")
    307 
    308     def write(self, string):

UnsupportedOperation: IOStream has no fileno.

print (curses.initscr(), curses.endwin())

(<_curses.curses window object at 0x7fef5c891a98>, None)

In 4: from pwnlib import *

Out 3:
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-2-5eb1d9485f4c> in <module>()
----> 1 from pwnlib import *

/usr/local/lib/python2.7/dist-packages/pwnlib/__init__.py in <module>()
     38 
     39 for module in __all__:
---> 40     importlib.import_module('.%s' % module, 'pwnlib')

/usr/lib/python2.7/importlib/__init__.pyc in import_module(name, package)
     35             level += 1
     36         name = _resolve_name(name[level:], package, level)
---> 37     __import__(name)
     38     return sys.modules[name]

/usr/local/lib/python2.7/dist-packages/pwnlib/args.py in <module>()
     46 import sys
     47 
---> 48 from . import term
     49 from .context import context
     50 

/usr/local/lib/python2.7/dist-packages/pwnlib/term/__init__.py in <module>()
      1 import sys
      2 
----> 3 from . import completer
      4 from . import key
      5 from . import keymap

/usr/local/lib/python2.7/dist-packages/pwnlib/term/completer.py in <module>()
      2 import re
      3 
----> 4 from . import readline
      5 
      6 

/usr/local/lib/python2.7/dist-packages/pwnlib/term/readline.py in <module>()
----> 1 from . import keyconsts as kc
      2 from . import keymap as km
      3 from . import term
      4 from . import text
      5 

ImportError: cannot import name keyconsts

​```

[Stolas]

This issue exists due to:

import curses
curses.setupterm() 

A bit of Googling gave me the following issue within another project: ContinuumIO/anaconda-issues#455

Starting ipython like TERMINFO=/usr/share/terminfo ipython seems to work.
However within a notebook TERMINFO=/usr/share/terminfo ipython notebook --no-browser it does not.

As I have also ran into this issue with pwntools it is really a Jupyter issue instead.
What I think that pwntools could do is check whether curses is available and if not to skip the animations etc. That way Jupyter users and Windows users can use pwntools.

[zachriggle]

This is something that can probably be fixed on the Pwntools side, but I don't want to spend the time sorting out Jupyter to fix it.

We'd certainly welcome a Pull Request to fix the issue.

[CherryWorm]

So I've taken a quick look at this, and there seem to be 2 fairly simple fixes. The problem with @Stolas's try is that jupyter uses different environment variables for each kernel, and the kernels don't inherit the environment variables of the jupyter process itself. However one can use the %env instruction to set these variables inside of the notebooks.

The first fix is to add

%env TERMINFO=/usr/share/terminfo

at the top of the notebook, this tells curses where it can find terminfo. I don't know how exactly curses interacts with each feature of pwntools, but I've tested pretty much all functions of the log module, all seemed to have been working.

I'm slightly more confident in this second fix (though the first one should work as well): pwntools already has a check implemented, which disables curses if the PWNLIB_NOTERM environment variable is set. This means adding

%env PWNLIB_NOTERM=true

at the top of a notebook will do exactly what @Stolas proposed.

[Zoruk]

With

%env PWNLIB_NOTERM=true

I still have an error:

ImportError                               Traceback (most recent call last)
<ipython-input-17-308ef5371a41> in <module>()
----> 1 from pwn import *

/home/zoruk/.virtualenvs/pwn/lib/python2.7/site-packages/pwntools-3.14.0.dev0-py2.7.egg/pwn/__init__.py in <module>()
      2 from __future__ import absolute_import
      3 
----> 4 from pwn.toplevel import *
      5 
      6 pwnlib.args.initialize()

/home/zoruk/.virtualenvs/pwn/lib/python2.7/site-packages/pwntools-3.14.0.dev0-py2.7.egg/pwn/toplevel.py in <module>()
     18 from pprint import pprint
     19 
---> 20 import pwnlib
     21 from pwnlib import *
     22 from pwnlib.asm import *

/home/zoruk/.virtualenvs/pwn/lib/python2.7/site-packages/pwntools-3.14.0.dev0-py2.7.egg/pwnlib/__init__.py in <module>()
     41 
     42 for module in __all__:
---> 43     importlib.import_module('.%s' % module, 'pwnlib')

/usr/lib64/python2.7/importlib/__init__.py in import_module(name, package)
     35             level += 1
     36         name = _resolve_name(name[level:], package, level)
---> 37     __import__(name)
     38     return sys.modules[name]

/home/zoruk/.virtualenvs/pwn/lib/python2.7/site-packages/pwntools-3.14.0.dev0-py2.7.egg/pwnlib/args.py in <module>()
     60 import sys
     61 
---> 62 from pwnlib import term
     63 from pwnlib.context import context
     64 

/home/zoruk/.virtualenvs/pwn/lib/python2.7/site-packages/pwntools-3.14.0.dev0-py2.7.egg/pwnlib/term/__init__.py in <module>()
      4 import sys
      5 
----> 6 from pwnlib.term import completer
      7 from pwnlib.term import key
      8 from pwnlib.term import keymap

/home/zoruk/.virtualenvs/pwn/lib/python2.7/site-packages/pwntools-3.14.0.dev0-py2.7.egg/pwnlib/term/completer.py in <module>()
      5 import re
      6 
----> 7 from pwnlib.term import readline
      8 
      9 

/home/zoruk/.virtualenvs/pwn/lib/python2.7/site-packages/pwntools-3.14.0.dev0-py2.7.egg/pwnlib/term/readline.py in <module>()
      4 from __future__ import division
      5 
----> 6 from pwnlib.term import keyconsts as kc
      7 from pwnlib.term import keymap as km
      8 from pwnlib.term import term

ImportError: cannot import name keyconsts

Any idea how to fix it ?

[mmsatari]

The two fixes above didn't work for me:

%env TERMINFO=/usr/share/terminfo
%env PWNLIB_NOTERM=true

I debugged a little and I realized that it's because it cannot open stdout, maybe because it's set to /dev/null?
here are jupyter notebooks's open fds

{0: 'pipe:[2231783]', 1: '/dev/null', 2: '/dev/pts/0',...}

so I set stdout to 0 which seems to be the pipe between the webui and ipython kernel and it worked.

sys.stdout.fileno = lambda :0

The next challenge is to get the stdin work because interactive mode doesn't work.