pip / setuptools support

[bcumming]

Add a setup.py for installing Arbor directly with pip/setuptools.

Implement a setuptools extension for CMake in setup.py. To be honest, I don't understand entirely how it works, but setuptools is obtuse enough that I don't feel at all guilty about this.

Define additional flags for optionally enabling GPUs, MPI, Vectorization and micro-architecture targets, for more adventurous users.

The documentation is updated with a "howto pip" for more casual users who don't want anything to do with CMake.

Fixes #958 .

[Helveg]

So currently GPU backends, MPI & Vectorization would not be available from pip? I think at least MPI for parallelization if frequently used (especially by NEURON users) and might better be included as default

[bcumming]

They are available! See the docs:
https://github.com/bcumming/arbor/blob/pip/doc/python.rst#installing
They are not enabled by default, because we want to make the basic pip install arbor workflow as foolproof and least likely to fail as possible.

[Helveg]

Ok, how would that work from PyPI? Or is the default one the only one that will be available as the python package?

(I thought wheels could do all the building and compilation beforehand on a machine with similar architecture as the target)

[bcumming]

I don't understand exactly how all the "wheeling and dealing" works... however I think that for anything but the basic version (no extras), you wouldn't want to use a pre-compiled version. Even for the basic version, users probably want to build it each time.

From PyPI it would look like

# vanilla version (multithreading support only)
# this might use precompiled wheels.... but I don't know if that is a good idea
pip install arbor

# install with mpi+GPU
pip install --install-option="--mpi" --install-option="--gpu" arbor

[bcumming]

I have seen an alternative approach:

pip install arbor[mpi4py]

where you can provide optional dependencies (in this case mpi4py).
The problem is that these dependencies are PyPI packages, so you inject an "artificial" dependency on a python package (mpi4py) to allow a more compact representation of the underlying dependency (MPI)... and we would still have to compile.

[bcumming]

I think at least MPI for parallelization if frequently used (especially by NEURON users) and might better be included as default

MPI is not available on every user's system by default, and furthermore it is very hard to ensure that CMake called inside the setup.py correctly finds and configures MPI. It often requires the user setting additional environment variables, loading modules, etc.

My personal position is that it makes it much harder for a new user to get started without issues.

[bcumming]

Sometimes I get arbitrary errors when using pip related to the following issue:
pypa/pip#2677
It looks like customized install_command phase may not be run in the wheel building phase (depends on the version of pip, and maybe the phase of the moon).

The errors don't stop pip from working: it goes ahead and builds and installs the package, but without making the wheel.

[Helveg]

Test results:

• Install to pyenv version: Package is correctly placed in the site-packages of the pyenv version.
• Install to a pyenv virtualenv: Package is correctly placed in the virtualenv site-packages.

I can't use the Python package virtualenv (because pyenv shims it) or Python's venv, but it looks good. @bcumming I suppose this error is one of them? cl_opt should hold the flags that don't get passed?

robin@robin-ZenBook-UX533FN:~/arbor$ git checkout pip
Branch 'pip' set up to track remote branch 'pip' from 'bcumming'.
Switched to a new branch 'pip'
robin@robin-ZenBook-UX533FN:~/arbor$ pip install .
Processing /home/robin/arbor
Building wheels for collected packages: arbor
  Building wheel for arbor (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /home/robin/.pyenv/versions/3.7.6/bin/python3.7 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-onmoqq5e/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-onmoqq5e/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-lmsv7eg4
       cwd: /tmp/pip-req-build-onmoqq5e/
  Complete output (32 lines):
  /home/robin/.pyenv/versions/3.7.6/lib/python3.7/site-packages/setuptools/dist.py:474: UserWarning: Normalizing '0.2.3-dev' to '0.2.3.dev0'
    normalized_version,
  running bdist_wheel
  running build
  running build_ext
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-req-build-onmoqq5e/setup.py", line 153, in <module>
      'Bug Reports': 'https://github.com/arbor-sim/arbor/issues',
    File "/home/robin/.pyenv/versions/3.7.6/lib/python3.7/site-packages/setuptools/__init__.py", line 145, in setup
      return distutils.core.setup(**attrs)
    File "/home/robin/.pyenv/versions/3.7.6/lib/python3.7/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/home/robin/.pyenv/versions/3.7.6/lib/python3.7/distutils/dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "/home/robin/.pyenv/versions/3.7.6/lib/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/home/robin/.pyenv/versions/3.7.6/lib/python3.7/site-packages/wheel/bdist_wheel.py", line 192, in run
      self.run_command('build')
    File "/home/robin/.pyenv/versions/3.7.6/lib/python3.7/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/home/robin/.pyenv/versions/3.7.6/lib/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/home/robin/.pyenv/versions/3.7.6/lib/python3.7/distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "/home/robin/.pyenv/versions/3.7.6/lib/python3.7/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/home/robin/.pyenv/versions/3.7.6/lib/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/tmp/pip-req-build-onmoqq5e/setup.py", line 81, in run
      '-DARB_WITH_MPI={}'.format('on' if cl_opt['mpi'] else 'off'),
  NameError: name 'cl_opt' is not defined
  ----------------------------------------
  ERROR: Failed building wheel for arbor
  Running setup.py clean for arbor
Failed to build arbor
Installing collected packages: arbor
    Running setup.py install for arbor ... done
Successfully installed arbor-0.2.3.dev0

[Helveg]

pip install git+https://github.com/arbor/arbor.git 🎉 🎈