On new debian 8 installs using google_compute_engine virtualenv installed boto is broken

[gcarothers]

Not exactly sure what's going on here but we are now seeing:

  File "/home/lex/local/lib/python2.7/site-packages/boto/__init__.py", line 1216, in <module>
    boto.plugin.load_plugins(config)
  File "/home/lex/local/lib/python2.7/site-packages/boto/plugin.py", line 93, in load_plugins
    _import_module(file)
  File "/home/lex/local/lib/python2.7/site-packages/boto/plugin.py", line 75, in _import_module
    return imp.load_module(name, file, filename, data)
  File "/usr/lib/python2.7/dist-packages/google_compute_engine/boto/boto_config.py", line 29, in <module>
    from google_compute_engine import config_manager
ImportError: No module named google_compute_engine

Note that this is a virtualenv installed copy of boto which ends up failing with an error in google_compute_engines vendoring(?) of boto?

[zmarano]

Hi, we'll see if we can figure out what is going on here. Can you describe why you are using a virtual env for boto and also how are you setting this up (which version etc) so we can reproduce?

[gcarothers]

# This file is automatically created at boot time by the /usr/lib/python
# 2.7/dist-packages/google_compute_engine/boto/boto_config.pyc script.
# Do not edit this file directly. If you need to add items to this file,
# create or edit /etc/boto.cfg.template instead and then re-run the
# script.

[GSUtil]
default_project_id = 45098631325
default_api_version = 2

[GoogleCompute]
service_account = default

[Plugin]
plugin_directory = /usr/lib/python2.7/dist-packages/google_compute_engine/boto

Yeah so that setups loading plugins from a package that doesn't exist in the virtualenv

[gcarothers]

Grab a normal GCE image from the current debian-8, create a virtualenv, install boto, try and import boto. Kaboom.

[gcarothers]

The old compute_auth.py plugin worked fine inside a virtualenv, the new plugin does not.

[illfelder]

To summarize the issue:

• The Linux guest environment is installed via the google-compute-engine package on GCE images.
• The package is located in: /usr/lib/python2.7/dist-packages/google_compute_engine
• The file that sets up credentials for boto is a part of the google-compute-engine package.
• Running a virtualenv removes Python dist-packages from the PATH.

In order to use boto from inside a virtualenv, you can set your Python path with the command:
export PYTHONPATH="/usr/lib/python2.7/dist-packages/google_compute_engine"

[gcarothers]

You can also create a virtualenv that includes the system site packages (/usr/lib/python2.7/dist-packages/). However the previous boto configuration worked perfectly fine as the plugin it specified had no dependencies except boto itself. The new boto plugin requires google_compute_engine which can't easily be installed in the virtualenv as this package isn't available from pypi.

[illfelder]

One of the reasons behind the improved packaging is to enable PyPI distribution of the Linux guest environment, which is currently in progress.

[gcarothers]

In that case I'd recommend noting a work around for anyone else running their own code using boto on GCE create any virtualenvs with virtualenv --system-site-packages otherwise the on import exceptions are quite startling.

[illfelder]

I've updated the README to include troubleshooting instructions for using boto.

[illfelder]

We now have a PyPI package for the Linux guest environment.

The package is available using: pip install google-compute-engine.

[gabrielspmoreira]

I also got an error with boto installation in Python 3 on Google Dataproc instances running Debian 8.
I initialize a Dataproc cluster, install Python 3 and install google-compute-engine package with pip...

apt-get install -y python3 python3-dev python3-pip
pip3 install -v google-compute-engine

when I try to import boto in python 3 ...
python3 -c "import boto"

I get the following error:

File "/usr/local/lib/python3.4/dist-packages/boto/plugin.py", line 75, in _import_module
return imp.load_module(name, file, filename, data)
File "/usr/lib/python3.4/imp.py", line 235, in load_module
return load_source(name, filename, file)
File "/usr/lib/python3.4/imp.py", line 171, in load_source
module = methods.load()
File "/usr/share/google/boto/boto_plugins/compute_auth.py", line 18, in
import urllib2
ImportError: No module named 'urllib2'

After a long time of investigation, I guessed the problem. Boto saves its config in /etc/boto.cfg .
In Dataproc instances, the [Plugin] section of the config file is as follows, pointing to a directory with Python 2.7 code (which refers urllib2, not available in Python 3)

[Plugin]
plugin_directory = /usr/share/google/boto/boto_plugins

The /etc/boto.cfg file is usually managed using scripts, so I just included this command in my bash script just after google-compute-engine package install, to update /etc/boto.cfg with the correct plugin_directory (/usr/local/lib/python3.4/dist-packages/google_compute_engine/boto)

python3 -c "from google_compute_engine.boto.boto_config import BotoConfig; BotoConfig()"

Just sharing in the case anyone runs into the same problem involving this setup: Dataproc (Image version v1.0), Python3 and boto.

[illfelder]

The Dataproc images have not updated to the new guest environment. When they next update their images, they should hopefully pick up the new guest environment and work the same as the standard Debian 8 images on Google Compute Engine.

[domenkozar]

Been hitting this (on travis-ci 14.04). Seems like google-compute-engine has a cyclic dependency on boto.

Global system-wide side effects are bad (lead into such trouble), this should be fixed at design level.

[alvinlindstam]

The suggested solution (adding system packages to the virtualenv or installing google-compute-engine into it) are not sufficient. Python virtualenvs are isolated from system packages (by default) for a reason, you want to be able to control where modules are imported from in a virtualenv. You should also not have to pip install a gce library into a virtualenv that doesn't use/need it just to be able to run boto.

I do find this to be an issue with boto itself, but google also has a responsibility here. I don't know why you can't just define the boto config file locally for projects using this, but if that's not the case can't you at least catch the import errors of google_compute_engine in the plugin you inject do nothing if google_compute_engine is not available?