VirtualEnv and Anaconda – Frustrations

by Gilbert Keith

Having anaconda on your mac is nice (IPython Notebook!)

But it totally breaks things if you are trying to use virtualenv on top of that.

Whenever you create a virtualenv, you get a bad looking error which is not very easy to parse.

GKs-MacBook-Air:~ gk$ virtualenv venv
New python executable in venv/bin/python
Installing setuptools, pip...
  Complete output from command /Users/gk/playtestr/bin/python -c "import sys, pip; sys...d\"] + sys.argv[1:]))" setuptools pip:
  Ignoring indexes: https://pypi.python.org/simple/
Exception:
Traceback (most recent call last):
  File "/Users/gk/anaconda/lib/python2.7/site-packages/virtualenv_support/pip-1.5.4-py2.py3-none-any.whl/pip/basecommand.py", line 122, in main
    status = self.run(options, args)
  File "/Users/gk/anaconda/lib/python2.7/site-packages/virtualenv_support/pip-1.5.4-py2.py3-none-any.whl/pip/commands/install.py", line 236, in run
    session = self._build_session(options)
  File "/Users/gk/anaconda/lib/python2.7/site-packages/virtualenv_support/pip-1.5.4-py2.py3-none-any.whl/pip/basecommand.py", line 52, in _build_session
    session = PipSession()
  File "/Users/gk/anaconda/lib/python2.7/site-packages/virtualenv_support/pip-1.5.4-py2.py3-none-any.whl/pip/download.py", line 216, in __init__
    super(PipSession, self).__init__(*args, **kwargs)
  File "/Users/gk/anaconda/lib/python2.7/site-packages/virtualenv_support/pip-1.5.4-py2.py3-none-any.whl/pip/_vendor/requests/sessions.py", line 200, in __init__
    self.headers = default_headers()
  File "/Users/gk/anaconda/lib/python2.7/site-packages/virtualenv_support/pip-1.5.4-py2.py3-none-any.whl/pip/_vendor/requests/utils.py", line 550, in default_headers
    'User-Agent': default_user_agent(),
  File "/Users/gk/anaconda/lib/python2.7/site-packages/virtualenv_support/pip-1.5.4-py2.py3-none-any.whl/pip/_vendor/requests/utils.py", line 519, in default_user_agent
    _implementation = platform.python_implementation()
  File "/Users/gk/anaconda/lib/python2.7/platform.py", line 1499, in python_implementation
    return _sys_version()[0]
  File "/Users/gk/anaconda/lib/python2.7/platform.py", line 1464, in _sys_version
    repr(sys_version))
ValueError: failed to parse CPython sys.version: '2.7.5 (default, Aug 25 2013, 00:04:04) \n[GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.0.68)]'

Storing debug log for failure in /Users/gk/.pip/pip.log
----------------------------------------
...Installing setuptools, pip...done.
Traceback (most recent call last):
  File "/Users/gk/anaconda/bin/virtualenv", line 11, in 
    sys.exit(main())
  File "/Users/gk/anaconda/lib/python2.7/site-packages/virtualenv.py", line 824, in main
    symlink=options.symlink)
  File "/Users/gk/anaconda/lib/python2.7/site-packages/virtualenv.py", line 992, in create_environment
    install_wheel(to_install, py_executable, search_dirs)
  File "/Users/gk/anaconda/lib/python2.7/site-packages/virtualenv.py", line 960, in install_wheel
    'PIP_NO_INDEX': '1'
  File "/Users/gk/anaconda/lib/python2.7/site-packages/virtualenv.py", line 902, in call_subprocess
    % (cmd_desc, proc.returncode))
OSError: Command /Users/gk/venv/bin/python -c "import sys, pip; sys...d\"] + sys.argv[1:]))" setuptools pip failed with error code 2

I started off trying to figure out what the hell CPython was, and if I could get rid of it. Turns out, it’s a fool’s errand. Don’t bother with my silly ways.

This is what CPython is (and hopefully you see why messing with it is bad) – From StackOverflow:

So what is CPython

CPython is the original Python implementation. It is the implementation you download from Python.org. People call it CPython to distinguish it from other, later, Python implementations, and to distinguish the implementation of the language engine from the Python programming language itself.

The latter part is where your confusion comes from; you need to keep Python-the-language separate from whatever runs the Python code.

CPython happens to be implemented in C. That is just an implementation detail really. CPython compiles your python code into bytecode (transparently) and interprets that bytecode in a evaluation loop.

What about Jython, etc.

Jython, IronPython and PyPy are the current ‘other’ implementations of the Python programming language; these are implemented in Java, C# and RPython (a subset of Python), respectively. Jython compiles your Python code to Java bytecode, so your Python code can run on the JVM. IronPython lets you run Python on the Microsoft CLR. And PyPy, being implemented in (a subset of) Python, lets you run Python code faster than CPython, which rightly should blow your mind. 🙂

Actually compiling to C

So CPython does itself not translate your Python code to C. It instead runs a interpreter loop. There is a project that does translate Python-ish code to C, and that is called Cython. Cython adds a few extensions to the Python language, and lets you compile your code to C extensions, code that plugs into the CPython interpreter.

Here’s a basic explanation of what’s going on:

  • I’m trying to create a virtualenv for an app
  • Virtualenv does some calls to underlying libraries for its thing
  • There’s something wrong with one of my libraries

After about an hour of searching, I came across this on the python.org bug tracker.

Digging through the whole thing, it became more clear what was going on – Anaconda had introduced something which breaks everything.

The user wesmadrigal had said he’d fixed it:

I just commented out the _sys_version_parser regular expression in 
anaconda/lib/python2.7/platform.py on line 1363 and replaced it with the
_sys_version_parser from /usr/lib/python2.7/platform.py and everything worked fine.

What he’s saying is that if you do this:

GKs-MacBook-Air:~ gk$ python
Python 2.7.6 |Anaconda 1.9.1 (x86_64)| (default, Jan 10 2014, 11:23:15) 
[GCC 4.0.1 (Apple Inc. build 5493)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import platform
>>> platform.__file__
'/Users/gk/anaconda/lib/python2.7/platform.pyc'

and open up the .py file (not the .pyc!) referenced therein, and search for ‘_sys_version_parser’, it should look like this:

_sys_version_parser = re.compile(
    r'([\w.+]+)\s*'
    '\(#?([^,]+),\s*([\w ]+),\s*([\w :]+)\)\s*'
    '\[([^\]]+)\]?')

and not like this:

_sys_version_parser = re.compile(
    r'([\w.+]+)\s*'
    '\|[^|]*\|\s*' # version extra
    '\(#?([^,]+),\s*([\w ]+),\s*([\w :]+)\)\s*'
    '\[([^\]]+)\]?')

It looks like wesmadrigal has submitted a bug to the folks at Continuum who write code for Anaconda.

So if you run into this error, hopefully the above will fix things.

I don’t know if

conda update conda

or

conda update anaconda

will break this again, though. Gotta keep an eye out.