Ian Bicking: a blog :: Workingenv is dead, long live Virtualenv!

{ 2007 10 10 }

Workingenv is dead, long live Virtualenv!

A lot of people have found workingenv useful, but it’s always been a bit fragile. If you’ve seen the .../site.py is not a setuptools-generated site.py; please remove it. message, you know what I mean.

For a while I tried to refactor and improve workingenv, but it didn’t go very well. So I decided to ditch it completely and revisit the ideas of virtual-python.py. That script works by copying the Python executable, and in doing so change sys.prefix — it’s pretty consistent that all other paths in Python derive from sys.prefix.

The result is virtualenv, which I think is now featureful enough for general use. It might still be buggy, but it’s worked well for some of us, and I expect the bugs to all be much easier than they were in workingenv.

Unlike virtual-python.py, virtualenv works on Windows and in the latest release also Mac framework builds. It also handles site-packages better, so you can manage some of your packages using packages from your OS distribution (e.g., debs or rpms), while also installing environment-local packages.

So, in summary: use virtualenv, don’t use workingenv. And if you were using the --requirements option to workingenv, the package PoachEggs lets you do a similar batch installation after you’ve created the environment.

Automatically generated list of related posts:

pyinstall is dead, long live pip! I’ve finished renaming pyinstall to its new name: pip. The...
Monkeypatching and dead ends Bill de hÓra and then Patrick Logan picked up on...
2 Python Environment Experiments two experiments in the Python environment. The first is virtualenv,...

Posted by Ian Bicking on Wednesday, October 10th, 2007, at 10:00 am, and filed under Python.

Comments have a feed.

33 Comments

Brian Beck says:

October 10, 2007 at 12:40 pm

To get easy_installed libraries in a working-env to show up in a mod_python deployment, I’ve been using the activate_workingenv.py script from the Pylons wiki (for example I run this in the settings.py file in Django apps, alright so it’s a little weird). Any other solutions, e.g. involving PythonImport, totally failed to work for me.

It sounds like this’ll be easier with virtual-env – what do you think it might look like to have, say, two mod_python apps using different virtual-envs?

Reply
Ian Bicking says:

October 10, 2007 at 12:47 pm

Well, it’ll be a little tricky. It would be nice if there was a clear way to handle this in mod_python or mod_wsgi. Maybe we’ll figure something out.

Anyway, in mod_* you can’t really change the Python interpreter to change the environment, so the normal way of “activating” the virtual environment won’t work (i.e., you can’t change your PATH or your interpreter). But that really matters most for installation and management of packages, which you probably will be doing from the command-line anyway.

If you just want to get the libraries from the virtual environment, not install anything, import site; site.addsitedir('/path/to/env/lib/pythonX.Y') should do it.

Reply
Graham Dumpleton says:

October 10, 2007 at 4:54 pm

The best you can do with mod_python and mod_wsgi is point the whole Apache server at a different virtualenv directory. This must though be for the exact same Python version that mod_python and mod_wsgi was originally compiled for as it is still using the embedded Python interpreter that it was compiled with and is ignoring the python executable in the virtualenv directory.

This all means that you cannot have different applications running inside of mod_python and mod_wsgi using different virtualenv environments. This is the case even for mod_wsgi daemon mode, as the Python interpreter is only initialised once in the Apache parent process and the daemons all inherit that instance.

If using mod_python you can point it at a different virtualenv by setting and exporting PYTHONEXECUTABLE environment variable in the Apache envvars file. In mod_wsgi you can use the WSGIPythonExecutable directive in the Apache configuration file. The WSGIPythonExecutable directive only sets the reference point though so Python can find where the library directory is actually located.

I have some ideas for how to better support virtualenv with daemon mode of mod_wsgi, but it effectively involves creating a hybrid system where mod_wsgi would run up separate Python process which it would transparently manage and direct to load up and run mod_wsgi specific modules to handle the in Python process side of the daemon mechanism. Note sure when I will get around to it though as want to get transient daemon process support added first.

Other than that I really need to research what the perceived problems are with workingenv and see whether it is still an adequate solution for more constrained environment of mod_wsgi.

Reply
Ian Bicking says:

October 10, 2007 at 5:06 pm

Graham: I don’t think it’s as complicated as all that. All the complexity of workingenv and virtualenv are around distutils and having scripts use the right libraries and paths. But just getting the libraries loaded on sys.path is pretty simple. As long as you aren’t installing packages from inside the mod_wsgi environment, all you have to do is get the libraries loaded. And all you have to do to load the libraries is that one line of code I gave (well, two lines).

That still means you have to have separate Python environments with different sys.path’s. From what I understand, you can have that with mod_wsgi, as there are separate interpreters. It’s just that all the interpreters are forks of a single original interpreter. And that’s fine.

Reply
Graham Dumpleton says:

October 10, 2007 at 5:18 pm

Agree, and using daemon mode of mod_wgsi means you can isolate applications wanting to use incompatible versions of C extension modules for Python.

In respect of hybrid mode am actually looking a bit further in trying to address one of the other complaints about solutions such as mod_python and mod_wsgi for embedding Python in Apache. That is the restriction that one must use the version (major/minor) of Python that the Apache module was compiled for and you cant use anything else. Some people have been quite critical of this as they want to be able to use different versions of Python for different applications. Yes they could use mod_proxy with Python applications managed by separate supervisor, but am thinking about how one could support this whereby Apache was used as the supervisor, thereby avoiding the need to install a separate supervisor system with all its dependencies.

If support for this was ever added, then it becomes very simple to use virtualenv as in designating the mod_wsgi daemon process you define the location of the python executable in the virtualenv and instead of just a fork for daemon and using internal mod_wsgi server support, it does a subsequent exec and runs up mod_wsgi server support in the separate Python process.

So, I don’t discount that doing necessary sys.path fixups is easy, and that someone has written instructions on how to do it in mod_wsgi already indicates it does work, I just still haven’t had time to sit down and look at it all properly, but I will. :-)

Reply
michele says:

October 10, 2007 at 5:37 pm

Great work Ian.

Regarding activating a workingenv/virtualenv, activate_workingenv.py is really over engineered. I faced this problem two weeks ago and ended up with the very same solution you gave (site.addsitedir), it’s simple and I can confirm that it works really great (I’m using it inside postgres for some plpythonu functions).

Reply
Brian Beck says:

October 11, 2007 at 3:01 pm

Thanks for the tip about site.addsitedir, I guess I didn’t realize that active_workingenv.py even solved a different problem.

Reply
Dobes Vandermeer says:

October 11, 2007 at 7:31 pm

activate.bat doesn’t seem to work in windows much. I was going to diagnose the problem further but looking at the script I can see this has no hope in hell of working and just needs an overhaul. All the percent signs and backslashed are doubled, the _OLD_VIRTUAL_PROMPT is used before it is set, and it sets the path to %VIRTUAL_ENV%\bin, even though python and easy_install are being installed in Scripts … Aside from that, I would prefer to see it installing things into bin/ for cross-platform compatibility, too, instead of Scripts.

C:\work>test_env\Scripts\activate.bat

(test_env) %PROMPT%deactivate

‘deactivate’ is not recognized as an internal or external command, operable program or batch file.

(test_env) %PROMPT%python

‘python’ is not recognized as an internal or external command, operable program or batch file.

(test_env) %PROMPT%easy_install

‘easy_install’ is not recognized as an internal or external command, operable program or batch file.

[Note: fixed in trunk]

Reply
Stephen Day says:

October 16, 2007 at 8:21 am

Has anyone managed to use this (or workingenv.py) for running a Windows service? I’m not sure how to get the service to use the proper python.exe.

Reply
Martin says:

October 16, 2007 at 9:29 am

I’ve not been able to use a ‘python setup.py develop’ with virtualenv on macosx, something that used to work with workingenv.py and a ‘user mac os x setup’ of easyinstall. It failes with:

[Errno 13] Permission denied: ‘/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/test-easy-install-27420.write-test’

Why does it want to write to the framework directory? Other than that, easy_install works like it should. Great work!

Reply
Ian Bicking says:

October 16, 2007 at 9:37 am

Just to be sure, I double-checked python setup.py develop on Linux, and it works fine there. Of course you must use the right Python executable for it to work at all. Otherwise it’s possible that there’s something funny there on Macs in particular, as there is some differences in the code paths there.

Reply

robert says:

October 21, 2007 at 3:56 pm

Problem installing mercurial with virtualenv v0.9.1 on ubuntu 7.04/python2.5:

(ve_msprod)robert@lightspeed:~$ ve_msprod/bin/easy_install mercurial
Searching for mercurial
Reading <a class="pypi.python.org/simple/mercurial/" rel="nofollow">pypi.python.org/simple/mercurial/</a>
Couldn't find index page for 'mercurial' (maybe misspelled?)
Scanning index of all packages (this may take a while)
Reading <a class="pypi.python.org/simple/" rel="nofollow">pypi.python.org/simple/</a>
Reading <a class="pypi.python.org/simple/Mercurial/" rel="nofollow">pypi.python.org/simple/Mercurial/</a>
Reading <a class="www.selenic.com/mercurial" rel="nofollow">www.selenic.com/mercurial</a>
Best match: mercurial 0.9.5
Downloading <a class="www.selenic.com/mercurial/release/mercurial-0.9.5.tar.gz" rel="nofollow">www.selenic.com/mercurial/release/mercurial-0.9.5.tar.gz</a>
Processing mercurial-0.9.5.tar.gz
Running mercurial-0.9.5/setup.py -q bdist_egg --dist-dir /tmp/easy_install-mpa28t/mercurial-0.9.5/egg-dist-tmp-RFJG3c
Traceback (most recent call last):
  File "ve_msprod/bin/easy_install", line 8, in 
    load_entry_point('setuptools==0.6c7', 'console_scripts', 'easy_install')()
  File "/home/robert/ve_msprod/lib/python2.5/site-packages/setuptools-0.6c7-py2.5.egg/setuptools/command/easy_install.py", line 1670, in main
  File "/home/robert/ve_msprod/lib/python2.5/site-packages/setuptools-0.6c7-py2.5.egg/setuptools/command/easy_install.py", line 1659, in with_ei_usage
  File "/home/robert/ve_msprod/lib/python2.5/site-packages/setuptools-0.6c7-py2.5.egg/setuptools/command/easy_install.py", line 1674, in 
  File "distutils/core.py", line 151, in setup
  File "distutils/dist.py", line 974, in run_commands
  File "distutils/dist.py", line 994, in run_command
  File "/home/robert/ve_msprod/lib/python2.5/site-packages/setuptools-0.6c7-py2.5.egg/setuptools/command/easy_install.py", line 211, in run
  File "/home/robert/ve_msprod/lib/python2.5/site-packages/setuptools-0.6c7-py2.5.egg/setuptools/command/easy_install.py", line 446, in easy_install
  File "/home/robert/ve_msprod/lib/python2.5/site-packages/setuptools-0.6c7-py2.5.egg/setuptools/command/easy_install.py", line 471, in install_item
  File "/home/robert/ve_msprod/lib/python2.5/site-packages/setuptools-0.6c7-py2.5.egg/setuptools/command/easy_install.py", line 655, in install_eggs
  File "/home/robert/ve_msprod/lib/python2.5/site-packages/setuptools-0.6c7-py2.5.egg/setuptools/command/easy_install.py", line 930, in build_and_install
  File "/home/robert/ve_msprod/lib/python2.5/site-packages/setuptools-0.6c7-py2.5.egg/setuptools/command/easy_install.py", line 919, in run_setup
  File "/home/robert/ve_msprod/lib/python2.5/site-packages/setuptools-0.6c7-py2.5.egg/setuptools/sandbox.py", line 27, in run_setup
  File "/home/robert/ve_msprod/lib/python2.5/site-packages/setuptools-0.6c7-py2.5.egg/setuptools/sandbox.py", line 63, in run
  File "/home/robert/ve_msprod/lib/python2.5/site-packages/setuptools-0.6c7-py2.5.egg/setuptools/sandbox.py", line 29, in 
  File "setup.py", line 16, in 
  File "/tmp/easy_install-mpa28t/mercurial-0.9.5/mercurial/version.py", line 16, in 
  File "/tmp/easy_install-mpa28t/mercurial-0.9.5/mercurial/util.py", line 1308, in 
TypeError: Error when calling the metaclass bases
    first argument must be callable

[I believe this is just a conflict between easyinstall and Mercurial, as Mercurial is subclassing an object that easyinstall monkeypatches]

robert says:

October 21, 2007 at 10:48 pm

> conflict between easyinstall and Mercurial

It is looking that way… I have a workingenv based setup with a successful egg install of mercurial, but I cannot successfully install it again w/o seeing the same problem – it is likely/possible that easy_install was upgraded at some point in the last week or two.

As an aside are there any issues with copying the installed packages from a workingenv-based ‘lib/python2.5′ to a virtualenv-based ‘lib/python2.5/site-packages’ and updating the easy_install.pth accordingly?

Reply
d2m says:

November 1, 2007 at 2:16 am

Problem installing virtualenv on MacOSX 10.3.9 (PPC), python 2.5.1 Framework build:

in create_environment the python interpreter is copied and installed. The necessary install_name_tool executable is available from the Apple developer tools package only (which is quite a big dependency of 1GB in size and you’ve also got to have some spare time and a CD/DVD around to install it).

Do you think this dependency could be worked around ?

Reply
gkamp says:

November 5, 2007 at 1:54 am

Tried to use virtualenv on MacOsX 10.4 and 10.5 (with preinstalled easyinstall. I had to run it with admin rights (In order to have write permissions for /usr/local/bin and /Library/Python). At first glance everything but one thing looks good: Seems like easyinstall is missing from the bin dir in the virtualenv. Am i missing something

Reply
agc says:

November 9, 2007 at 10:36 pm

I also am seeing the behavior that gkamp (in above post) is seeing. I’m running Leopard (os x 10.5). Also, before using virtualenv I had to install XCode, (as d2m mentions above) is this behavior required/expected? Thanks for any info.

Reply
Graham Dumpleton says:

November 11, 2007 at 2:13 am

FWIW, I have also had various issues on MacOS X with virtualenv and packages still wanting to write into system directories where root privileges would be required. It got frustrating enough that I gave up and went back to using what I call a poor man’s Python virtual environment. That is, use the inbuilt feature of Python to define the PYTHONHOME environment variable to be where the Python installation is installed. You can find more details on my blog about it.

Reply
Ian Bicking says:

November 11, 2007 at 10:21 pm

These Mac problems should be fixed in the newest release.

Reply
agc says:

November 12, 2007 at 10:49 pm

Thanks Ian, looking forward to it.

Reply
gkamp says:

November 12, 2007 at 11:44 pm

Gave it a try yesterday evening. Looks good to me.

Reply

Graham Dumpleton says:

November 13, 2007 at 3:32 am

Version 0.9.2 still doesn’t work on MacOS X 10.4 with OS supplied Python 2.3.

$ easy_install "TurboGears==1.0.4b2"
'import site' failed; use -v for traceback
Traceback (most recent call last):
  File "/usr/local/wsgi/virtualenv/ENV1/bin/easy_install", line 5, in ?
    from pkg_resources import load_entry_point
ImportError: No module named pkg_resources

This time it appears to be because ‘set’ was used in site.py when Python 2.3 doesn’t support ‘set’. Using ‘python -v’ with easy_install yields:

'import site' failed; traceback:
Traceback (most recent call last):
  File "/usr/local/wsgi/virtualenv/ENV1/bin/../lib/python2.3/site.py", line 432, in ?
    main()
  File "/usr/local/wsgi/virtualenv/ENV1/bin/../lib/python2.3/site.py", line 412, in main
    paths_in_sys = removeduppaths()
  File "/usr/local/wsgi/virtualenv/ENV1/bin/../lib/python2.3/site.py", line 87, in removeduppaths
    known_paths = set()
NameError: global name 'set' is not defined

Graham Dumpleton says:

November 13, 2007 at 5:12 am

BTW, any chance you can document –no-site-packages option on PyPi site where documentation is for virtualenv. I see this as an important option when using mod_wsgi for shared hosting as in that sort of situation you want to base the underlying Python environment used by mod_wsgi off a virtual environment which has an empty site-packages. Specific applications would then add on top using their respective virtual environments using site.addsitedir() based mechanisms. If however the application owners didn’t use –no-site-packages option and a required module was in global site-packages, and thus easy_install didn’t add a copy to their own virtual environment, that module can then show up as missing when their application is run since mod_wsgi will not be using the globally installed Python environment.

Reply
Sean Upton says:

November 13, 2007 at 5:11 pm

On Win32, packages that distutils need to build C modules for as part of easy_install within virtualenv require (at least for me), the workaround of copying c:\Python25\libs and c:\Python25\include – since there is not an include symlink li