Installing Python Modules¶

Best case: trivial installation¶

In the best case, someone will have prepared a special version of the module distribution you want to install that is targeted specifically at your platform and is installed just like any other software on your platform. For example, the module developer might make an executable installer available for Windows users, an RPM package for users of RPM-based Linux systems (Red Hat, SuSE, Mandrake, and many others), a Debian package for users of Debian-based Linux systems, and so forth.

In that case, you would download the installer appropriate to your platform and do the obvious thing with it: run it if it’s an executable installer, rpm --install it if it’s an RPM, etc. You don’t need to run Python or a setup script, you don’t need to compile anything—you might not even need to read any instructions (although it’s always a good idea to do so anyways).

Of course, things will not always be that easy. You might be interested in a module distribution that doesn’t have an easy-to-use installer for your platform. In that case, you’ll have to start with the source distribution released by the module’s author/maintainer. Installing from a source distribution is not too hard, as long as the modules are packaged in the standard way. The bulk of this document is about building and installing modules from standard source distributions.

The new standard: Distutils¶

If you download a module source distribution, you can tell pretty quickly if it was packaged and distributed in the standard way, i.e. using the Distutils. First, the distribution’s name and version number will be featured prominently in the name of the downloaded archive, e.g. foo-1.0.tar.gz or widget-0.9.7.zip. Next, the archive will unpack into a similarly-named directory: foo-1.0 or widget-0.9.7. Additionally, the distribution will contain a setup script setup.py, and a file named README.txt or possibly just README, which should explain that building and installing the module distribution is a simple matter of running

python setup.py install

If all these things are true, then you already know how to build and install the modules you’ve just downloaded: Run the command above. Unless you need to install things in a non-standard way or customize the build process, you don’t really need this manual. Or rather, the above command is everything you need to get out of this manual.

Platform variations¶

You should always run the setup command from the distribution root directory, i.e. the top-level subdirectory that the module source distribution unpacks into. For example, if you’ve just downloaded a module source distribution foo-1.0.tar.gz onto a Unix system, the normal thing to do is:

gunzip -c foo-1.0.tar.gz | tar xf -    # unpacks into directory foo-1.0
cd foo-1.0
python setup.py install

On Windows, you’d probably download foo-1.0.zip. If you downloaded the archive file to C:\Temp, then it would unpack into C:\Temp\foo-1.0; you can use either a archive manipulator with a graphical user interface (such as WinZip) or a command-line tool (such as unzip or pkunzip) to unpack the archive. Then, open a command prompt window (“DOS box”), and run:

cd c:\Temp\foo-1.0
python setup.py install

Splitting the job up¶

Running setup.py install builds and installs all modules in one run. If you prefer to work incrementally—especially useful if you want to customize the build process, or if things are going wrong—you can use the setup script to do one thing at a time. This is particularly helpful when the build and install will be done by different users—for example, you might want to build a module distribution and hand it off to a system administrator for installation (or do it yourself, with super-user privileges).

For example, you can build everything in one step, and then install everything in a second step, by invoking the setup script twice:

python setup.py build
python setup.py install

If you do this, you will notice that running the install command first runs the build command, which—in this case—quickly notices that it has nothing to do, since everything in the build directory is up-to-date.

You may not need this ability to break things down often if all you do is install modules downloaded off the ‘net, but it’s very handy for more advanced tasks. If you get into distributing your own Python modules and extensions, you’ll run lots of individual Distutils commands on their own.

How building works¶

As implied above, the build command is responsible for putting the files to install into a build directory. By default, this is build under the distribution root; if you’re excessively concerned with speed, or want to keep the source tree pristine, you can change the build directory with the --build-base option. For example:

python setup.py build --build-base=/tmp/pybuild/foo-1.0

(Or you could do this permanently with a directive in your system or personal Distutils configuration file; see section Distutils Configuration Files.) Normally, this isn’t necessary.

The default layout for the build tree is as follows:

--- build/ --- lib/
or
--- build/ --- lib.<plat>/
               temp.<plat>/

where <plat> expands to a brief description of the current OS/hardware platform and Python version. The first form, with just a lib directory, is used for “pure module distributions”—that is, module distributions that include only pure Python modules. If a module distribution contains any extensions (modules written in C/C++), then the second form, with two <plat> directories, is used. In that case, the temp.plat directory holds temporary files generated by the compile/link process that don’t actually get installed. In either case, the lib (or lib.plat) directory contains all Python modules (pure Python and extensions) that will be installed.

In the future, more directories will be added to handle Python scripts, documentation, binary executables, and whatever else is needed to handle the job of installing Python modules and applications.

How installation works¶

After the build command runs (whether you run it explicitly, or the install command does it for you), the work of the install command is relatively simple: all it has to do is copy everything under build/lib (or build/lib.plat) to your chosen installation directory.

If you don’t choose an installation directory—i.e., if you just run setup.py install—then the install command installs to the standard location for third-party Python modules. This location varies by platform and by how you built/installed Python itself. On Unix (and Mac OS X, which is also Unix-based), it also depends on whether the module distribution being installed is pure Python or contains extensions (“non-pure”):

Notes:

1. Most Linux distributions include Python as a standard part of the system, so prefix and exec-prefix are usually both /usr on Linux. If you build Python yourself on Linux (or any Unix-like system), the default prefix and exec-prefix are /usr/local.
2. The default installation directory on Windows was C:\Program Files\Python under Python 1.6a1, 1.5.2, and earlier.

prefix and exec-prefix stand for the directories that Python is installed to, and where it finds its libraries at run-time. They are always the same under Windows, and very often the same under Unix and Mac OS X. You can find out what your Python installation uses for prefix and exec-prefix by running Python in interactive mode and typing a few simple commands. Under Unix, just type python at the shell prompt. Under Windows, choose Start ‣ Programs ‣ Python X.Y ‣ Python (command line). Once the interpreter is started, you type Python code at the prompt. For example, on my Linux system, I type the three Python statements shown below, and get the output as shown, to find out my prefix and exec-prefix:

Python 2.4 (#26, Aug  7 2004, 17:19:02)
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.prefix
'/usr'
>>> sys.exec_prefix
'/usr'

If you don’t want to install modules to the standard location, or if you don’t have permission to write there, then you need to read about alternate installations in section Alternate Installation. If you want to customize your installation directories more heavily, see section Custom Installation on custom installations.

Alternate installation: the home scheme¶

The idea behind the “home scheme” is that you build and maintain a personal stash of Python modules. This scheme’s name is derived from the idea of a “home” directory on Unix, since it’s not unusual for a Unix user to make their home directory have a layout similar to /usr/ or /usr/local/. This scheme can be used by anyone, regardless of the operating system their installing for.

Installing a new module distribution is as simple as

python setup.py install --home=<dir>

where you can supply any directory you like for the --home option. On Unix, lazy typists can just type a tilde (~); the install command will expand this to your home directory:

python setup.py install --home=~

The --home option defines the installation base directory. Files are installed to the following directories under the installation base as follows:

Alternate installation: Unix (the prefix scheme)¶

The “prefix scheme” is useful when you wish to use one Python installation to perform the build/install (i.e., to run the setup script), but install modules into the third-party module directory of a different Python installation (or something that looks like a different Python installation). If this sounds a trifle unusual, it is—that’s why the “home scheme” comes first. However, there are at least two known cases where the prefix scheme will be useful.

First, consider that many Linux distributions put Python in /usr, rather than the more traditional /usr/local. This is entirely appropriate, since in those cases Python is part of “the system” rather than a local add-on. However, if you are installing Python modules from source, you probably want them to go in /usr/local/lib/python2.X rather than /usr/lib/python2.X. This can be done with

/usr/bin/python setup.py install --prefix=/usr/local

Another possibility is a network filesystem where the name used to write to a remote directory is different from the name used to read it: for example, the Python interpreter accessed as /usr/local/bin/python might search for modules in /usr/local/lib/python2.X, but those modules would have to be installed to, say, /mnt/@server/export/lib/python2.X. This could be done with

/usr/local/bin/python setup.py install --prefix=/mnt/@server/export

In either case, the --prefix option defines the installation base, and the --exec-prefix option defines the platform-specific installation base, which is used for platform-specific files. (Currently, this just means non-pure module distributions, but could be expanded to C libraries, binary executables, etc.) If --exec-prefix is not supplied, it defaults to --prefix. Files are installed as follows:

There is no requirement that --prefix or --exec-prefix actually point to an alternate Python installation; if the directories listed above do not already exist, they are created at installation time.

Incidentally, the real reason the prefix scheme is important is simply that a standard Unix installation uses the prefix scheme, but with --prefix and --exec-prefix supplied by Python itself as sys.prefix and sys.exec_prefix. Thus, you might think you’ll never use the prefix scheme, but every time you run python setup.py install without any other options, you’re using it.

Note that installing extensions to an alternate Python installation has no effect on how those extensions are built: in particular, the Python header files (Python.h and friends) installed with the Python interpreter used to run the setup script will be used in compiling extensions. It is your responsibility to ensure that the interpreter used to run extensions installed in this way is compatible with the interpreter used to build them. The best way to do this is to ensure that the two interpreters are the same version of Python (possibly different builds, or possibly copies of the same build). (Of course, if your --prefix and --exec-prefix don’t even point to an alternate Python installation, this is immaterial.)

Alternate installation: Windows (the prefix scheme)¶

Windows has no concept of a user’s home directory, and since the standard Python installation under Windows is simpler than under Unix, the --prefix option has traditionally been used to install additional packages in separate locations on Windows.

python setup.py install --prefix="\Temp\Python"

to install modules to the \Temp\Python directory on the current drive.

The installation base is defined by the --prefix option; the --exec-prefix option is not supported under Windows. Files are installed as follows:

Modifying Python’s Search Path¶

When the Python interpreter executes an import statement, it searches for both Python code and extension modules along a search path. A default value for the path is configured into the Python binary when the interpreter is built. You can determine the path by importing the sys module and printing the value of sys.path.

$ python
Python 2.2 (#11, Oct  3 2002, 13:31:27)
[GCC 2.96 20000731 (Red Hat Linux 7.3 2.96-112)] on linux2
Type ``help'', ``copyright'', ``credits'' or ``license'' for more information.
>>> import sys
>>> sys.path
['', '/usr/local/lib/python2.3', '/usr/local/lib/python2.3/plat-linux2',
 '/usr/local/lib/python2.3/lib-tk', '/usr/local/lib/python2.3/lib-dynload',
 '/usr/local/lib/python2.3/site-packages']
>>>

The null string in sys.path represents the current working directory.

The expected convention for locally installed packages is to put them in the .../site-packages/ directory, but you may want to install Python modules into some arbitrary directory. For example, your site may have a convention of keeping all software related to the web server under /www. Add-on Python modules might then belong in /www/python, and in order to import them, this directory must be added to sys.path. There are several different ways to add the directory.

The most convenient way is to add a path configuration file to a directory that’s already on Python’s path, usually to the .../site-packages/ directory. Path configuration files have an extension of .pth, and each line must contain a single path that will be appended to sys.path. (Because the new paths are appended to sys.path, modules in the added directories will not override standard modules. This means you can’t use this mechanism for installing fixed versions of standard modules.)

Paths can be absolute or relative, in which case they’re relative to the directory containing the .pth file. See the documentation of the site module for more information.

A slightly less convenient way is to edit the site.py file in Python’s standard library, and modify sys.path. site.py is automatically imported when the Python interpreter is executed, unless the -S switch is supplied to suppress this behaviour. So you could simply edit site.py and add two lines to it:

import sys
sys.path.append('/www/python/')

However, if you reinstall the same major version of Python (perhaps when upgrading from 2.2 to 2.2.2, for example) site.py will be overwritten by the stock version. You’d have to remember that it was modified and save a copy before doing the installation.

There are two environment variables that can modify sys.path. PYTHONHOME sets an alternate value for the prefix of the Python installation. For example, if PYTHONHOME is set to /www/python, the search path will be set to ['', '/www/python/lib/pythonX.Y/', '/www/python/lib/pythonX.Y/plat-linux2', ...].

The PYTHONPATH variable can be set to a list of paths that will be added to the beginning of sys.path. For example, if PYTHONPATH is set to /www/python:/opt/py, the search path will begin with ['/www/python', '/opt/py']. (Note that directories must exist in order to be added to sys.path; the site module removes paths that don’t exist.)

Finally, sys.path is just a regular Python list, so any Python application can modify it by adding or removing entries.

Location and names of config files¶

The names and locations of the configuration files vary slightly across platforms. On Unix and Mac OS X, the three configuration files (in the order they are processed) are:

And on Windows, the configuration files are:

Notes:

1. Strictly speaking, the system-wide configuration file lives in the directory where the Distutils are installed; under Python 1.6 and later on Unix, this is as shown. For Python 1.5.2, the Distutils will normally be installed to prefix/lib/python1.5/site-packages/distutils, so the system configuration file should be put there under Python 1.5.2.
2. On Unix, if the HOME environment variable is not defined, the user’s home directory will be determined with the getpwuid() function from the standard pwd module.
3. I.e., in the current directory (usually the location of the setup script).
4. (See also note (1).) Under Python 1.6 and later, Python’s default “installation prefix” is C:\Python, so the system configuration file is normally C:\Python\Lib\distutils\distutils.cfg. Under Python 1.5.2, the default prefix was C:\Program Files\Python, and the Distutils were not part of the standard library—so the system configuration file would be C:\Program Files\Python\distutils\distutils.cfg in a standard Python 1.5.2 installation under Windows.
5. On Windows, if the HOME environment variable is not defined, no personal configuration file will be found or used. (In other words, the Distutils make no attempt to guess your home directory on Windows.)

Syntax of config files¶

The Distutils configuration files all have the same syntax. The config files are grouped into sections. There is one section for each Distutils command, plus a global section for global options that affect every command. Each section consists of one option per line, specified as option=value.

For example, the following is a complete config file that just forces all commands to run quietly by default:

[global]
verbose=0

If this is installed as the system config file, it will affect all processing of any Python module distribution by any user on the current system. If it is installed as your personal config file (on systems that support them), it will affect only module distributions processed by you. And if it is used as the setup.cfg for a particular module distribution, it affects only that distribution.

You could override the default “build base” directory and make the build* commands always forcibly rebuild all files with the following:

[build]
build-base=blib
force=1

which corresponds to the command-line arguments

python setup.py build --build-base=blib --force

except that including the build command on the command-line means that command will be run. Including a particular command in config files has no such implication; it only means that if the command is run, the options in the config file will apply. (Or if other commands that derive values from it are run, they will use the values in the config file.)

You can find out the complete list of options for any command using the --help option, e.g.:

python setup.py build --help

and you can find out the complete list of global options by using --help without a command:

python setup.py --help

See also the “Reference” section of the “Distributing Python Modules” manual.

Tweaking compiler/linker flags¶

Compiling a Python extension written in C or C++ will sometimes require specifying custom flags for the compiler and linker in order to use a particular library or produce a special kind of object code. This is especially true if the extension hasn’t been tested on your platform, or if you’re trying to cross-compile Python.

In the most general case, the extension author might have foreseen that compiling the extensions would be complicated, and provided a Setup file for you to edit. This will likely only be done if the module distribution contains many separate extension modules, or if they often require elaborate sets of compiler flags in order to work.

A Setup file, if present, is parsed in order to get a list of extensions to build. Each line in a Setup describes a single module. Lines have the following structure:

module ... [sourcefile ...] [cpparg ...] [library ...]

Let’s examine each of the fields in turn.

• module is the name of the extension module to be built, and should be a valid Python identifier. You can’t just change this in order to rename a module (edits to the source code would also be needed), so this should be left alone.
• sourcefile is anything that’s likely to be a source code file, at least judging by the filename. Filenames ending in .c are assumed to be written in C, filenames ending in .C, .cc, and .c++ are assumed to be C++, and filenames ending in .m or .mm are assumed to be in Objective C.
• cpparg is an argument for the C preprocessor, and is anything starting with -I, -D, -U or -C.
• library is anything ending in .a or beginning with -l or -L.

If a particular platform requires a special library on your platform, you can add it by editing the Setup file and running python setup.py build. For example, if the module defined by the line

foo foomodule.c

must be linked with the math library libm.a on your platform, simply add -lm to the line:

foo foomodule.c -lm

Arbitrary switches intended for the compiler or the linker can be supplied with the -Xcompiler arg and -Xlinker arg options:

foo foomodule.c -Xcompiler -o32 -Xlinker -shared -lm

The next option after -Xcompiler and -Xlinker will be appended to the proper command line, so in the above example the compiler will be passed the -o32 option, and the linker will be passed -shared. If a compiler option requires an argument, you’ll have to supply multiple -Xcompiler options; for example, to pass -x c++ the Setup file would have to contain -Xcompiler -x -Xcompiler c++.

Compiler flags can also be supplied through setting the CFLAGS environment variable. If set, the contents of CFLAGS will be added to the compiler flags specified in the Setup file.

Using non-Microsoft compilers on Windows¶

coff2omf python25.lib python25_bcpp.lib

python setup.py build --compiler=bcpp

pexports python25.dll >python25.def

/cygwin/bin/dlltool --dllname python25.dll --def python25.def --output-lib libpython25.a

python setup.py build --compiler=cygwin

python setup.py build --compiler=mingw32