Python Distutils Tips

From CAC Documentation wiki
Jump to: navigation, search

Python Distutils Tips

by Drew Dolgert, 17 Feb 2010

You have read how to Install Python Modules with Distutils and how to create your own packages for Distributing Python Modules. These directions are enough to build and distribute single-package distributions in most cases. How can distutils help with configuration of the package according to the environment?

Most of Distutils functionality is in distutils.core's setup, Extension, and Distribution. If you look in the Distutils API reference, there are a few other kinds of functionality available:

  • Compiler and linker control - distutils.ccompiler lets you, for instance, add include directories and invoke the compiler.
  • Dependency checking - distutuls.dep_util
  • Dry-run capability - Available through several interfaces.
  • Autoconf-like detection of the environment
    • distutils.ccompiler also has a has_function method and find_library_file.
    • distutils.sysconfig provides standard directory names, such as /usr/lib/python2.5/site-packages.
  • Lots of other capabilities that seem to overlap with standard modules such as shutil, zlib, and getopt.

How Does Distutils Work?

Distutils is designed to execute high-level commands to build, install, and package Python modules. The distutils.core.setup() function is a convenience method to create the over-arching Distribution instance which negotiates among your distutils configuration files, keyword arguments to setup() and command-line arguments to configure the high-level commands. All of the action happens in those commands, and commands like build have sub-commands, build_py, build_clib, build_ext, and build_scripts. The build command, itself, just stores options for its sub-commands and decides which of those sub-commands are appropriate to run.

The command-line help shows the structure of the library: global, high-level command, sub-commands. Basic help comes from the Distutils class, which tells you only about global options. Any of these options can come immediately after setup.py on the command-line.

python setup.py --help
 --verbose (-v)      run verbosely (default)
 --quiet (-q)        run quietly (turns verbosity off)
 --dry-run (-n)      don't actually do anything
 --help (-h)         show detailed help message
 --command-packages  list of packages that provide distutils commands
Information display options (just display information, ignore any commands)
 --help-commands     list all available commands
 --name              print package name
 --version (-V)      print package version
 --fullname          print <package name>-<version>
 --author            print the author's name
 --author-email      print the author's email address
 --maintainer        print the maintainer's name
 --maintainer-email  print the maintainer's email address
 --contact           print the maintainer's name if known, else the author's
 --contact-email     print the maintainer's email address if known, else the
                     author's
 --url               print the URL for this package
 --license           print the license of the package
 --licence           alias for --license
 --description       print the package description
 --long-description  print the long package description
 --platforms         print the list of platforms
 --classifiers       print the list of classifiers
 --keywords          print the list of keywords
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
  or: setup.py --help [cmd1 cmd2 ...]
  or: setup.py --help-commands
  or: setup.py cmd --help

The list of commands, from python setup.py --help-commands, shows both commands and their sub-commands.

Standard commands:
 build            build everything needed to install
 build_py         "build" pure Python modules (copy to build directory)
 build_ext        build C/C++ extensions (compile/link to build directory)
 build_clib       build C/C++ libraries used by Python extensions
 build_scripts    "build" scripts (copy and fixup #! line)
 clean            clean up temporary files from 'build' command
 install          install everything from build directory
 install_lib      install all Python modules (extensions and pure Python)
 install_headers  install C/C++ header files
 install_scripts  install scripts (Python or otherwise)
 install_data     install data files
 sdist            create a source distribution (tarball, zip file, etc.)
 register         register the distribution with the Python package index
 bdist            create a built (binary) distribution
 bdist_dumb       create a "dumb" built distribution
 bdist_rpm        create an RPM distribution
 bdist_wininst    create an executable installer for MS Windows

The install command has sub-commands build, install_lib, install_headers, install_scripts, and install_data. If you execute python setup.py install, it will call build if dependencies are out of date, and that will, in turn, call the correct sub-class of build if it's appropriate.

When you call setup, you give each command its command-line arguments following that command, so you might use

python setup.py build --debug install --prefix=${HOME}

where the debug option is just for the build command (rather, its sub-commands), and the prefix option is for install. It will complain if options are out of order.

If we ask for help on the build command, setup.py prints a section just for it. These options list only those that the build command collects for its sub-commands to use.

Options for 'build' command:
 --build-base (-b)  base directory for build library
 --build-purelib    build directory for platform-neutral distributions
 --build-platlib    build directory for platform-specific distributions
 --build-lib        build directory for all distribution (defaults to either
                    build-purelib or build-platlib
 --build-scripts    build directory for scripts
 --build-temp (-t)  temporary build directory
 --compiler (-c)    specify the compiler type
 --debug (-g)       compile extensions and libraries with debugging
                    information
 --force (-f)       forcibly build everything (ignore file timestamps)
 --executable (-e)  specify final destination interpreter path (build.py)
 --help-compiler    list available compilers

If you ask for more specific options, for the build_py command, for example, you see that you can ask that it precompile py files to pyc files.

python setup.py build_py --help
Options for 'build_py' command:
 --build-lib (-d)  directory to "build" (copy) to
 --compile (-c)    compile .py to .pyc
 --no-compile      don't compile .py files [default]
 --optimize (-O)   also compile with optimization: -O1 for "python -O", -O2
                   for "python -OO", and -O0 to disable [default: -O0]
 --force (-f)      forcibly build everything (ignore file timestamps)

Maybe more interesting are the commands for build_ext, to compile C extensions to Python.

Options for 'build_ext' command:
 --build-lib (-b)     directory for compiled extension modules
 --build-temp (-t)    directory for temporary files (build by-products)
 --inplace (-i)       ignore build-lib and put compiled extensions into the
                      source directory alongside your pure Python modules
 --include-dirs (-I)  list of directories to search for header files
                      (separated by ':')
 --define (-D)        C preprocessor macros to define
 --undef (-U)         C preprocessor macros to undefine
 --libraries (-l)     external C libraries to link with
 --library-dirs (-L)  directories to search for external C libraries
                      (separated by ':')
 --rpath (-R)         directories to search for shared C libraries at runtime
 --link-objects (-O)  extra explicit link objects to include in the link
 --debug (-g)         compile/link with debugging information
 --force (-f)         forcibly build everything (ignore file timestamps)
 --compiler (-c)      specify the compiler type
 --swig-cpp           make SWIG create C++ files (default is C)
 --swig-opts          list of SWIG command line options
 --swig               path to the SWIG executable
 --help-compiler      list available compilers

The -rpath option lets you hard-code into the shared library the LD_LIBRARY_PATH where the loader will look for its dependencies when it loads this extension.

You can't call

python setup.py build --swig-cpp

because swig-cpp is an option for build's sub-command. You have to invoke the sub-command explicitly to get access to that option

python setup.py build_ext --swig-cpp

or specify those options programmatically in the setup file.

How to find if a library exists

Don't ask whether a library exists because the only function for that is find_library_file():

compiler=distutils.ccompiler.new_compiler()
lib_dirs=['/usr/lib','/usr/local/lib',os.path.expanduser('~/lib')]
if compiler.find_library_file(lib_dirs,'rt'):
  user_macros.append(('HAVE_POSIX_TIMER','1'))

How are you supposed to get lib_dirs correct? The distutils C compiler knows how to invoke cc, but it does not know default library locations. There are usually two ways to control those locations, either by setting the LIBRARY_PATH variable or by adding a -L switch to the linker command line. Distutils doesn't track this behavior. A better way to check whether a library exists is to try to compile it.

compiler=distutils.ccompiler.new_compiler()
if compiler.has_function('timer_create',libraries=('rt',)):
  user_macros.append(('HAVE_POSIX_TIMER','1'))

Finding include directories for extension modules

When you build an extension module, the build command invokes the build_ext command which invokes the current compiler, and adds include paths and shared library paths for the system copy of Python, but you may have to add other paths. To find Python includes in the user directory, such as $HOME/include/Python2.5, use

userinc=distutils.sysconfig.get_python_inc(prefix=os.path.expanduser('~'))

There is a similar call for get_python_lib().

The Extension class thinks my file is C, but it's C++

Because the Extension class builds a shared library, it is possible that it will link but not load properly when Python tries to import it. You can tell whether the compiler recognizes that a file has a C++ extension by calling ccompiler.detect_language(filename). If it does not, then either add "language='c++'" to arguments for the Extension constructor or add extra_link_args=["-lstdc++"], which a more brute force approach.

How do I provide arguments to the setup script without breaking it?

If you want to specify your own command-line arguments to affect configuration before Distutils does its work, you will find that Distutils will complain about your argument. You have to pull your command-line argument out of the argument list and explicitly specify that list for Distutils.

copy_args=sys.argv[1:]
if '--mock' in copy_args:
  custom_macros.append(('MOCK_CALLS','1'))
  copy_args.remove('--mock')
setup(..., script_args=copy_args, define_macros=custom_macros,...)


What if I just want to use Distutils to compile some extension modules in the current directory?

It seems like this would be easy. There is an option for the build_ext command called --inplace, or -I, which does just this. It builds the module and puts the shared library right next to it. If you specify a typical setup file, such as

import os
from distutils.core import setup, Extension
chase_da_ext=Extension(
    "acert.chase_da._chase_da", sources=["chase_da/_chase_da.c"],
    libraries=['chase_da'],
    define_macros=[('HAVE_TIMER','1')],
    include_dirs=[os.path.expanduser('~/include')])
setup(
    name="chase_da",
    version="0.1",
    description="Wrapper module for Chase Waveform Generator",
    ext_modules=[chase_da_ext],
        packages=['acert.chase_da'],
    package_dir = {'acert.chase_da' : 'chase_da'},
)

then you can build the module in place using the command-line

python setup.py build_ext --inplace

There is no argument to the Extension constructor to specify the module should be built in place, though, and you could not specify

python setup.py build --inplace # ERROR, unknown argument.

because the inplace argument is only for the build_ext command.