Git Product home page Git Product logo

unladen-swallow's Introduction

This is Python version 2.6.4
============================

Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009
Python Software Foundation.
All rights reserved.

Copyright (c) 2000 BeOpen.com.
All rights reserved.

Copyright (c) 1995-2001 Corporation for National Research Initiatives.
All rights reserved.

Copyright (c) 1991-1995 Stichting Mathematisch Centrum.
All rights reserved.


License information
-------------------

See the file "LICENSE" for information on the history of this
software, terms & conditions for usage, and a DISCLAIMER OF ALL
WARRANTIES.

This Python distribution contains no GNU General Public Licensed
(GPLed) code so it may be used in proprietary projects just like prior
Python distributions.  There are interfaces to some GNU code but these
are entirely optional.

All trademarks referenced herein are property of their respective
holders.


What's new in this release?
---------------------------

See the file "Misc/NEWS".


If you don't read instructions
------------------------------

Congratulations on getting this far. :-)

To start building right away (on UNIX): type "./configure" in the
current directory and when it finishes, type "make".  This creates an
executable "./python"; to install in /usr/local, first do "su root"
and then "make install".

The section `Build instructions' below is still recommended reading.


What is Python anyway?
----------------------

Python is an interpreted, interactive object-oriented programming
language suitable (amongst other uses) for distributed application
development, scripting, numeric computing and system testing.  Python
is often compared to Tcl, Perl, Java, JavaScript, Visual Basic or
Scheme.  To find out more about what Python can do for you, point your
browser to http://www.python.org/.


How do I learn Python?
----------------------

The official tutorial is still a good place to start; see
http://docs.python.org/ for online and downloadable versions, as well
as a list of other introductions, and reference documentation.

There's a quickly growing set of books on Python.  See
http://wiki.python.org/moin/PythonBooks for a list.


Documentation
-------------

All documentation is provided online in a variety of formats.  In
order of importance for new users: Tutorial, Library Reference,
Language Reference, Extending & Embedding, and the Python/C API.  The
Library Reference is especially of immense value since much of
Python's power is described there, including the built-in data types
and functions!

All documentation is also available online at the Python web site
(http://docs.python.org/, see below).  It is available online for occasional
reference, or can be downloaded in many formats for faster access.  The
documentation is downloadable in HTML, PostScript, PDF, LaTeX, and
reStructuredText (2.6+) formats; the LaTeX and reStructuredText versions are
primarily for documentation authors, translators, and people with special
formatting requirements.


Web sites
---------

New Python releases and related technologies are published at
http://www.python.org/.  Come visit us!

There's also a Python community web site at
http://starship.python.net/.


Newsgroups and Mailing Lists
----------------------------

Read comp.lang.python, a high-volume discussion newsgroup about
Python, or comp.lang.python.announce, a low-volume moderated newsgroup
for Python-related announcements.  These are also accessible as
mailing lists: see http://www.python.org/community/lists.html for an
overview of these and many other Python-related mailing lists.

Archives are accessible via the Google Groups Usenet archive; see
http://groups.google.com/.  The mailing lists are also archived, see
http://www.python.org/community/lists.html for details.


Bug reports
-----------

To report or search for bugs, please use the Python Bug
Tracker at http://bugs.python.org.


Patches and contributions
-------------------------

To submit a patch or other contribution, please use the Python Patch
Manager at http://bugs.python.org.  Guidelines
for patch submission may be found at http://www.python.org/dev/patches/.

If you have a proposal to change Python, you may want to send an email to the
comp.lang.python or python-ideas mailing lists for inital feedback. A Python
Enhancement Proposal (PEP) may be submitted if your idea gains ground. All
current PEPs, as well as guidelines for submitting a new PEP, are listed at
http://www.python.org/dev/peps/.


Questions
---------

For help, if you can't find it in the manuals or on the web site, it's
best to post to the comp.lang.python or the Python mailing list (see
above).  If you specifically don't want to involve the newsgroup or
mailing list, send questions to [email protected] (a group of volunteers
who answer questions as they can).  The newsgroup is the most
efficient way to ask public questions.


Build instructions
==================

Before you can build Python, you must first configure it.
Fortunately, the configuration and build process has been automated
for Unix and Linux installations, so all you usually have to do is
type a few commands and sit back.  There are some platforms where
things are not quite as smooth; see the platform specific notes below.
If you want to build for multiple platforms sharing the same source
tree, see the section on VPATH below.

Start by running the script "./configure", which determines your
system configuration and creates the Makefile.  (It takes a minute or
two -- please be patient!)  You may want to pass options to the
configure script -- see the section below on configuration options and
variables.  When it's done, you are ready to run make.

To build Python, you normally type "make" in the toplevel directory.
If you have changed the configuration, the Makefile may have to be
rebuilt.  In this case you may have to run make again to correctly
build your desired target.  The interpreter executable is built in the
top level directory.

Once you have built a Python interpreter, see the subsections below on
testing and installation.  If you run into trouble, see the next
section.

Previous versions of Python used a manual configuration process that
involved editing the file Modules/Setup.  While this file still exists
and manual configuration is still supported, it is rarely needed any
more: almost all modules are automatically built as appropriate under
guidance of the setup.py script, which is run by Make after the
interpreter has been built.


Troubleshooting
---------------

See also the platform specific notes in the next section.

If you run into other trouble, see the FAQ
(http://www.python.org/doc/faq) for hints on what can go wrong, and
how to fix it.

If you rerun the configure script with different options, remove all
object files by running "make clean" before rebuilding.  Believe it or
not, "make clean" sometimes helps to clean up other inexplicable
problems as well.  Try it before sending in a bug report!

If the configure script fails or doesn't seem to find things that
should be there, inspect the config.log file.

If you get a warning for every file about the -Olimit option being no
longer supported, you can ignore it.  There's no foolproof way to know
whether this option is needed; all we can do is test whether it is
accepted without error.  On some systems, e.g. older SGI compilers, it
is essential for performance (specifically when compiling eval.cc,
which has more basic blocks than the default limit of 1000).  If the
warning bothers you, edit the Makefile to remove "-Olimit 1500" from
the OPT variable.

If you get failures in test_long, or sys.maxint gets set to -1, you
are probably experiencing compiler bugs, usually related to
optimization.  This is a common problem with some versions of gcc, and
some vendor-supplied compilers, which can sometimes be worked around
by turning off optimization.  Consider switching to stable versions
(gcc 2.95.2, gcc 3.x, or contact your vendor.)

From Python 2.0 onward, all Python C code is ANSI C.  Compiling using
old K&R-C-only compilers is no longer possible.  ANSI C compilers are
available for all modern systems, either in the form of updated
compilers from the vendor, or one of the free compilers (gcc).

If "make install" fails mysteriously during the "compiling the library"
step, make sure that you don't have any of the PYTHONPATH or PYTHONHOME
environment variables set, as they may interfere with the newly built
executable which is compiling the library.

Unsupported systems
-------------------

A number of features are not supported in Python 2.5 anymore. Some
support code is still present, but will be removed in Python 2.6. 
If you still need to use current Python versions on these systems,
please send a message to [email protected] indicating that you
volunteer to support this system. For a more detailed discussion 
regarding no-longer-supported and resupporting platforms, as well
as a list of platforms that became or will be unsupported, see PEP 11.

More specifically, the following systems are not supported any
longer:
- SunOS 4
- DYNIX
- dgux
- Minix
- NeXT
- Irix 4 and --with-sgi-dl
- Linux 1
- Systems defining __d6_pthread_create (configure.in)
- Systems defining PY_PTHREAD_D4, PY_PTHREAD_D6,
  or PY_PTHREAD_D7 in thread_pthread.h
- Systems using --with-dl-dld
- Systems using --without-universal-newlines
- MacOS 9

The following systems are still supported in Python 2.5, but
support will be dropped in 2.6:
- Systems using --with-wctype-functions
- Win9x, WinME

Warning on install in Windows 98 and Windows Me
-----------------------------------------------

Following Microsoft's closing of Extended Support for
Windows 98/ME (July 11, 2006), Python 2.6 will stop
supporting these platforms. Python development and
maintainability becomes easier (and more reliable) when
platform specific code targeting OSes with few users
and no dedicated expert developers is taken out. The
vendor also warns that the OS versions listed above
"can expose customers to security risks" and recommends
upgrade.

Platform specific notes
-----------------------

(Some of these may no longer apply.  If you find you can build Python
on these platforms without the special directions mentioned here,
submit a documentation bug report to SourceForge (see Bug Reports
above) so we can remove them!)

Unix platforms: If your vendor still ships (and you still use) Berkeley DB
        1.85 you will need to edit Modules/Setup to build the bsddb185
        module and add a line to sitecustomize.py which makes it the
        default.  In Modules/Setup a line like

            bsddb185 bsddbmodule.c

        should work.  (You may need to add -I, -L or -l flags to direct the
        compiler and linker to your include files and libraries.)

XXX I think this next bit is out of date:

64-bit platforms: The modules audioop, and imageop don't work.
        The setup.py script disables them on 64-bit installations.
        Don't try to enable them in the Modules/Setup file.  They
        contain code that is quite wordsize sensitive.  (If you have a
        fix, let us know!)

Solaris: When using Sun's C compiler with threads, at least on Solaris
        2.5.1, you need to add the "-mt" compiler option (the simplest
        way is probably to specify the compiler with this option as
        the "CC" environment variable when running the configure
        script).

        When using GCC on Solaris, beware of binutils 2.13 or GCC
        versions built using it.  This mistakenly enables the
        -zcombreloc option which creates broken shared libraries on
        Solaris.  binutils 2.12 works, and the binutils maintainers
        are aware of the problem.  Binutils 2.13.1 only partially
        fixed things.  It appears that 2.13.2 solves the problem
        completely.  This problem is known to occur with Solaris 2.7
        and 2.8, but may also affect earlier and later versions of the
        OS.

        When the dynamic loader complains about errors finding shared
        libraries, such as

        ld.so.1: ./python: fatal: libstdc++.so.5: open failed:
        No such file or directory

        you need to first make sure that the library is available on
        your system. Then, you need to instruct the dynamic loader how
        to find it. You can choose any of the following strategies:

        1. When compiling Python, set LD_RUN_PATH to the directories
           containing missing libraries.
        2. When running Python, set LD_LIBRARY_PATH to these directories.
        3. Use crle(8) to extend the search path of the loader.
        4. Modify the installed GCC specs file, adding -R options into the
           *link: section.

        The complex object fails to compile on Solaris 10 with gcc 3.4 (at
        least up to 3.4.3).  To work around it, define Py_HUGE_VAL as
        HUGE_VAL(), e.g.:

          make CPPFLAGS='-D"Py_HUGE_VAL=HUGE_VAL()" -I. -I$(srcdir)/Include'
          ./python setup.py CPPFLAGS='-D"Py_HUGE_VAL=HUGE_VAL()"'

Linux:  A problem with threads and fork() was tracked down to a bug in
        the pthreads code in glibc version 2.0.5; glibc version 2.0.7
        solves the problem.  This causes the popen2 test to fail;
        problem and solution reported by Pablo Bleyer.

Red Hat Linux: Red Hat 9 built Python2.2 in UCS-4 mode and hacked
        Tcl to support it. To compile Python2.3 with Tkinter, you will
        need to pass --enable-unicode=ucs4 flag to ./configure.

        There's an executable /usr/bin/python which is Python
        1.5.2 on most older Red Hat installations; several key Red Hat tools
        require this version.  Python 2.1.x may be installed as
        /usr/bin/python2.  The Makefile installs Python as
        /usr/local/bin/python, which may or may not take precedence
        over /usr/bin/python, depending on how you have set up $PATH.

FreeBSD 3.x and probably platforms with NCurses that use libmytinfo or
        similar: When using cursesmodule, the linking is not done in
        the correct order with the defaults.  Remove "-ltermcap" from
        the readline entry in Setup, and use as curses entry: "curses
        cursesmodule.c -lmytinfo -lncurses -ltermcap" - "mytinfo" (so
        called on FreeBSD) should be the name of the auxiliary library
        required on your platform.  Normally, it would be linked
        automatically, but not necessarily in the correct order.

BSDI:   BSDI versions before 4.1 have known problems with threads,
        which can cause strange errors in a number of modules (for
        instance, the 'test_signal' test script will hang forever.)
        Turning off threads (with --with-threads=no) or upgrading to
        BSDI 4.1 solves this problem.

DEC Unix: Run configure with --with-dec-threads, or with
        --with-threads=no if no threads are desired (threads are on by
        default).  When using GCC, it is possible to get an internal
        compiler error if optimization is used.  This was reported for
        GCC 2.7.2.3 on selectmodule.c.  Manually compile the affected
        file without optimization to solve the problem.

DEC Ultrix: compile with GCC to avoid bugs in the native compiler,
        and pass SHELL=/bin/sh5 to Make when installing.

AIX:    A complete overhaul of the shared library support is now in
        place.  See Misc/AIX-NOTES for some notes on how it's done.
        (The optimizer bug reported at this place in previous releases
        has been worked around by a minimal code change.) If you get
        errors about pthread_* functions, during compile or during
        testing, try setting CC to a thread-safe (reentrant) compiler,
        like "cc_r".  For full C++ module support, set CC="xlC_r" (or
        CC="xlC" without thread support).

AIX 5.3: To build a 64-bit version with IBM's compiler, I used the
        following:

        export PATH=/usr/bin:/usr/vacpp/bin
        ./configure --with-gcc="xlc_r -q64" --with-cxx="xlC_r -q64" \
                    --disable-ipv6 AR="ar -X64"
        make

HP-UX:  When using threading, you may have to add -D_REENTRANT to the
        OPT variable in the top-level Makefile; reported by Pat Knight,
        this seems to make a difference (at least for HP-UX 10.20)
        even though pyconfig.h defines it. This seems unnecessary when
        using HP/UX 11 and later - threading seems to work "out of the
        box".

HP-UX ia64: When building on the ia64 (Itanium) platform using HP's
        compiler, some experience has shown that the compiler's
        optimiser produces a completely broken version of python
        (see http://www.python.org/sf/814976). To work around this,
        edit the Makefile and remove -O from the OPT line.

        To build a 64-bit executable on an Itanium 2 system using HP's
        compiler, use these environment variables:

                CC=cc
                CXX=aCC
                BASECFLAGS="+DD64"
                LDFLAGS="+DD64 -lxnet"

        and call configure as:

                ./configure --without-gcc

        then *unset* the environment variables again before running
        make.  (At least one of these flags causes the build to fail
        if it remains set.)  You still have to edit the Makefile and
        remove -O from the OPT line.

HP PA-RISC 2.0: A recent bug report (http://www.python.org/sf/546117)
        suggests that the C compiler in this 64-bit system has bugs
        in the optimizer that break Python.  Compiling without
        optimization solves the problems.

SCO:    The following apply to SCO 3 only; Python builds out of the box
        on SCO 5 (or so we've heard).

        1) Everything works much better if you add -U__STDC__ to the
        defs.  This is because all the SCO header files are broken.
        Anything that isn't mentioned in the C standard is
        conditionally excluded when __STDC__ is defined.

        2) Due to the U.S. export restrictions, SCO broke the crypt
        stuff out into a separate library, libcrypt_i.a so the LIBS
        needed be set to:

                LIBS=' -lsocket -lcrypt_i'

UnixWare: There are known bugs in the math library of the system, as well as
        problems in the handling of threads (calling fork in one
        thread may interrupt system calls in others). Therefore, test_math and
        tests involving threads will fail until those problems are fixed.

QNX:    Chris Herborth ([email protected]) writes:
        configure works best if you use GNU bash; a port is available on
        ftp.qnx.com in /usr/free.  I used the following process to build,
        test and install Python 1.5.x under QNX:

        1) CONFIG_SHELL=/usr/local/bin/bash CC=cc RANLIB=: \
            ./configure --verbose --without-gcc --with-libm=""

        2) edit Modules/Setup to activate everything that makes sense for
           your system... tested here at QNX with the following modules:

                array, audioop, binascii, cPickle, cStringIO, cmath,
                crypt, curses, errno, fcntl, gdbm, grp, imageop,
                _locale, math, md5, new, operator, parser, pcre,
                posix, pwd, readline, regex, reop,
                select, signal, socket, soundex, strop, struct,
                syslog, termios, time, timing, zlib, audioop, imageop

        3) make SHELL=/usr/local/bin/bash

           or, if you feel the need for speed:

           make SHELL=/usr/local/bin/bash OPT="-5 -Oil+nrt"

        4) make SHELL=/usr/local/bin/bash test

           Using GNU readline 2.2 seems to behave strangely, but I
           think that's a problem with my readline 2.2 port.  :-\

        5) make SHELL=/usr/local/bin/bash install

        If you get SIGSEGVs while running Python (I haven't yet, but
        I've only run small programs and the test cases), you're
        probably running out of stack; the default 32k could be a
        little tight.  To increase the stack size, edit the Makefile
        to read: LDFLAGS = -N 48k

BeOS:   See Misc/BeOS-NOTES for notes about compiling/installing
        Python on BeOS R3 or later.  Note that only the PowerPC
        platform is supported for R3; both PowerPC and x86 are
        supported for R4.

Cray T3E: Mark Hadfield ([email protected]) writes:
        Python can be built satisfactorily on a Cray T3E but based on
        my experience with the NIWA T3E (2002-05-22, version 2.2.1)
        there are a few bugs and gotchas. For more information see a
        thread on comp.lang.python in May 2002 entitled "Building
        Python on Cray T3E".

        1) Use Cray's cc and not gcc. The latter was reported not to
           work by Konrad Hinsen. It may work now, but it may not.

        2) To set sys.platform to something sensible, pass the
           following environment variable to the configure script:

             MACHDEP=unicosmk

        2) Run configure with option "--enable-unicode=ucs4".

        3) The Cray T3E does not support dynamic linking, so extension
           modules have to be built by adding (or uncommenting) lines
           in Modules/Setup. The minimum set of modules is

             posix, new, _sre, unicodedata

           On NIWA's vanilla T3E system the following have also been
           included successfully:

             _codecs, _locale, _socket, _symtable, _testcapi, _weakref
             array, binascii, cmath, cPickle, crypt, cStringIO, dbm
             errno, fcntl, grp, math, md5, operator, parser, pcre, pwd
             regex, rotor, select, struct, strop, syslog, termios
             time, timing, xreadlines

        4) Once the python executable and library have been built, make
           will execute setup.py, which will attempt to build remaining
           extensions and link them dynamically. Each of these attempts
           will fail but should not halt the make process. This is
           normal.

        5) Running "make test" uses a lot of resources and causes
           problems on our system. You might want to try running tests
           singly or in small groups.

SGI:    SGI's standard "make" utility (/bin/make or /usr/bin/make)
        does not check whether a command actually changed the file it
        is supposed to build.  This means that whenever you say "make"
        it will redo the link step.  The remedy is to use SGI's much
        smarter "smake" utility (/usr/sbin/smake), or GNU make.  If
        you set the first line of the Makefile to #!/usr/sbin/smake
        smake will be invoked by make (likewise for GNU make).

        WARNING: There are bugs in the optimizer of some versions of
        SGI's compilers that can cause bus errors or other strange
        behavior, especially on numerical operations.  To avoid this,
        try building with "make OPT=".

OS/2:   If you are running Warp3 or Warp4 and have IBM's VisualAge C/C++
        compiler installed, just change into the pc\os2vacpp directory
        and type NMAKE.  Threading and sockets are supported by default
        in the resulting binaries of PYTHON15.DLL and PYTHON.EXE.

Monterey (64-bit AIX): The current Monterey C compiler (Visual Age)
        uses the OBJECT_MODE={32|64} environment variable to set the
        compilation mode to either 32-bit or 64-bit (32-bit mode is
        the default).  Presumably you want 64-bit compilation mode for
        this 64-bit OS.  As a result you must first set OBJECT_MODE=64
        in your environment before configuring (./configure) or
        building (make) Python on Monterey.

Reliant UNIX: The thread support does not compile on Reliant UNIX, and
        there is a (minor) problem in the configure script for that
        platform as well.  This should be resolved in time for a
        future release.

MacOSX: The tests will crash on both 10.1 and 10.2 with SEGV in
        test_re and test_sre due to the small default stack size.  If
        you set the stack size to 2048 before doing a "make test" the
        failure can be avoided.  If you're using the tcsh or csh shells,
        use "limit stacksize 2048" and for the bash shell (the default
        as of OSX 10.3), use "ulimit -s 2048".

        On naked Darwin you may want to add the configure option
        "--disable-toolbox-glue" to disable the glue code for the Carbon
        interface modules. The modules themselves are currently only built
        if you add the --enable-framework option, see below.

        On a clean OSX /usr/local does not exist. Do a
        "sudo mkdir -m 775 /usr/local"
        before you do a make install. It is probably not a good idea to
        do "sudo make install" which installs everything as superuser,
        as this may later cause problems when installing distutils-based
        additions.

        Some people have reported problems building Python after using "fink"
        to install additional unix software. Disabling fink (remove all 
        references to /sw from your .profile or .login) should solve this.

        You may want to try the configure option "--enable-framework"
        which installs Python as a framework. The location can be set
        as argument to the --enable-framework option (default
        /Library/Frameworks). A framework install is probably needed if you
        want to use any Aqua-based GUI toolkit (whether Tkinter, wxPython,
        Carbon, Cocoa or anything else).

        You may also want to try the configure option "--enable-universalsdk"
        which builds Python as a universal binary with support for the 
        i386 and PPC architetures. This requires Xcode 2.1 or later to build.

        See Mac/README for more information on framework and 
        universal builds.

Cygwin: With recent (relative to the time of writing, 2001-12-19)
        Cygwin installations, there are problems with the interaction
        of dynamic linking and fork().  This manifests itself in build
        failures during the execution of setup.py.

        There are two workarounds that both enable Python (albeit
        without threading support) to build and pass all tests on
        NT/2000 (and most likely XP as well, though reports of testing
        on XP would be appreciated).

        The workarounds:

        (a) the band-aid fix is to link the _socket module statically
        rather than dynamically (which is the default).

        To do this, run "./configure --with-threads=no" including any
        other options you need (--prefix, etc.).  Then in Modules/Setup
        uncomment the lines:

        #SSL=/usr/local/ssl
        #_socket socketmodule.c \
        #       -DUSE_SSL -I$(SSL)/include -I$(SSL)/include/openssl \
        #       -L$(SSL)/lib -lssl -lcrypto

        and remove "local/" from the SSL variable.  Finally, just run
        "make"!

        (b) The "proper" fix is to rebase the Cygwin DLLs to prevent
        base address conflicts.  Details on how to do this can be
        found in the following mail:

           http://sources.redhat.com/ml/cygwin/2001-12/msg00894.html

        It is hoped that a version of this solution will be
        incorporated into the Cygwin distribution fairly soon.

        Two additional problems:

        (1) Threading support should still be disabled due to a known
        bug in Cygwin pthreads that causes test_threadedtempfile to
        hang.

        (2) The _curses module does not build.  This is a known
        Cygwin ncurses problem that should be resolved the next time
        that this package is released.

        On older versions of Cygwin, test_poll may hang and test_strftime
        may fail.

        The situation on 9X/Me is not accurately known at present.
        Some time ago, there were reports that the following
        regression tests failed:

            test_pwd
            test_select (hang)
            test_socket

        Due to the test_select hang on 9X/Me, one should run the
        regression test using the following:

            make TESTOPTS='-l -x test_select' test

        News regarding these platforms with more recent Cygwin
        versions would be appreciated!

Windows: When executing Python scripts on the command line using file type
        associations (i.e. starting "script.py" instead of "python script.py"),
        redirects may not work unless you set a specific registry key.  See
        the Knowledge Base article <http://support.microsoft.com/kb/321788>.


Configuring the bsddb and dbm modules
-------------------------------------

Beginning with Python version 2.3, the PyBsddb package
<http://pybsddb.sf.net/> was adopted into Python as the bsddb package,
exposing a set of package-level functions which provide
backwards-compatible behavior.  Only versions 3.3 through 4.4 of
Sleepycat's libraries provide the necessary API, so older versions
aren't supported through this interface.  The old bsddb module has
been retained as bsddb185, though it is not built by default.  Users
wishing to use it will have to tweak Modules/Setup to build it.  The
dbm module will still be built against the Sleepycat libraries if
other preferred alternatives (ndbm, gdbm) are not found.

Building the sqlite3 module
---------------------------

To build the sqlite3 module, you'll need the sqlite3 or libsqlite3
packages installed, including the header files. Many modern operating
systems distribute the headers in a separate package to the library -
often it will be the same name as the main package, but with a -dev or
-devel suffix. 

The version of pysqlite2 that's including in Python needs sqlite3 3.0.8
or later. setup.py attempts to check that it can find a correct version.

Configuring threads
-------------------

As of Python 2.0, threads are enabled by default.  If you wish to
compile without threads, or if your thread support is broken, pass the
--with-threads=no switch to configure.  Unfortunately, on some
platforms, additional compiler and/or linker options are required for
threads to work properly.  Below is a table of those options,
collected by Bill Janssen.  We would love to automate this process
more, but the information below is not enough to write a patch for the
configure.in file, so manual intervention is required.  If you patch
the configure.in file and are confident that the patch works, please
send in the patch.  (Don't bother patching the configure script itself
-- it is regenerated each time the configure.in file changes.)

Compiler switches for threads
.............................

The definition of _REENTRANT should be configured automatically, if
that does not work on your system, or if _REENTRANT is defined
incorrectly, please report that as a bug.

    OS/Compiler/threads                     Switches for use with threads
    (POSIX is draft 10, DCE is draft 4)     compile & link

    SunOS 5.{1-5}/{gcc,SunPro cc}/solaris   -mt
    SunOS 5.5/{gcc,SunPro cc}/POSIX         (nothing)
    DEC OSF/1 3.x/cc/DCE                    -threads
            ([email protected])
    Digital UNIX 4.x/cc/DCE                 -threads
            ([email protected])
    Digital UNIX 4.x/cc/POSIX               -pthread
            ([email protected])
    AIX 4.1.4/cc_r/d7                       (nothing)
            ([email protected])
    AIX 4.1.4/cc_r4/DCE                     (nothing)
            ([email protected])
    IRIX 6.2/cc/POSIX                       (nothing)
            ([email protected])


Linker (ld) libraries and flags for threads
...........................................

    OS/threads                          Libraries/switches for use with threads

    SunOS 5.{1-5}/solaris               -lthread
    SunOS 5.5/POSIX                     -lpthread
    DEC OSF/1 3.x/DCE                   -lpthreads -lmach -lc_r -lc
            ([email protected])
    Digital UNIX 4.x/DCE                -lpthreads -lpthread -lmach -lexc -lc
            ([email protected])
    Digital UNIX 4.x/POSIX              -lpthread -lmach -lexc -lc
            ([email protected])
    AIX 4.1.4/{draft7,DCE}              (nothing)
            ([email protected])
    IRIX 6.2/POSIX                      -lpthread
            ([email protected])


Building a shared libpython
---------------------------

Starting with Python 2.3, the majority of the interpreter can be built
into a shared library, which can then be used by the interpreter
executable, and by applications embedding Python. To enable this feature,
configure with --enable-shared.

If you enable this feature, the same object files will be used to create
a static library.  In particular, the static library will contain object
files using position-independent code (PIC) on platforms where PIC flags
are needed for the shared library.


Configuring additional built-in modules
---------------------------------------

Starting with Python 2.1, the setup.py script at the top of the source
distribution attempts to detect which modules can be built and
automatically compiles them.  Autodetection doesn't always work, so
you can still customize the configuration by editing the Modules/Setup
file; but this should be considered a last resort.  The rest of this
section only applies if you decide to edit the Modules/Setup file.
You also need this to enable static linking of certain modules (which
is needed to enable profiling on some systems).

This file is initially copied from Setup.dist by the configure script;
if it does not exist yet, create it by copying Modules/Setup.dist
yourself (configure will never overwrite it).  Never edit Setup.dist
-- always edit Setup or Setup.local (see below).  Read the comments in
the file for information on what kind of edits are allowed.  When you
have edited Setup in the Modules directory, the interpreter will
automatically be rebuilt the next time you run make (in the toplevel
directory).

Many useful modules can be built on any Unix system, but some optional
modules can't be reliably autodetected.  Often the quickest way to
determine whether a particular module works or not is to see if it
will build: enable it in Setup, then if you get compilation or link
errors, disable it -- you're either missing support or need to adjust
the compilation and linking parameters for that module.

On SGI IRIX, there are modules that interface to many SGI specific
system libraries, e.g. the GL library and the audio hardware.  These
modules will not be built by the setup.py script.

In addition to the file Setup, you can also edit the file Setup.local.
(the makesetup script processes both).  You may find it more
convenient to edit Setup.local and leave Setup alone.  Then, when
installing a new Python version, you can copy your old Setup.local
file.


Setting the optimization/debugging options
------------------------------------------

If you want or need to change the optimization/debugging options for
the C compiler, assign to the OPT variable on the toplevel make
command; e.g. "make OPT=-g" will build a debugging version of Python
on most platforms.  The default is OPT=-O; a value for OPT in the
environment when the configure script is run overrides this default
(likewise for CC; and the initial value for LIBS is used as the base
set of libraries to link with).

When compiling with GCC, the default value of OPT will also include
the -Wall and -Wstrict-prototypes options.

Additional debugging code to help debug memory management problems can
be enabled by using the --with-pydebug option to the configure script.

For flags that change binary compatibility, use the EXTRA_CFLAGS
variable.


Profiling
---------

If you want C profiling turned on, the easiest way is to run configure
with the CC environment variable to the necessary compiler
invocation.  For example, on Linux, this works for profiling using
gprof(1):

    CC="gcc -pg" ./configure

Note that on Linux, gprof apparently does not work for shared
libraries.  The Makefile/Setup mechanism can be used to compile and
link most extension modules statically.


Coverage checking
-----------------

For C coverage checking using gcov, run "make coverage".  This will
build a Python binary with profiling activated, and a ".gcno" and
".gcda" file for every source file compiled with that option.  With
the built binary, now run the code whose coverage you want to check.
Then, you can see coverage statistics for each individual source file
by running gcov, e.g.

    gcov -o Modules zlibmodule

This will create a "zlibmodule.c.gcov" file in the current directory
containing coverage info for that source file.

This works only for source files statically compiled into the
executable; use the Makefile/Setup mechanism to compile and link
extension modules you want to coverage-check statically.


Testing
-------

To test the interpreter, type "make test" in the top-level directory.
This runs the test set twice (once with no compiled files, once with
the compiled files left by the previous test run).  The test set
produces some output.  You can generally ignore the messages about
skipped tests due to optional features which can't be imported.
If a message is printed about a failed test or a traceback or core
dump is produced, something is wrong.  On some Linux systems (those
that are not yet using glibc 6), test_strftime fails due to a
non-standard implementation of strftime() in the C library. Please
ignore this, or upgrade to glibc version 6.

IMPORTANT: If the tests fail and you decide to mail a bug report,
*don't* include the output of "make test".  It is useless.  Run the
failing test manually, as follows:

        ./python ./Lib/test/test_whatever.py

(substituting the top of the source tree for '.' if you built in a
different directory).  This runs the test in verbose mode.


Installing
----------

To install the Python binary, library modules, shared library modules
(see below), include files, configuration files, and the manual page,
just type

        make install

This will install all platform-independent files in subdirectories of
the directory given with the --prefix option to configure or to the
`prefix' Make variable (default /usr/local).  All binary and other
platform-specific files will be installed in subdirectories if the
directory given by --exec-prefix or the `exec_prefix' Make variable
(defaults to the --prefix directory) is given.

If DESTDIR is set, it will be taken as the root directory of the
installation, and files will be installed into $(DESTDIR)$(prefix),
$(DESTDIR)$(exec_prefix), etc.

All subdirectories created will have Python's version number in their
name, e.g. the library modules are installed in
"/usr/local/lib/python<version>/" by default, where <version> is the
<major>.<minor> release number (e.g. "2.1").  The Python binary is
installed as "python<version>" and a hard link named "python" is
created.  The only file not installed with a version number in its
name is the manual page, installed as "/usr/local/man/man1/python.1"
by default.

If you want to install multiple versions of Python see the section below
entitled "Installing multiple versions".

The only thing you may have to install manually is the Python mode for
Emacs found in Misc/python-mode.el.  (But then again, more recent
versions of Emacs may already have it.)  Follow the instructions that
came with Emacs for installation of site-specific files.

On Mac OS X, if you have configured Python with --enable-framework, you
should use "make frameworkinstall" to do the installation. Note that this
installs the Python executable in a place that is not normally on your
PATH, you may want to set up a symlink in /usr/local/bin.


Installing multiple versions
----------------------------

On Unix and Mac systems if you intend to install multiple versions of Python
using the same installation prefix (--prefix argument to the configure
script) you must take care that your primary python executable is not
overwritten by the installation of a different versio.  All files and
directories installed using "make altinstall" contain the major and minor
version and can thus live side-by-side.  "make install" also creates
${prefix}/bin/python which refers to ${prefix}/bin/pythonX.Y.  If you intend
to install multiple versions using the same prefix you must decide which
version (if any) is your "primary" version.  Install that version using
"make install".  Install all other versions using "make altinstall".

For example, if you want to install Python 2.5, 2.6 and 3.0 with 2.6 being
the primary version, you would execute "make install" in your 2.6 build
directory and "make altinstall" in the others.


Configuration options and variables
-----------------------------------

Some special cases are handled by passing options to the configure
script.

WARNING: if you rerun the configure script with different options, you
must run "make clean" before rebuilding.  Exceptions to this rule:
after changing --prefix or --exec-prefix, all you need to do is remove
Modules/getpath.o.

--with(out)-gcc: The configure script uses gcc (the GNU C compiler) if
        it finds it.  If you don't want this, or if this compiler is
        installed but broken on your platform, pass the option
        --without-gcc.  You can also pass "CC=cc" (or whatever the
        name of the proper C compiler is) in the environment, but the
        advantage of using --without-gcc is that this option is
        remembered by the config.status script for its --recheck
        option.

--prefix, --exec-prefix: If you want to install the binaries and the
        Python library somewhere else than in /usr/local/{bin,lib},
        you can pass the option --prefix=DIRECTORY; the interpreter
        binary will be installed as DIRECTORY/bin/python and the
        library files as DIRECTORY/lib/python/*.  If you pass
        --exec-prefix=DIRECTORY (as well) this overrides the
        installation prefix for architecture-dependent files (like the
        interpreter binary).  Note that --prefix=DIRECTORY also
        affects the default module search path (sys.path), when
        Modules/config.c is compiled.  Passing make the option
        prefix=DIRECTORY (and/or exec_prefix=DIRECTORY) overrides the
        prefix set at configuration time; this may be more convenient
        than re-running the configure script if you change your mind
        about the install prefix.

--with-readline: This option is no longer supported.  GNU
        readline is automatically enabled by setup.py when present.

--with-threads: On most Unix systems, you can now use multiple
        threads, and support for this is enabled by default.  To
        disable this, pass --with-threads=no.  If the library required
        for threads lives in a peculiar place, you can use
        --with-thread=DIRECTORY.  IMPORTANT: run "make clean" after
        changing (either enabling or disabling) this option, or you
        will get link errors!  Note: for DEC Unix use
        --with-dec-threads instead.

--with-sgi-dl: On SGI IRIX 4, dynamic loading of extension modules is
        supported by the "dl" library by Jack Jansen, which is
        ftp'able from ftp://ftp.cwi.nl/pub/dynload/dl-1.6.tar.Z.
        This is enabled (after you've ftp'ed and compiled the dl
        library) by passing --with-sgi-dl=DIRECTORY where DIRECTORY
        is the absolute pathname of the dl library.  (Don't bother on
        IRIX 5, it already has dynamic linking using SunOS style
        shared libraries.)  THIS OPTION IS UNSUPPORTED.

--with-dl-dld: Dynamic loading of modules is rumored to be supported
        on some other systems: VAX (Ultrix), Sun3 (SunOS 3.4), Sequent
        Symmetry (Dynix), and Atari ST.  This is done using a
        combination of the GNU dynamic loading package
        (ftp://ftp.cwi.nl/pub/dynload/dl-dld-1.1.tar.Z) and an
        emulation of the SGI dl library mentioned above (the emulation
        can be found at
        ftp://ftp.cwi.nl/pub/dynload/dld-3.2.3.tar.Z).  To
        enable this, ftp and compile both libraries, then call
        configure, passing it the option
        --with-dl-dld=DL_DIRECTORY,DLD_DIRECTORY where DL_DIRECTORY is
        the absolute pathname of the dl emulation library and
        DLD_DIRECTORY is the absolute pathname of the GNU dld library.
        (Don't bother on SunOS 4 or 5, they already have dynamic
        linking using shared libraries.)  THIS OPTION IS UNSUPPORTED.

--with-libm, --with-libc: It is possible to specify alternative
        versions for the Math library (default -lm) and the C library
        (default the empty string) using the options
        --with-libm=STRING and --with-libc=STRING, respectively.  For
        example, if your system requires that you pass -lc_s to the C
        compiler to use the shared C library, you can pass
        --with-libc=-lc_s. These libraries are passed after all other
        libraries, the C library last.

--with-libs='libs': Add 'libs' to the LIBS that the python interpreter
        is linked against.

--with-cxx-main=<compiler>: If you plan to use C++ extension modules,
        then -- on some platforms -- you need to compile python's main()
        function with the C++ compiler. With this option, make will use
        <compiler> to compile main() *and* to link the python executable.
        It is likely that the resulting executable depends on the C++
        runtime library of <compiler>. (The default is --without-cxx-main.)

        There are platforms that do not require you to build Python
        with a C++ compiler in order to use C++ extension modules.
        E.g., x86 Linux with ELF shared binaries and GCC 3.x, 4.x is such
        a platform. We recommend that you configure Python
        --without-cxx-main on those platforms because a mismatch
        between the C++ compiler version used to build Python and to
        build a C++ extension module is likely to cause a crash at
        runtime.

        The Python installation also stores the variable CXX that
        determines, e.g., the C++ compiler distutils calls by default
        to build C++ extensions. If you set CXX on the configure command
        line to any string of non-zero length, then configure won't
        change CXX. If you do not preset CXX but pass
        --with-cxx-main=<compiler>, then configure sets CXX=<compiler>.
        In all other cases, configure looks for a C++ compiler by
        some common names (c++, g++, gcc, CC, cxx, cc++, cl) and sets
        CXX to the first compiler it finds. If it does not find any
        C++ compiler, then it sets CXX="".

        Similarly, if you want to change the command used to link the
        python executable, then set LINKCC on the configure command line.


--with-pydebug:  Enable additional debugging code to help track down
        memory management problems.  This allows printing a list of all
        live objects when the interpreter terminates.

--with(out)-universal-newlines: enable reading of text files with
        foreign newline convention (default: enabled). In other words,
        any of \r, \n or \r\n is acceptable as end-of-line character.
        If enabled import and execfile will automatically accept any newline
        in files. Python code can open a file with open(file, 'U') to
        read it in universal newline mode. THIS OPTION IS UNSUPPORTED.

--with-tsc: Profile using the Pentium timestamping counter (TSC).

--with-system-ffi:  Build the _ctypes extension module using an ffi
        library installed on the system.


Building for multiple architectures (using the VPATH feature)
-------------------------------------------------------------

If your file system is shared between multiple architectures, it
usually is not necessary to make copies of the sources for each
architecture you want to support.  If the make program supports the
VPATH feature, you can create an empty build directory for each
architecture, and in each directory run the configure script (on the
appropriate machine with the appropriate options).  This creates the
necessary subdirectories and the Makefiles therein.  The Makefiles
contain a line VPATH=... which points to a directory containing the
actual sources.  (On SGI systems, use "smake -J1" instead of "make" if
you use VPATH -- don't try gnumake.)

For example, the following is all you need to build a minimal Python
in /usr/tmp/python (assuming ~guido/src/python is the toplevel
directory and you want to build in /usr/tmp/python):

        $ mkdir /usr/tmp/python
        $ cd /usr/tmp/python
        $ ~guido/src/python/configure
        [...]
        $ make
        [...]
        $

Note that configure copies the original Setup file to the build
directory if it finds no Setup file there.  This means that you can
edit the Setup file for each architecture independently.  For this
reason, subsequent changes to the original Setup file are not tracked
automatically, as they might overwrite local changes.  To force a copy
of a changed original Setup file, delete the target Setup file.  (The
makesetup script supports multiple input files, so if you want to be
fancy you can change the rules to create an empty Setup.local if it
doesn't exist and run it with arguments $(srcdir)/Setup Setup.local;
however this assumes that you only need to add modules.)

Also note that you can't use a workspace for VPATH and non VPATH builds. The
object files left behind by one version confuses the other.


Building on non-UNIX systems
----------------------------

For Windows (2000/NT/ME/98/95), assuming you have MS VC++ 7.1, the
project files are in PCbuild, the workspace is pcbuild.dsw.  See
PCbuild\readme.txt for detailed instructions.

For other non-Unix Windows compilers, in particular MS VC++ 6.0 and
for OS/2, enter the directory "PC" and read the file "readme.txt".

For the Mac, a separate source distribution will be made available,
for use with the CodeWarrior compiler.  If you are interested in Mac
development, join the PythonMac Special Interest Group
(http://www.python.org/sigs/pythonmac-sig/, or send email to
[email protected]).

Of course, there are also binary distributions available for these
platforms -- see http://www.python.org/.

To port Python to a new non-UNIX system, you will have to fake the
effect of running the configure script manually (for Mac and PC, this
has already been done for you).  A good start is to copy the file
pyconfig.h.in to pyconfig.h and edit the latter to reflect the actual
configuration of your system.  Most symbols must simply be defined as
1 only if the corresponding feature is present and can be left alone
otherwise; however the *_t type symbols must be defined as some
variant of int if they need to be defined at all.

For all platforms, it's important that the build arrange to define the
preprocessor symbol NDEBUG on the compiler command line in a release
build of Python (else assert() calls remain in the code, hurting
release-build performance).  The Unix, Windows and Mac builds already
do this.


Miscellaneous issues
====================

Emacs mode
----------

There's an excellent Emacs editing mode for Python code; see the file
Misc/python-mode.el.  Originally written by the famous Tim Peters, it
is now maintained by the equally famous Barry Warsaw (it's no
coincidence that they now both work on the same team).  The latest
version, along with various other contributed Python-related Emacs
goodies, is online at http://www.python.org/emacs/python-mode.  And
if you are planning to edit the Python C code, please pick up the
latest version of CC Mode http://www.python.org/emacs/cc-mode; it
contains a "python" style used throughout most of the Python C source
files.  (Newer versions of Emacs or XEmacs may already come with the
latest version of python-mode.)


Tkinter
-------

The setup.py script automatically configures this when it detects a
usable Tcl/Tk installation.  This requires Tcl/Tk version 8.0 or
higher.

For more Tkinter information, see the Tkinter Resource page:
http://www.python.org/topics/tkinter/

There are demos in the Demo/tkinter directory.

Note that there's a Python module called "Tkinter" (capital T) which
lives in Lib/lib-tk/Tkinter.py, and a C module called "_tkinter"
(lower case t and leading underscore) which lives in
Modules/_tkinter.c.  Demos and normal Tk applications import only the
Python Tkinter module -- only the latter imports the C _tkinter
module.  In order to find the C _tkinter module, it must be compiled
and linked into the Python interpreter -- the setup.py script does
this.  In order to find the Python Tkinter module, sys.path must be
set correctly -- normal installation takes care of this.


Distribution structure
----------------------

Most subdirectories have their own README files.  Most files have
comments.

Demo/           Demonstration scripts, modules and programs
Doc/            Documentation sources (reStructuredText)
Grammar/        Input for the parser generator
Include/        Public header files
LICENSE         Licensing information
Lib/            Python library modules
Mac/            Macintosh specific resources
Makefile.pre.in Source from which config.status creates the Makefile.pre
Misc/           Miscellaneous useful files
Modules/        Implementation of most built-in modules
Objects/        Implementation of most built-in object types
PC/             Files specific to PC ports (DOS, Windows, OS/2)
PCbuild/        Build directory for Microsoft Visual C++
Parser/         The parser and tokenizer and their input handling
Python/         The byte-compiler and interpreter
README          The file you're reading now
RISCOS/         Files specific to RISC OS port
Tools/          Some useful programs written in Python
pyconfig.h.in   Source from which pyconfig.h is created (GNU autoheader output)
configure       Configuration shell script (GNU autoconf output)
configure.in    Configuration specification (input for GNU autoconf)
install-sh      Shell script used to install files
setup.py        Python script used to build extension modules

The following files will (may) be created in the toplevel directory by
the configuration and build processes:

Makefile        Build rules
Makefile.pre    Build rules before running Modules/makesetup
buildno         Keeps track of the build number
config.cache    Cache of configuration variables
pyconfig.h      Configuration header
config.log      Log from last configure run
config.status   Status from last run of the configure script
getbuildinfo.o  Object file from Modules/getbuildinfo.c
libpython<version>.a    The library archive
python          The executable interpreter
reflog.txt      Output from running the regression suite with the -R flag 
tags, TAGS      Tags files for vi and Emacs


That's all, folks!
------------------


--Guido van Rossum (home page: http://www.python.org/~guido/)

unladen-swallow's People

Watchers

 avatar

unladen-swallow's Issues

Add a regex benchmark suite

Python currently doesn't have a good-quality regex benchmark suite that can
be run automatically, have statistics drawn from it, etc. We need such a
thing before starting work on regex performance.

Possible resources:
- Fredrik Lundh's original benchmarks for SRE:
http://mail.python.org/pipermail/python-dev/2000-August/007797.html
- V8's JS regex benchmarks:
http://v8.googlecode.com/svn/data/benchmarks/v3/regexp.js

Ideally we would do a search of the Python regexes in Google Code Search or
similar corpus and distill some representative set from them. V8's may be
good enough, though.

Original issue reported on code.google.com by collinw on 14 Apr 2009 at 11:37

Review vmgen branch for merge into trunk

Purpose of code changes on this branch:
To use vmgen to generate the main interpretation loop in PyEval_EvalFrameEx().

When reviewing my code changes, please focus on:
Readability, understandability, and anything stupid I did.

After the review, I'll merge this branch into:
/trunk

Original issue reported on code.google.com by [email protected] on 31 Jan 2009 at 1:00

Tune hotness function

We should add a sys.optimize decorator so that known-important functions
don't have to hit the hotness threshold before we optimize them. This would
be particularly useful for the slowspitfire benchmark: the function that
does all the work will never be "hot" by our current heuristics.

Currently, if we force compilation via -L2, slowspitfire shows a ~10% gain
over 2009Q1, but -L2 hurts start-up time. A decorator like this is similar
to the way Spitfire uses Psyco (ie, explicitly flagging functions for
optimization).

Original issue reported on code.google.com by collinw on 5 Jun 2009 at 11:47

Offload JIT compilation to secondary threads

Execution should not block on compiling a function with LLVM or
reoptimizing it with new data. We should send these work units to separate
worker threads, allowing the main threads to carry on unimpeded.

The first implementation could/should probably just be a simple FIFO queue
of work items with a single worker thread. We can add heuristics and
additional worker threads as the data warrants.

Original issue reported on code.google.com by collinw on 28 May 2009 at 9:57

compilation fails with IBM xlc in ceval

In ceval.c, compilation fails with IBM's xlc at the following point:

Python/ceval.c-854-     static Opcode labels[] = {
Python/ceval.c:855:#include "ceval-labels.i"
Python/ceval.c-856-     };

with a message for each entry in ceval-labels.i that is of the form:

"Include/ceval-labels.i", line 1.1: 1506-221 (S) Initializer must be a
valid constant expression. 

I think it's actually just upset about taking the address of a label,
something gcc complains about if you try to compile with -pedantic. It's
not a showstopper as it is still possible to compile ceval with gcc and
finish the build with xlc.

Original issue reported on code.google.com by [email protected] on 27 Apr 2009 at 3:45

Share code objects between identical functions

This idea is more speculative: every "def foo(): pass" function has the
same bytecode. It might reduce memory usage if we should share the
implementation of these functions between function definitions. This could
be done by hashing the function's parameters and bytecode.

Issues:
- Hashing the AST might be better than hashing the bytecode.
- If two functions have the same bytecode but different parameter names,
they probably can't take advantage of this. In the example above, "def
foo(): pass" and "def bar(a): pass" can't share an implementation

I'd be interested to know, for an application like 2to3 or Django's test
suite, how many functions are really the same (modulo their name).

Original issue reported on code.google.com by collinw on 6 Jun 2009 at 12:07

Python compiles dead code

CPython and Unladen Swallow both currently emit bytecode/machine code for
code following a return, raise, break or continue statement, like so:

def foo():
  return 5
  for i in range(4):
    print i

Eliminating this code isn't critical (it isn't very common, and LLVM is
smart enough to eliminate it for us), but the need to support this has
spread into the LLVM compiler, which has to allow for this with special
deadcode blocks. This is ugly and should be fixed.

This could be done in a couple of ways: the dead code could be pruned from
the AST (or not even make it into the AST), or could be ignored by the
bytecode compiler.

Original issue reported on code.google.com by collinw on 13 May 2009 at 10:40

LLVM-generated functions need to handle signals

Currently, LLVM-generated functions don't ever check for pending signals to
handle the way the interpreter does. Among other problems, this makes it
impossible to KeyboardInterrupt Unladen Swallow when running with -L[012].

evlogimenos has agreed to work on this (and thread switching, since they're
related).

Original issue reported on code.google.com by collinw on 28 May 2009 at 12:21

Compiling large functions runs out of memory

The following program:

import sys
copies = int(sys.argv[1])
print "Running %d copies..." % copies

longexpr = 'x = x or ' + '-x' * 2500
code = ('''
def f(x):
''' + '''    %s
''' * copies + '''
    # the expressions above have no effect, x == argument
    while x:
        x -= 1
    return x
''') % ((longexpr,) * copies)
exec code
print f(5)



demonstrates non-linear memory use and running times with Unladen Swallow
r573. This was extracted from test_compile, which takes more than 4GB of
memory which exhausts a 32-bit address space.

The memory use below is from watching the "real memory" column in the apple
activity monitor and taking the highest number I saw.

$ time ./python.exe -S -L0 ./use_lots_of_memory.py  1
Running 1 copies...
0

real    0m37.979s
user    0m36.972s
sys 0m0.602s
memory 174MB

$ time ./python.exe -S -L0 ./use_lots_of_memory.py  2
Running 2 copies...
0

real    1m15.750s
user    1m13.479s
sys 0m1.368s
memory 491MB  (delta 317)

$ time ./python.exe -S -L0 ./use_lots_of_memory.py  3
Running 3 copies...
0

real    2m6.118s
user    2m2.631s
sys 0m2.328s
memory 944MB  (delta 453)

$ time ./python.exe -S -L0 ./use_lots_of_memory.py  4
Running 4 copies...
0

real    5m59.303s
user    3m10.135s
sys 0m14.223s
memory 1500MB  (delta 556)


Watching memory use of the "3" case, it seems to rise to ~30-40MB up to and
during SimplifyCFG, then rise by ~3 MB per second through CodeGenAndEmitDAG
up to about 150MB, and then rise by ~50MB per second through
LiveVariables::runOnMachineFunction up to about 750MB. LiveIntervals seems
to account for most of the rest of the memory use.

Original issue reported on code.google.com by [email protected] on 22 May 2009 at 2:59

test_bsddb3 is flaky

test_bsddb3 fails sometimes, passes other times, resulting in spurious
failure reports. There may be patches upstream to make it more stable.

Original issue reported on code.google.com by collinw on 15 May 2009 at 9:50

Referenceleaks in LLVM mode

When running regrtest -R:: (which requires a pydebug build) with -L0 or
higher, some of the tests end up leaking references:

test_codecs leaked [27, 27, 27, 27] references, sum=108
test_copy leaked [2, 2, 2, 2] references, sum=8
test_datetime leaked [2, 2, 2, 2] references, sum=8
test_decimal leaked [2, 2, 2, 2] references, sum=8
test_difflib leaked [56, 56, 56, 56] references, sum=224
test_generators leaked [296, 296, 296, 296] references, sum=1184
test_grammar leaked [3, 3, 3, 3] references, sum=12
test_io leaked [2, 2, 2, 2] references, sum=8
test_itertools leaked [26, 26, 26, 26] references, sum=104
test_lib2to3 leaked [22, 22, 22, 22] references, sum=88

These tests do not leak without -L. Considering test_llvm does not leak, I
suspect a leak in an error path that test_llvm fails to test.

Original issue reported on code.google.com by [email protected] on 20 May 2009 at 1:12

Teach the JIT to recompile things

We currently use llvm::ExecutionEngine::getPointerToFunction(func) to
translate LLVM IR into machine code. getPointerToFunction caches the
returned machine code for each input function, and will not regenerate it
even if the function has changed. This means we can't re-optimize a
function after calling it through LLVM.

ExecutionEngine::recompileAndRelinkFunction(func) is available to force
LLVM to regenerate the machine code, but it overwrites the original machine
code with a jump to the new block, which means we can't use it while any
stack frame in any thread is still executing the code.

More information on fixing this is at
http://wiki.llvm.org/Provide_more_control_over_and_access_to_JIT%27s_output

Original issue reported on code.google.com by [email protected] on 28 May 2009 at 10:47

Measure: function call overhead

We need to have more accurate measurements for the total time taken from
when we start a CALL_FUNCTION opcode to the time when the body of the
function starts executing. This should work regardless of whether we're
dispatching to an interpreted function or an LLVM function; C functions can
be fudged a bit (right before calling the function pointer?)


- Data should be stored in a vector and stats printed out at Python-shutdown.
- This should include whether the execution is in machine code or the
interpreter.
- Should be a special build (controlled by #ifdef's).
- Use TSCs?

Original issue reported on code.google.com by collinw on 30 May 2009 at 1:46

Make perf.py --track_memory work on other platforms

Currently, perf.py's --track_memory option works by reading Linux 2.6 smaps
files from /proc/. This obviously doesn't work on non-Linux platforms, or
even Linux platforms pre-2.6.16.

Darwin is the most important of the currently-unsupported platforms.
/usr/bin/time -l (lowercase ell) on Darwin will provide the maximum rss for
the process.

Original issue reported on code.google.com by collinw on 27 May 2009 at 2:52

Get full stack traces from LLVM JITed code

Steps to reproduce:
1. Modify a C function in Python that you know will be called from LLVM so
that it will segfault.  (printf("%d\n", *((int*)NULL));)
2. Build Python --with-pydebug.
3. Load up the binary in GDB.
4. Run a python script that will call the bad function with -L2.
5. Examine the backtrace.

The stack trace should show your C function and maybe some frames above it,
and then the heap address of some LLVM generated code with no name for it.
 If you're lucky, eventually the trace will get past there and make it back
down to main.  If you're unlucky, you get nothing.

As a work around, you can set the environment variable
PYTHONLLVMFLAGS="--disable-fp-elim" to your python binary, and that should
give you a full stack trace that is only missing information for LLVM
generated code.  Eventually we'd like to enable this by default for debug
builds, but right now there's no easy way to toggle that option.  The right
thing to do would probably be to submit a patch to LLVM so we can toggle
that option.

Original issue reported on code.google.com by [email protected] on 8 Jun 2009 at 5:50

Persist LLVM IR to .pyc files

Currently, the compiler emits both CPython bytecode and LLVM IR and
attaches both to all code objects it creates. However, the marshal format
only understands bytecode and drops the LLVM IR on the floor when it saves
.pyc files. Disabling .pyc files entirely seems to slow down regrtest to
unacceptable levels so r323 just checks that the LLVM IR is present when it
tries to run through the LLVM JIT and raises a SystemError if it's not
present. We'll need to fix this before the JIT is really a viable way to
run things.

Original issue reported on code.google.com by [email protected] on 24 Mar 2009 at 7:34

f_lasti is probably broken

We have altered the definition of f_lasti: pray we don't alter it any
further; but regardless, we should make it work again for pdb.

Original issue reported on code.google.com by [email protected] on 22 Apr 2009 at 3:26

perf.py needs to track memory usage

perf.py should grow a --memory option that will profile memory usage of the
different benchmarks. Memory usage would be summarized in the same way that
running time is.

Original issue reported on code.google.com by collinw on 18 May 2009 at 8:50

Re-make-ing rebuilds too much

Running make twice in a row takes forever, even if you didn't change
anything: make wants to rebuild Python/llvm_inline_functions.bc, which
cascades into causing the python binary and libpython to both be relinked.

A simple no-op make takes almost 30 seconds, all to do nothing.

Original issue reported on code.google.com by collinw on 28 May 2009 at 4:07

Review only_why_reports_exceptions change-branch

Purpose of code changes on this branch: To investigate removing all ways of
reporting errors except for 'why'.

2to3 seems to be slightly faster with this patch; pybench seems to be
slightly slower. We'll need to measure this on real hardware and compilers,
rather than my laptop.

There are a couple bugs in EvalFrameEx where it doesn't signal an error
when it should. Since those are functionality changes, I'll do them in a
separate change.

After the review, I'll merge this branch into:
/trunk

... Does code know how to diff branches? We'll see.

Original issue reported on code.google.com by [email protected] on 8 Dec 2008 at 12:17

Cannot pickle cPickle.Pickler objects

In Python 2.6, you can pickle cPickle.Pickler and Unpickler items. I seem
to have broken that in Unladen Swallow.

Original issue reported on code.google.com by collinw on 22 Apr 2009 at 4:42

Inline calls to Py_INCREF/Py_DECREF

In our LLVM code, Py_INCREF/Py_DECREF are function calls instead of the
nice speedy macros that the interpreter can take advantage of. These calls
should be inlined in the LLVM IR.

jyasskin is already working on this.

Original issue reported on code.google.com by collinw on 27 May 2009 at 10:15

Add a ./configure flag to disable LLVM

In order to maintain support for smaller platforms like cell phones, we
should include a ./configure flag to disable LLVM entirely. Since we'll be
keeping around the eval loop, this should be pretty straightforward to
implement.

Original issue reported on code.google.com by collinw on 13 Apr 2009 at 10:33

Some tests are flaky

test_bsddb3 fails sometimes, passes other times, resulting in spurious
failure reports. There may be patches upstream to make it more stable.

Original issue reported on code.google.com by collinw on 15 May 2009 at 9:49

Review opcodes_to_functions branch

Purpose of code changes on this branch: simplify eval loop by converting a
number of infrequent to builtin functions. These builtins are prefixed with
#@, e.g., #@import_star.

This shows a 1% improvement in 2to3 performance on all tested platforms.
This should also contribute positively to the vmgen branch.


After the review, I'll merge this branch into:
/trunk


Original issue reported on code.google.com by collinw on 5 Jan 2009 at 7:34

Simple functions are way bigger than they need to be. Fix that.

We currently know of a couple inefficiencies:

1. Because we always load the stack pointer, LLVM can't optimize away the
unwind loop, even in simple functions that, say, return once from the outer
scope. For non-generators, we can do better.
2. The block stack is stored in the frame, which again prevents LLVM from
optimizing away most of the unwind block.

This issue will hold a summary of this kind of issue, but we may split it
up when we get around to optimizing these things.

Original issue reported on code.google.com by [email protected] on 21 Apr 2009 at 10:01

Speed up regular expressions

CPython's regex implementation is slower than it could be. There are a lot
of techniques we could use to make things faster:

- JIT compile regexes. V8 and SquirrelFish Extreme have had good luck with
this approach.
- Thompson NFAs are much faster for some regexes, but don't support the
full range of Python regex syntax. This would necessitate a multi-engine
system that picks the fastest engine for the given regex.
- Apply VM optimizations from the main Python eval loop to the regex eval
loop. This isn't likely to give us the speed we want, but may help in a
multi-engine approach if one of those engines is the existing regex VM.


Longer-form, with links:
http://code.google.com/p/unladen-swallow/wiki/ProjectPlan#Regular_Expressions

Original issue reported on code.google.com by collinw on 14 Apr 2009 at 11:30

sqlite3 fails to import

It looks like sqlite3 fails to import due to not being able to track down 
the old so file. I have attached the strace -fi but I had to ****** some 
stuff for security/privacy reasons

Original issue reported on code.google.com by [email protected] on 11 May 2009 at 7:46

Attachments:

Turn BUILD_TUPLE/BUILD_LIST into memcpy operations.

The core of BUILD_TUPLE/BUILD_LIST (and also other opcodes that build
tuples or lists out of stack items, like CALL_FUNCTION might do) is
essentially a memcpy() from the stack to the body of the (newly created)
tuple/list object. LLVM doesn't seem to optimize this by itself, so we
should turn this into an @llvm.memcpy call.

Original issue reported on code.google.com by [email protected] on 21 Apr 2009 at 10:43

test_urllib2_localnet leaks references

regrtest.py -R:: test_urllib2_localnet currently leaks three references per
run. This problem is present in Unladen Swallow trunk and
release-2009Q1-maint, as well as mainline CPython trunk.

I've found the line that triggers the leak and am tracking down the cause.

Original issue reported on code.google.com by collinw on 15 May 2009 at 7:02

llc produces unused variables

llc produces some unused variables in its output (at least for
initial_llvm_module.cc):

Python/initial_llvm_module.cc:662: warning: unused variable ‘PointerTy_80’
Python/initial_llvm_module.cc:664: warning: unused variable ‘PointerTy_81’
Python/initial_llvm_module.cc:675: warning: unused variable ‘PointerTy_82’
Python/initial_llvm_module.cc:693: warning: unused variable ‘PointerTy_86’
Python/initial_llvm_module.cc:704: warning: unused variable ‘PointerTy_88’
Python/initial_llvm_module.cc:716: warning: unused variable ‘PointerTy_90’
Python/initial_llvm_module.cc:724: warning: unused variable ‘PointerTy_92’

We silence the warnings for now, but we should see if we can fix llc instead.

Original issue reported on code.google.com by [email protected] on 12 May 2009 at 9:06

Huge expressions are really slow to compile under LLVM

Lib/test/test_compile.py takes 1.7s for python2.6, 33s for trunk unladen
swallow (which emits some LLVM bitcode), and 12.5min for the llvm-working
branch (which emits full LLVM bitcode). The problem is test_extended_arg
which emits expressions containing 2500 subtractions.

Original issue reported on code.google.com by [email protected] on 30 Mar 2009 at 10:36

Add fine-grained JIT command line options

We should have a --jit option (approximate) that controls when and how
often the reoptimizer will run.

--jit=once will compile a function as heavily as possible the first time
it's found hot, then ignore it thereafter. This may mean that bad
predictions can't be corrected; we'll have to see.

--jit=never disables the LLVM integration, forcing all code to run through
the bytecode interpreter.

--jit=everything is equivalent to the current -L[012] options: all
functions are compiled to machine code when they are defined. As functions
are found to be hotter and hotter, they may be reoptimized.

The -O option should be extended to take numeric arguments the way gcc
does. -O[0123] will control the initial optimization used by --jit=once and
--jit=everything.

I'm open to debate on what other options --jit should support. Should
--jit=everything disable reoptimization?

Original issue reported on code.google.com by collinw on 30 May 2009 at 1:35

Add a script to install LLVM with the right options

Now that we have the --with-llvm option, we should have a script in
Util/llvm to configure, make and install LLVM with the proper options. We
pass a bunch of options from Python's ./configure to LLVM's ./configure,
and those should be automated into a simple script. Otherwise, you'll have
to dig through Python's ./configure to find the right options every time,
and that's a pain in the ass.

This will make --with-llvm a lot easier to use.

Original issue reported on code.google.com by collinw on 29 May 2009 at 4:47

Please review the faster-pickling branch

Purpose of code changes on this branch: speed up cPickle.

Pickle (complex):
Min: 1.023 -> 0.409: 150.36% faster
Avg: 1.053 -> 0.410: 157.17% faster
Significant (t=1102.029662, a=0.95)

Pickle (simple):
Min: 1.223 -> 0.868: 40.83% faster
Avg: 1.229 -> 0.876: 40.20% faster
Significant (t=695.483070, a=0.95)

Unpickle (complex):
Min: 0.738 -> 0.536: 37.71% faster
Avg: 0.746 -> 0.547: 36.24% faster
Significant (t=122.112665, a=0.95)

Unpickle (simple):
Min: 0.756 -> 0.486: 55.60% faster
Avg: 0.774 -> 0.493: 56.91% faster
Significant (t=331.578243, a=0.95)


When reviewing my code changes, please focus on: anything stupid I did,
style issues, things that might block merger back to mainline.

Let me know if you'd rather review this on Rietveld.


After the review, I'll merge this branch into:
/trunk



Original issue reported on code.google.com by collinw on 28 Feb 2009 at 9:49

Look into faster string concentration for templating languages?

A central part of templating languages is the way they combine multiple strings 
into one string 
during template runtime.

The current most often used pattern seems to be a BufferIO class like the one 
from spitfire 
(http://code.google.com/p/spitfire/source/browse/trunk/spitfire/runtime/template
.py):

class BufferIO(list):
  write = list.append

  def getvalue(self):
    return ''.join(self)

The constrains of templating languages have even more minimal requirements than 
the built-in 
list used above, namely they care only about appending of multiple Unicode 
strings and a one-
time combination of those into a single Unicode string. As no slicing, 
retrieval of individual 
values or anything like it is required, I'm wondering if a more fine-tuned 
version of this could 
result in noticeable differences for the spitfire test cases. Using a 
collections.deque instead of a 
list here, doesn't produce any real difference.

Original issue reported on code.google.com by [email protected] on 28 Mar 2009 at 2:29

TSC support is broken.

Building with TSC (the WITH_TSC define, enabled by --with-tsc) which
instruments CPython with CPU counters, is currently broken (at least by the
vmgen patch, which does not include the right #ifdefs in call_function and
the CALL_FUNCTION_VAR_KW opcode implementation.)

Original issue reported on code.google.com by [email protected] on 14 Apr 2009 at 1:34

Add fuzz-based testing

As we do deeper and deeper surgery on the compiler, I'd like to have a
fuzzer or some other kind of randomized testing to look for corner cases in
our implementation.

Before writing our own, we should try to reuse Fusil
(http://fusil.hachoir.org/svn/trunk/), which has been shown to find bugs in
CPython already. Other Python implementations may already have something
like this; if so, we should reuse that.

Eventually, this would become part of the continuous build, forever
searching the space of random Python programs for crashes.

Original issue reported on code.google.com by collinw on 14 Apr 2009 at 11:46

test_distutils leaks references

regrtest.py -R:: test_distutils leaks 103-105 references per run. This
problem is present in both trunk and release-2009Q1-maint, though trunk
leaks 103 references and release-2009Q1-maint leaks 105 references.

Original issue reported on code.google.com by collinw on 15 May 2009 at 7:03

Make the stackpointer not escape

CALL_FUNCTION, CALL_FUNCTION_VAR_KW and UNPACK_SEQUENCE make the
stackpointer escape LLVM's pervue because their implementations (in C
functions) manipulate the stack directly. Getting rid of the direct
manipulation would allow LLVM to optimize more stack operations (and make
it easier to move away from the stack machine.)

Original issue reported on code.google.com by [email protected] on 21 Apr 2009 at 10:41

cPickle.Unpickler objects cannot be reused

The Unpickler doesn't re-read the from the file object when it runs out of
data, meaning Unpicklers can't be reused in long-lived streaming pickle
sessions. This also needs tests.

Original issue reported on code.google.com by collinw on 26 Mar 2009 at 3:28

LLVM LOAD_CONST implementation should skip co_consts

Currently, the LLVM implementation of the LOAD_CONST opcode indexes into
the code object's co_consts tuple. Since these are *constants*, the
generated machine code should just load the object's address, skipping
co_consts entirely.

Original issue reported on code.google.com by collinw on 29 May 2009 at 3:52

Measure: JIT compilation time

We should measure how long execution blocks for when sending a code object
to be compiled/optimized by LLVM. This will be useful for proving that
offloading compilation/optimization to worker threads is valuable.

- These times should be stored in a vector and statistics displayed at
Python-shutdown.
- This should be a special build (ie, controlled by #ifdef's).
- Timing done using TSCs?

Original issue reported on code.google.com by collinw on 30 May 2009 at 1:39

Consider use of CMake for build system

CMake would allow developers to choose their build system/IDE they use to 
build unladen-swallow. What say you?

Is anyone on the project familiar with CMake?

Original issue reported on code.google.com by [email protected] on 27 Mar 2009 at 9:22

LLVM-generated functions need to switch threads

Currently, LLVM-generated functions don't yield the GIL to other threads
(unless they're doing IO) the way the interpreter does. Among other
problems, this greatly reduces threading fairness when running with -L[012].

evlogimenos has agreed to work on this (and signal handling, since they're
related).

Original issue reported on code.google.com by collinw on 28 May 2009 at 12:30

cannot install 4Suite-XML

I couldn't install 4Suite-XML with either the 2009Q1 version or the trunk
version. It complains about KeyError: 'EXTENDED_ARG'. Attached is the
output from running easy_install 4Suite-XML.

Original issue reported on code.google.com by [email protected] on 4 May 2009 at 3:59

Attachments:

Need support for debugging LLVM-generated machine code

We currently don't have a good way of debugging the machine code that comes
out of LLVM's JIT compiler.

To the best of my knowledge, LLVM doesn't emit debug information for JITted
code. Even if it did, there's no way to tell gdb to read this information.
However, there is support in the LLVM and gdb communities for fixing both
of these issues.

References:
- http://lists.cs.uiuc.edu/pipermail/llvmdev/2009-March/021255.html
- http://lists.cs.uiuc.edu/pipermail/llvmdev/2009-April/021421.html
- http://lists.cs.uiuc.edu/pipermail/llvmdev/2009-April/021424.html
- http://wiki.llvm.org/HowTo:_Tell_GDB_about_JITted_code

Original issue reported on code.google.com by collinw on 27 May 2009 at 9:42

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.