pombreda / unladen-swallow Goto Github PK
View Code? Open in Web Editor NEWAutomatically exported from code.google.com/p/unladen-swallow
License: Other
Automatically exported from code.google.com/p/unladen-swallow
License: Other
This is Python version 2.6.4 ============================ Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009 Python Software Foundation. All rights reserved. Copyright (c) 2000 BeOpen.com. All rights reserved. Copyright (c) 1995-2001 Corporation for National Research Initiatives. All rights reserved. Copyright (c) 1991-1995 Stichting Mathematisch Centrum. All rights reserved. License information ------------------- See the file "LICENSE" for information on the history of this software, terms & conditions for usage, and a DISCLAIMER OF ALL WARRANTIES. This Python distribution contains no GNU General Public Licensed (GPLed) code so it may be used in proprietary projects just like prior Python distributions. There are interfaces to some GNU code but these are entirely optional. All trademarks referenced herein are property of their respective holders. What's new in this release? --------------------------- See the file "Misc/NEWS". If you don't read instructions ------------------------------ Congratulations on getting this far. :-) To start building right away (on UNIX): type "./configure" in the current directory and when it finishes, type "make". This creates an executable "./python"; to install in /usr/local, first do "su root" and then "make install". The section `Build instructions' below is still recommended reading. What is Python anyway? ---------------------- Python is an interpreted, interactive object-oriented programming language suitable (amongst other uses) for distributed application development, scripting, numeric computing and system testing. Python is often compared to Tcl, Perl, Java, JavaScript, Visual Basic or Scheme. To find out more about what Python can do for you, point your browser to http://www.python.org/. How do I learn Python? ---------------------- The official tutorial is still a good place to start; see http://docs.python.org/ for online and downloadable versions, as well as a list of other introductions, and reference documentation. There's a quickly growing set of books on Python. See http://wiki.python.org/moin/PythonBooks for a list. Documentation ------------- All documentation is provided online in a variety of formats. In order of importance for new users: Tutorial, Library Reference, Language Reference, Extending & Embedding, and the Python/C API. The Library Reference is especially of immense value since much of Python's power is described there, including the built-in data types and functions! All documentation is also available online at the Python web site (http://docs.python.org/, see below). It is available online for occasional reference, or can be downloaded in many formats for faster access. The documentation is downloadable in HTML, PostScript, PDF, LaTeX, and reStructuredText (2.6+) formats; the LaTeX and reStructuredText versions are primarily for documentation authors, translators, and people with special formatting requirements. Web sites --------- New Python releases and related technologies are published at http://www.python.org/. Come visit us! There's also a Python community web site at http://starship.python.net/. Newsgroups and Mailing Lists ---------------------------- Read comp.lang.python, a high-volume discussion newsgroup about Python, or comp.lang.python.announce, a low-volume moderated newsgroup for Python-related announcements. These are also accessible as mailing lists: see http://www.python.org/community/lists.html for an overview of these and many other Python-related mailing lists. Archives are accessible via the Google Groups Usenet archive; see http://groups.google.com/. The mailing lists are also archived, see http://www.python.org/community/lists.html for details. Bug reports ----------- To report or search for bugs, please use the Python Bug Tracker at http://bugs.python.org. Patches and contributions ------------------------- To submit a patch or other contribution, please use the Python Patch Manager at http://bugs.python.org. Guidelines for patch submission may be found at http://www.python.org/dev/patches/. If you have a proposal to change Python, you may want to send an email to the comp.lang.python or python-ideas mailing lists for inital feedback. A Python Enhancement Proposal (PEP) may be submitted if your idea gains ground. All current PEPs, as well as guidelines for submitting a new PEP, are listed at http://www.python.org/dev/peps/. Questions --------- For help, if you can't find it in the manuals or on the web site, it's best to post to the comp.lang.python or the Python mailing list (see above). If you specifically don't want to involve the newsgroup or mailing list, send questions to [email protected] (a group of volunteers who answer questions as they can). The newsgroup is the most efficient way to ask public questions. Build instructions ================== Before you can build Python, you must first configure it. Fortunately, the configuration and build process has been automated for Unix and Linux installations, so all you usually have to do is type a few commands and sit back. There are some platforms where things are not quite as smooth; see the platform specific notes below. If you want to build for multiple platforms sharing the same source tree, see the section on VPATH below. Start by running the script "./configure", which determines your system configuration and creates the Makefile. (It takes a minute or two -- please be patient!) You may want to pass options to the configure script -- see the section below on configuration options and variables. When it's done, you are ready to run make. To build Python, you normally type "make" in the toplevel directory. If you have changed the configuration, the Makefile may have to be rebuilt. In this case you may have to run make again to correctly build your desired target. The interpreter executable is built in the top level directory. Once you have built a Python interpreter, see the subsections below on testing and installation. If you run into trouble, see the next section. Previous versions of Python used a manual configuration process that involved editing the file Modules/Setup. While this file still exists and manual configuration is still supported, it is rarely needed any more: almost all modules are automatically built as appropriate under guidance of the setup.py script, which is run by Make after the interpreter has been built. Troubleshooting --------------- See also the platform specific notes in the next section. If you run into other trouble, see the FAQ (http://www.python.org/doc/faq) for hints on what can go wrong, and how to fix it. If you rerun the configure script with different options, remove all object files by running "make clean" before rebuilding. Believe it or not, "make clean" sometimes helps to clean up other inexplicable problems as well. Try it before sending in a bug report! If the configure script fails or doesn't seem to find things that should be there, inspect the config.log file. If you get a warning for every file about the -Olimit option being no longer supported, you can ignore it. There's no foolproof way to know whether this option is needed; all we can do is test whether it is accepted without error. On some systems, e.g. older SGI compilers, it is essential for performance (specifically when compiling eval.cc, which has more basic blocks than the default limit of 1000). If the warning bothers you, edit the Makefile to remove "-Olimit 1500" from the OPT variable. If you get failures in test_long, or sys.maxint gets set to -1, you are probably experiencing compiler bugs, usually related to optimization. This is a common problem with some versions of gcc, and some vendor-supplied compilers, which can sometimes be worked around by turning off optimization. Consider switching to stable versions (gcc 2.95.2, gcc 3.x, or contact your vendor.) From Python 2.0 onward, all Python C code is ANSI C. Compiling using old K&R-C-only compilers is no longer possible. ANSI C compilers are available for all modern systems, either in the form of updated compilers from the vendor, or one of the free compilers (gcc). If "make install" fails mysteriously during the "compiling the library" step, make sure that you don't have any of the PYTHONPATH or PYTHONHOME environment variables set, as they may interfere with the newly built executable which is compiling the library. Unsupported systems ------------------- A number of features are not supported in Python 2.5 anymore. Some support code is still present, but will be removed in Python 2.6. If you still need to use current Python versions on these systems, please send a message to [email protected] indicating that you volunteer to support this system. For a more detailed discussion regarding no-longer-supported and resupporting platforms, as well as a list of platforms that became or will be unsupported, see PEP 11. More specifically, the following systems are not supported any longer: - SunOS 4 - DYNIX - dgux - Minix - NeXT - Irix 4 and --with-sgi-dl - Linux 1 - Systems defining __d6_pthread_create (configure.in) - Systems defining PY_PTHREAD_D4, PY_PTHREAD_D6, or PY_PTHREAD_D7 in thread_pthread.h - Systems using --with-dl-dld - Systems using --without-universal-newlines - MacOS 9 The following systems are still supported in Python 2.5, but support will be dropped in 2.6: - Systems using --with-wctype-functions - Win9x, WinME Warning on install in Windows 98 and Windows Me ----------------------------------------------- Following Microsoft's closing of Extended Support for Windows 98/ME (July 11, 2006), Python 2.6 will stop supporting these platforms. Python development and maintainability becomes easier (and more reliable) when platform specific code targeting OSes with few users and no dedicated expert developers is taken out. The vendor also warns that the OS versions listed above "can expose customers to security risks" and recommends upgrade. Platform specific notes ----------------------- (Some of these may no longer apply. If you find you can build Python on these platforms without the special directions mentioned here, submit a documentation bug report to SourceForge (see Bug Reports above) so we can remove them!) Unix platforms: If your vendor still ships (and you still use) Berkeley DB 1.85 you will need to edit Modules/Setup to build the bsddb185 module and add a line to sitecustomize.py which makes it the default. In Modules/Setup a line like bsddb185 bsddbmodule.c should work. (You may need to add -I, -L or -l flags to direct the compiler and linker to your include files and libraries.) XXX I think this next bit is out of date: 64-bit platforms: The modules audioop, and imageop don't work. The setup.py script disables them on 64-bit installations. Don't try to enable them in the Modules/Setup file. They contain code that is quite wordsize sensitive. (If you have a fix, let us know!) Solaris: When using Sun's C compiler with threads, at least on Solaris 2.5.1, you need to add the "-mt" compiler option (the simplest way is probably to specify the compiler with this option as the "CC" environment variable when running the configure script). When using GCC on Solaris, beware of binutils 2.13 or GCC versions built using it. This mistakenly enables the -zcombreloc option which creates broken shared libraries on Solaris. binutils 2.12 works, and the binutils maintainers are aware of the problem. Binutils 2.13.1 only partially fixed things. It appears that 2.13.2 solves the problem completely. This problem is known to occur with Solaris 2.7 and 2.8, but may also affect earlier and later versions of the OS. When the dynamic loader complains about errors finding shared libraries, such as ld.so.1: ./python: fatal: libstdc++.so.5: open failed: No such file or directory you need to first make sure that the library is available on your system. Then, you need to instruct the dynamic loader how to find it. You can choose any of the following strategies: 1. When compiling Python, set LD_RUN_PATH to the directories containing missing libraries. 2. When running Python, set LD_LIBRARY_PATH to these directories. 3. Use crle(8) to extend the search path of the loader. 4. Modify the installed GCC specs file, adding -R options into the *link: section. The complex object fails to compile on Solaris 10 with gcc 3.4 (at least up to 3.4.3). To work around it, define Py_HUGE_VAL as HUGE_VAL(), e.g.: make CPPFLAGS='-D"Py_HUGE_VAL=HUGE_VAL()" -I. -I$(srcdir)/Include' ./python setup.py CPPFLAGS='-D"Py_HUGE_VAL=HUGE_VAL()"' Linux: A problem with threads and fork() was tracked down to a bug in the pthreads code in glibc version 2.0.5; glibc version 2.0.7 solves the problem. This causes the popen2 test to fail; problem and solution reported by Pablo Bleyer. Red Hat Linux: Red Hat 9 built Python2.2 in UCS-4 mode and hacked Tcl to support it. To compile Python2.3 with Tkinter, you will need to pass --enable-unicode=ucs4 flag to ./configure. There's an executable /usr/bin/python which is Python 1.5.2 on most older Red Hat installations; several key Red Hat tools require this version. Python 2.1.x may be installed as /usr/bin/python2. The Makefile installs Python as /usr/local/bin/python, which may or may not take precedence over /usr/bin/python, depending on how you have set up $PATH. FreeBSD 3.x and probably platforms with NCurses that use libmytinfo or similar: When using cursesmodule, the linking is not done in the correct order with the defaults. Remove "-ltermcap" from the readline entry in Setup, and use as curses entry: "curses cursesmodule.c -lmytinfo -lncurses -ltermcap" - "mytinfo" (so called on FreeBSD) should be the name of the auxiliary library required on your platform. Normally, it would be linked automatically, but not necessarily in the correct order. BSDI: BSDI versions before 4.1 have known problems with threads, which can cause strange errors in a number of modules (for instance, the 'test_signal' test script will hang forever.) Turning off threads (with --with-threads=no) or upgrading to BSDI 4.1 solves this problem. DEC Unix: Run configure with --with-dec-threads, or with --with-threads=no if no threads are desired (threads are on by default). When using GCC, it is possible to get an internal compiler error if optimization is used. This was reported for GCC 2.7.2.3 on selectmodule.c. Manually compile the affected file without optimization to solve the problem. DEC Ultrix: compile with GCC to avoid bugs in the native compiler, and pass SHELL=/bin/sh5 to Make when installing. AIX: A complete overhaul of the shared library support is now in place. See Misc/AIX-NOTES for some notes on how it's done. (The optimizer bug reported at this place in previous releases has been worked around by a minimal code change.) If you get errors about pthread_* functions, during compile or during testing, try setting CC to a thread-safe (reentrant) compiler, like "cc_r". For full C++ module support, set CC="xlC_r" (or CC="xlC" without thread support). AIX 5.3: To build a 64-bit version with IBM's compiler, I used the following: export PATH=/usr/bin:/usr/vacpp/bin ./configure --with-gcc="xlc_r -q64" --with-cxx="xlC_r -q64" \ --disable-ipv6 AR="ar -X64" make HP-UX: When using threading, you may have to add -D_REENTRANT to the OPT variable in the top-level Makefile; reported by Pat Knight, this seems to make a difference (at least for HP-UX 10.20) even though pyconfig.h defines it. This seems unnecessary when using HP/UX 11 and later - threading seems to work "out of the box". HP-UX ia64: When building on the ia64 (Itanium) platform using HP's compiler, some experience has shown that the compiler's optimiser produces a completely broken version of python (see http://www.python.org/sf/814976). To work around this, edit the Makefile and remove -O from the OPT line. To build a 64-bit executable on an Itanium 2 system using HP's compiler, use these environment variables: CC=cc CXX=aCC BASECFLAGS="+DD64" LDFLAGS="+DD64 -lxnet" and call configure as: ./configure --without-gcc then *unset* the environment variables again before running make. (At least one of these flags causes the build to fail if it remains set.) You still have to edit the Makefile and remove -O from the OPT line. HP PA-RISC 2.0: A recent bug report (http://www.python.org/sf/546117) suggests that the C compiler in this 64-bit system has bugs in the optimizer that break Python. Compiling without optimization solves the problems. SCO: The following apply to SCO 3 only; Python builds out of the box on SCO 5 (or so we've heard). 1) Everything works much better if you add -U__STDC__ to the defs. This is because all the SCO header files are broken. Anything that isn't mentioned in the C standard is conditionally excluded when __STDC__ is defined. 2) Due to the U.S. export restrictions, SCO broke the crypt stuff out into a separate library, libcrypt_i.a so the LIBS needed be set to: LIBS=' -lsocket -lcrypt_i' UnixWare: There are known bugs in the math library of the system, as well as problems in the handling of threads (calling fork in one thread may interrupt system calls in others). Therefore, test_math and tests involving threads will fail until those problems are fixed. QNX: Chris Herborth ([email protected]) writes: configure works best if you use GNU bash; a port is available on ftp.qnx.com in /usr/free. I used the following process to build, test and install Python 1.5.x under QNX: 1) CONFIG_SHELL=/usr/local/bin/bash CC=cc RANLIB=: \ ./configure --verbose --without-gcc --with-libm="" 2) edit Modules/Setup to activate everything that makes sense for your system... tested here at QNX with the following modules: array, audioop, binascii, cPickle, cStringIO, cmath, crypt, curses, errno, fcntl, gdbm, grp, imageop, _locale, math, md5, new, operator, parser, pcre, posix, pwd, readline, regex, reop, select, signal, socket, soundex, strop, struct, syslog, termios, time, timing, zlib, audioop, imageop 3) make SHELL=/usr/local/bin/bash or, if you feel the need for speed: make SHELL=/usr/local/bin/bash OPT="-5 -Oil+nrt" 4) make SHELL=/usr/local/bin/bash test Using GNU readline 2.2 seems to behave strangely, but I think that's a problem with my readline 2.2 port. :-\ 5) make SHELL=/usr/local/bin/bash install If you get SIGSEGVs while running Python (I haven't yet, but I've only run small programs and the test cases), you're probably running out of stack; the default 32k could be a little tight. To increase the stack size, edit the Makefile to read: LDFLAGS = -N 48k BeOS: See Misc/BeOS-NOTES for notes about compiling/installing Python on BeOS R3 or later. Note that only the PowerPC platform is supported for R3; both PowerPC and x86 are supported for R4. Cray T3E: Mark Hadfield ([email protected]) writes: Python can be built satisfactorily on a Cray T3E but based on my experience with the NIWA T3E (2002-05-22, version 2.2.1) there are a few bugs and gotchas. For more information see a thread on comp.lang.python in May 2002 entitled "Building Python on Cray T3E". 1) Use Cray's cc and not gcc. The latter was reported not to work by Konrad Hinsen. It may work now, but it may not. 2) To set sys.platform to something sensible, pass the following environment variable to the configure script: MACHDEP=unicosmk 2) Run configure with option "--enable-unicode=ucs4". 3) The Cray T3E does not support dynamic linking, so extension modules have to be built by adding (or uncommenting) lines in Modules/Setup. The minimum set of modules is posix, new, _sre, unicodedata On NIWA's vanilla T3E system the following have also been included successfully: _codecs, _locale, _socket, _symtable, _testcapi, _weakref array, binascii, cmath, cPickle, crypt, cStringIO, dbm errno, fcntl, grp, math, md5, operator, parser, pcre, pwd regex, rotor, select, struct, strop, syslog, termios time, timing, xreadlines 4) Once the python executable and library have been built, make will execute setup.py, which will attempt to build remaining extensions and link them dynamically. Each of these attempts will fail but should not halt the make process. This is normal. 5) Running "make test" uses a lot of resources and causes problems on our system. You might want to try running tests singly or in small groups. SGI: SGI's standard "make" utility (/bin/make or /usr/bin/make) does not check whether a command actually changed the file it is supposed to build. This means that whenever you say "make" it will redo the link step. The remedy is to use SGI's much smarter "smake" utility (/usr/sbin/smake), or GNU make. If you set the first line of the Makefile to #!/usr/sbin/smake smake will be invoked by make (likewise for GNU make). WARNING: There are bugs in the optimizer of some versions of SGI's compilers that can cause bus errors or other strange behavior, especially on numerical operations. To avoid this, try building with "make OPT=". OS/2: If you are running Warp3 or Warp4 and have IBM's VisualAge C/C++ compiler installed, just change into the pc\os2vacpp directory and type NMAKE. Threading and sockets are supported by default in the resulting binaries of PYTHON15.DLL and PYTHON.EXE. Monterey (64-bit AIX): The current Monterey C compiler (Visual Age) uses the OBJECT_MODE={32|64} environment variable to set the compilation mode to either 32-bit or 64-bit (32-bit mode is the default). Presumably you want 64-bit compilation mode for this 64-bit OS. As a result you must first set OBJECT_MODE=64 in your environment before configuring (./configure) or building (make) Python on Monterey. Reliant UNIX: The thread support does not compile on Reliant UNIX, and there is a (minor) problem in the configure script for that platform as well. This should be resolved in time for a future release. MacOSX: The tests will crash on both 10.1 and 10.2 with SEGV in test_re and test_sre due to the small default stack size. If you set the stack size to 2048 before doing a "make test" the failure can be avoided. If you're using the tcsh or csh shells, use "limit stacksize 2048" and for the bash shell (the default as of OSX 10.3), use "ulimit -s 2048". On naked Darwin you may want to add the configure option "--disable-toolbox-glue" to disable the glue code for the Carbon interface modules. The modules themselves are currently only built if you add the --enable-framework option, see below. On a clean OSX /usr/local does not exist. Do a "sudo mkdir -m 775 /usr/local" before you do a make install. It is probably not a good idea to do "sudo make install" which installs everything as superuser, as this may later cause problems when installing distutils-based additions. Some people have reported problems building Python after using "fink" to install additional unix software. Disabling fink (remove all references to /sw from your .profile or .login) should solve this. You may want to try the configure option "--enable-framework" which installs Python as a framework. The location can be set as argument to the --enable-framework option (default /Library/Frameworks). A framework install is probably needed if you want to use any Aqua-based GUI toolkit (whether Tkinter, wxPython, Carbon, Cocoa or anything else). You may also want to try the configure option "--enable-universalsdk" which builds Python as a universal binary with support for the i386 and PPC architetures. This requires Xcode 2.1 or later to build. See Mac/README for more information on framework and universal builds. Cygwin: With recent (relative to the time of writing, 2001-12-19) Cygwin installations, there are problems with the interaction of dynamic linking and fork(). This manifests itself in build failures during the execution of setup.py. There are two workarounds that both enable Python (albeit without threading support) to build and pass all tests on NT/2000 (and most likely XP as well, though reports of testing on XP would be appreciated). The workarounds: (a) the band-aid fix is to link the _socket module statically rather than dynamically (which is the default). To do this, run "./configure --with-threads=no" including any other options you need (--prefix, etc.). Then in Modules/Setup uncomment the lines: #SSL=/usr/local/ssl #_socket socketmodule.c \ # -DUSE_SSL -I$(SSL)/include -I$(SSL)/include/openssl \ # -L$(SSL)/lib -lssl -lcrypto and remove "local/" from the SSL variable. Finally, just run "make"! (b) The "proper" fix is to rebase the Cygwin DLLs to prevent base address conflicts. Details on how to do this can be found in the following mail: http://sources.redhat.com/ml/cygwin/2001-12/msg00894.html It is hoped that a version of this solution will be incorporated into the Cygwin distribution fairly soon. Two additional problems: (1) Threading support should still be disabled due to a known bug in Cygwin pthreads that causes test_threadedtempfile to hang. (2) The _curses module does not build. This is a known Cygwin ncurses problem that should be resolved the next time that this package is released. On older versions of Cygwin, test_poll may hang and test_strftime may fail. The situation on 9X/Me is not accurately known at present. Some time ago, there were reports that the following regression tests failed: test_pwd test_select (hang) test_socket Due to the test_select hang on 9X/Me, one should run the regression test using the following: make TESTOPTS='-l -x test_select' test News regarding these platforms with more recent Cygwin versions would be appreciated! Windows: When executing Python scripts on the command line using file type associations (i.e. starting "script.py" instead of "python script.py"), redirects may not work unless you set a specific registry key. See the Knowledge Base article <http://support.microsoft.com/kb/321788>. Configuring the bsddb and dbm modules ------------------------------------- Beginning with Python version 2.3, the PyBsddb package <http://pybsddb.sf.net/> was adopted into Python as the bsddb package, exposing a set of package-level functions which provide backwards-compatible behavior. Only versions 3.3 through 4.4 of Sleepycat's libraries provide the necessary API, so older versions aren't supported through this interface. The old bsddb module has been retained as bsddb185, though it is not built by default. Users wishing to use it will have to tweak Modules/Setup to build it. The dbm module will still be built against the Sleepycat libraries if other preferred alternatives (ndbm, gdbm) are not found. Building the sqlite3 module --------------------------- To build the sqlite3 module, you'll need the sqlite3 or libsqlite3 packages installed, including the header files. Many modern operating systems distribute the headers in a separate package to the library - often it will be the same name as the main package, but with a -dev or -devel suffix. The version of pysqlite2 that's including in Python needs sqlite3 3.0.8 or later. setup.py attempts to check that it can find a correct version. Configuring threads ------------------- As of Python 2.0, threads are enabled by default. If you wish to compile without threads, or if your thread support is broken, pass the --with-threads=no switch to configure. Unfortunately, on some platforms, additional compiler and/or linker options are required for threads to work properly. Below is a table of those options, collected by Bill Janssen. We would love to automate this process more, but the information below is not enough to write a patch for the configure.in file, so manual intervention is required. If you patch the configure.in file and are confident that the patch works, please send in the patch. (Don't bother patching the configure script itself -- it is regenerated each time the configure.in file changes.) Compiler switches for threads ............................. The definition of _REENTRANT should be configured automatically, if that does not work on your system, or if _REENTRANT is defined incorrectly, please report that as a bug. OS/Compiler/threads Switches for use with threads (POSIX is draft 10, DCE is draft 4) compile & link SunOS 5.{1-5}/{gcc,SunPro cc}/solaris -mt SunOS 5.5/{gcc,SunPro cc}/POSIX (nothing) DEC OSF/1 3.x/cc/DCE -threads ([email protected]) Digital UNIX 4.x/cc/DCE -threads ([email protected]) Digital UNIX 4.x/cc/POSIX -pthread ([email protected]) AIX 4.1.4/cc_r/d7 (nothing) ([email protected]) AIX 4.1.4/cc_r4/DCE (nothing) ([email protected]) IRIX 6.2/cc/POSIX (nothing) ([email protected]) Linker (ld) libraries and flags for threads ........................................... OS/threads Libraries/switches for use with threads SunOS 5.{1-5}/solaris -lthread SunOS 5.5/POSIX -lpthread DEC OSF/1 3.x/DCE -lpthreads -lmach -lc_r -lc ([email protected]) Digital UNIX 4.x/DCE -lpthreads -lpthread -lmach -lexc -lc ([email protected]) Digital UNIX 4.x/POSIX -lpthread -lmach -lexc -lc ([email protected]) AIX 4.1.4/{draft7,DCE} (nothing) ([email protected]) IRIX 6.2/POSIX -lpthread ([email protected]) Building a shared libpython --------------------------- Starting with Python 2.3, the majority of the interpreter can be built into a shared library, which can then be used by the interpreter executable, and by applications embedding Python. To enable this feature, configure with --enable-shared. If you enable this feature, the same object files will be used to create a static library. In particular, the static library will contain object files using position-independent code (PIC) on platforms where PIC flags are needed for the shared library. Configuring additional built-in modules --------------------------------------- Starting with Python 2.1, the setup.py script at the top of the source distribution attempts to detect which modules can be built and automatically compiles them. Autodetection doesn't always work, so you can still customize the configuration by editing the Modules/Setup file; but this should be considered a last resort. The rest of this section only applies if you decide to edit the Modules/Setup file. You also need this to enable static linking of certain modules (which is needed to enable profiling on some systems). This file is initially copied from Setup.dist by the configure script; if it does not exist yet, create it by copying Modules/Setup.dist yourself (configure will never overwrite it). Never edit Setup.dist -- always edit Setup or Setup.local (see below). Read the comments in the file for information on what kind of edits are allowed. When you have edited Setup in the Modules directory, the interpreter will automatically be rebuilt the next time you run make (in the toplevel directory). Many useful modules can be built on any Unix system, but some optional modules can't be reliably autodetected. Often the quickest way to determine whether a particular module works or not is to see if it will build: enable it in Setup, then if you get compilation or link errors, disable it -- you're either missing support or need to adjust the compilation and linking parameters for that module. On SGI IRIX, there are modules that interface to many SGI specific system libraries, e.g. the GL library and the audio hardware. These modules will not be built by the setup.py script. In addition to the file Setup, you can also edit the file Setup.local. (the makesetup script processes both). You may find it more convenient to edit Setup.local and leave Setup alone. Then, when installing a new Python version, you can copy your old Setup.local file. Setting the optimization/debugging options ------------------------------------------ If you want or need to change the optimization/debugging options for the C compiler, assign to the OPT variable on the toplevel make command; e.g. "make OPT=-g" will build a debugging version of Python on most platforms. The default is OPT=-O; a value for OPT in the environment when the configure script is run overrides this default (likewise for CC; and the initial value for LIBS is used as the base set of libraries to link with). When compiling with GCC, the default value of OPT will also include the -Wall and -Wstrict-prototypes options. Additional debugging code to help debug memory management problems can be enabled by using the --with-pydebug option to the configure script. For flags that change binary compatibility, use the EXTRA_CFLAGS variable. Profiling --------- If you want C profiling turned on, the easiest way is to run configure with the CC environment variable to the necessary compiler invocation. For example, on Linux, this works for profiling using gprof(1): CC="gcc -pg" ./configure Note that on Linux, gprof apparently does not work for shared libraries. The Makefile/Setup mechanism can be used to compile and link most extension modules statically. Coverage checking ----------------- For C coverage checking using gcov, run "make coverage". This will build a Python binary with profiling activated, and a ".gcno" and ".gcda" file for every source file compiled with that option. With the built binary, now run the code whose coverage you want to check. Then, you can see coverage statistics for each individual source file by running gcov, e.g. gcov -o Modules zlibmodule This will create a "zlibmodule.c.gcov" file in the current directory containing coverage info for that source file. This works only for source files statically compiled into the executable; use the Makefile/Setup mechanism to compile and link extension modules you want to coverage-check statically. Testing ------- To test the interpreter, type "make test" in the top-level directory. This runs the test set twice (once with no compiled files, once with the compiled files left by the previous test run). The test set produces some output. You can generally ignore the messages about skipped tests due to optional features which can't be imported. If a message is printed about a failed test or a traceback or core dump is produced, something is wrong. On some Linux systems (those that are not yet using glibc 6), test_strftime fails due to a non-standard implementation of strftime() in the C library. Please ignore this, or upgrade to glibc version 6. IMPORTANT: If the tests fail and you decide to mail a bug report, *don't* include the output of "make test". It is useless. Run the failing test manually, as follows: ./python ./Lib/test/test_whatever.py (substituting the top of the source tree for '.' if you built in a different directory). This runs the test in verbose mode. Installing ---------- To install the Python binary, library modules, shared library modules (see below), include files, configuration files, and the manual page, just type make install This will install all platform-independent files in subdirectories of the directory given with the --prefix option to configure or to the `prefix' Make variable (default /usr/local). All binary and other platform-specific files will be installed in subdirectories if the directory given by --exec-prefix or the `exec_prefix' Make variable (defaults to the --prefix directory) is given. If DESTDIR is set, it will be taken as the root directory of the installation, and files will be installed into $(DESTDIR)$(prefix), $(DESTDIR)$(exec_prefix), etc. All subdirectories created will have Python's version number in their name, e.g. the library modules are installed in "/usr/local/lib/python<version>/" by default, where <version> is the <major>.<minor> release number (e.g. "2.1"). The Python binary is installed as "python<version>" and a hard link named "python" is created. The only file not installed with a version number in its name is the manual page, installed as "/usr/local/man/man1/python.1" by default. If you want to install multiple versions of Python see the section below entitled "Installing multiple versions". The only thing you may have to install manually is the Python mode for Emacs found in Misc/python-mode.el. (But then again, more recent versions of Emacs may already have it.) Follow the instructions that came with Emacs for installation of site-specific files. On Mac OS X, if you have configured Python with --enable-framework, you should use "make frameworkinstall" to do the installation. Note that this installs the Python executable in a place that is not normally on your PATH, you may want to set up a symlink in /usr/local/bin. Installing multiple versions ---------------------------- On Unix and Mac systems if you intend to install multiple versions of Python using the same installation prefix (--prefix argument to the configure script) you must take care that your primary python executable is not overwritten by the installation of a different versio. All files and directories installed using "make altinstall" contain the major and minor version and can thus live side-by-side. "make install" also creates ${prefix}/bin/python which refers to ${prefix}/bin/pythonX.Y. If you intend to install multiple versions using the same prefix you must decide which version (if any) is your "primary" version. Install that version using "make install". Install all other versions using "make altinstall". For example, if you want to install Python 2.5, 2.6 and 3.0 with 2.6 being the primary version, you would execute "make install" in your 2.6 build directory and "make altinstall" in the others. Configuration options and variables ----------------------------------- Some special cases are handled by passing options to the configure script. WARNING: if you rerun the configure script with different options, you must run "make clean" before rebuilding. Exceptions to this rule: after changing --prefix or --exec-prefix, all you need to do is remove Modules/getpath.o. --with(out)-gcc: The configure script uses gcc (the GNU C compiler) if it finds it. If you don't want this, or if this compiler is installed but broken on your platform, pass the option --without-gcc. You can also pass "CC=cc" (or whatever the name of the proper C compiler is) in the environment, but the advantage of using --without-gcc is that this option is remembered by the config.status script for its --recheck option. --prefix, --exec-prefix: If you want to install the binaries and the Python library somewhere else than in /usr/local/{bin,lib}, you can pass the option --prefix=DIRECTORY; the interpreter binary will be installed as DIRECTORY/bin/python and the library files as DIRECTORY/lib/python/*. If you pass --exec-prefix=DIRECTORY (as well) this overrides the installation prefix for architecture-dependent files (like the interpreter binary). Note that --prefix=DIRECTORY also affects the default module search path (sys.path), when Modules/config.c is compiled. Passing make the option prefix=DIRECTORY (and/or exec_prefix=DIRECTORY) overrides the prefix set at configuration time; this may be more convenient than re-running the configure script if you change your mind about the install prefix. --with-readline: This option is no longer supported. GNU readline is automatically enabled by setup.py when present. --with-threads: On most Unix systems, you can now use multiple threads, and support for this is enabled by default. To disable this, pass --with-threads=no. If the library required for threads lives in a peculiar place, you can use --with-thread=DIRECTORY. IMPORTANT: run "make clean" after changing (either enabling or disabling) this option, or you will get link errors! Note: for DEC Unix use --with-dec-threads instead. --with-sgi-dl: On SGI IRIX 4, dynamic loading of extension modules is supported by the "dl" library by Jack Jansen, which is ftp'able from ftp://ftp.cwi.nl/pub/dynload/dl-1.6.tar.Z. This is enabled (after you've ftp'ed and compiled the dl library) by passing --with-sgi-dl=DIRECTORY where DIRECTORY is the absolute pathname of the dl library. (Don't bother on IRIX 5, it already has dynamic linking using SunOS style shared libraries.) THIS OPTION IS UNSUPPORTED. --with-dl-dld: Dynamic loading of modules is rumored to be supported on some other systems: VAX (Ultrix), Sun3 (SunOS 3.4), Sequent Symmetry (Dynix), and Atari ST. This is done using a combination of the GNU dynamic loading package (ftp://ftp.cwi.nl/pub/dynload/dl-dld-1.1.tar.Z) and an emulation of the SGI dl library mentioned above (the emulation can be found at ftp://ftp.cwi.nl/pub/dynload/dld-3.2.3.tar.Z). To enable this, ftp and compile both libraries, then call configure, passing it the option --with-dl-dld=DL_DIRECTORY,DLD_DIRECTORY where DL_DIRECTORY is the absolute pathname of the dl emulation library and DLD_DIRECTORY is the absolute pathname of the GNU dld library. (Don't bother on SunOS 4 or 5, they already have dynamic linking using shared libraries.) THIS OPTION IS UNSUPPORTED. --with-libm, --with-libc: It is possible to specify alternative versions for the Math library (default -lm) and the C library (default the empty string) using the options --with-libm=STRING and --with-libc=STRING, respectively. For example, if your system requires that you pass -lc_s to the C compiler to use the shared C library, you can pass --with-libc=-lc_s. These libraries are passed after all other libraries, the C library last. --with-libs='libs': Add 'libs' to the LIBS that the python interpreter is linked against. --with-cxx-main=<compiler>: If you plan to use C++ extension modules, then -- on some platforms -- you need to compile python's main() function with the C++ compiler. With this option, make will use <compiler> to compile main() *and* to link the python executable. It is likely that the resulting executable depends on the C++ runtime library of <compiler>. (The default is --without-cxx-main.) There are platforms that do not require you to build Python with a C++ compiler in order to use C++ extension modules. E.g., x86 Linux with ELF shared binaries and GCC 3.x, 4.x is such a platform. We recommend that you configure Python --without-cxx-main on those platforms because a mismatch between the C++ compiler version used to build Python and to build a C++ extension module is likely to cause a crash at runtime. The Python installation also stores the variable CXX that determines, e.g., the C++ compiler distutils calls by default to build C++ extensions. If you set CXX on the configure command line to any string of non-zero length, then configure won't change CXX. If you do not preset CXX but pass --with-cxx-main=<compiler>, then configure sets CXX=<compiler>. In all other cases, configure looks for a C++ compiler by some common names (c++, g++, gcc, CC, cxx, cc++, cl) and sets CXX to the first compiler it finds. If it does not find any C++ compiler, then it sets CXX="". Similarly, if you want to change the command used to link the python executable, then set LINKCC on the configure command line. --with-pydebug: Enable additional debugging code to help track down memory management problems. This allows printing a list of all live objects when the interpreter terminates. --with(out)-universal-newlines: enable reading of text files with foreign newline convention (default: enabled). In other words, any of \r, \n or \r\n is acceptable as end-of-line character. If enabled import and execfile will automatically accept any newline in files. Python code can open a file with open(file, 'U') to read it in universal newline mode. THIS OPTION IS UNSUPPORTED. --with-tsc: Profile using the Pentium timestamping counter (TSC). --with-system-ffi: Build the _ctypes extension module using an ffi library installed on the system. Building for multiple architectures (using the VPATH feature) ------------------------------------------------------------- If your file system is shared between multiple architectures, it usually is not necessary to make copies of the sources for each architecture you want to support. If the make program supports the VPATH feature, you can create an empty build directory for each architecture, and in each directory run the configure script (on the appropriate machine with the appropriate options). This creates the necessary subdirectories and the Makefiles therein. The Makefiles contain a line VPATH=... which points to a directory containing the actual sources. (On SGI systems, use "smake -J1" instead of "make" if you use VPATH -- don't try gnumake.) For example, the following is all you need to build a minimal Python in /usr/tmp/python (assuming ~guido/src/python is the toplevel directory and you want to build in /usr/tmp/python): $ mkdir /usr/tmp/python $ cd /usr/tmp/python $ ~guido/src/python/configure [...] $ make [...] $ Note that configure copies the original Setup file to the build directory if it finds no Setup file there. This means that you can edit the Setup file for each architecture independently. For this reason, subsequent changes to the original Setup file are not tracked automatically, as they might overwrite local changes. To force a copy of a changed original Setup file, delete the target Setup file. (The makesetup script supports multiple input files, so if you want to be fancy you can change the rules to create an empty Setup.local if it doesn't exist and run it with arguments $(srcdir)/Setup Setup.local; however this assumes that you only need to add modules.) Also note that you can't use a workspace for VPATH and non VPATH builds. The object files left behind by one version confuses the other. Building on non-UNIX systems ---------------------------- For Windows (2000/NT/ME/98/95), assuming you have MS VC++ 7.1, the project files are in PCbuild, the workspace is pcbuild.dsw. See PCbuild\readme.txt for detailed instructions. For other non-Unix Windows compilers, in particular MS VC++ 6.0 and for OS/2, enter the directory "PC" and read the file "readme.txt". For the Mac, a separate source distribution will be made available, for use with the CodeWarrior compiler. If you are interested in Mac development, join the PythonMac Special Interest Group (http://www.python.org/sigs/pythonmac-sig/, or send email to [email protected]). Of course, there are also binary distributions available for these platforms -- see http://www.python.org/. To port Python to a new non-UNIX system, you will have to fake the effect of running the configure script manually (for Mac and PC, this has already been done for you). A good start is to copy the file pyconfig.h.in to pyconfig.h and edit the latter to reflect the actual configuration of your system. Most symbols must simply be defined as 1 only if the corresponding feature is present and can be left alone otherwise; however the *_t type symbols must be defined as some variant of int if they need to be defined at all. For all platforms, it's important that the build arrange to define the preprocessor symbol NDEBUG on the compiler command line in a release build of Python (else assert() calls remain in the code, hurting release-build performance). The Unix, Windows and Mac builds already do this. Miscellaneous issues ==================== Emacs mode ---------- There's an excellent Emacs editing mode for Python code; see the file Misc/python-mode.el. Originally written by the famous Tim Peters, it is now maintained by the equally famous Barry Warsaw (it's no coincidence that they now both work on the same team). The latest version, along with various other contributed Python-related Emacs goodies, is online at http://www.python.org/emacs/python-mode. And if you are planning to edit the Python C code, please pick up the latest version of CC Mode http://www.python.org/emacs/cc-mode; it contains a "python" style used throughout most of the Python C source files. (Newer versions of Emacs or XEmacs may already come with the latest version of python-mode.) Tkinter ------- The setup.py script automatically configures this when it detects a usable Tcl/Tk installation. This requires Tcl/Tk version 8.0 or higher. For more Tkinter information, see the Tkinter Resource page: http://www.python.org/topics/tkinter/ There are demos in the Demo/tkinter directory. Note that there's a Python module called "Tkinter" (capital T) which lives in Lib/lib-tk/Tkinter.py, and a C module called "_tkinter" (lower case t and leading underscore) which lives in Modules/_tkinter.c. Demos and normal Tk applications import only the Python Tkinter module -- only the latter imports the C _tkinter module. In order to find the C _tkinter module, it must be compiled and linked into the Python interpreter -- the setup.py script does this. In order to find the Python Tkinter module, sys.path must be set correctly -- normal installation takes care of this. Distribution structure ---------------------- Most subdirectories have their own README files. Most files have comments. Demo/ Demonstration scripts, modules and programs Doc/ Documentation sources (reStructuredText) Grammar/ Input for the parser generator Include/ Public header files LICENSE Licensing information Lib/ Python library modules Mac/ Macintosh specific resources Makefile.pre.in Source from which config.status creates the Makefile.pre Misc/ Miscellaneous useful files Modules/ Implementation of most built-in modules Objects/ Implementation of most built-in object types PC/ Files specific to PC ports (DOS, Windows, OS/2) PCbuild/ Build directory for Microsoft Visual C++ Parser/ The parser and tokenizer and their input handling Python/ The byte-compiler and interpreter README The file you're reading now RISCOS/ Files specific to RISC OS port Tools/ Some useful programs written in Python pyconfig.h.in Source from which pyconfig.h is created (GNU autoheader output) configure Configuration shell script (GNU autoconf output) configure.in Configuration specification (input for GNU autoconf) install-sh Shell script used to install files setup.py Python script used to build extension modules The following files will (may) be created in the toplevel directory by the configuration and build processes: Makefile Build rules Makefile.pre Build rules before running Modules/makesetup buildno Keeps track of the build number config.cache Cache of configuration variables pyconfig.h Configuration header config.log Log from last configure run config.status Status from last run of the configure script getbuildinfo.o Object file from Modules/getbuildinfo.c libpython<version>.a The library archive python The executable interpreter reflog.txt Output from running the regression suite with the -R flag tags, TAGS Tags files for vi and Emacs That's all, folks! ------------------ --Guido van Rossum (home page: http://www.python.org/~guido/)
Python currently doesn't have a good-quality regex benchmark suite that can
be run automatically, have statistics drawn from it, etc. We need such a
thing before starting work on regex performance.
Possible resources:
- Fredrik Lundh's original benchmarks for SRE:
http://mail.python.org/pipermail/python-dev/2000-August/007797.html
- V8's JS regex benchmarks:
http://v8.googlecode.com/svn/data/benchmarks/v3/regexp.js
Ideally we would do a search of the Python regexes in Google Code Search or
similar corpus and distill some representative set from them. V8's may be
good enough, though.
Original issue reported on code.google.com by collinw
on 14 Apr 2009 at 11:37
Purpose of code changes on this branch:
To use vmgen to generate the main interpretation loop in PyEval_EvalFrameEx().
When reviewing my code changes, please focus on:
Readability, understandability, and anything stupid I did.
After the review, I'll merge this branch into:
/trunk
Original issue reported on code.google.com by [email protected]
on 31 Jan 2009 at 1:00
We should add a sys.optimize decorator so that known-important functions
don't have to hit the hotness threshold before we optimize them. This would
be particularly useful for the slowspitfire benchmark: the function that
does all the work will never be "hot" by our current heuristics.
Currently, if we force compilation via -L2, slowspitfire shows a ~10% gain
over 2009Q1, but -L2 hurts start-up time. A decorator like this is similar
to the way Spitfire uses Psyco (ie, explicitly flagging functions for
optimization).
Original issue reported on code.google.com by collinw
on 5 Jun 2009 at 11:47
[deleted issue]
Execution should not block on compiling a function with LLVM or
reoptimizing it with new data. We should send these work units to separate
worker threads, allowing the main threads to carry on unimpeded.
The first implementation could/should probably just be a simple FIFO queue
of work items with a single worker thread. We can add heuristics and
additional worker threads as the data warrants.
Original issue reported on code.google.com by collinw
on 28 May 2009 at 9:57
In ceval.c, compilation fails with IBM's xlc at the following point:
Python/ceval.c-854- static Opcode labels[] = {
Python/ceval.c:855:#include "ceval-labels.i"
Python/ceval.c-856- };
with a message for each entry in ceval-labels.i that is of the form:
"Include/ceval-labels.i", line 1.1: 1506-221 (S) Initializer must be a
valid constant expression.
I think it's actually just upset about taking the address of a label,
something gcc complains about if you try to compile with -pedantic. It's
not a showstopper as it is still possible to compile ceval with gcc and
finish the build with xlc.
Original issue reported on code.google.com by [email protected]
on 27 Apr 2009 at 3:45
This idea is more speculative: every "def foo(): pass" function has the
same bytecode. It might reduce memory usage if we should share the
implementation of these functions between function definitions. This could
be done by hashing the function's parameters and bytecode.
Issues:
- Hashing the AST might be better than hashing the bytecode.
- If two functions have the same bytecode but different parameter names,
they probably can't take advantage of this. In the example above, "def
foo(): pass" and "def bar(a): pass" can't share an implementation
I'd be interested to know, for an application like 2to3 or Django's test
suite, how many functions are really the same (modulo their name).
Original issue reported on code.google.com by collinw
on 6 Jun 2009 at 12:07
CPython and Unladen Swallow both currently emit bytecode/machine code for
code following a return, raise, break or continue statement, like so:
def foo():
return 5
for i in range(4):
print i
Eliminating this code isn't critical (it isn't very common, and LLVM is
smart enough to eliminate it for us), but the need to support this has
spread into the LLVM compiler, which has to allow for this with special
deadcode blocks. This is ugly and should be fixed.
This could be done in a couple of ways: the dead code could be pruned from
the AST (or not even make it into the AST), or could be ignored by the
bytecode compiler.
Original issue reported on code.google.com by collinw
on 13 May 2009 at 10:40
Currently, LLVM-generated functions don't ever check for pending signals to
handle the way the interpreter does. Among other problems, this makes it
impossible to KeyboardInterrupt Unladen Swallow when running with -L[012].
evlogimenos has agreed to work on this (and thread switching, since they're
related).
Original issue reported on code.google.com by collinw
on 28 May 2009 at 12:21
The following program:
import sys
copies = int(sys.argv[1])
print "Running %d copies..." % copies
longexpr = 'x = x or ' + '-x' * 2500
code = ('''
def f(x):
''' + ''' %s
''' * copies + '''
# the expressions above have no effect, x == argument
while x:
x -= 1
return x
''') % ((longexpr,) * copies)
exec code
print f(5)
demonstrates non-linear memory use and running times with Unladen Swallow
r573. This was extracted from test_compile, which takes more than 4GB of
memory which exhausts a 32-bit address space.
The memory use below is from watching the "real memory" column in the apple
activity monitor and taking the highest number I saw.
$ time ./python.exe -S -L0 ./use_lots_of_memory.py 1
Running 1 copies...
0
real 0m37.979s
user 0m36.972s
sys 0m0.602s
memory 174MB
$ time ./python.exe -S -L0 ./use_lots_of_memory.py 2
Running 2 copies...
0
real 1m15.750s
user 1m13.479s
sys 0m1.368s
memory 491MB (delta 317)
$ time ./python.exe -S -L0 ./use_lots_of_memory.py 3
Running 3 copies...
0
real 2m6.118s
user 2m2.631s
sys 0m2.328s
memory 944MB (delta 453)
$ time ./python.exe -S -L0 ./use_lots_of_memory.py 4
Running 4 copies...
0
real 5m59.303s
user 3m10.135s
sys 0m14.223s
memory 1500MB (delta 556)
Watching memory use of the "3" case, it seems to rise to ~30-40MB up to and
during SimplifyCFG, then rise by ~3 MB per second through CodeGenAndEmitDAG
up to about 150MB, and then rise by ~50MB per second through
LiveVariables::runOnMachineFunction up to about 750MB. LiveIntervals seems
to account for most of the rest of the memory use.
Original issue reported on code.google.com by [email protected]
on 22 May 2009 at 2:59
test_bsddb3 fails sometimes, passes other times, resulting in spurious
failure reports. There may be patches upstream to make it more stable.
Original issue reported on code.google.com by collinw
on 15 May 2009 at 9:50
When running regrtest -R:: (which requires a pydebug build) with -L0 or
higher, some of the tests end up leaking references:
test_codecs leaked [27, 27, 27, 27] references, sum=108
test_copy leaked [2, 2, 2, 2] references, sum=8
test_datetime leaked [2, 2, 2, 2] references, sum=8
test_decimal leaked [2, 2, 2, 2] references, sum=8
test_difflib leaked [56, 56, 56, 56] references, sum=224
test_generators leaked [296, 296, 296, 296] references, sum=1184
test_grammar leaked [3, 3, 3, 3] references, sum=12
test_io leaked [2, 2, 2, 2] references, sum=8
test_itertools leaked [26, 26, 26, 26] references, sum=104
test_lib2to3 leaked [22, 22, 22, 22] references, sum=88
These tests do not leak without -L. Considering test_llvm does not leak, I
suspect a leak in an error path that test_llvm fails to test.
Original issue reported on code.google.com by [email protected]
on 20 May 2009 at 1:12
We currently use llvm::ExecutionEngine::getPointerToFunction(func) to
translate LLVM IR into machine code. getPointerToFunction caches the
returned machine code for each input function, and will not regenerate it
even if the function has changed. This means we can't re-optimize a
function after calling it through LLVM.
ExecutionEngine::recompileAndRelinkFunction(func) is available to force
LLVM to regenerate the machine code, but it overwrites the original machine
code with a jump to the new block, which means we can't use it while any
stack frame in any thread is still executing the code.
More information on fixing this is at
http://wiki.llvm.org/Provide_more_control_over_and_access_to_JIT%27s_output
Original issue reported on code.google.com by [email protected]
on 28 May 2009 at 10:47
We need to have more accurate measurements for the total time taken from
when we start a CALL_FUNCTION opcode to the time when the body of the
function starts executing. This should work regardless of whether we're
dispatching to an interpreted function or an LLVM function; C functions can
be fudged a bit (right before calling the function pointer?)
- Data should be stored in a vector and stats printed out at Python-shutdown.
- This should include whether the execution is in machine code or the
interpreter.
- Should be a special build (controlled by #ifdef's).
- Use TSCs?
Original issue reported on code.google.com by collinw
on 30 May 2009 at 1:46
Currently, perf.py's --track_memory option works by reading Linux 2.6 smaps
files from /proc/. This obviously doesn't work on non-Linux platforms, or
even Linux platforms pre-2.6.16.
Darwin is the most important of the currently-unsupported platforms.
/usr/bin/time -l (lowercase ell) on Darwin will provide the maximum rss for
the process.
Original issue reported on code.google.com by collinw
on 27 May 2009 at 2:52
Steps to reproduce:
1. Modify a C function in Python that you know will be called from LLVM so
that it will segfault. (printf("%d\n", *((int*)NULL));)
2. Build Python --with-pydebug.
3. Load up the binary in GDB.
4. Run a python script that will call the bad function with -L2.
5. Examine the backtrace.
The stack trace should show your C function and maybe some frames above it,
and then the heap address of some LLVM generated code with no name for it.
If you're lucky, eventually the trace will get past there and make it back
down to main. If you're unlucky, you get nothing.
As a work around, you can set the environment variable
PYTHONLLVMFLAGS="--disable-fp-elim" to your python binary, and that should
give you a full stack trace that is only missing information for LLVM
generated code. Eventually we'd like to enable this by default for debug
builds, but right now there's no easy way to toggle that option. The right
thing to do would probably be to submit a patch to LLVM so we can toggle
that option.
Original issue reported on code.google.com by [email protected]
on 8 Jun 2009 at 5:50
Currently, the compiler emits both CPython bytecode and LLVM IR and
attaches both to all code objects it creates. However, the marshal format
only understands bytecode and drops the LLVM IR on the floor when it saves
.pyc files. Disabling .pyc files entirely seems to slow down regrtest to
unacceptable levels so r323 just checks that the LLVM IR is present when it
tries to run through the LLVM JIT and raises a SystemError if it's not
present. We'll need to fix this before the JIT is really a viable way to
run things.
Original issue reported on code.google.com by [email protected]
on 24 Mar 2009 at 7:34
We have altered the definition of f_lasti: pray we don't alter it any
further; but regardless, we should make it work again for pdb.
Original issue reported on code.google.com by [email protected]
on 22 Apr 2009 at 3:26
perf.py should grow a --memory option that will profile memory usage of the
different benchmarks. Memory usage would be summarized in the same way that
running time is.
Original issue reported on code.google.com by collinw
on 18 May 2009 at 8:50
Running make twice in a row takes forever, even if you didn't change
anything: make wants to rebuild Python/llvm_inline_functions.bc, which
cascades into causing the python binary and libpython to both be relinked.
A simple no-op make takes almost 30 seconds, all to do nothing.
Original issue reported on code.google.com by collinw
on 28 May 2009 at 4:07
Purpose of code changes on this branch: To investigate removing all ways of
reporting errors except for 'why'.
2to3 seems to be slightly faster with this patch; pybench seems to be
slightly slower. We'll need to measure this on real hardware and compilers,
rather than my laptop.
There are a couple bugs in EvalFrameEx where it doesn't signal an error
when it should. Since those are functionality changes, I'll do them in a
separate change.
After the review, I'll merge this branch into:
/trunk
... Does code know how to diff branches? We'll see.
Original issue reported on code.google.com by [email protected]
on 8 Dec 2008 at 12:17
In Python 2.6, you can pickle cPickle.Pickler and Unpickler items. I seem
to have broken that in Unladen Swallow.
Original issue reported on code.google.com by collinw
on 22 Apr 2009 at 4:42
In our LLVM code, Py_INCREF/Py_DECREF are function calls instead of the
nice speedy macros that the interpreter can take advantage of. These calls
should be inlined in the LLVM IR.
jyasskin is already working on this.
Original issue reported on code.google.com by collinw
on 27 May 2009 at 10:15
In order to maintain support for smaller platforms like cell phones, we
should include a ./configure flag to disable LLVM entirely. Since we'll be
keeping around the eval loop, this should be pretty straightforward to
implement.
Original issue reported on code.google.com by collinw
on 13 Apr 2009 at 10:33
test_bsddb3 fails sometimes, passes other times, resulting in spurious
failure reports. There may be patches upstream to make it more stable.
Original issue reported on code.google.com by collinw
on 15 May 2009 at 9:49
Purpose of code changes on this branch: simplify eval loop by converting a
number of infrequent to builtin functions. These builtins are prefixed with
#@, e.g., #@import_star.
This shows a 1% improvement in 2to3 performance on all tested platforms.
This should also contribute positively to the vmgen branch.
After the review, I'll merge this branch into:
/trunk
Original issue reported on code.google.com by collinw
on 5 Jan 2009 at 7:34
We've currently dropped optimizations like BINARY_ADD's special case for
PyInts and PyStrings. We'll want to add them back before doing real
performance comparisons.
Original issue reported on code.google.com by [email protected]
on 21 Apr 2009 at 10:03
We currently know of a couple inefficiencies:
1. Because we always load the stack pointer, LLVM can't optimize away the
unwind loop, even in simple functions that, say, return once from the outer
scope. For non-generators, we can do better.
2. The block stack is stored in the frame, which again prevents LLVM from
optimizing away most of the unwind block.
This issue will hold a summary of this kind of issue, but we may split it
up when we get around to optimizing these things.
Original issue reported on code.google.com by [email protected]
on 21 Apr 2009 at 10:01
CPython's regex implementation is slower than it could be. There are a lot
of techniques we could use to make things faster:
- JIT compile regexes. V8 and SquirrelFish Extreme have had good luck with
this approach.
- Thompson NFAs are much faster for some regexes, but don't support the
full range of Python regex syntax. This would necessitate a multi-engine
system that picks the fastest engine for the given regex.
- Apply VM optimizations from the main Python eval loop to the regex eval
loop. This isn't likely to give us the speed we want, but may help in a
multi-engine approach if one of those engines is the existing regex VM.
Longer-form, with links:
http://code.google.com/p/unladen-swallow/wiki/ProjectPlan#Regular_Expressions
Original issue reported on code.google.com by collinw
on 14 Apr 2009 at 11:30
It looks like sqlite3 fails to import due to not being able to track down
the old so file. I have attached the strace -fi but I had to ****** some
stuff for security/privacy reasons
Original issue reported on code.google.com by [email protected]
on 11 May 2009 at 7:46
Attachments:
The core of BUILD_TUPLE/BUILD_LIST (and also other opcodes that build
tuples or lists out of stack items, like CALL_FUNCTION might do) is
essentially a memcpy() from the stack to the body of the (newly created)
tuple/list object. LLVM doesn't seem to optimize this by itself, so we
should turn this into an @llvm.memcpy call.
Original issue reported on code.google.com by [email protected]
on 21 Apr 2009 at 10:43
regrtest.py -R:: test_urllib2_localnet currently leaks three references per
run. This problem is present in Unladen Swallow trunk and
release-2009Q1-maint, as well as mainline CPython trunk.
I've found the line that triggers the leak and am tracking down the cause.
Original issue reported on code.google.com by collinw
on 15 May 2009 at 7:02
llc produces some unused variables in its output (at least for
initial_llvm_module.cc):
Python/initial_llvm_module.cc:662: warning: unused variable ‘PointerTy_80’
Python/initial_llvm_module.cc:664: warning: unused variable ‘PointerTy_81’
Python/initial_llvm_module.cc:675: warning: unused variable ‘PointerTy_82’
Python/initial_llvm_module.cc:693: warning: unused variable ‘PointerTy_86’
Python/initial_llvm_module.cc:704: warning: unused variable ‘PointerTy_88’
Python/initial_llvm_module.cc:716: warning: unused variable ‘PointerTy_90’
Python/initial_llvm_module.cc:724: warning: unused variable ‘PointerTy_92’
We silence the warnings for now, but we should see if we can fix llc instead.
Original issue reported on code.google.com by [email protected]
on 12 May 2009 at 9:06
Lib/test/test_compile.py takes 1.7s for python2.6, 33s for trunk unladen
swallow (which emits some LLVM bitcode), and 12.5min for the llvm-working
branch (which emits full LLVM bitcode). The problem is test_extended_arg
which emits expressions containing 2500 subtractions.
Original issue reported on code.google.com by [email protected]
on 30 Mar 2009 at 10:36
We should have a --jit option (approximate) that controls when and how
often the reoptimizer will run.
--jit=once will compile a function as heavily as possible the first time
it's found hot, then ignore it thereafter. This may mean that bad
predictions can't be corrected; we'll have to see.
--jit=never disables the LLVM integration, forcing all code to run through
the bytecode interpreter.
--jit=everything is equivalent to the current -L[012] options: all
functions are compiled to machine code when they are defined. As functions
are found to be hotter and hotter, they may be reoptimized.
The -O option should be extended to take numeric arguments the way gcc
does. -O[0123] will control the initial optimization used by --jit=once and
--jit=everything.
I'm open to debate on what other options --jit should support. Should
--jit=everything disable reoptimization?
Original issue reported on code.google.com by collinw
on 30 May 2009 at 1:35
Now that we have the --with-llvm option, we should have a script in
Util/llvm to configure, make and install LLVM with the proper options. We
pass a bunch of options from Python's ./configure to LLVM's ./configure,
and those should be automated into a simple script. Otherwise, you'll have
to dig through Python's ./configure to find the right options every time,
and that's a pain in the ass.
This will make --with-llvm a lot easier to use.
Original issue reported on code.google.com by collinw
on 29 May 2009 at 4:47
Purpose of code changes on this branch: speed up cPickle.
Pickle (complex):
Min: 1.023 -> 0.409: 150.36% faster
Avg: 1.053 -> 0.410: 157.17% faster
Significant (t=1102.029662, a=0.95)
Pickle (simple):
Min: 1.223 -> 0.868: 40.83% faster
Avg: 1.229 -> 0.876: 40.20% faster
Significant (t=695.483070, a=0.95)
Unpickle (complex):
Min: 0.738 -> 0.536: 37.71% faster
Avg: 0.746 -> 0.547: 36.24% faster
Significant (t=122.112665, a=0.95)
Unpickle (simple):
Min: 0.756 -> 0.486: 55.60% faster
Avg: 0.774 -> 0.493: 56.91% faster
Significant (t=331.578243, a=0.95)
When reviewing my code changes, please focus on: anything stupid I did,
style issues, things that might block merger back to mainline.
Let me know if you'd rather review this on Rietveld.
After the review, I'll merge this branch into:
/trunk
Original issue reported on code.google.com by collinw
on 28 Feb 2009 at 9:49
A central part of templating languages is the way they combine multiple strings
into one string
during template runtime.
The current most often used pattern seems to be a BufferIO class like the one
from spitfire
(http://code.google.com/p/spitfire/source/browse/trunk/spitfire/runtime/template
.py):
class BufferIO(list):
write = list.append
def getvalue(self):
return ''.join(self)
The constrains of templating languages have even more minimal requirements than
the built-in
list used above, namely they care only about appending of multiple Unicode
strings and a one-
time combination of those into a single Unicode string. As no slicing,
retrieval of individual
values or anything like it is required, I'm wondering if a more fine-tuned
version of this could
result in noticeable differences for the spitfire test cases. Using a
collections.deque instead of a
list here, doesn't produce any real difference.
Original issue reported on code.google.com by [email protected]
on 28 Mar 2009 at 2:29
Building with TSC (the WITH_TSC define, enabled by --with-tsc) which
instruments CPython with CPU counters, is currently broken (at least by the
vmgen patch, which does not include the right #ifdefs in call_function and
the CALL_FUNCTION_VAR_KW opcode implementation.)
Original issue reported on code.google.com by [email protected]
on 14 Apr 2009 at 1:34
As we do deeper and deeper surgery on the compiler, I'd like to have a
fuzzer or some other kind of randomized testing to look for corner cases in
our implementation.
Before writing our own, we should try to reuse Fusil
(http://fusil.hachoir.org/svn/trunk/), which has been shown to find bugs in
CPython already. Other Python implementations may already have something
like this; if so, we should reuse that.
Eventually, this would become part of the continuous build, forever
searching the space of random Python programs for crashes.
Original issue reported on code.google.com by collinw
on 14 Apr 2009 at 11:46
regrtest.py -R:: test_distutils leaks 103-105 references per run. This
problem is present in both trunk and release-2009Q1-maint, though trunk
leaks 103 references and release-2009Q1-maint leaks 105 references.
Original issue reported on code.google.com by collinw
on 15 May 2009 at 7:03
This is a summary bug to track everything I'm trying while optimizing IR
generation and codegen. This doesn't cover more aggressive optimizations
like type feedback and const-prop-based devirtualization. It does cover
issues 16, 17, and 18.
Original issue reported on code.google.com by [email protected]
on 4 Jun 2009 at 3:52
CALL_FUNCTION, CALL_FUNCTION_VAR_KW and UNPACK_SEQUENCE make the
stackpointer escape LLVM's pervue because their implementations (in C
functions) manipulate the stack directly. Getting rid of the direct
manipulation would allow LLVM to optimize more stack operations (and make
it easier to move away from the stack machine.)
Original issue reported on code.google.com by [email protected]
on 21 Apr 2009 at 10:41
The Unpickler doesn't re-read the from the file object when it runs out of
data, meaning Unpicklers can't be reused in long-lived streaming pickle
sessions. This also needs tests.
Original issue reported on code.google.com by collinw
on 26 Mar 2009 at 3:28
Currently, the LLVM implementation of the LOAD_CONST opcode indexes into
the code object's co_consts tuple. Since these are *constants*, the
generated machine code should just load the object's address, skipping
co_consts entirely.
Original issue reported on code.google.com by collinw
on 29 May 2009 at 3:52
We should measure how long execution blocks for when sending a code object
to be compiled/optimized by LLVM. This will be useful for proving that
offloading compilation/optimization to worker threads is valuable.
- These times should be stored in a vector and statistics displayed at
Python-shutdown.
- This should be a special build (ie, controlled by #ifdef's).
- Timing done using TSCs?
Original issue reported on code.google.com by collinw
on 30 May 2009 at 1:39
CMake would allow developers to choose their build system/IDE they use to
build unladen-swallow. What say you?
Is anyone on the project familiar with CMake?
Original issue reported on code.google.com by [email protected]
on 27 Mar 2009 at 9:22
Currently, LLVM-generated functions don't yield the GIL to other threads
(unless they're doing IO) the way the interpreter does. Among other
problems, this greatly reduces threading fairness when running with -L[012].
evlogimenos has agreed to work on this (and signal handling, since they're
related).
Original issue reported on code.google.com by collinw
on 28 May 2009 at 12:30
I couldn't install 4Suite-XML with either the 2009Q1 version or the trunk
version. It complains about KeyError: 'EXTENDED_ARG'. Attached is the
output from running easy_install 4Suite-XML.
Original issue reported on code.google.com by [email protected]
on 4 May 2009 at 3:59
Attachments:
We currently don't have a good way of debugging the machine code that comes
out of LLVM's JIT compiler.
To the best of my knowledge, LLVM doesn't emit debug information for JITted
code. Even if it did, there's no way to tell gdb to read this information.
However, there is support in the LLVM and gdb communities for fixing both
of these issues.
References:
- http://lists.cs.uiuc.edu/pipermail/llvmdev/2009-March/021255.html
- http://lists.cs.uiuc.edu/pipermail/llvmdev/2009-April/021421.html
- http://lists.cs.uiuc.edu/pipermail/llvmdev/2009-April/021424.html
- http://wiki.llvm.org/HowTo:_Tell_GDB_about_JITted_code
Original issue reported on code.google.com by collinw
on 27 May 2009 at 9:42
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.