copterfly / modwsgi Goto Github PK

All mod_wsgi daemon process still shows as httpd which makes it hard to 
identify what each is 
doing. Only way besides enabling info level error logging in Apache is to run 
daemons as 
different users.

To make it easier, add a 'display-name' option to WSGIDaemonProcess and try to 
use various 
techniques to set the active process name to be that value.

Some references as to how to do this on different platforms are:

NETBSD and MacOSX: 
http://netbsd.gw.com/cgi-bin/man-cgi?setprogname++NetBSD-current

FREEBSD: http://www.freebsd.org/cgi/man.cgi?
query=setproctitle&apropos=0&sektion=0&manpath=FreeBSD+4.5-RELEASE&format=html

LINUX: http://lightconsulting.com/~thalakan/process-title-notes.html

If WSGIDaemonProcess directive appears in Apache configuration before the
User/Group directives, the value for uid/gid of Apache process will not yet
be set and will find default of -1/-1. This will result in daemon process
changing to uid/gid of -1/-1, which may subsequently cause child process
initialisation code to crash in Python setup somewhere due to lack of
permissions or some other problem with Python not liking the user/group
that process is running as.

Setup code may have to record whether user/group options to
WSGIDaemonProcess were set, and if not, then only lookup values
corresponding to User/Group after configuration has all been processed.

Interim work around is to always ensure that User/Group come before
WSGIDaemonProcess in Apache configuration files.

At present, if Python code executed as part of initial script import, or by
WSGI application itself calls sys.exit() or explicitly raises a SystemExit
exception, then it will cause the Apache process to exit. This is because
error messages for Python exceptions are currently generated by calling
PyErr_Print() which has a side effect of exiting the program if SystemExit
exception is detected.

At the minimum, must capture when SystemExit exception has been raised, log
an Apache warning message indicating it has been ignored and otherwise
continue processing requests.

When using mod_wsgi embedded mode with a HTTPS site everything runs okay.
When using mod_wsgi daemon mode with a HTTPS site however, certain
configurations or Apache versions will result in daemon process crashing
when a request arrives. This is believed to only affect Apache 2.2 at this
point in time. For discussions on this issue see:

  http://groups.google.com/group/modwsgi/browse_frm/thread/3cba73baaa00a437

At present issue appears to be that SSL input/output filters are being
installed when we don't want them to be. This is causing a crash due to not
all data being setup to support SSL, plus, the Apache child process isn't
going to talk SSL to daemon process anyway, so there would be a
disagreement on data format.

After setting inactivity_timeout apache consumes 100% CPU protracted 

OS: Fedora8
mod_wsgi: 2.0c3

after downgrade to 2.0c2 all runs normal

More testing of these functions required in varied use cases, especially
where calls to the two functions are intermixed resulting in data being
retained within the internal buffer.

The CGI specification allows for a CGI script to return a 'Location' header
which refers to a location within the local web server. Quoting the RFC:

"""If the Location value is a path, then the server will generate the
response that it would have produced in response to a request containing
the URL"""

In Apache this is honoured when the Status returned by the CGI script is
also 200. The end result is that rather than sending a redirect back to the
web client, Apache will trigger a new sub request against the path (as a
GET request) and return the result of that to the web client.

Although the WSGI specification doesn't mention any requirement for a WSGI
adapter for a web server to do the same thing, it may be an interesting
extension to add to mod_wsgi. Although nearly all WSGI adapters would not
currently support this, it should already be available where a CGI/WSGI is
currently being used if that CGI implementation adheres to the CGI
specification.

On FreeBSD, plus Ubuntu and RedHat Linux (but NOT Mac OS X), code dependent
on popen() is failing in mod_wsgi daemon processes.

For example, for code:

from commands import getoutput
def application(environ, start_response):
   status = '200 OK'
   output = getoutput('uptime')
   response_headers = [('Content-type', 'text/plain'),
                       ('Content-Length', str(len(output)))]
   start_response(status, response_headers)
   return [output]

this will fail with:

[Tue Oct 16 15:51:17 2007] [info] mod_wsgi (pid=23422): Create
interpreter '127.0.1.1|/wsgi/commands.py'.
[Tue Oct 16 15:51:17 2007] [info] [client 127.0.0.1] mod_wsgi
(pid=23422, process='grahamd',
application='127.0.1.1|/wsgi/commands.py'): Loading WSGI script
'/home/grahamd/scripts/wsgi/commands.py'.
[Tue Oct 16 15:51:17 2007] [error] [client 127.0.0.1] mod_wsgi
(pid=23422): Exception occurred processing WSGI script
'/home/grahamd/scripts/wsgi/commands.py'.
[Tue Oct 16 15:51:17 2007] [error] [client 127.0.0.1] Traceback (most
recent call last):
[Tue Oct 16 15:51:17 2007] [error] [client 127.0.0.1]   File
"/home/grahamd/scripts/wsgi/commands.py", line 4, in application
[Tue Oct 16 15:51:17 2007] [error] [client 127.0.0.1]     output =
getoutput('uptime')
[Tue Oct 16 15:51:17 2007] [error] [client 127.0.0.1]   File
"commands.py", line 44, in getoutput
[Tue Oct 16 15:51:17 2007] [error] [client 127.0.0.1]     return
getstatusoutput(cmd)[1]
[Tue Oct 16 15:51:17 2007] [error] [client 127.0.0.1]   File
"commands.py", line 55, in getstatusoutput
[Tue Oct 16 15:51:17 2007] [error] [client 127.0.0.1]     sts = pipe.close()
[Tue Oct 16 15:51:17 2007] [error] [client 127.0.0.1] IOError: [Errno
10] No child processes

The reason being that underlying system pclose() function is not able to
get the return status of the sub process as something else appears to have
already consumed the status.

The culprit may centre around how mod_wsgi in daemon mode uses sigwait() to
determine when the daemon process should shutdown, although initial
requests don't show this as being the case. Other possibilities are the way
that in daemon mode signals are blocked from being received by worker
threads, but again, initial tests don't show this as the cause. All up, not
sure at all what is the actual cause and why it would only affect some
operating systems and not others.

Only workaround at this time is to use popen() direct rather than through
wrappers and ignore this error message from close() call on pipe. For example:

def getstatusoutput(cmd):
   """Return (status, output) of executing cmd in a shell."""
   import os
   pipe = os.popen('{ ' + cmd + '; } 2>&1', 'r')
   text = pipe.read()
   try:
       sts = None
       sts = pipe.close()
   except:
       pass
   if sts is None: sts = 0
   if text[-1:] == '\n': text = text[:-1]
   return sts, text

At present the wsgi.log object will call ap_log_rerror() for every call to
the write() function even if there is no newline in the data. This means
that if someone does:

  log = environ['wsgi.log']
  print >> log, 1, 2, 3, 4

that ap_log_rerror will be called five times, once for each number being
printed and then for the newline.

The result of this is that the output in the log files can be hard to read
as line breaks are introduced when none existed originally.

The log object should instead perform buffering and only call
ap_log_rerror() when it has a complete line of data.

To ensure that any data which doesn't have a trailing newline is still
passed through to the Apache log file, the log object should be flushed at
the end of the specific request it was created for.

Hi.

I've tried to build this on OS/X and have run into a minor issue.

$ ./configure --with-apxs=/usr/local/apache2/bin/apxs 
--with-python=/usr/local/bin/python

$ apachectl configtest
httpd: Syntax error on line 415 of /usr/local/apache2/conf/httpd.conf: Syntax 
error on line 1 of 
/usr/local/apache2/conf.d/00_django.conf: Cannot load 
/usr/local/apache2/modules/
mod_wsgi.so into server: Symbol not found: _PyList_Type\n  Referenced from: 
/usr/local/
apache2/modules/mod_wsgi.so\n  Expected in: flat namespace\n

apache 2.3 HEAD
python 2.5 (downloaded from python.org)

running OS/X 10.4 
$ uname -a
Darwin Aeaea.local 8.10.3 Darwin Kernel Version 8.10.3: Wed Jun 27 23:29:36 PDT 
2007; 
root:xnu-792.23.3~1/RELEASE_I386 i386 i386

The bottom section of the Trac guide (the part about multiple environments) is 
very over-
engineered. All that is needed is to set trac.env_parent_dir, it will do the 
indexing and such 
automatically.

What steps will reproduce the problem?
unknown.

What is the expected output? What do you see instead?
TypeError is produced when status is '200 OK'

What version of the product are you using? On what operating system?
Revision: 240 on RedHat Linux ES5


Please provide any additional information below.
strtol has errno==ERANGE with valid value in some cases. 
man strtol says to check value for LONG_MIN or LONG_MAX before checking
errno.  Fix attached.

Should add to 'mod_wsgi' module the variables:

  process_group
  application_group

This is so imported modules can determine if they should enable features
that may only work in or out of main Python interpreter, or only in Apache
child processes or daemon processes.

Possibly should also add:

  multithread
  multiprocess
  run_once

By rights there should be a standard 'wsgi' module but WSGI specification
doesn't require such a thing when it really should.

At present, details of any exceptions generated in Python code are output
to the Apache error log file by calling PyErr_Print() and the
PyErr_Flush(). This works because Apache links the underlying stderr stream
to the Apache log file. Doing this is suboptimal as any association with a
specific request is lost.

At minimum, need to capture exception details and traceback, format it
along with details of process ID, server and WSGI application SCRIPT_NAME
and send it directly to the Apache error log using ap_log_rerror(). This
will have the result of also capturing date and time of request plugs any
details of client making the request such as an IP address.

Also need to look at whether sending error messages from scripts to the
standard Apache error log is the best approach. In mod_cgi it is possible
to have a distinct error log file for scripts. It may be better to also
have this ability so that all WSGI application errors can be logged
separately if desired.

I would very like mod_wsgi support for Authorization similar to
mod_python's  PythonAuthzHandler. 

The aim is to be able to filter access to URLs by using a Python script
with custom logic, implementing a sort of ACLs for web-based resources. My
use-case would be a WebDAV-enabled SVN server, where I could allow/deny
access to resources for my team mates according to our custom access config
file (not using SVN's authz).

Add new option to WSGIDaemonProcess which allows an inactivity timeout
value to be specified. The effect of this would be that if no requests are
received by the daemon process within that period, the process would be
automatically shutdown.

Because daemon processes created using WSGIDaemonProcess are persistent,
this would mean the process will start up again straight away, but memory
usage would drop down to initial levels. To avoid unnecessary cycling of
processes, the shutdown would only occur if prior to the period of
inactivity a request had actually been received since the process had been
started. In other words, no point restarting process if it hasn't handled
any requests yet.

One possibility for the name of the option would be 'inactivity-timeout'.

When a script file is changed, it can currently trigger Module (Script), or
Interpreter reloading. When daemon processes are being used, should support
Process reloading. That is, if the script file modification time has
changed since last time application handled a request, the daemon process
should be shutdown and restarted.

Note that this is a bit tricky, as the Apache child process communicating
with the daemon process will have to wait for an acknowledgment from the
daemon process to say it will handle the request before it sends the actual
content of the request. If the daemon process indicates that it will not
handle the request and is restarting, then the Apache child process will
need to close the socket connection to the daemon process and create a new
one. If there are multiple daemon processes in the process group, the
Apache child process may have to do this as many times as there are daemon
processes in the group, until it gets a process which has finished
restarting. Some sort of upper limit of attempts would need to be
implemented. It should be more than the number of daemon processes in the
process group plus some margin.

This Process reloading mechanism is important for commodity web hosting
where sites a load balanced across a number of machines in a cluster, as
one cannot rely on an application sending itself a signal to restart, nor
on a control panel, as this can only affect the process on one machine and
not others. Thus, needs to be a way for it to automatically be detected.

What steps will reproduce the problem?
1. Using python 2.4.4 compiled from source installed as 
/usr/local/bin/python2.4, apache 2.2.4 
installed from portage
2. Downloaded mod_wsgi from 1.0 tag in svn
3. ./configure --with-python=/usr/local/bin/python2.4 --prefix=/usr/local
4. make

What is the expected output? What do you see instead?
mod_wsgi.c:5022: warning: passing argument 1 of 'ap_strstr' discards qualifiers 
from pointer 
target type
mod_wsgi.c:5029: warning: passing argument 1 of 'ap_strstr' discards qualifiers 
from pointer 
target type
mod_wsgi.c:5038: warning: passing argument 1 of 'ap_strstr' discards qualifiers 
from pointer 
target type
/usr/bin/libtool --silent --mode=link x86_64-pc-linux-gnu-gcc -o mod_wsgi.la  
-rpath /usr/
lib64/apache2/modules -module -avoid-version    mod_wsgi.lo 
-L/usr/local/lib/python2.4/
config -lpython2.4 -lpthread -ldl -lutil
/usr/lib/gcc/x86_64-pc-linux-gnu/4.1.2/../../../../x86_64-pc-linux-gnu/bin/ld: 
/usr/local/
lib/python2.4/config/libpython2.4.a(node.o): relocation R_X86_64_32 against `a 
local symbol' 
can not be used when making a shared object; recompile with -fPIC
/usr/local/lib/python2.4/config/libpython2.4.a: could not read symbols: Bad 
value
collect2: ld returned 1 exit status
apxs:Error: Command failed with rc=65536
.
make: *** [mod_wsgi.la] Error 1

What version of the product are you using? On what operating system?
mod_wsgi 1.0 (tagged svn)
apache 2.2.4
python 2.4.4
Gentoo Linux on AMD64, running under Xen:
Linux frodo 2.6.16.29-xen #3 SMP Sun Oct 15 13:15:34 BST 2006 x86_64 Dual-Core 
AMD 
Opteron(tm) Processor 2212 HE AuthenticAMD GNU/Linux

Currently when a script is loaded/reloaded nothing is logged. For debugging
purposes should be able to optionally enable logging of such events.
Directive to enable this could be called WSGIScriptDebugging. Need to
determine whether it should default to On or Off.

Note that any such logging should include details of process ID,
application group and SCRIPT_NAME corresponding to application as well as
path to script file itself.

Although support for daemon processes already exists, the number of daemon
processes must be statically defined and the processes persist. Such long
running daemon processes aren't really acceptable in commodity web hosting
setups, nor is the need to statically defined how many daemon process
groups are to be started.

What is required is a means of creating daemon processes on demand. This
should be achievable with minimal configuration that doesn't have to be
changed over time. The user/group of the daemon process should be able to
be defined dynamically through request notes or request environment so it
can be driven by custom virtual hosting modules or rewrite rules and
rewrite maps. An inactivity timeout should be able to be defined which
causes the daemon process to shutdown automatically if not used.

An example of how this may be defined is:

  WSGITransientProcess users user=%{ENV:HTTP_USER} inactivity-timeout=300

This would define a process group 'users' but which is qualified by a user
name dynamically provided through HTTP_USER request environment.

If now a WSGI application is defined and delegated to this process group:

  WSGIScriptAlias /usermin /some/path/usermin.wsgi

  <Location /usermin>
  AuthType Basic
  AuthName "Control Panel"
  AuthUserFile /usr/local/apache/passwd/passwords
  Require valid-user

  WSGIProcessGroup users
  <Location>

where access is controlled through an authentication scheme, when user logs
in an accesses the application, the HTTP_USER variable set by the
authentication scheme would result in a unique 'users' daemon process for
the user being created, ie. running as that user. The process would
shutdown after five minutes on inactivity. This sort of configuration would
be good for site control panels.

Rather than naming a transient process group, one could instead perhaps
bind it to a virtual host (server). The idea here would be that any request
coming in for a specific virtual host (server) would be handled in a
transient daemon process running as the owner of the virtual host.

  WSGITransientProcess %{SERVER} user=%{ENV:VHOST_USER}

  WSGIProcessGroup %{SERVER}

The owner of the virtual host would need to be provided from a rewrite map
or equivalent.

Exact configuration would need to be worked out, but idea here is that one
could get away with a single configuration line and there would be no need
to add additional configuration when new hosts are added. Any external
databases may need to be updated though to indicate user name of owner of
host etc so that this can drive configuration when expanded automatically
on receipt of a request.

What steps will reproduce the problem?
1. try to use WSGIDaemonProcess ... umask=0007  (same for 0077)

What is the expected output? What do you see instead?
Invalid umask.

What version of the product are you using? On what operating system?
1.0rc1 on Debian etch.

Please provide any additional information below.

umask=0002 works.

Django can serve static files using simple function that checks static file
modification time and sets appropriate headers:

{{{
#!python

statobj = os.stat(fullpath)
    if not was_modified_since(request.META.get('HTTP_IF_MODIFIED_SINCE'),
                              statobj[stat.ST_MTIME], statobj[stat.ST_SIZE]):
        return HttpResponseNotModified()
    mimetype = mimetypes.guess_type(fullpath)[0]
    contents = open(fullpath, 'rb').read()
    response = HttpResponse(contents, mimetype=mimetype)
    response["Last-Modified"] = http_date(statobj[stat.ST_MTIME])
    return response
}}}

Can mod_wsgi do the same? Usual use case of mod_wsgi is to set it up to
serve static files while Django serves dynamic content. 

Regards,
Max.

user and group stuff does not work correctly. It does something, but not
enough as it seems:

What steps will reproduce the problem?
1. WSGIDaemonProcess localhost user=tw group=tw
2. restart apache2
3. moin can not access stuff although it is tw.tw owned
4. chmod stuff to www-data.www-data and it works

What version of the product are you using? On what operating system?
mod_wsgi 1.0rc1 on Debian Linux etch
apache 2.2.3-4

Apache is started by root (1 apache2 process shows in ps aux with owner
root, others with owner www-data, 1 other with owner tw).

When using Pylons and the 'Interpreter' option is used for the
WSGIReloadMechanism directive, the Apache process will crash when creating
the new sub interpreter instance subsequent to destroying the prior instance.

The stack trace for where the crash occurs is:

#0  0x98ab7bd8 in PyObject_GetAttr ()
#1  0x013fac30 in __Pyx_GetName (dict=0x13b8610, name=0x13b8be0) at
src/protocols/_speedups.c:1862
#2  0x013fcacc in __pyx_f_9_speedups_Protocol__adapt__
(__pyx_self=0x13b8610, __pyx_args=0xa8a8bab8, __pyx_kwds=0x4c69) at
src/protocols/_speedups.c:1083
#3  0x98a8e8e0 in PyObject_Call ()
#4  0x98a96af4 in PyMethod_New ()
#5  0x98a8e8e0 in PyObject_Call ()
#6  0x98aed9a0 in PyEval_CallObjectWithKeywords ()
#7  0x013fe700 in __pyx_f_9_speedups__adapt (__pyx_v_obj=0x0,
__pyx_v_protocol=0x165f5d0, __pyx_v_default=0x48d3c0) at
src/protocols/_speedups.c:497
#8  0x013fb958 in __pyx_f_9_speedups_adapt (__pyx_self=0x13b8610,
__pyx_args=0xa8a8bab8, __pyx_kwds=0x4c69) at src/protocols/_speedups.c:641
#9  0x98aedc5c in PyEval_GetFuncDesc ()
#10 0x98aeb414 in PyEval_EvalCode ()

Note that this crash does not appear to be a problem with mod_wsgi but with
the pyrex generated code present in the PyProtocols package which is used
by Pylons. In short, that code does not appear to cope with an interpreter
being destroyed and the module then being reused in a new interpreter.

Chances of getting either PyProtocols or pyrex changed, if they are the
problem, so that it will work in this sort of circumstance is probably
quite small unless some enthusiastic Pylons users want to push the issue.

Some libraries are unusable when run by mod_wsgi and another apache module
at the same time and they expects different versions. A prominent example
is the mysql client library used by both the MySQL-python module and the
PHP module in apache. Misleading, unusable tracebacks are produced to the
apache error logs.

A post at the mod_python FAQ describes this weird issue:
http://www.modpython.org/FAQ/faqw.py?req=all#2.13

It is somehow complex to reproduce:

1. Install a Debian Sarge server with Apache2 and PHP (4 or 5), ensure that
mysql is enabled in PHP and the client libraries are included in your
distribution.

2. Install mod_wsgi 1.0, other versions could behave the same way, since it
seems to be an apache library issue.

3. Write a WSGI app using mysql through MySQL-python-1.2.2. By using
SQLObject you can make this very fast. Creating a table object such as
below would cause the error without doing anything with the DB:

__connection__='mysql://someuser:passwd:localhost/somedb'
class SomeTable:
    class sqlmeta:
        fromDatabase=True

The script will run fine from the command line with the same version of
python, but won't run when called from the apache process (when you try to
see it's output in the browser).

You will get unusable, partial tracebacks and 500 Internal server error
responses. Usually it takes hours for an experienced programmer to catch
the real cause of this kind of bug, which is really dangerous thing in an
enterprise environment.

I'm not sure that it can be fixed in mod_wsgi or mod_python, since this
could be caused by the library handling of apache or Linux. The bug can be
produced by various other libraries such as expat or libpng when used by
both mod_wsgi and the php module.

OS: Linux, Debian Sarge 3.1, up to date (but not the current stable anymore)
Python 2.5.1, compiled from official sources
mod_wsgi 1.0 (compiled on 13th April 2007 as a recent version)
MySQL-python-1.2.2
SQLObject 0.9.1 (not essential, but convenient to reproduce the bug)
mysql  Ver 14.7 Distrib 4.1.11
mysqld  Ver 4.1.11-Debian_4sarge7
Apache/2.0.54 MPM prefork
PHP 4.3.10-19 (cli) (built: Mar  5 2007 20:46:29)
Zend Engine v1.3.0

Typical apache error log lines:

[Sun Aug 26 00:30:05 2007] [notice] [client <IP>] mod_wsgi (pid=20824):
Target WSGI script '/var/www/ourhost.hu/ds/datasource.py' cannot be loaded
as Python module.
[Sun Aug 26 00:30:05 2007] [notice] [client <IP>] mod_wsgi (pid=20824):
Exception occurred within WSGI script '/var/www/ourhost.hu/ds/datasource.py'.
[Sun Aug 26 00:30:05 2007] [error] [client <IP>] Traceback (most recent
call last):
[Sun Aug 26 00:30:05 2007] [error] [client <IP>]   File
"/var/www/ourhost.hu/ds/datasource.py", line 6, in <module>
[Sun Aug 26 00:30:05 2007] [error] [client <IP>]     from befi import *
[Sun Aug 26 00:30:05 2007] [error] [client <IP>]   File
"/var/www/ourhost.hu/ds/befi.py", line 6, in <module>
[Sun Aug 26 00:30:05 2007] [error] [client <IP>]     from config_db import *
[Sun Aug 26 00:30:05 2007] [error] [client <IP>]   File
"/var/www/ourhost.hu/ds/config_db.py", line 7, in <module>
[Sun Aug 26 00:30:05 2007] [error] [client <IP>]     class Alap(SO.SQLObject):
[Sun Aug 26 00:30:05 2007] [error] [client <IP>]   File
"/usr/local/lib/python2.5/site-packages/SQLObject-0.9.1-py2.5.egg/sqlobject/decl
arative.py",
line 121, in __new__
[Sun Aug 26 00:30:05 2007] [error] [client <IP>]     cls.__classinit__(cls,
new_attrs)
[Sun Aug 26 00:30:05 2007] [error] [client <IP>]   File
"/usr/local/lib/python2.5/site-packages/SQLObject-0.9.1-py2.5.egg/sqlobject/main
.py",
line 792, in __classinit__
[Sun Aug 26 00:30:05 2007] [error] [client <IP>]    
cls.sqlmeta.addColumnsFromDatabase()
[Sun Aug 26 00:30:05 2007] [error] [client <IP>]   File
"/usr/local/lib/python2.5/site-packages/SQLObject-0.9.1-py2.5.egg/sqlobject/main
.py",
line 450, in addColumnsFromDatabase
[Sun Aug 26 00:30:05 2007] [error] [client <IP>]     for columnDef in
conn.columnsFromSchema(sqlmeta.table, soClass):
[Sun Aug 26 00:30:05 2007] [error] [client <IP>]   File
"/usr/local/lib/python2.5/site-packages/SQLObject-0.9.1-py2.5.egg/sqlobject/mysq
l/mysqlconnection.py",
line 212, in columnsFromSchema

Note, that no real exception info is produced, only an unusable traceback.
This points to the line of code where the python script first tries to use
the mysql library, usually something related to an SQL cursor. But this is
irrelevant and can be anything in that library, since a wrong version is
used not the one expected by python.

The WSGI specification defines the file_wrapper extension. It is not
mandatory for a WSGI server adapter to implement it, but should look at
doing it at some point if practical.

What steps will reproduce the problem?
1. In a webpy application request handler, write 

web.debug.write('REMOTE_USER "%s"\n" % 
        web.ctx.environ.get("REMOTE_USER", "")
    )

2. Configure .htaccess to require valid user. Don't declare the
WSGIPassAuthorization directive.
3. Observe user name in Apache 2.2.4 error log.

What is the expected output? What do you see instead?

I thought that HTTP_AUTHORIZATION will be passed to webpy's web.wsgiserver.

What version of the product are you using? On what operating system?

Subversion checkout, revision 555 (2007-09-23).

Please provide any additional information below.

webpy 0.210-1 in Ubuntu gutsy

Hi,

In mod_wsgi.c (rev 655), line 2508, a Py_DECR(item) is missing if we want
the reference to item decreased in all cases.

What steps will reproduce the problem?
1. WSGIDaemonProcess ... maximum-requests=100
2. put a global counter into a trivial wsgi script
3. keep reloading the page, see the counter going past 100, process does
not get restarted

What version of the product are you using? On what operating system?
1.0rc1

The WSGI specification allows one to do:

  try:
    # regular application code here
    status = "200 Froody"
    response_headers = [("content-type","text/plain")]
    start_response(status, response_headers)
    return ["normal body goes here"]
  except:
    # XXX should trap runtime issues like MemoryError, KeyboardInterrupt
    #     in a separate handler before this bare 'except:'...
    status = "500 Oops"
    response_headers = [("content-type","text/plain")]
    start_response(status, response_headers, sys.exc_info())
    return ["error body goes here"]

That is, try and replace the response headers with an alternate set. If no
content has been sent yet then the header replacement should succeed. If
content has already been sent, then the exc_info should instead be raised
and presumably the response will be cut off, with details of exception
being logged in some way.

In the latter case where content has already been sent, mod_wsgi will
currently crash the process at some point as it is forgetting to increment
the reference counts on exc_info components before restoring it as the
current exception state.

This paradigm isn't something that is common in WSGI applications and
especially if using a high level framework which isn't designed for WSGI
but just uses it as a hosting mechanism, you shouldn't encounter the problem. 

When a browser client closes a socket connection before a complete response
has been written by the application, an error message is written to the
Apache error log. For example:

[Mon Sep 03 11:35:24 2007] [error] [client 213.197.179.210] mod_wsgi
(pid=26069): Exception occurred within WSGI script '/home/dalius/wsgi/
blog.wsgi'., referer: http://blog.sandbox.lt/lt?page_nr=1
[Mon Sep 03 11:35:24 2007] [error] [client 213.197.179.210] IOError:
client connection closed, referer: http://blog.sandbox.lt/lt?page_nr=1

For certain types of applications which use lots of AJAX stuff and parallel
requests from a page, one can get a lot of these if users are traversing
off the page before the page is completely updated.

Other applications where this is a problem is those which stream continuous
data, as one will get this message for every request when user drops
connection.

Thus need to look at option which would allow this specific type of error
message in logs to be suppressed to stop logs filling up and to get rid of
what is an innocuous message not indicative of an actual problem.

Although there should be an Apache configuration directive, there should
also be a way of an application disabling it so users in commodity web
hosting can also switch it off. One option there is to use CGI 1.1 type
Script-Control header in response.

On FreeBSD 5.4, Python 2.4 and Apache 2.0.58, building modwsgi fails as 
following error:

{{{
> ./configure
checking for apxs2... no
checking for apxs... /usr/local/sbin/apxs
checking for python... /usr/local/bin/python
checking Apache version... 2.0.58
configure: creating ./config.status
config.status: creating Makefile
> make
/usr/local/sbin/apxs -c -I/usr/local/include/python2.4 -DNDEBUG 
-D__wchar_t=wchar_t -
DTHREAD_STACK_SIZE=0x20000  mod_wsgi.c -L/usr/local/lib/python2.4/config 
-lpython2.4 -
lutil
/usr/local/share/apache2/build/libtool --silent --mode=compile cc -prefer-pic 
-O -pipe -
DAP_HAVE_DESIGNATED_INITIALIZER -D_REENTRANT -D_THREAD_SAFE  
-I/usr/local/include/
apache2  -I/usr/local/include/apache2   -I/usr/local/include/apache2 
-I/usr/local/include -I/
usr/local/include/python2.4 -DNDEBUG -D__wchar_t=wchar_t 
-DTHREAD_STACK_SIZE=0x20000  
-c -o mod_wsgi.lo mod_wsgi.c && touch mod_wsgi.slo
mod_wsgi.c: In function `wsgi_python_child_init':
mod_wsgi.c:2042: error: `wsgi_interp_lock' undeclared (first use in this 
function)
mod_wsgi.c:2042: error: (Each undeclared identifier is reported only once
mod_wsgi.c:2042: error: for each function it appears in.)
mod_wsgi.c:2042: error: `APR_THREAD_MUTEX_UNNESTED' undeclared (first use in 
this function)
mod_wsgi.c:2043: error: `wsgi_module_lock' undeclared (first use in this 
function)
apxs:Error: Command failed with rc=65536
.
*** Error code 1

Stop in /home/ymasuda/src/modwsgi.
> 
}}}

This seems to be caused by misuse of #ifdef directive instead of #if in 
mod_wsgi.c 2041:
{{{
#ifdef APR_HAS_THREADS
 ...
}}}

The attached file should fix the problem.

The definition of wsgi_execute_remote is protected by
MOD_WSGI_WITH_DAEMONS, but its definition and call are protected instead by
non-equivalent conditions. MOD_WSGI_WITH_DAEMONS should be used to protect
all three.

This bug is flushed out under Unix and Apache 2+ when APR_HAS_THREADS is
false (default on FreeBSD, e.g.). In this case MOD_WSGI_WITH_DAEMONS is not
set though the non-equivalent conditions still evaluate true.

See this thread for more:

http://groups.google.com/group/modwsgi/browse_frm/thread/7b12933ad592df25/

When using Pylons and the 'Interpreter' option is used for the
WSGIReloadMechanism directive, the Apache process will crash when creating
the new sub interpreter instance subsequent to destroying the prior instance.

The stack trace for where the crash occurs is:

#0  0x98ab7bd8 in PyObject_GetAttr ()
#1  0x013fac30 in __Pyx_GetName (dict=0x13b8610, name=0x13b8be0) at
src/protocols/_speedups.c:1862
#2  0x013fcacc in __pyx_f_9_speedups_Protocol__adapt__
(__pyx_self=0x13b8610, __pyx_args=0xa8a8bab8, __pyx_kwds=0x4c69) at
src/protocols/_speedups.c:1083
#3  0x98a8e8e0 in PyObject_Call ()
#4  0x98a96af4 in PyMethod_New ()
#5  0x98a8e8e0 in PyObject_Call ()
#6  0x98aed9a0 in PyEval_CallObjectWithKeywords ()
#7  0x013fe700 in __pyx_f_9_speedups__adapt (__pyx_v_obj=0x0,
__pyx_v_protocol=0x165f5d0, __pyx_v_default=0x48d3c0) at
src/protocols/_speedups.c:497
#8  0x013fb958 in __pyx_f_9_speedups_adapt (__pyx_self=0x13b8610,
__pyx_args=0xa8a8bab8, __pyx_kwds=0x4c69) at src/protocols/_speedups.c:641
#9  0x98aedc5c in PyEval_GetFuncDesc ()
#10 0x98aeb414 in PyEval_EvalCode ()

Note that this crash does not appear to be a problem with mod_wsgi but with
the pyrex generated code present in the PyProtocols package which is used
by Pylons. In short, that code does not appear to cope with an interpreter
being destroyed and the module then being reused in a new interpreter.

Chances of getting either PyProtocols or pyrex changed, if they are the
problem, so that it will work in this sort of circumstance is probably
quite small unless some enthusiastic Pylons users want to push the issue.

The WSGI specification doesn't really say in what way sys.stdin, sys.stdout
and sys.stderr can be used. For some environments which WSGI can be used
in, for example CGI, the sys.stdin and sys.stdout streams actually have
special significance as they are how data is read from and written back to
the server. In the interest of highlighting portability issues with WSGI
scripts, use of sys.stdin and sys.stdout should probably be precluded and
raise an error.

With sys.stderr, although anything sent here gets to the Apache error log
if flushed it doesn't get time stamped like if ap_log_error() is used. As
such a special sys.stderr should perhaps be provided which buffers output
until complete lines are received or output flushed explicitly and the
buffered output only then sent to the log but via ap_log_error(). Only
issue with this is that each thread would need to have its own buffer area
and would need to ensure somehow that buffer areas are cleaned up if
threads finish, or if thread linked to request, that it is flushed and
cleaned up at end of request.

For further discussion of this, see Python WEB-SIG posting at:


http://groups.google.com.au/group/python-web-sig/browse_thread/thread/512874b8ac
16626e?hl=en

A WSGI app I am writing sometimes causes mod_wsgi to segfault.  I spent
some time narrowing the problem and found that the following simple script
will trigger the segfault:


def application(environ, start_response):
    global save
    save = start_response
    status = '200 OK'
    output = 'Hello World!'
    environ['wsgi.errors'].write('hello')
    response_headers = [('Content-type', 'text/plain'),
                        ('Content-Length', str(len(output)))]
    start_response(status, response_headers)
    return [output]


This is just a "hello, world" script with two modifications: it saves the
start_response callback and it writes to the error log without a
terminating newline.  The first request for this script returns the "Hello
World!" response to the browser just fine, but the "hello" message does not
appear in Apache's log.  A segfault occurs the second time I make a request
for the script.

If I do not store the start_response callback, or I add a newline to the
error message, or I call environ['wsgi.errors'].flush() before returning,
no problem occurs.  This suggests to me that mod_wsgi is supposed to flush
the error stream before the request is finished, but that is not happening,
and the flush is occurring after the request has been destroyed.  In a gdb
post-mortem session, I noticed that a certain PyObject* had the address
"0xb", an invalid address, which suggests that perhaps the destroyed
request object had already been reused for another object.

In reality, my app is slightly misbehaved in that it really should not
store the start_response callback.  That was accidental and I will fix it.
 However, no WSGI app should be able to trigger a segfault this way.

OS: OpenSuse 10.2 inside a Linux VServer
Apache: 2.2.3
Python: 2.5 and 2.5.1 (problem occurred with both versions)
mod_wsgi: 1.0c2

In mod_wsgi the sys.stderr (and sys.stdout if not restricted) stream
objects are replaced with alternatives which will ensure that any output is
flushed out into the Apache error logs whenever a new line is encountered
in the output.

Where sys.stderr is being used to output messages by multiple threads at
the same time and each is emitting a string which is not terminated by a
new line, but where the string contains an embedded new line in the middle
of the string, there is a possibility that part of the output will be lost
or duplicated. In the worst case scenario, an attempt could be made to call
free() with a NULL pointer, which could result in the process crashing.

For this to occur the separate threads must have entered the write()
function at the same time and for the above condition to be satisfied.
Since the probability of this is very low, at present that the problem can
occur is theoretical based on code inspection and no problem has actually
reported by any user.

Note that technically this problem could manifest with wsgi.errors stream
object passed in WSGI environment, but even smaller probability that it
would be encountered in that case, as typically only the thread handling
the request would be accessing that stream object.

Patch for mod_wsgi.c attached if wanting to address problem before next
release.

The WSGI specification says:

"""The start_response callable must not actually transmit the response
headers. Instead, it must store them for the server or gateway to transmit
only after the first iteration of the application return value that yields
a non-empty string, or upon the application's first invocation of the
write() callable. In other words, response headers must not be sent until
there is actual body data available, or until the application's returned
iterable is exhausted. (The only possible exception to this rule is if the
response headers explicitly include a Content-Length of zero.)"""

In mod_wsgi it is not complying with this 100% as it will transmit the
headers even if the first string yielded is an empty string.

In general use this is not likely to present any problems. The only time it
may be an issue is where start_response() is called a second time in an
attempt to change the type of response that is sent before any actual
non-empty strings have been yielded.

This would not be a common practice so you would be unlucky to be affected
by this.

As an idea to controlling memory usage, provide a means of monitoring
memory usage and force a restart of the daemon processes when process
memory usage exceeds some set value.

Such a mechanism is provide with mod_perl:

  http://search.cpan.org/dist/mod_perl/lib/Apache/SizeLimit.pm
  http://books.google.com/books?q=set_min_shared_size&btnG=Search+Books

although it use is restricted to certain platforms. On Linux the smaps
information from the proc filesystem can be used. On true BSD systems (not
Mac OS X), getrusage() can be used.

Although the mod_perl implementation allows actions to be triggered off the
amount of shared memory dropping below a certain value, this is problematic
on some systems when a process is swapped. Thus concentrate just on total
process memory size, ie., unshared + shared. Although this counts shared, a
single value is probably easier to comprehend by shared web hosting setups,
who would be the primary target for this feature.

The name of the option for WSGIDaemonProcess might be 'maximum-memory'.

Note, would probably only look at implementing this for daemon processes
and not for main Apache child processes.

Hi,

I've been trying to setup mod_wsgi 1.0 for Turbogears 1.0.3.2 + SA 0.3.10.
I've just copied the sample in the wiki but can't it working.

I get the error below in my apache2 error.log. I'm probably doing something
wrong. Any advice


 mod_wsgi (pid=14475): Target WSGI script '/srv/Beta/beta.wsgi' cannot be
loaded as Python module.
 mod_wsgi (pid=14475): Exception occurred within WSGI script
'/srv/Beta/beta.wsgi'.
 Traceback (most recent call last):
   File "/srv/Beta/beta.wsgi", line 21, in ?
     cherrypy.server.start(init_only=True, server_class=None)
   File
"/usr/lib/python2.4/site-packages/CherryPy-2.2.1-py2.4.egg/cherrypy/_cpserver.py
",
line 72, in start
     Engine.start(self)
   File
"/usr/lib/python2.4/site-packages/CherryPy-2.2.1-py2.4.egg/cherrypy/_cpengine.py
",
line 104, in start
     self._start()
   File
"/usr/lib/python2.4/site-packages/CherryPy-2.2.1-py2.4.egg/cherrypy/_cpserver.py
",
line 78, in _start
     Engine._start(self)
   File
"/usr/lib/python2.4/site-packages/CherryPy-2.2.1-py2.4.egg/cherrypy/_cpengine.py
",
line 108, in _start
     func()
   File
"/usr/lib/python2.4/site-packages/TurboGears-1.0.3.2-py2.4.egg/turbogears/startu
p.py",
line 234, in startTurboGears
     ext.start_extension()
   File
"/usr/lib/python2.4/site-packages/TurboGears-1.0.3.2-py2.4.egg/turbogears/visit/
api.py",
line 64, in start_extension
     ext.start_extension()
   File
"/usr/lib/python2.4/site-packages/TurboGears-1.0.3.2-py2.4.egg/turbogears/visit/
api.py",
line 64, in start_extension
     _manager = _create_visit_manager(timeout)
   File
"/usr/lib/python2.4/site-packages/TurboGears-1.0.3.2-py2.4.egg/turbogears/visit/
api.py",
line 45, in _create_visit_manager
     return plugin(timeout)
   File
"/usr/lib/python2.4/site-packages/TurboGears-1.0.3.2-py2.4.egg/turbogears/visit/
savisit.py",
line 30, in __init__
     bind_meta_data()
   File
"/usr/lib/python2.4/site-packages/TurboGears-1.0.3.2-py2.4.egg/turbogears/databa
se.py",
line 66, in bind_meta_data
     get_engine()
   File
"/usr/lib/python2.4/site-packages/TurboGears-1.0.3.2-py2.4.egg/turbogears/databa
se.py",
line 42, in get_engine
     dburi = alch_args.pop('dburi')
 KeyError: 'pop(): dictionary is empty'
     _manager = _create_visit_manager(timeout)
   File
"/usr/lib/python2.4/site-packages/TurboGears-1.0.3.2-py2.4.egg/turbogears/visit/
api.py",
line 45, in _create_visit_manager
     return plugin(timeout)
   File
"/usr/lib/python2.4/site-packages/TurboGears-1.0.3.2-py2.4.egg/turbogears/visit/
savisit.py",
line 30, in __init__
     bind_meta_data()
   File
"/usr/lib/python2.4/site-packages/TurboGears-1.0.3.2-py2.4.egg/turbogears/databa
se.py",
line 66, in bind_meta_data
     get_engine()
   File
"/usr/lib/python2.4/site-packages/TurboGears-1.0.3.2-py2.4.egg/turbogears/databa
se.py",
line 42, in get_engine
     dburi = alch_args.pop('dburi')
 KeyError: 'pop(): dictionary is empty'

At present the application callable must be a single variable name. Might
be more flexible to allow it to be a dotted object path and for mod_wsgi to
interpret the path, traversing any object structures to determine the final
application callable.

In conjunction with WSGIScriptCallable and mod_rewrite, this would allow a
URL to be mapped by mod_rewrite into an object structure within a script
file. For example, with a Directory directive context you might use:

  RewriteEngine On
  RewriteRule ^([^./]+)/([^./])/ - [E=SCRIPT_CALLABLE:%1.%2]

  WSGIScriptCallable %{ENV:SCRIPT_CALLABLE}

Issues with allowing this though are somehow protecting what can be
traversed into if allowing URL to dictate path and also that SCRIPT_NAME
wouldn't reflect that more of the URL has been consumed.

With Apache 2.0.61 when I attempt to load the mod_wsgi (v1.1) module I get
this error:

mod_wsgi.so: undefined symbol: rl_line_buffer

What steps will reproduce the problem?
1. Add "LoadModule wsgi_module modules/mod_wsgi.so" to httpd.conf
2. Try to "apachectl start"
3.

If I load both mod_python & mod_wsgi then I don't get the error. I also
tried building from SVN and I receive the same error,

What steps will reproduce the problem?
1. compile mod_wsgi 2.0c4 with apache 2.2.6
2. setup:
   AuthType Basic
   AuthName "Top Secret"
   AuthBasicProvider wsgi
   WSGIAuthUserScript /usr/local/hacr/etc/apache2/auth.wsgi
   WSGIAuthGroupScript /usr/local/hacr/etc/apache2/auth.wsgi
   require group secret-agents

with:

def groups_for_user(environ, user):
    if user == 'robrien':
        return ['secret-agents']
    return ['']


3. try to connect:

and I have tried with the following in 4 combinations (on/off, off/on,
on/on, off/off) to no avail:

   AuthzDefaultAuthoritative Off
   AuthzUserAuthoritative Off


What is the expected output? 

Login succeed

What do you see instead?

access to /trac/test/login failed, reason: require directives present and
no Authoritative handler., referer: https://localhost/trac



What version of the product are you using? On what operating system?

mod_wsgi 2.0c4
apache 2.2.6
RHEL 5/Mac OSX 10.4.11
python 2.5.1




Please provide any additional information below.

Dec 19 01:22:41 mack auth.wsgi: Auth attempt by: robrien
Dec 19 01:22:41 mack auth.wsgi: Got None robrien
Dec 19 01:22:41 mack auth.wsgi: checkCache: got robrien
Dec 19 01:22:41 mack auth.wsgi: robrien expired: False
Dec 19 01:22:41 mack auth.wsgi: robrien authenticated successfully
Dec 19 01:22:41 mack auth.wsgi: [error] [client ::1] access to
/trac/test/login failed, reason: require directives present and no
Authoritative handler., referer: https://localhost/trac/test

Hi,

I was wondering if you kind folks could distribute modwsgi as a .deb package, 
along with the tarball 
and whatnot. It would be much appreciated.

Chris

If the python interpreter is not loaded into the Apache process, then the
conflicts between mod_python, other apache modules, and other shared will
not occur. This has been a common source of problems and is very difficult
for end-users to diagnose.

Further, with such a change, mod_wsgi would be easier to extend to work
with multiple versions of Python, which has been a common request.

Maybe it is possible for mod_wsgi to work in the following manner: (1) The
apache module is not linked to any code that requires Python at all. Thus,
all conflicts within Apache are avoided. (2) A "template" daemon process is
forked. That template daemon process dynamically load a shared library that
is linked to a particular version of Python. (3) Then more processes to
handle requests, as needed.

In between steps (2) and (3), a special initialization callable can be
executed. That initialization callable could pre-load all Python modules
used by the application, and/or initialize other expensive data structures
that can be shared between processes. This should significantly reduce the
unshared writable memory used by each daemon process on any OS that uses
copy-on-write for fork().

If using Apache 1.3/2.0, the wrong SCRIPT_NAME and consequently application
group name can be derived when the URL has repeating slashes appearing in
it after the mount point of the WSGI application. In particular, a trailing
slash will be incorrectly added to the SCRIPT_NAME and the application
group name when it shouldn't be. This will result in two instances of an
application being loaded into the same process, with which being used
dependent on whether repeating slashes appeared in the URL.

This issue has been fixed in revision 601 of the 1.X branch.

Broader description of the issue found in:

  http://code.google.com/p/modwsgi/wiki/ChangesInVersion0102

A patch for this specific problem is also attached to this issue.

If the Apache mod_logio module is loaded at the same time as mod_wsgi and
daemon mode of mod_wsgi is used then the daemon process will crash on first
attempt to write response content back to Apache child process from which
the request was proxied.

If the response was returned as one complete string, this would only occur
after content has been sent and so web pages would appear to work fine,
except for being very slow. The long time to get a response is because the
crashing daemon process means that a new process has to be created for each
request, resulting in CGI like speeds.

A fix which is hoped will address this problem was committed as revision
539 on the 1.X branch.

Build scripts currently only support UNIX platform. Need to implement some
build scripts for Windows platform.

The simplest way of doing this may be to use Python distutils mechanism in
much the same way that mod_python does for building mod_python on Windows.

Add feature whereby if an individual request takes longer than some set
period, that the daemon process will be restarted. This will help to pick
up stuck requests which would otherwise make that thread and/or process
unusable thereafter if the request never actually returns.

Code currently uses:

  ap_setup_client_block(r, REQUEST_CHUNKED_ERROR)

as read policy for request content. This means that chunked transfer
encoding isn't supported as there will be no overall content length header.

Ultimately need to support this as WSGI specification requires it. Should
be noted though that very few if any WSGI server adapters currently support
chunked transfer encoding on request content. The only one of those looked
at that tries to support it is CherryPy WSGI server, except that it handles
it by reading all the content up front and then creates a content length
for it. This isn't really a practical solution as it triggers 100 continue
responses before an application may be ready to process data, doesn't allow
streaming of reads by an application thereby avoiding buffering and doesn't
allow end to end communication whereby an application responds to client
data with some response and a client in turn sends more data in response to
the data returned by the application.

As far as implementation goes, issues are that when performing call to
Apache ap_get_client_block() function, the read size must be at least
enough to hold any chunk data length information and any trailer
extensions. This means that when WSGI application uses read() with a size
argument, may need to actually ask for more than the requested amount of
data to satisfy the requirement. We cant return more data than requested,
so will need to buffer internally any extra above what is deemed the
minimum size we can acceptably read.

When application calls read() with no argument, have no idea how much more
data there might be, so can only fall back to reading data in some defined
block size. There should be some minimal default but allow it to be set to
be some higher value with a directive. To make implementation easier, may
make more sense to enforce that this blocked reading be used with read()
even when chunked transfer encoding isn't used. This would be in place of
attempting to read all remaining data based on the content length of the
request. This shouldn't be a problem as a WSGI application should keep
calling read() until an empty string is returned and shouldn't be assuming
that it only has to call it once to have all data returned.