From rogerbinns on June 09, 2009 18:42:44

Please note that APSW in no way interferes with access to your database. It does not
parse your queries or try to second guess what is going on. The behaviour you see is
exactly what SQLite does. APSW is just forwarding your Python calls to the SQLite C API.

There are reasons why SQLite will not call a registered busy handler, mainly because
it decides there would be a deadlock. See the fourth paragraph at http://www.sqlite.org/c3ref/busy_handler.html

Status: Invalid
Labels: -Type-Defect Type-Other Component-Docs

from apsw.

From [email protected] on June 12, 2009 14:25:46

Sorry to bring this up again, but I managed to isolate the problem and it seems to me
that it is actually caused by apsw (or Python itself, since the situation cannot
occur when using the C API). I'm not saying that this is a bug, but I think it is at
least worth being documented.

Would you have the time to look at the attached testcase? Can you explain why the
first VACUUM does not raise an error, while the the second one does? It certainly
caught me by surprise...

Attachment: vacuum.py

from apsw.

From rogerbinns on June 12, 2009 17:28:20

The code works just fine for me.

The most likely cause of the problem you see is that when you raise the exception the
stack frame from dostuff is still available (eg as sys.exc_info()) in particular as
part of the traceback object. This means the 'cur' cursor inside dostuff has not
been deallocated and in particular is sitting at a result row (no=33). The vacuum is
then failed because of the outstanding cursor blocking it.

But as I said I don't see the problem with Python 2.6 and APSW/SQLite 3.6.14.

from apsw.

From [email protected] on June 12, 2009 18:05:47

You're right, after updating to the most recent APSW version the problem does not
occur any longer.

Note that the dostuff() frame is indeed still available, both with the old and new
APSW version (I encapsulated the cursor class and added print statements into del
and init). So it seems that in the end this was really an issue with apxs, only
that it has already been corrected :-).

from apsw.

From rogerbinns on June 12, 2009 19:13:06

Which version were you using that showed the issue? (I'd like to add something to
the test suite to catch this should it ever happen again, but need a test case that
actually fails). Please give Python, APSW, SQLite and platform.

Note that adding del methods changes the behaviour of the Python garbage
collector. In particular it becomes less likely to gc the object, and more likely to
take longer to do it.

The only change that could possibly affect garbage collection in the last two years
was in 3.6.11 which was to cause references to be released on close being called
rather than always waiting for the destructor. However you don't have any close
calls so it doesn't affect the test code.

from apsw.

From [email protected] on June 12, 2009 19:28:34

Try the above example with:

$ python --version
Python 2.6.2

$ apt-cache show python-apsw | grep Version
Version: 3.5.9- r2 -1ubuntu1
Python-Version: 2.5, 2.6

$ apt-cache show libsqlite3-0 | grep Version
Version: 3.6.10-1ubuntu0.2
Version: 3.6.10-1

i.e. the standard versions from Ubuntu Jaunty. Then the test should fail.

Let me know if that works.

from apsw.

From [email protected] on June 12, 2009 19:29:05

$ uname -a
Linux vostro 2.6.28-11-generic #42-Ubuntu SMP Fri Apr 17 01:57:59 UTC 2009 i686 GNU/Linux

from apsw.

From rogerbinns on June 12, 2009 19:33:45

Thanks - I can reproduce it with those versions and so now can add a test.

from apsw.

From rogerbinns on June 12, 2009 21:00:28

Despite much trying and tweaking I can't find a reliable test for this as it depends
very closely on exactly when gc is run, how much garbage there is etc. Any test
would need to give identical behaviour across all supported Python versions (2.3 to 3.1).

from apsw.

From [email protected] on June 13, 2009 08:43:04

AFAIK it cannot depend on gc, since the object is eliminated by reference counting.
You can see this by raising an artifical exception and explicitly invocing the gc:

This fails:

try:
dostuff(conn)
except TypeError:
pass

gc.collect()
print "done with dostuff"
cur = conn.cursor()
cur.execute("VACUUM")

This works:

try:
dostuff(conn)
except TypeError:
pass

try:
raise RuntimeError
except RuntimeError:
pass
print "done with dostuff"
cur = conn.cursor()
cur.execute("VACUUM")

Can you tell me a concrete Python version where this testcase does not work? Here it
worked at least with 2.5.4 and 2.6.2.

from apsw.

From rogerbinns on June 15, 2009 00:03:34

I got rid of my test code as it didn't work and was getting messy :-) I also wanted
to exclude the exception raising bit since I didn't want the test to depend on the
vagaries of how that is handled internally in Python. I was only testing in Python
2.6 between the two APSW versions.

I'll try again ...

from apsw.