arachnid / bdbdatastore Goto Github PK

When BDB-JE cursors read a record, they lock that record for as long as they're positioned on it. This causes locking issues when someone tries to update based on the results of a query they've executed.

Options for fixing this include:

• Use READ_UNCOMMITTED isolation. Ugh.
• Advance the cursor after reading a record, ensuring it's always positioned on a not-yet-returned record. Solves the locking issue, but may introduce unnecessary contention.
• Close the cursor after each read (storing the last location) and reopen it for the next read.
• Pester BDB-JE authors to introduce a 'read only' locking mode.

A query such as TestModel.all().order('-foo').fetch(10) returns 0 results even if some exist. This was due to BDBDatastore erroneously trying to use a single property index with a forward scan in this case.

Key only queries were added in SDK 1.2.2, and need to be added to BDBDatastore.

I've been experiencing quite reliably that fetch(1000) for a quite small model takes a lot more time than just iterating through the results (which is also very slow.) There are about 500 instances of the model in datastore total. First the jvm process peaks a little, but then python peaks royally for a long time--sometimes for minutes after which I've just terminated it.

I could well try to trace the problem, but only if someone told me how to get the frickin dev_server to allow pydev to use sockets. :)

Trying to delete some records at /_ah/admin, I got the exception:

Traceback (most recent call last):
File "/Applications/_dock/dev/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/webapp/init.py", line 503, in call
handler.post(*groups)
File "/Applications/_dock/dev/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/admin/init.py", line 655, in post
datastore.Delete(datastore.Key(key))
File "/Applications/_dock/dev/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 268, in Delete
raise _ToDatastoreError(err)
File "/Applications/_dock/dev/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 2020, in _ToDatastoreError
raise errorserr.application_error
TransactionFailedError: Operation was terminated to resolve a deadlock.

The query planner erroneously generates a single index requirement when other already existing indexes would suffice. This occurs when a query with multiple equality filters is executed.

Older entities need to be updated to add 'multiple' fields if not already present.

The field 'multiple' on properties is now required. getSchema does not set this field.

Currently, getBounds erroneously returns a single property value repeatedly when called for a query with multiple equalities. It needs reworking to support multiple equalities.

I have run into a problem now though with PolyModels
I can create them fine, however if I fetch the entity I get the
following error.

c1=psc.models.Course.all().fetch(1)[0]
c2=psc.models.Campus.all().fetch(1)[0]
c2

c1

cc = psc.models.CourseCampus(key_name="abc")
cc.src_id = str(c1.key())
cc.target_id = str(c2.key())
cc.src = c1
cc.target = c2
cc.put()
datastore_types.Key.from_path(u'Reference', u'abc',_app_id_namespace=u'psc-dev1')
xx = psc.models.CourseCampus.get_by_key_name("abc")
Traceback (most recent call last):
File "", line 1, in
File "/home/timh/google_appengine_1.2.4/google/appengine/ext/db/
init.py", line 900, in get_by_key_name
return get(*keys)
File "/home/timh/google_appengine_1.2.4/google/appengine/ext/db/
init.py", line 1105, in get
model = cls1.from_entity(entity)
File "/home/timh/google_appengine_1.2.4/google/appengine/ext/db/
polymodel.py", line 310, in from_entity
raise db.KindError('No implementation for class '%s'' % key)
TypeError: not all arguments converted during string formatting

Exactly the same code works fine with standard SDK datastore

The socket_apiproxy_stub erroneously tries to close the socket at the end of the request even if it was never created.

I get different results in BDBDatastore and default datastore for the following query:

results = Query(Model).filter('created <', datetime.datetime(2009, 5, 3, 1, 59, 17)).order('-created').order('__key__').fetch(10)

There's an entity stored with created = datetime.datetime(2009, 5, 3, 1, 59, 17). It is returned as the first result where it should not.

When looking for indexes to satisfy a query, BDBDatastore checks all indexes. For more efficiency, indexes could be indexed by property type and sorted list of fields.