The use of checkXXX prevents the tests from being run by class name; instead you much specify an individual method. The test suite customizes the test names to use check instead of the standard test.

For example, python -m relstorage.tests.testpostgresql HPPostgreSQLTests does nothing instead of running all the test methods declared in HPPostgreSQLTests as one would expect.

Having to specify methods individually leads to a lot of extra typing.

Regarding #18 (comment) in #18, @seanupton posted a message about a possible lock protocol violation to the ZODB-dev list, which sadly didn't get any replies.

I traced through the transaction handling code to try to see what might be going on, and this is what I came up with:

1. When a transaction is committed, it calls tpc_begin on the registered managers. This will be a ZODB Connection, which passes the call down to its storage. At this time, RelStorge acquires the commit lock.
2. Some other stuff happens.
3. The transaction calls tpc_vote on all managers. Again, the connection passes this down to the storage. In RelStorge's case, tpc_vote calls _vote which calls _finish_store which can raise a ConflictError.
4. The transaction calls tpc_finish which again is passed down to the RelStorage where the commit lock is released in a finally block.

If any error (except:) happens between steps 1 through 4, inclusive, the transaction invokes its _cleanup method, which unconditionally calls tpc_abort on all the registered managers. The Connection passes this on to RelStorage, which again uses a finally block to release the commit lock. (Perhaps older transaction packages didn't catch all errors in this fashion?)

My conclusion is that under most circumstances, this should not result in a lock being held beyond the lifetime of a transaction. I do see a few possible holes, however:

1. The _cleanup function only catches Exception. So if a resource manager's abort or tpc_abort method raises BaseException and not Exception, tpc_abort for RelStorge might not get called.
2. The Connection's tpc_abort first aborts any active savepoint it has before passing the abort on to the storage. If this raises an exception, then tpc_abort for RelStorage might not get called.
3. Maybe some other case with savepoints? I haven't walked through all of that logic.

Case 1 should be unlikely but can't be ruled out. It wouldn't be logged by the normal handlers so it might not be immediately obvious it happened. Case 2 should also be unlikely but can't be ruled out.

All that said, I don't really see any serious downside to having RelStorage abort early on an error during voting. It should have tpc_abort get called, but there are a couple of holes, so it's just slightly safer. One possible downside I could imagine is in an exotic multi-db or multi-resource-manager situation where some of the external locks (RelStorages) get released but other's don't; that could make future transactions deadlock in weird ways, but should be no worse that the current situation.

Perhaps we should change the default. Thoughts?

Recent-ish versions of MySQL spew warnings all over the place during the test suite (and probably during production):

relstorage/relstorage/adapters/txncontrol.py:140: Warning: Invalid utf8 character string: '808182'

There is a mysql bug detailing the steps needed to resolve this.

Since there are no blockers open and several improvements done - what about a final 1.6.0 release?
We're using 1.6.0b3 in production since a year for many sites w/o any problems.
If you need any help releasing I'am happy to help out.

On attempted install the use of 'file' in setup.py:41 causes the install to bomb out. It should be 'open' right?

Beyond this how compatible is RelStorage with python3

At startup, this would be used to pre-populate the local in-memory cache.

In order to do this efficiently, we need to keep track of the OIDs that were in use (in the local cache) by the instance when it shut down so we'd know what data to get. We could do this manually by parsing the cache keys that are found in the local cache and writing them to disk.

From there, we'd need to get the object states somehow. Again, we could do this manually by launching a series of queries at the RDBMS based on the OIDs we discovered in step one.

However, instead of doing all this manually, we could simply use zeo.cache.ClientCache.

While this claims to be "a simple in-memory cache", in reality all the data is on the disk, and all that's in memory are some relatively small integer maps. It takes care of managing its disk size and if we load and store to it like we do our local cache, it will keep an LRU structure on disk for us.

At startup, we would open the ClientCache, and then we'd use our normal Poller to detect any OIDs and TIDs to invalidate (we'd need to cap it to implement "drop rather than verify" if we've been disconnected for a very long time) and pass those on to the cache. Finally, we'd call cache.contents() to read all the OID/TID/state triples off disk and populate our in-memory cache in one go (we could even skip this step, potentially, although that would complicate future loads).

Another benefit is that we would gain access to the ZEO tracing tools to examine our cache efficiency.

This should be pretty simple; the only tricky part is making sure each RDMBS storage loads the right current data. I think all we need to do is pass storage._prev_polled_tid (guaranteed to be current as if the start of the transaction) down through StorageCache.load() (new parameter) and into ClientCache.load(oid, before) or maybe ClientCache.loadBefore(oid, before). (The StorageCache might already have the info it needs given in StorageCache.after_poll.)

To make this really useful, we'd like to be able to configure a directory to hold up-to a specified number of cache files and have the client automatically choose the next available cache file (perhaps copying it from the newest cache file if there isn't one). This is better than the ZEO approach of requiring each client to have a distinct name in the configuration file; that falls down when using forking servers like gunicorn. Ideally this idea make it back to ZEO.

using as source a filesystem+blobstorage where blobstorage is not allowed access due to missing permissions, blobs are silently ignored. I'd expect here at least a warning.

Hi all

I'm getting an error when shutting down my server in my development environment.
Plone 4.3.3
RelStorage = 1.5.1

sauna and get the same issue when CTRL + C'ing the insance.

[instance]
<= instance_base
recipe = plone.recipe.zope2instance
http-address = 8080
blob-dir ${buildout:var-dir}/blobstorage
dsn dbname = ${buildout:database-name} host=${buildout:database-host} user=${buildout:database-user}

^C2014-06-17 14:15:21 INFO SignalHandler Caught signal SIGINT
2014-06-17 14:15:21 INFO Z2 Shutting down
Traceback (most recent call last):
  File "/home/neilf/development/website/website-website.wa.gov.au-v2/parts/instance/bin/interpreter", line 328, in <module>
    exec(compile(__file__f.read(), __file__, "exec"))
  File "/home/neilf/development/website/buildout-cache/eggs/Zope2-2.13.22-py2.7.egg/Zope2/Startup/run.py", line 76, in <module>
    run()
  File "/home/neilf/development/website/buildout-cache/eggs/Zope2-2.13.22-py2.7.egg/Zope2/Startup/run.py", line 26, in run
    starter.run()
  File "/home/neilf/development/website/buildout-cache/eggs/Zope2-2.13.22-py2.7.egg/Zope2/Startup/__init__.py", line 108, in run
    self.shutdown()
  File "/home/neilf/development/website/buildout-cache/eggs/Zope2-2.13.22-py2.7.egg/Zope2/Startup/__init__.py", line 113, in shutdown
    db.close()
  File "/home/neilf/development/website/buildout-cache/eggs/ZODB3-3.10.5-py2.7-linux-x86_64.egg/ZODB/DB.py", line 624, in close
    @self._connectionMap
  File "/home/neilf/development/website/buildout-cache/eggs/ZODB3-3.10.5-py2.7-linux-x86_64.egg/ZODB/DB.py", line 506, in _connectionMap
    self.pool.map(f)
  File "/home/neilf/development/website/buildout-cache/eggs/ZODB3-3.10.5-py2.7-linux-x86_64.egg/ZODB/DB.py", line 206, in map
    self.all.map(f)
  File "/home/neilf/development/website/buildout-cache/eggs/transaction-1.1.1-py2.7.egg/transaction/weakset.py", line 58, in map
    f(elt)
  File "/home/neilf/development/website/buildout-cache/eggs/ZODB3-3.10.5-py2.7-linux-x86_64.egg/ZODB/DB.py", line 628, in _
    c._release_resources()
  File "/home/neilf/development/website/buildout-cache/eggs/ZODB3-3.10.5-py2.7-linux-x86_64.egg/ZODB/Connection.py", line 1075, in _release_resources
    c._storage.release()
AttributeError: 'NoneType' object has no attribute 'release'

With MySQL after having changed all remaining MyISAM tables (not crash safe) to InnoDB, at the first write one transaction holds an exclusive lock and that transaction is never closed (commit or rollback). When using MyISAM it seems to work but that doesn't scale and it's not safe either. I think this is just because MyISAM just ignore any transaction related statements as it's not supported.

So back to InnoDB, it's very bad to have large transactions especially when the thread holding that transaction is sleeping (this is the case here). It's bad as it locks other statements but also because it consumes undo logs (history list to purge later will also increase dramatically and may freeze the full db).

So if we check the processlist::

| 72 | user | xx.xx.xx.xx:54312 | zope | Sleep | 97 | | NULL | 1 | 1 |
| 73 | user | xx.xx.xx.xx:54319 | zope | Query | 97 | update | REPLACE INTO new_oid VALUES(1504) | 0 | 0 |
| 74 | user | xx.xx.xx.xx:54320 | zope | Sleep | 97 | | NULL | 0 | 0 |

So we can see that thread 73 updating new_oid is taking a lot of time (97sec in this case)... (this replace one record !)

If we check in InnoDB status output at that time, we can see::

---TRANSACTION 6700, ACTIVE 94 sec inserting
mysql tables in use 1, locked 1
LOCK WAIT 2 lock struct(s), heap size 360, 1 row lock(s)
MySQL thread id 73, OS thread handle 0x7fbd71b17700, query id 4124 xx.xx.xx.xx user update
------- TRX HAS BEEN WAITING 94 SEC FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 10 page no 3 n bits 576 index PRIMARY of table zope.new_oid trx id 6700 lock_mode X locks rec but not gap waiting

TABLE LOCK table zope.new_oid trx id 6700 lock mode IX
RECORD LOCKS space id 10 page no 3 n bits 576 index PRIMARY of table zope.new_oid trx id 6700 lock_mode X locks rec but not gap waiting
---TRANSACTION 6699, ACTIVE 94 sec
MySQL thread id 72, OS thread handle 0x7fbd71b58700, query id 4115 xx.xx.xx.xx user cleaning up
Trx read view will not see trx with id >= 6700, sees < 6690
---TRANSACTION 6690, ACTIVE 98 sec
2 lock struct(s), heap size 360, 251 row lock(s), undo log entries 498
MySQL thread id 74, OS thread handle 0x7fbd71ad6700, query id 4122 xx.xx.xx.xx user cleaning up
TABLE LOCK table zope.new_oid trx id 6690 lock mode IX
RECORD LOCKS space id 10 page no 3 n bits 576 index PRIMARY of table zope.new_oid trx id 6690 lock_mode X locks rec but not gap

So the transaction 6690 open by thread id 74, is open since 98 secs and holds a lock on the table... why is this transaction not committed as in the processlist we can see it's in "sleep" mode ?

We would like to implement support for a few new databases (MS SQL Server for production #29, SQLite for testing). Maybe we'd even like to support some sort of simple plug-in system so database support can be contributed and distributed as third-party packages. This currently seems challenging due to the organization of the code.

Currently, code to support a specific database is scattered in several places:

• adapters/<dbname>py
  The main knowledge about how to connect to a database and pull together all the other db-specific pieces. Each file defines an implementation of IRelStorageAdapter and a subclass of AbstractConnectionManager. (oracle.py also defines another subclass of OracleScriptRunner from adapters/scriptrunner.py)
• adapters/batch.py
  Each supported database has a subclass of RowBatcher here.
• adapters/locker.py
  Each supported database has a subclass of Locker here.
• adapters/oidallocator.py
  Each supported database has a unique implementation of IOIDAllocator here.
• adapters/packundo.py
  Each database has two subclasses of a PackUndo class.
• adapters/scriptrunner.py
  Has a subclass of ScriptRunner for oracle.
• adapters/stats
  Has a custom object (with no interface or superclass) for each database.
• adapters/txncontrol.py
  Has a subclass of TransactionControl for each object.

There are two files that are particularly difficult.

• adapters/mover.py
  This contains one class, ObjectMover, that has three versions of each method, one for each supported database. At runtime, it selects a set of methods to use based on the database adapter in use.
• adapters/schema.py
  In addition to a number of strings named for each database, there are two large strings (one history-free, one history-preserving) that have a custom syntax to make the specific statement to run to create the same logical object for each database.

I believe these files (especially the last two) are organized like this to keep similar objects together across database in order to make it easier to maintain parity between them.

However, it has a couple of less-nice consequences too. A person attempting to understand what's specific about MySQL, say, has to visit all of these other files, and wade through unrelated code. Similarly, trying to add support for a new database requires editing every file in the adapters directory, which carries the possibility of unintended changes for other databases.

Maybe we can find a nice middle ground.

This will be necessary for gevent support. Both psycopg2 and psycopg2cffi call through to the native libpq, which manages its own native sockets and can't be portably monkey patched.

It looks like the best supported is probably pg8000. If there's a better one, please let me know!

Is there a plan to support MS SQL Server? What would it take, coding and possibly $$$ wise? Thank you

I forgot to do that for 2.0b1 and had to upload two wheels 😦

Currently, only a single conflict is read at a time and we loop until all conflicts are gone. This introduces a network roundtrip to the DB for every conflict. The reason for this is to avoid loading all the pickled data for all the conflicts into memory at once.

However, once #38 is resolved, that will no longer be a concern and we can eliminate N-1 network round trips, increasing performance in the case of many conflicts.

Upsides: increased performance.

Downsides: changing conflict resolution code which historically has been tricky. However, coverage data shows this to be well tested.

Right now, none of the functional tests exercise memcache or relstorage.pylibmc_wrapper.

It would probably be good to have a configuration that did so, at least on the CI server.

This would let us break the long README down into smaller, easier pieces (and maybe auto-generate some of them?). Plus we'd get better formatting.

The remaining README on PyPI would be much simpler.

check-manifest and pyroma complain about a few hygiene things.

Hi people.
I am using Plone and I am interisted to make use of RelStorage along with MariaDB. Instead of the standard InnoDB / XtraDB Engine, I'd love to make use of a different one at the time RelStorage is creating the tables.

Could somebody of you please tell me how to accomplish that.

best, Tamer Higazi

This class is part of the stdlib starting with 2.6 and above and will let us remove a chunk of custom code+custom tests.

A sqlite backend could be useful for testing, especially for people that are making changes to the higher level, database-independent parts of RelStorage. This way they don't have to have a complete RDBMS up and running locally.

It might also serve as an interesting comparison against FileStorage, especially when used with ZEO 😉 I know that there was once a Berkeley DB based-storage.

This requires some investigation to make sure the semantics we rely on are fully supported.

Depends on #77.

Stacking demostrage with relstorage does not reconnect when encountering disconnected_exceptions

zope.conf:

<zodb_db main>
    # Main database
    cache-size 100000
    <demostorage>
    %import relstorage
    <relstorage base>
        commit-lock-timeout 240
        blob-dir /opt/plone/responsive/var/blobcache
        blob-cache-size 10gb
        shared-blob-dir false
        <mysql>
            passwd secretpassword
            db zodb
            host localhost
            user zope
            port 3306
        </mysql>
    </relstorage>
    # Blob-enabled FileStorage database
    <blobstorage changes>
      blob-dir /opt/plone/responsive/var/demostorage/blobstorage
      # FileStorage database
      <filestorage>
        path /opt/plone/responsive/var/demostorage/Data.fs
      </filestorage>
    </blobstorage>
    </demostorage>
    mount-point /
</zodb_db>

traceback:

2014-05-27T09:47:06 ERROR ZODB.Connection Couldn't load state for 0x1e7708
Traceback (most recent call last):
  File "/opt/plone/buildout/eggs/ZODB3-3.10.3-py2.6-linux-x86_64.egg/ZODB/Connection.py", line 860, in setstate
    self._setstate(obj)
  File "/opt/plone/buildout/eggs/ZODB3-3.10.3-py2.6-linux-x86_64.egg/ZODB/Connection.py", line 901, in _setstate
    p, serial = self._storage.load(obj._p_oid, '')
  File "/opt/plone/buildout/eggs/ZODB3-3.10.3-py2.6-linux-x86_64.egg/ZODB/DemoStorage.py", line 159, in load
    return self.base.load(oid, version)
  File "/opt/plone/buildout/eggs/RelStorage-1.5.1-py2.6.egg/relstorage/storage.py", line 457, in load
    state, tid_int = cache.load(cursor, oid_int)
  File "/opt/plone/buildout/eggs/RelStorage-1.5.1-py2.6.egg/relstorage/cache.py", line 279, in load
    state, tid_int = self.adapter.mover.load_current(cursor, oid_int)
  File "/opt/plone/buildout/eggs/RelStorage-1.5.1-py2.6.egg/relstorage/adapters/mover.py", line 130, in mysql_load_current
    cursor.execute(stmt, (oid,))
  File "/opt/plone/buildout/eggs/MySQL_python-1.2.3-py2.6-linux-x86_64.egg/MySQLdb/cursors.py", line 174, in execute
    self.errorhandler(self, exc, value)
  File "/opt/plone/buildout/eggs/MySQL_python-1.2.3-py2.6-linux-x86_64.egg/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue
OperationalError: (2006, 'MySQL server has gone away')

The same thing is happening in base ZODB over in zopefoundation/ZODB#72

It would be nice if RelStorage was able to run its tests automatically on PRs/commits.

I think this should be (mostly) doable. Travis CI offers support for MySQL and PostgreSQL databases, and Appveyor offers SQL Server (which isn't supported at all yet).

I can take a look to see what it would take to put this together and open a PR. It might involve adding tox support as well.

Reference

Things I think of as I work on this.

• The test extra needs to install zope.testing
• Tests (reltestbase) must transition to zodbpickle. Code in relstorage.storage, etc, should as well.

currently zodbconvert.py succesfully converting Data.fs to Relstorage.
But What about the Data.fs.index?

There are numerous methods, try/catch statements and the like dedicated to supporting old ZODB releases, mostly 3.9 and below.

ZODB 3.10 was released in October of 2010, nearly six years ago, and 4.0b1 is three years old.

The code contains a comment that "RelStorage needs to continue to support ZODB 3.8 and 3.7 for a few years". That comment was written in late 2010 or early 2011.

I suspect we can safely move on now.

Some of this was already incidentally done during the Python 3, PyPy, and ZODB5-forward compatibility efforts, so it's unlikely that current RelStorage would function on ZODB < 3.9 at the oldest as it is.

Using MAX instead of ORDER BY DESC LIMIT 1 produces a much better query plan, even in large databases, so theoretically it should perform better. In practice I haven't been able to measure a difference.

See also a comment in #87.

This is for a now a RFC. After looking deeper into PostgreSQL and our Relstorage issues I started to wonder if Relstorage could not leverage the MVCC support in PostgreSQL directly.
I have NOT yet done a deeper look into Relstorage with regard to this, but I plan to
@saily @jensens

optparse is deprecated and argparse is easier to use.

This affects zodbpack.py and zodbconvert.py.

packing the ZODB gives an error when the setting pack-gc=true . It works fine when pack-gc=false

Relstorage versions working: 1.5.1 and 1.6.0b1
Relstorage versions not working: 1.6.0b2 and 1.6.0b3

database: Oracle11

bin/zodbpack bin/zodbpack.cfg                                                                        
2016-04-22 16:38:17,956 [zodbpack] INFO Opening storage (RelStorageFactory)...
2016-04-22 16:38:18,215 [zodbpack] INFO Packing storage (RelStorageFactory).
2016-04-22 16:38:18,314 [relstorage] INFO pack: analyzing transactions committed Fri Apr 22 16:24:57 2016 or before
2016-04-22 16:38:18,361 [relstorage.adapters.packundo] INFO pre_pack: start with gc enabled
2016-04-22 16:38:18,505 [relstorage.adapters.packundo] INFO analyzing references from objects in 8418 new transaction(s)
2016-04-22 16:38:18,513 [relstorage.adapters.packundo] ERROR pre_pack: failed
Traceback (most recent call last):
  File "/demo/eggs/RelStorage-1.6.0b2-py2.7.egg/relstorage/adapters/packundo.py", line 504, in pre_pack
    conn, cursor, pack_tid, get_references)
  File "/demo/eggs/RelStorage-1.6.0b2-py2.7.egg/relstorage/adapters/packundo.py", line 598, in _pre_pack_with_gc
    self.fill_object_refs(conn, cursor, get_references)
  File "/demo/eggs/RelStorage-1.6.0b2-py2.7.egg/relstorage/adapters/packundo.py", line 404, in fill_object_refs
    self._add_refs_for_tid(cursor, tid, get_references)
  File "/demo/eggs/RelStorage-1.6.0b2-py2.7.egg/relstorage/adapters/packundo.py", line 443, in _add_refs_for_tid
    state = state.read()
ProgrammingError: LOB variable no longer valid after subsequent fetch

zodbpack.cfg

<relstorage>
  pack-gc true
  <oracle>
    dsn  (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = host)(PORT = 1234)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = PLONE)))
    user plone
    password pwd
  </oracle>
</relstorage>

Currently, RelStorage effectively has an internal ordered list of supported alternative DB API drivers for those adapters that support multiple drivers (MySQL and PostgreSQL). It always uses the first one that it can import.

However, "explicit is better than implicit" and as the lists grow ( #85 #79 ), the chances of getting an undesirable driver do too. The consequences can be dramatic. For example, PyMySQL is reported to be 30% slower than MySQL-python under CPython, but the exact opposite is true under PyPy. The pure-python drivers support gevent, the native ones do not, leading to very hard to debug issues if you're using gevent greenlets to handle concurrency.

So it would be good if there was a way to specify the desired driver in the configuration. The default of "auto" would be the current behaviour. Any specified value would fail to initialize the storage if it cannot be found.

This may be enhanced by #77 or similar refactoring.

It's gevent compatible but also implemented in C for speed.

We've been using it in production for years. It can be substantially faster than PyMySQL, while still being gevent compatible. Here's a table (stolen from ) that shows this:

Hi,

Recently we realized Relstorage seems to start and never close a database transaction after the first write. This database connection is reported to be <IDLE> in transaction. See the query below.

Note: this idling transaction is only started after adding an object to a "container" - in our testing setup, it was adding an object() to the Zope application root. Storing an attribute on the app root, seems to not result in leaving an idleing transaction.

We suspect that this idling transaction keeps postgres from successfully running its auto vacuuming task.

Is this expected behaviour? Do I need to provide more details?

Kind regards, jw

postgres=# SELECT procpid, current_timestamp - xact_start AS xact_runtime, current_query
FROM pg_stat_activity ORDER BY xact_start;
 procpid |      xact_runtime      |                                 current_query                                 
---------+------------------------+-------------------------------------------------------------------------------
   25473 | 2 days 04:21:24.419343 | <IDLE> in transaction
   13115 | 1 day 16:33:19.276329  | <IDLE> in transaction
   32072 | 1 day 14:35:30.829614  | <IDLE> in transaction
    1577 | 00:00:00.631836        | <IDLE> in transaction
   26561 | 00:00:00               | SELECT procpid, current_timestamp - xact_start AS xact_runtime, current_query
                                  : FROM pg_stat_activity ORDER BY xact_start;

Some thoughts:

• Replace easy_install with pip
• Ensure other best practices are current and default values match, etc.
• Dependencies: Python 3 will be different than Python 2 (mysql)
• Document incremental option for zodbconvert
• Update supported list of ZODB versions
• Rename file to README.rst for better rendering and editing.

We've been carrying around this patch:

from relstorage.storage import RelStorage

_RelStorage_release = RelStorage.release

def _release(self):
    try:
        _RelStorage_release(self)
    finally:
        for client in self._cache.clients_local_first:
            disconnect = getattr(client, 'disconnect_all', None)
            if disconnect:
                logger.debug("Explicitly disconnecting memcache %s", client)
                disconnect()

RelStorage.release = _release

With this description:

RelStorage can use a memcache instance to provide tremendous speed benefits. Relstorage also implements :class:.IMVCCStorage, which means its new_instance and release methods are called frequently. These methods can allocate another memcache connection, but they never explicitly release it, instead relying on Python's reference counting to release it. However, if there is a cycle, the connection's reference count may never go to zero, leaving behind a stale, useless open connection.
If we're on PyPy, the connections can hang around longer than desired waiting for a GC.

And this TODO:

This same approach can be used for memcache client pooling.

This is because the new pickled data is already guaranteed to be in the local cache (an AutoTemporaryFile/SpooledTemporaryFile).

Upsides: less network IO (and at least in some DBs, the pickled data is base64 encoded, so lower overhead to decode). So theoretically it should be faster, especially if there are many conflicts.

Downsides: changing the conflict resolution code. On the other hand, this code path is pretty well tested.

This is necessary for zc.zlibstorage to work. Currently, the implementation is a no-op.

We've been carrying around this patch:

        from relstorage.storage import RelStorage

        def registerDB(self, db):
                # We know the current implementation is a no-op,
                # so we simply replace it. This needs to be checked
                # when the version changes.
                try:
                        super(RelStorage, self).registerDB(db)
                        raise TypeError("Internals changed, check patch")
                except AttributeError:
                        # We expect the MRO to be
                        # [<class 'relstorage.storage.RelStorage'>,
                        #  <class ZODB.UndoLogCompatible.UndoLogCompatible>,
                        #  <class 'ZODB.ConflictResolution.ConflictResolvingStorage'>,
                        #  <type 'object'>]
                        # So when ConflictResolvingStorage calls super(), it raises
                        # this attribute error
                        pass

        RelStorage.registerDB = registerDB

NOTE: The AttributeError is no longer raised as of ZODB 4.3.0.

When packing on a relstorage with a corrupted ZODB (broken index with BTree buckets), we got Unpickling errors. We got plone back working by reindexing the broken catalogs. But packing still was an issue.

On Filestorage the packing went fine, Relstorage still gave unpickling errors on the broken buckets. As far as I understood the code of Filestorage and Relstorage, the issue is caused by the following.
Filestorage loads the root object, finds the objects it refers to and does the same for these objects, putting that info in a list. All objects that are not in this list, can be removed by pack.
Relstorage seems to load all transactions in pre_pack, unpickling all objects in these transactions, and build a list of referenced objects with that information, so pack can decide which objects to remove.

The fundamental difference seems to be in the creation of the list. Filestorage does not access the old, broken index objects. Relstorage seems to access these broken objects anyway.
Our temporary workaround is by patching the ZODB/serialize.py, by putting the two u.noload() in referencesf() in a try/except block, where we just move on (pass) in case of an exception. This ignores the broken index bucket object, the correct list for pack seems to be build and the broken index bucket objects are removed from the ZODB.

Could you provide some feedback if my mental model is correct (or totally not), eventually a suggestion for a better workaround?
I guess the cleanest solution would be if Relstorage could use the same approach as Filestorage to build up the list of what to keep and what to remove. But I also guess that’s not the best solution because of performance or other issues.

Why are all connections left in state idle in transaction? is this a general requirement to support zodb/zope or would it be possible to release that transaction?

History-free storage was done in #47 . The documentation for IExternalGC implies that the process may be more complicated for history-preserving storages ("This method marks an object as deleted via a new object revision...The object will be removed from the storage when all not-delete records are removed."), due to the use of non-local information, but I haven't really looked into it.

However, I'm under the impression from something that I read somewhere that many uses of RelStorage are history-free, so I currently personally prioritize this fairly lowly and don't intend to hold up the 1.7 release for it. But it's still nice for completeness and for those that need history preservation.

This requires implementing one method that deletes an individual object (revision? I've only done this in history-free schemas).

Right now this would have to be integrated, which means a lot of plumbing through the adapter layer, but zc.zodbdgc can be changed to use zope.interface to request an alternate implementation (adapter).

Using master plus ZODB 4.X, zodbconvert output of a database where the target RelStorage already held several million objects, and the source FileStorage did as well:

2014-06-23 06:10:50,597 [zodbconvert] INFO Storages opened successfully.
2014-06-23 06:10:50,597 [zodbconvert] INFO Clearing old data...
^C

The ^C was pressed six hours later. I'm still waiting for the process to be aborted. The resource usage on the machine was light (no swapping). It seems as if the code to clear the database hadn't completed, and I'd contend that it should have within six hours.

1. Starting size of blobstorage directory: 119M
2. Add a ~1GB file to portal. blobstorage size: ~1.3GB
3. Delete file from zope and run zodbpack
4. Postgres db size is fine, but blobstorage directory remains ~1.3GB

Perhaps I'm missing something but my understanding was that packing was supposed to remove blobs on the file system that no longer have any references pointed to them.

buildout.cfg snippet:

rel-storage =
keep-history false
type postgresql
blob-dir /sprj/btp_zope_plone4/blobstorage/maize
dsn dbname='${databases:zodb}' user='${users:zodb}' host='${hosts:zodb}' password='${passwords:zodb}' port='${ports:zodb}'

zpack.cfg snippet:

<relstorage>
pack-gc true
<postgresql>
dsn dbname='xxx' user='xxx' host='xxx' password='xxx' port='xxx'
</postgresql>
keep-history false
</relstorage>

I did find this issue zopefoundation/ZEO#2 but my understanding is that this is not analogous to RelStorage because with shared-blob-dir = true (default for RelStorage) here there is no blobcache and setting a blob cache size value would be ignored. How do prevent the blobstorage dir from growing indefinitely?

Tested with 1.6.0b3 and 1.5.0.

Hi,

we are using here relstorage at a site with about 300k plone content items (dexterity based). So, it performs better than ZEO-Server. But we now got lots of conflict errors with relstorage (both, mysql and postgresql) on hi write scenarios, i.e. when an import is running and editors are working. We already have this distributed to different zope instances and machines.

At least on mysql it seems the locking strategy is very paranoid. The whole database i locked explicit. Max, the database expert of our service provider, nailed it down to these locks, see https://github.com/zodb/relstorage/blob/master/relstorage/adapters/locker.py#L111 It means other db connections cant write to the DB while it is locked. Also at transaction commit time it waits at worst case the given timeout.

But as far as i understand it is sufficient to take care only about modified objects, those joined to the transaction and identified by oid. Max proposed to keep a table of locked oid's instead of locking the whole thing. According to him using the usal mysql transactions its then possible get information about modified objects on oid level.

I didnt digged deeper for now on this topic. But in my opinion current heavy locking behaviour is a real stopper for hi-write scenarios. It really blocks writing at some point.

Btw. Postgresql had an about 10-20% better write performance on hi write load. We cloned our complete environment (4 vms) and on thd db-vm we replaced mysql by postresql. But on hi-write it had the same problems. As I see from code it does table based locking and it seems since we have only a few tables this is finally the same as mysql database locks.

I'd like to read some opinions about this, also some background why this method of locking was choosen. I'am sure there is a good reason. May be simplicity? But if more complexity stops blocking on write I'd prefer it.

We would like to contribute making this work better if its possible.

regards Jens

This will make them more accessible.

Some of them may belong as doc comments in the source, possibly.

If it doesn't make sense to convert them, it probably makes sense to drop them (they'll still be in the historical versions).

Today I tried to incrementally convert a Data.fs into a Postrgres-backed Relstorage using the HEAD revision of relstorage master branch. The initial conversion succeeded normally. A second run using the --incremental option raised an error however:

$ ./bin/zodbconvert parts/etc/zodbconvert.ini --incremental
2013-07-19 14:02:05,687 [ZODB.blob] WARNING (34583) Blob dir /home/jw/dev/var/blobstorage/ has insecure mode setting
2013-07-19 14:02:05,732 [ZODB.blob] WARNING (34583) Blob dir /home/jw/dev/var/cache/blob/ has insecure mode setting
2013-07-19 14:02:05,733 [zodbconvert] INFO Storages opened successfully.
Traceback (most recent call last):
File "./bin/zodbconvert", line 20, in
sys.exit(relstorage.zodbconvert.main())
File "/home/jw/dev/src//Relstorage/relstorage/zodbconvert.py", line 99, in main
destination._load_cursor)
File "/home/jw/dev/src/Relstorage/relstorage/adapters/txncontrol.py", line 73, in get_tid
cursor.execute(stmt)
AttributeError: 'NoneType' object has no attribute 'execute'

It seems the code for the --incremental code that was put in a fairly long time ago was never really tested?

This feature would be a most helpful addition to zodbconvert.

Please let me know if I can provide additional details or testing.

Some common refactoring of the existing tests Postgres resolver tests should make this easy to do.

In the words of RelStorage's creator Shane Hathaway, " the "poll-interval" option is an unnecessary feature. I think it's harmful, in fact." ZODB creator Jim Fulton agrees, writing "I think just
polling in poll_invalidations would make it a lot easier for me to reason about. :) I'd also remove sync from this dance altogether at the IMVCCStorage layer."

Our organization has been using the recommended cluster configuration of a cache-server with a 60s poll-interval, as documented. We have run into a few rare, hard-to-debug instances across the cluster where it's obvious that the caches are out of sync, and so we looked into poll-interval more closely.

What the documentation doesn't mention is that if there's any problem with memcache at all, such as the connection being dropped, or a certain key being evicted, then the polling that's required won't take place. This is confounded by the fact that the python-memcached module, at least, nearly silently drops broken connections, and relstorage moves on to the next cache at that point.

Plus, when polling is required (which happens for every connection whenever there's been a commit by any other connection) it just adds overhead: we make a call to memcache, which says yes, then we go ahead and make the call to the DB; the memcache call is superfluous. The situation would be different if there were practically 0 writes, but in a large cluster given that there are hundreds of connections, even a tiny number of writes will cause lots of overhead for all the other connections.

So this proposes to remove the poll-interval option as a loaded footgun. For the 2.0 release, we will continue to accept it in the configuration but will warn if it's non-zero and ignore it. Future releases may silently ignore it and/or drop it altogether.

This is necessary to work with zc.zlibstorage. (This could actually be a bug in ConflictResolvingStorage)

We've been carrying around this patch:

        from relstorage.storage import RelStorage

        # It could be argued
        # that ConflictResolvingStorage needs to implement this method
        # and RelStorage call super
        orig_new_instance = RelStorage.new_instance
        def new_instance(self):
                new_instance = orig_new_instance(self)
                new_instance._crs_transform_record_data = self._crs_transform_record_data
                new_instance._crs_untransform_record_data = self._crs_untransform_record_data
                return new_instance
        RelStorage.new_instance = new_instance

I've a 80GB relstorage zodb running on MySQL 5.6. I'm using Relstorage 1.5.1 with Plone 4.3.1.

I've configured periodical packing which fails after analysing table structure.

After digging into that i figured out this select statement: https://github.com/zodb/relstorage/blob/master/relstorage/adapters/packundo.py#L85 which returns a huge result set and oom-killer invokes a kill.

zodbpack.conf:

<relstorage>
    pack-gc true
    create-schema false
    keep-history false
    shared-blob-dir true
    blob-dir /var/blob_storage/zodb_plone
    commit-lock-timeout 600
    <mysql>
        db     zodb_plone
        host   db.host.com
        user   plone
        passwd *****
        connect_timeout 30
    </mysql>
</relstorage>

process log:

[zope@pl51 plone]$ bin/zodbpack etc/zodbpack.conf
2013-10-07 08:47:03,282 [zodbpack] INFO Opening storage (RelStorageFactory)...
2013-10-07 08:47:03,312 [ZODB.blob] WARNING (43732) Blob dir /var/blob_storage/zodb_plone/ has insecure mode setting
2013-10-07 08:47:03,313 [zodbpack] INFO Packing storage (RelStorageFactory).
2013-10-07 08:47:03,389 [relstorage] INFO pack: analyzing transactions committed Tue Oct  1 15:56:49 2013 or before
2013-10-07 08:47:03,392 [relstorage.adapters.packundo] INFO pre_pack: filling the pack_object table
2013-10-07 08:53:24,517 [relstorage.adapters.packundo] INFO pre_pack: downloading pack_object and object_ref.
Killed

machine memory:

[zope@pl51 ~]$ free -m
             total       used       free     shared    buffers     cached
Mem:         15943        424      15519          0         16         32
-/+ buffers/cache:        375      15568
Swap:         2047         63       1984

oom-killer kills zodbpack at 3.6 GB
any ideas?

It's not documented and it's been the default since at least ZODB 3.10.