zopefoundation / persistent Goto Github PK

View Code? Open in Web Editor NEW

46.0 68.0 28.0 1.03 MB

automatic persistence for Python objects

Home Page: https://pypi.org/project/persistent/

License: Other

Python 70.85% C 28.48% Shell 0.67%

maintained

persistent's Introduction

`persistent`: automatic persistence for Python objects

This package contains a generic persistence implementation for Python. It forms the core protocol for making objects interact "transparently" with a database such as the ZODB.

Please see the Sphinx documentation (docs/index.rst) for further information, or view the documentation at Read The Docs, for either the latest (https://persistent.readthedocs.io/en/latest/) or stable release (https://persistent.readthedocs.io/en/stable/).

Note

Use of this standalone persistent release is not recommended or supported with ZODB < 3.11. ZODB 3.10 and earlier bundle their own version of the persistent package.

persistent's People

Contributors

Stargazers

Watchers

persistent's Issues

Test failure in test_hash_equal on py2.6 windows

Commit 2a8313c breaks tests on winbot

see here:

http://winbot.zope.org/builders/persistent_py_265_32/builds/971/steps/test/logs/stdio

Issue with _p_mtime when mocking time with python-libfaketime

python-libfaketime is a great python wrapper for libfaketime. It catches system-calls related to date and time, and allows you to mock them as you want:

>>> # after eval $(python-libfaketime)
>>> import libfaketime, datetime
>>> with libfaketime.fake_time("2015-01-01"):
...     datetime.datetime.now()
datetime.datetime(2015, 1, 1, 0, 0)

The code of python-libfaketime is outrageously short and simple (< 300 LOC), and for what I have played with, it works great in many contexts, including C extensions. However, I encountered an unexpected behavior with ZODB and the persistent _p_mtime attribute.

If a ZODB database object is created in a libfaketime context, after editing some persistent objects, _p_mtime value is the mocked time, as expected:

>>> import datetime, libfaketime, ZODB
>>> with libfaketime.fake_time("2015-01-01"):
...     db = ZODB.DB(None)
...     connection = db.open()
...     connection.root()["a"] = 1
...     connection.transaction_manager.commit()
...     timestamp = connection.root()._p_mtime
...     db.close()
...     datetime.datetime.fromtimestamp(timestamp)
datetime.datetime(2015, 1, 1, 0, 0)

But if the ZODB database object is created outside the libfaketime context, _p_mtime value is not mocked:

>>> import datetime, libfaketime, ZODB
>>> db = ZODB.DB(None)
>>> with libfaketime.fake_time("2015-01-01"):
...     connection = db.open()
...     connection.root()["a"] = 1
...     connection.transaction_manager.commit()
...     timestamp = connection.root()._p_mtime
...     db.close()
...     datetime.datetime.fromtimestamp(timestamp)
datetime.datetime(2019, 8, 25, 10, 25, 15, 652691)

I would have expected the latter example to print a mocked time since the modification and the commit are done in the libfaketime context.

So I know that bug report related to different libraries that were not designed to work together are sometimes tricky, but I though someone here might have a clue.

What happens during the ZODB.DB creation that would need to happen in a libfaketime context?
Is it something we can easily fix in ZODB?

Thank you for your help

Automatic PyPI uploads are failing: `zope.wheelbuilder` needs a verified primary email

See this build for 4.3.0:

HTTPError: 400 Client Error: User 'zope.wheelbuilder' does not have a verified primary email address. Please add a verified primary email before attempting to upload to PyPI. See https://pypi.org/help/#verified-email for more information.for more information. for url: https://upload.pypi.org/legacy/

Warehouse issue: pypi/warehouse#3632

DeprecationWarning: PY_SSIZE_T_CLEAN will be required for '#' formats

I've been testing zodbbrowser on Python 3.8.0a3, and I noticed new deprecation warnings:

/home/mg/src/zodbbrowser/.tox/py38/lib/python3.8/site-packages/ZODB/FileStorage/FileStorage.py:316: DeprecationWarning: PY_SSIZE_T_CLEAN will be required for '#' formats
  self._ts = tid = TimeStamp(tid)
/home/mg/src/zodbbrowser/.tox/py38/lib/python3.8/site-packages/ZODB/FileStorage/FileStorage.py:1154: DeprecationWarning: PY_SSIZE_T_CLEAN will be required for '#' formats
  d.update({"time": TimeStamp(h.tid).timeTime(),
/home/mg/src/zodbbrowser/.tox/py38/lib/python3.8/site-packages/zodbbrowser/browser.py:483: DeprecationWarning: PY_SSIZE_T_CLEAN will be required for '#' formats
  ts = TimeStamp(d.tid).timeTime()
...

There are many more, but they all mention TimeStamp(...), so I assume that's where the bug is.

These were introduced by https://bugs.python.org/issue36381 to warn about an upcoming Python C API change. The meaning of PY_SSIZE_T_CLEAN is described in https://python.readthedocs.io/en/stable/c-api/arg.html#strings-and-buffers.

What ZODB needs to do is:

find all invocations of PyArg_Parse and related functions that use # formats
make sure the type of the length argument is a Py_ssize_t
#define PY_SSIZE_T_CLEAN above the #include <Python.h>.

PY_SSIZE_T_CLEAN was introduced in Python 2.5 so there shouldn't be any backwards-compatibility issues.

Must `_p_mtime` activate a ghost?

The C implementation does forcibly activate it, while the pure-Python version (until #20) did not.

See: #20

Python3: AssertionError: TypeError not raised by <lambda>

When you run the tests under python3 I get the following error:

$ python3 setup.py test -q
running test
running egg_info
...
======================================================================
FAIL: test_assign_p_estimated_size_wrong_type (persistent.tests.test_persistence.CPersistentTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/src/RPM/BUILD/python3-module-persistent-4.0.6/persistent/tests/test_persistence.py", line 506, in test_assign_p_estimated_size_wrong_type
    lambda : setattr(inst, '_p_estimated_size', int(1)))
AssertionError: TypeError not raised by <lambda>

======================================================================
FAIL: test_assign_p_estimated_size_wrong_type (persistent.tests.test_persistence.PyPersistentTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/src/RPM/BUILD/python3-module-persistent-4.0.6/persistent/tests/test_persistence.py", line 506, in test_assign_p_estimated_size_wrong_type
    lambda : setattr(inst, '_p_estimated_size', int(1)))
AssertionError: TypeError not raised by <lambda>

Solutions I have not found it. Maybe this is caused by code translation file persistent/tests/test_persistence.py utility 2to3:

$ find ./ -name '*.py' -print0 | xargs -0i 2to3 -w '{}'
...
RefactoringTool: Refactored ./persistent/tests/test_persistence.py
RefactoringTool: Files that were modified:
RefactoringTool: ./persistent/tests/test_persistence.py
--- ./persistent/tests/test_persistence.py      (original)
+++ ./persistent/tests/test_persistence.py      (refactored)
@@ -498,12 +498,12 @@
         self.assertRaises(TypeError,
                           lambda : setattr(inst, '_p_estimated_size', None))
         try:
-            long
+            int
         except NameError:
             pass
         else:
             self.assertRaises(TypeError,
-                          lambda : setattr(inst, '_p_estimated_size', long(1)))
+                          lambda : setattr(inst, '_p_estimated_size', int(1)))

     def test_assign_p_estimated_size_negative(self):
         inst = self._makeOne()
@@ -1282,8 +1282,8 @@
     def test_w_alternate_metaclass(self):
         class alternateMeta(type):
             pass
-        class alternate(object):
-            __metaclass__ = alternateMeta
+        class alternate(object, metaclass=alternateMeta):
+            pass
         class mixedMeta(alternateMeta, type):
             pass
         # no raise
RefactoringTool: Skipping implicit fixer: buffer
...

macOS wheels are installing the 'terryfy' package

I doubt we want to do that.

$ wget https://files.pythonhosted.org/packages/ca/7b/770efeb480aa09bf3ede5470a887058ba3f37b9507c0af1cd774a64295bb/persistent-4.4.2-cp27-cp27m-macosx_10_6_intel.whl
..
Connecting to files.pythonhosted.org (files.pythonhosted.org)|151.101.1.63|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 158066 (154K) [binary/octet-stream]
Saving to: ‘persistent-4.4.2-cp27-cp27m-macosx_10_6_intel.whl’
...
2018-10-21 10:08:38 (1.29 MB/s) - ‘persistent-4.4.2-cp27-cp27m-macosx_10_6_intel.whl’ saved [158066/158066]

$ unzip -l per*whl
Archive:  persistent-4.4.2-cp27-cp27m-macosx_10_6_intel.whl
  Length      Date    Time    Name
---------  ---------- -----   ----
     2070  08-28-2018 15:15   persistent/__init__.py
...
     5114  08-28-2018 15:15   persistent-4.4.2.data/headers/cPersistence.h
     2639  08-28-2018 15:15   persistent-4.4.2.data/headers/ring.h
       31  08-28-2018 15:15   terryfy/__init__.py
      811  08-28-2018 15:15   terryfy/bdist_wheel.py
      720  08-28-2018 15:15   terryfy/cp_suff_real_libs.py
     1099  08-28-2018 15:15   terryfy/fuse_suff_real_libs.py
     1708  08-28-2018 15:15   terryfy/monkeyexec.py
     1018  08-28-2018 15:15   terryfy/repath_lib_names.py
     2148  08-28-2018 15:15   terryfy/test_travisparse.py
     1763  08-28-2018 15:15   terryfy/travisparse.py
     9118  08-28-2018 15:15   terryfy/wafutils.py
       19  08-28-2018 15:16   persistent-4.4.2.dist-info/top_level.txt
      109  08-28-2018 15:16   persistent-4.4.2.dist-info/WHEEL
    13598  08-28-2018 15:16   persistent-4.4.2.dist-info/METADATA
     4156  08-28-2018 15:16   persistent-4.4.2.dist-info/RECORD
---------                     -------
   593495                     50 files

$ pip uninstall persistent
Uninstalling persistent-4.4.2:
  Would remove:
    //include/site/python3.7/persistent/cPersistence.h
    //include/site/python3.7/persistent/ring.h
    //site-packages/persistent-4.4.2.dist-info/*
    //site-packages/persistent/*
    //site-packages/terryfy/*

Remove the _b and _u compatibility functions

We have native literals for all of those now.

Upstream tarball contains pycache/*.o files

The persistent-4.1.0.tar.gz file on PyPI appears to contain __pycache__/*.o files from cffi. This is problematic for downstream consumers because the binary files don't correspond directly to source files. For example, this violates Debian policy, so the upstream tarball must be repacked to be made compliant with policy.

I'm guessing this is just an artifact of the release process rather than something you actually want to do.

PersistentList slicing returns a non-persistent list on python3

Since __setslice__() and the likes have been killed in python3 (in favor of __getitem__(slice(...)), it seems that PersistentList must be updated, else slices return regular lists and the DB gets dangerously filled with non-persistent mutables:

Python 3.7.2 (tags/v3.7.2:9a3ffc0492, Dec 23 2018, 23:09:28) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from persistent.list import PersistentList
>>> a = PersistentList([1, 2,3])
>>> isinstance(a, PersistentList)
True
>>> isinstance(a[1:], PersistentList)
False

please adhere to distutils valid version numbers

py 2.7 distutils says:

class StrictVersion (Version):

    """Version numbering for anal retentives and software idealists.
    Implements the standard interface for version number classes as
    described above.  A version number consists of two or three
    dot-separated numeric components, with an optional "pre-release" tag
    on the end.  The pre-release tag consists of the letter 'a' or 'b'
    followed by a number.  If the numeric components of two version
    numbers are equal, then one with a pre-release tag will always
    be deemed earlier (lesser) than one without.

    The following are valid version numbers (shown in the order that
    would be obtained by sorting according to the supplied cmp function):

        0.4       0.4.0  (these two are equivalent)
        0.4.1
        0.5a1
        0.5b3
        0.5
        0.9.6
        1.0
        1.0.4a3
        1.0.4b1
        1.0.4

    The following are examples of invalid version numbers:

        1
        2.7.2.2
        1.3.a4
        1.3pl1
        1.3c4

    The rationale for this version numbering system will be explained
    in the distutils documentation.
    """

winbot uses that, fails on recent 4.2.4.2 versions

super doesn't work on PersistentMapping

I have a Class inheriting from three base classes. First
PersistentMapping and two of my own classes.

In init of this class I call super(MyClass, self).init() but the
init methods of my two classes get never called.

If i move PersistentMapping to inherit last. The init methods of
both classes are called.

is cPickleCache tested?

Looking at the tests in test_picklecache, it looks like they're only testing the Python version.

Am I missing something?

Regression: persistent 4.4.0 cannot be installed on pypy

$ virtualenv -p pypy env
$ env/bin/pip install persistent==4.4.0

Building wheels for collected packages: persistent
  Running setup.py bdist_wheel for persistent ... error
  Complete output from command /tmp/env/bin/pypy -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-JxosYs/persistent/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-pRuF5D --python-tag pp2510:
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-2.7
  creating build/lib.linux-x86_64-2.7/persistent
  copying persistent/__init__.py -> build/lib.linux-x86_64-2.7/persistent
  copying persistent/interfaces.py -> build/lib.linux-x86_64-2.7/persistent
  copying persistent/list.py -> build/lib.linux-x86_64-2.7/persistent
  copying persistent/_compat.py -> build/lib.linux-x86_64-2.7/persistent
  copying persistent/wref.py -> build/lib.linux-x86_64-2.7/persistent
  copying persistent/picklecache.py -> build/lib.linux-x86_64-2.7/persistent
  copying persistent/timestamp.py -> build/lib.linux-x86_64-2.7/persistent
  copying persistent/persistence.py -> build/lib.linux-x86_64-2.7/persistent
  copying persistent/ring.py -> build/lib.linux-x86_64-2.7/persistent
  copying persistent/dict.py -> build/lib.linux-x86_64-2.7/persistent
  copying persistent/_ring_build.py -> build/lib.linux-x86_64-2.7/persistent
  copying persistent/mapping.py -> build/lib.linux-x86_64-2.7/persistent
  creating build/lib.linux-x86_64-2.7/persistent/tests
  copying persistent/tests/__init__.py -> build/lib.linux-x86_64-2.7/persistent/tests
  copying persistent/tests/test_wref.py -> build/lib.linux-x86_64-2.7/persistent/tests
  copying persistent/tests/test_mapping.py -> build/lib.linux-x86_64-2.7/persistent/tests
  copying persistent/tests/test_ring.py -> build/lib.linux-x86_64-2.7/persistent/tests
  copying persistent/tests/test_docs.py -> build/lib.linux-x86_64-2.7/persistent/tests
  copying persistent/tests/utils.py -> build/lib.linux-x86_64-2.7/persistent/tests
  copying persistent/tests/attrhooks.py -> build/lib.linux-x86_64-2.7/persistent/tests
  copying persistent/tests/test_timestamp.py -> build/lib.linux-x86_64-2.7/persistent/tests
  copying persistent/tests/test_picklecache.py -> build/lib.linux-x86_64-2.7/persistent/tests
  copying persistent/tests/test_list.py -> build/lib.linux-x86_64-2.7/persistent/tests
  copying persistent/tests/test_persistence.py -> build/lib.linux-x86_64-2.7/persistent/tests
  copying persistent/tests/cucumbers.py -> build/lib.linux-x86_64-2.7/persistent/tests
  running egg_info
  writing persistent.egg-info/PKG-INFO
  writing dependency_links to persistent.egg-info/dependency_links.txt
  writing requirements to persistent.egg-info/requires.txt
  writing top-level names to persistent.egg-info/top_level.txt
  reading manifest file 'persistent.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  warning: no previously-included files matching '*.dll' found anywhere in distribution
  warning: no previously-included files matching '*.pyc' found anywhere in distribution
  warning: no previously-included files matching '*.pyo' found anywhere in distribution
  warning: no previously-included files matching '*.so' found anywhere in distribution
  warning: no previously-included files matching 'coverage.xml' found anywhere in distribution
  no previously-included directories found matching 'docs/_build'
  no previously-included directories found matching 'persistent/__pycache__'
  writing manifest file 'persistent.egg-info/SOURCES.txt'
  copying persistent/_compat.h -> build/lib.linux-x86_64-2.7/persistent
  copying persistent/_timestamp.c -> build/lib.linux-x86_64-2.7/persistent
  copying persistent/cPersistence.c -> build/lib.linux-x86_64-2.7/persistent
  copying persistent/cPersistence.h -> build/lib.linux-x86_64-2.7/persistent
  copying persistent/cPickleCache.c -> build/lib.linux-x86_64-2.7/persistent
  copying persistent/ring.c -> build/lib.linux-x86_64-2.7/persistent
  copying persistent/ring.h -> build/lib.linux-x86_64-2.7/persistent
  running build_ext
  generating cffi module 'build/temp.linux-x86_64-2.7/persistent._ring.c'
  creating build/temp.linux-x86_64-2.7
  building 'persistent._ring' extension
  creating build/temp.linux-x86_64-2.7/build
  creating build/temp.linux-x86_64-2.7/build/temp.linux-x86_64-2.7
  cc -pthread -DNDEBUG -O2 -fPIC -I/tmp/pip-install-JxosYs/persistent/persistent -I/tmp/env/include -c build/temp.linux-x86_64-2.7/persistent._ring.c -o build/temp.linux-x86_64-2.7/build/temp.linux-x86_64-2.7/persistent._ring.o
  cc -pthread -shared build/temp.linux-x86_64-2.7/build/temp.linux-x86_64-2.7/persistent._ring.o -o build/lib.linux-x86_64-2.7/persistent/_ring.pypy-41-x86_64-linux-gnu.so
  installing to build/bdist.linux-x86_64/wheel
  running install
  running install_lib
  creating build/bdist.linux-x86_64
  creating build/bdist.linux-x86_64/wheel
  creating build/bdist.linux-x86_64/wheel/persistent
  copying build/lib.linux-x86_64-2.7/persistent/_ring.pypy-41-x86_64-linux-gnu.so -> build/bdist.linux-x86_64/wheel/persistent
  copying build/lib.linux-x86_64-2.7/persistent/cPickleCache.c -> build/bdist.linux-x86_64/wheel/persistent
  copying build/lib.linux-x86_64-2.7/persistent/cPersistence.h -> build/bdist.linux-x86_64/wheel/persistent
  copying build/lib.linux-x86_64-2.7/persistent/__init__.py -> build/bdist.linux-x86_64/wheel/persistent
  copying build/lib.linux-x86_64-2.7/persistent/interfaces.py -> build/bdist.linux-x86_64/wheel/persistent
  copying build/lib.linux-x86_64-2.7/persistent/list.py -> build/bdist.linux-x86_64/wheel/persistent
  copying build/lib.linux-x86_64-2.7/persistent/_compat.py -> build/bdist.linux-x86_64/wheel/persistent
  copying build/lib.linux-x86_64-2.7/persistent/cPersistence.c -> build/bdist.linux-x86_64/wheel/persistent
  copying build/lib.linux-x86_64-2.7/persistent/wref.py -> build/bdist.linux-x86_64/wheel/persistent
  copying build/lib.linux-x86_64-2.7/persistent/picklecache.py -> build/bdist.linux-x86_64/wheel/persistent
  copying build/lib.linux-x86_64-2.7/persistent/_timestamp.c -> build/bdist.linux-x86_64/wheel/persistent
  copying build/lib.linux-x86_64-2.7/persistent/ring.c -> build/bdist.linux-x86_64/wheel/persistent
  creating build/bdist.linux-x86_64/wheel/persistent/tests
  copying build/lib.linux-x86_64-2.7/persistent/tests/__init__.py -> build/bdist.linux-x86_64/wheel/persistent/tests
  copying build/lib.linux-x86_64-2.7/persistent/tests/test_wref.py -> build/bdist.linux-x86_64/wheel/persistent/tests
  copying build/lib.linux-x86_64-2.7/persistent/tests/test_mapping.py -> build/bdist.linux-x86_64/wheel/persistent/tests
  copying build/lib.linux-x86_64-2.7/persistent/tests/test_ring.py -> build/bdist.linux-x86_64/wheel/persistent/tests
  copying build/lib.linux-x86_64-2.7/persistent/tests/test_docs.py -> build/bdist.linux-x86_64/wheel/persistent/tests
  copying build/lib.linux-x86_64-2.7/persistent/tests/utils.py -> build/bdist.linux-x86_64/wheel/persistent/tests
  copying build/lib.linux-x86_64-2.7/persistent/tests/attrhooks.py -> build/bdist.linux-x86_64/wheel/persistent/tests
  copying build/lib.linux-x86_64-2.7/persistent/tests/test_timestamp.py -> build/bdist.linux-x86_64/wheel/persistent/tests
  copying build/lib.linux-x86_64-2.7/persistent/tests/test_picklecache.py -> build/bdist.linux-x86_64/wheel/persistent/tests
  copying build/lib.linux-x86_64-2.7/persistent/tests/test_list.py -> build/bdist.linux-x86_64/wheel/persistent/tests
  copying build/lib.linux-x86_64-2.7/persistent/tests/test_persistence.py -> build/bdist.linux-x86_64/wheel/persistent/tests
  copying build/lib.linux-x86_64-2.7/persistent/tests/cucumbers.py -> build/bdist.linux-x86_64/wheel/persistent/tests
  copying build/lib.linux-x86_64-2.7/persistent/_compat.h -> build/bdist.linux-x86_64/wheel/persistent
  copying build/lib.linux-x86_64-2.7/persistent/timestamp.py -> build/bdist.linux-x86_64/wheel/persistent
  copying build/lib.linux-x86_64-2.7/persistent/persistence.py -> build/bdist.linux-x86_64/wheel/persistent
  copying build/lib.linux-x86_64-2.7/persistent/ring.h -> build/bdist.linux-x86_64/wheel/persistent
  copying build/lib.linux-x86_64-2.7/persistent/ring.py -> build/bdist.linux-x86_64/wheel/persistent
  copying build/lib.linux-x86_64-2.7/persistent/dict.py -> build/bdist.linux-x86_64/wheel/persistent
  copying build/lib.linux-x86_64-2.7/persistent/_ring_build.py -> build/bdist.linux-x86_64/wheel/persistent
  copying build/lib.linux-x86_64-2.7/persistent/mapping.py -> build/bdist.linux-x86_64/wheel/persistent
  running install_headers
  creating build/bdist.linux-x86_64/wheel/persistent-4.4.0.data
  creating build/bdist.linux-x86_64/wheel/persistent-4.4.0.data/headers
  Traceback (most recent call last):
    File "<module>", line 1, in <module>
    File "/tmp/pip-install-JxosYs/persistent/setup.py", line 130, in <module>
      entry_points={},
    File "/tmp/env/site-packages/setuptools/__init__.py", line 140, in setup
      return distutils.core.setup(**attrs)
    File "/usr/lib/pypy/lib-python/2.7/distutils/core.py", line 151, in setup
      dist.run_commands()
    File "/usr/lib/pypy/lib-python/2.7/distutils/dist.py", line 953, in run_commands
      self.run_command(cmd)
    File "/usr/lib/pypy/lib-python/2.7/distutils/dist.py", line 972, in run_command
      cmd_obj.run()
    File "/tmp/env/site-packages/wheel/bdist_wheel.py", line 238, in run
      self.run_command('install')
    File "/usr/lib/pypy/lib-python/2.7/distutils/cmd.py", line 334, in run_command
      self.distribution.run_command(command)
    File "/usr/lib/pypy/lib-python/2.7/distutils/dist.py", line 972, in run_command
      cmd_obj.run()
    File "/tmp/env/site-packages/setuptools/command/install.py", line 61, in run
      return orig.install.run(self)
    File "/usr/lib/pypy/lib-python/2.7/distutils/command/install.py", line 625, in run
      self.run_command(cmd_name)
    File "/usr/lib/pypy/lib-python/2.7/distutils/cmd.py", line 334, in run_command
      self.distribution.run_command(command)
    File "/usr/lib/pypy/lib-python/2.7/distutils/dist.py", line 972, in run_command
      cmd_obj.run()
    File "/usr/lib/pypy/lib-python/2.7/distutils/command/install_headers.py", line 42, in run
      (out, _) = self.copy_file(header, self.install_dir)
    File "/usr/lib/pypy/lib-python/2.7/distutils/cmd.py", line 373, in copy_file
      dry_run=self.dry_run)
    File "/usr/lib/pypy/lib-python/2.7/distutils/file_util.py", line 108, in copy_file
      if not os.path.isfile(src):
    File "/tmp/env/lib-python/2.7/genericpath.py", line 37, in isfile
      st = os.stat(path)
  TypeError: expected string, got instance object
  
  ----------------------------------------
  Failed building wheel for persistent
  Running setup.py clean for persistent
Failed to build persistent
Installing collected packages: zope.interface, persistent
  Running setup.py install for persistent ... error
    Complete output from command /tmp/env/bin/pypy -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-JxosYs/persistent/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-0vS2xa/install-record.txt --single-version-externally-managed --compile --install-headers /tmp/env/include/site/python2.7/persistent:
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-2.7
    creating build/lib.linux-x86_64-2.7/persistent
    copying persistent/__init__.py -> build/lib.linux-x86_64-2.7/persistent
    copying persistent/interfaces.py -> build/lib.linux-x86_64-2.7/persistent
    copying persistent/list.py -> build/lib.linux-x86_64-2.7/persistent
    copying persistent/_compat.py -> build/lib.linux-x86_64-2.7/persistent
    copying persistent/wref.py -> build/lib.linux-x86_64-2.7/persistent
    copying persistent/picklecache.py -> build/lib.linux-x86_64-2.7/persistent
    copying persistent/timestamp.py -> build/lib.linux-x86_64-2.7/persistent
    copying persistent/persistence.py -> build/lib.linux-x86_64-2.7/persistent
    copying persistent/ring.py -> build/lib.linux-x86_64-2.7/persistent
    copying persistent/dict.py -> build/lib.linux-x86_64-2.7/persistent
    copying persistent/_ring_build.py -> build/lib.linux-x86_64-2.7/persistent
    copying persistent/mapping.py -> build/lib.linux-x86_64-2.7/persistent
    creating build/lib.linux-x86_64-2.7/persistent/tests
    copying persistent/tests/__init__.py -> build/lib.linux-x86_64-2.7/persistent/tests
    copying persistent/tests/test_wref.py -> build/lib.linux-x86_64-2.7/persistent/tests
    copying persistent/tests/test_mapping.py -> build/lib.linux-x86_64-2.7/persistent/tests
    copying persistent/tests/test_ring.py -> build/lib.linux-x86_64-2.7/persistent/tests
    copying persistent/tests/test_docs.py -> build/lib.linux-x86_64-2.7/persistent/tests
    copying persistent/tests/utils.py -> build/lib.linux-x86_64-2.7/persistent/tests
    copying persistent/tests/attrhooks.py -> build/lib.linux-x86_64-2.7/persistent/tests
    copying persistent/tests/test_timestamp.py -> build/lib.linux-x86_64-2.7/persistent/tests
    copying persistent/tests/test_picklecache.py -> build/lib.linux-x86_64-2.7/persistent/tests
    copying persistent/tests/test_list.py -> build/lib.linux-x86_64-2.7/persistent/tests
    copying persistent/tests/test_persistence.py -> build/lib.linux-x86_64-2.7/persistent/tests
    copying persistent/tests/cucumbers.py -> build/lib.linux-x86_64-2.7/persistent/tests
    running egg_info
    writing persistent.egg-info/PKG-INFO
    writing dependency_links to persistent.egg-info/dependency_links.txt
    writing requirements to persistent.egg-info/requires.txt
    writing top-level names to persistent.egg-info/top_level.txt
    reading manifest file 'persistent.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    warning: no previously-included files matching '*.dll' found anywhere in distribution
    warning: no previously-included files matching '*.pyc' found anywhere in distribution
    warning: no previously-included files matching '*.pyo' found anywhere in distribution
    warning: no previously-included files matching '*.so' found anywhere in distribution
    warning: no previously-included files matching 'coverage.xml' found anywhere in distribution
    no previously-included directories found matching 'docs/_build'
    no previously-included directories found matching 'persistent/__pycache__'
    writing manifest file 'persistent.egg-info/SOURCES.txt'
    copying persistent/_compat.h -> build/lib.linux-x86_64-2.7/persistent
    copying persistent/_timestamp.c -> build/lib.linux-x86_64-2.7/persistent
    copying persistent/cPersistence.c -> build/lib.linux-x86_64-2.7/persistent
    copying persistent/cPersistence.h -> build/lib.linux-x86_64-2.7/persistent
    copying persistent/cPickleCache.c -> build/lib.linux-x86_64-2.7/persistent
    copying persistent/ring.c -> build/lib.linux-x86_64-2.7/persistent
    copying persistent/ring.h -> build/lib.linux-x86_64-2.7/persistent
    running build_ext
    generating cffi module 'build/temp.linux-x86_64-2.7/persistent._ring.c'
    creating build/temp.linux-x86_64-2.7
    building 'persistent._ring' extension
    creating build/temp.linux-x86_64-2.7/build
    creating build/temp.linux-x86_64-2.7/build/temp.linux-x86_64-2.7
    cc -pthread -DNDEBUG -O2 -fPIC -I/tmp/pip-install-JxosYs/persistent/persistent -I/tmp/env/include -c build/temp.linux-x86_64-2.7/persistent._ring.c -o build/temp.linux-x86_64-2.7/build/temp.linux-x86_64-2.7/persistent._ring.o
    cc -pthread -shared build/temp.linux-x86_64-2.7/build/temp.linux-x86_64-2.7/persistent._ring.o -o build/lib.linux-x86_64-2.7/persistent/_ring.pypy-41-x86_64-linux-gnu.so
    running install_lib
    creating /tmp/env/site-packages/persistent
    copying build/lib.linux-x86_64-2.7/persistent/_ring.pypy-41-x86_64-linux-gnu.so -> /tmp/env/site-packages/persistent
    copying build/lib.linux-x86_64-2.7/persistent/cPickleCache.c -> /tmp/env/site-packages/persistent
    copying build/lib.linux-x86_64-2.7/persistent/cPersistence.h -> /tmp/env/site-packages/persistent
    copying build/lib.linux-x86_64-2.7/persistent/__init__.py -> /tmp/env/site-packages/persistent
    copying build/lib.linux-x86_64-2.7/persistent/interfaces.py -> /tmp/env/site-packages/persistent
    copying build/lib.linux-x86_64-2.7/persistent/list.py -> /tmp/env/site-packages/persistent
    copying build/lib.linux-x86_64-2.7/persistent/_compat.py -> /tmp/env/site-packages/persistent
    copying build/lib.linux-x86_64-2.7/persistent/cPersistence.c -> /tmp/env/site-packages/persistent
    copying build/lib.linux-x86_64-2.7/persistent/wref.py -> /tmp/env/site-packages/persistent
    copying build/lib.linux-x86_64-2.7/persistent/picklecache.py -> /tmp/env/site-packages/persistent
    copying build/lib.linux-x86_64-2.7/persistent/_timestamp.c -> /tmp/env/site-packages/persistent
    copying build/lib.linux-x86_64-2.7/persistent/ring.c -> /tmp/env/site-packages/persistent
    creating /tmp/env/site-packages/persistent/tests
    copying build/lib.linux-x86_64-2.7/persistent/tests/__init__.py -> /tmp/env/site-packages/persistent/tests
    copying build/lib.linux-x86_64-2.7/persistent/tests/test_wref.py -> /tmp/env/site-packages/persistent/tests
    copying build/lib.linux-x86_64-2.7/persistent/tests/test_mapping.py -> /tmp/env/site-packages/persistent/tests
    copying build/lib.linux-x86_64-2.7/persistent/tests/test_ring.py -> /tmp/env/site-packages/persistent/tests
    copying build/lib.linux-x86_64-2.7/persistent/tests/test_docs.py -> /tmp/env/site-packages/persistent/tests
    copying build/lib.linux-x86_64-2.7/persistent/tests/utils.py -> /tmp/env/site-packages/persistent/tests
    copying build/lib.linux-x86_64-2.7/persistent/tests/attrhooks.py -> /tmp/env/site-packages/persistent/tests
    copying build/lib.linux-x86_64-2.7/persistent/tests/test_timestamp.py -> /tmp/env/site-packages/persistent/tests
    copying build/lib.linux-x86_64-2.7/persistent/tests/test_picklecache.py -> /tmp/env/site-packages/persistent/tests
    copying build/lib.linux-x86_64-2.7/persistent/tests/test_list.py -> /tmp/env/site-packages/persistent/tests
    copying build/lib.linux-x86_64-2.7/persistent/tests/test_persistence.py -> /tmp/env/site-packages/persistent/tests
    copying build/lib.linux-x86_64-2.7/persistent/tests/cucumbers.py -> /tmp/env/site-packages/persistent/tests
    copying build/lib.linux-x86_64-2.7/persistent/_compat.h -> /tmp/env/site-packages/persistent
    copying build/lib.linux-x86_64-2.7/persistent/timestamp.py -> /tmp/env/site-packages/persistent
    copying build/lib.linux-x86_64-2.7/persistent/persistence.py -> /tmp/env/site-packages/persistent
    copying build/lib.linux-x86_64-2.7/persistent/ring.h -> /tmp/env/site-packages/persistent
    copying build/lib.linux-x86_64-2.7/persistent/ring.py -> /tmp/env/site-packages/persistent
    copying build/lib.linux-x86_64-2.7/persistent/dict.py -> /tmp/env/site-packages/persistent
    copying build/lib.linux-x86_64-2.7/persistent/_ring_build.py -> /tmp/env/site-packages/persistent
    copying build/lib.linux-x86_64-2.7/persistent/mapping.py -> /tmp/env/site-packages/persistent
    byte-compiling /tmp/env/site-packages/persistent/__init__.py to __init__.pypy-41.pyc
    byte-compiling /tmp/env/site-packages/persistent/interfaces.py to interfaces.pypy-41.pyc
    byte-compiling /tmp/env/site-packages/persistent/list.py to list.pypy-41.pyc
    byte-compiling /tmp/env/site-packages/persistent/_compat.py to _compat.pypy-41.pyc
    byte-compiling /tmp/env/site-packages/persistent/wref.py to wref.pypy-41.pyc
    byte-compiling /tmp/env/site-packages/persistent/picklecache.py to picklecache.pypy-41.pyc
    byte-compiling /tmp/env/site-packages/persistent/tests/__init__.py to __init__.pypy-41.pyc
    byte-compiling /tmp/env/site-packages/persistent/tests/test_wref.py to test_wref.pypy-41.pyc
    byte-compiling /tmp/env/site-packages/persistent/tests/test_mapping.py to test_mapping.pypy-41.pyc
    byte-compiling /tmp/env/site-packages/persistent/tests/test_ring.py to test_ring.pypy-41.pyc
    byte-compiling /tmp/env/site-packages/persistent/tests/test_docs.py to test_docs.pypy-41.pyc
    byte-compiling /tmp/env/site-packages/persistent/tests/utils.py to utils.pypy-41.pyc
    byte-compiling /tmp/env/site-packages/persistent/tests/attrhooks.py to attrhooks.pypy-41.pyc
    byte-compiling /tmp/env/site-packages/persistent/tests/test_timestamp.py to test_timestamp.pypy-41.pyc
    byte-compiling /tmp/env/site-packages/persistent/tests/test_picklecache.py to test_picklecache.pypy-41.pyc
    byte-compiling /tmp/env/site-packages/persistent/tests/test_list.py to test_list.pypy-41.pyc
    byte-compiling /tmp/env/site-packages/persistent/tests/test_persistence.py to test_persistence.pypy-41.pyc
    byte-compiling /tmp/env/site-packages/persistent/tests/cucumbers.py to cucumbers.pypy-41.pyc
    byte-compiling /tmp/env/site-packages/persistent/timestamp.py to timestamp.pypy-41.pyc
    byte-compiling /tmp/env/site-packages/persistent/persistence.py to persistence.pypy-41.pyc
    byte-compiling /tmp/env/site-packages/persistent/ring.py to ring.pypy-41.pyc
    byte-compiling /tmp/env/site-packages/persistent/dict.py to dict.pypy-41.pyc
    byte-compiling /tmp/env/site-packages/persistent/_ring_build.py to _ring_build.pypy-41.pyc
    byte-compiling /tmp/env/site-packages/persistent/mapping.py to mapping.pypy-41.pyc
    running install_headers
    creating /tmp/env/include/site
    error: could not create '/tmp/env/include/site': Permission denied
    
    ----------------------------------------
Command "/tmp/env/bin/pypy -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-JxosYs/persistent/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-0vS2xa/install-record.txt --single-version-externally-managed --compile --install-headers /tmp/env/include/site/python2.7/persistent" failed with error code 1 in /tmp/pip-install-JxosYs/persistent/

pip install persistent=4.3.0 succeeds.

pypy --version prints

Python 2.7.13 (5.10.0+dfsg-3build2, Feb 06 2018, 18:37:50)
[PyPy 5.10.0 with GCC 7.3.0]

I'm on Ubuntu 18.04 LTS. I see a similar problem with pypy3 (PyPy 5.9.0-beta0).

The setup.py `url` is broken

Currently it points to "http://www.zope.org/Products/ZODB", which is a 404.

Most zopefoundation projects use their github page as the url these days, which has the nice side-effect of putting some extra information on PyPI.

cPickleCache uses sizeof(X) when it means isinstance(X,persistent)

From cPickleCache.c's cc_add function:

    else if (v->ob_type->tp_basicsize < sizeof(cPersistentObject))
    {
        /* If it's not an instance of a persistent class, (ie Python
            classes that derive from persistent.Persistent, BTrees,
            etc), report an error.
            TODO:  checking sizeof() seems a poor test.
        */
        PyErr_SetString(PyExc_TypeError,
                        "Cache values must be persistent objects.");
        return -1;
    }

I think I agree with the code's own comment that 'checking sizeof() seems a poor test'. Why does it do that instead of using PyObject_IsInstance (which is what the Python implementation of PickleCache does)? Could that check be changed?

(Ref zopefoundation/Persistence#9)

Pure python Persistent does not call jar.register() without _p_oid but c-implementation does

from persistent import Persistent


class A(Persistent):
    name = None


class Jar(object):
    r = 0
    def register(self, obj):
        self.r += 1

j = Jar()

a = A()
a._p_jar = j

a.name = 'test'

print j.r

prints "0" when runs with PURE_PYTHON=1 and "1" - without.

Tests fails on Windows, Python3, 64bit machine, likely due to float == double on windows

The failures are

======================================================================
FAIL: test_hash_equal (persistent.tests.test_timestamp.PyAndCComparisonTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\projects\persistent\persistent\tests\test_timestamp.py", line 260, in test_hash_equal
    self.assertEqual(hash(c), hash(py))
AssertionError: 2875592705 != -1419374591

======================================================================
FAIL: test_hash_equal_constants (persistent.tests.test_timestamp.PyAndCComparisonTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\projects\persistent\persistent\tests\test_timestamp.py", line 327, in test_hash_equal_constants
    self.assertEqual(hash(c), -721379967)
AssertionError: 3573587329 != -721379967

The hashes are equal mod 2**32. The tests pass on winbot, which is running a 32-bit Python.

The failing AppVeyor build is using a 64-bit Python.

This looks like a genuine bug in the test suite:

    def test_hash_equal_constants(self):
        ...
        is_32_bit = persistent.timestamp.c_long == ctypes.c_int32 or _is_jython                                                                     
        ....
        if is_32_bit:
            self.assertEqual(hash(c), -721379967)
        else:
            self.assertEqual(hash(c), 1000006000001)

On 64-bit Windows long is 32-bits long.

zope.wheelbuilder needs to be registered as a maintainer of pypi

Travis and Appveyor can't upload the wheels to pypi because zope.wheelbuilder is not registered maintainer of the persistent package on pypi.

PyPy/Pure-Python: `PickleCache.new_ghost` destroys any information passed to `new`

Spurred from the work in #44, with a lengthy description in #44 (comment)

Both dicts and slots are affected and need a test case.

pyTimeStamp and TimeStamp have different sub-second precision

I don't think this is correct:

>>> from persistent.timestamp import TimeStamp, pyTimeStamp
>>> ts1 = TimeStamp(2001, 2, 3, 4, 5, 6.123456789)
>>> ts2 = pyTimeStamp(2001, 2, 3, 4, 5, 6.123456789)
>>> ts1 == ts2
True
>>> ts1.second() == ts2.second()
False
>>> ts1.second()
6.1234567780047655
>>> ts2.second()
6.123456789

Unable to build/install persistent 4.5.0 on Debian 9

I discovered this issue when upgrading from Plone 5.2RC3 (using persistent 4.4.3) to Plone 5.2RC4-pending using persistent 4.5.0.

plone/Products.CMFPlone#2900

I can reproduce this error using a Python 3.7.3 installation on a pretty fresh and update Debian 9 system. Python 3.7.3 is self-compiled without any special configuration of compile flags.

In the end there is something with the cffi module here?!

bin/pip install persistent==4.5.0
Collecting persistent==4.5.0
  Using cached https://files.pythonhosted.org/packages/15/30/cd03f249bf63dbed76f34822f7acb0b5c4a58cb776ce3818184a825695cc/persistent-4.5.0.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-xzgp9es_/persistent/setup.py", line 137, in <module>
        entry_points={})
      File "/home/ajung/sandboxes/ugent-portaal-plone-4x/lib/python3.7/site-packages/setuptools/__init__.py", line 145, in setup
        return distutils.core.setup(**attrs)
      File "/opt/python-3.7.3/lib/python3.7/distutils/core.py", line 108, in setup
        _setup_distribution = dist = klass(attrs)
      File "/home/ajung/sandboxes/ugent-portaal-plone-4x/lib/python3.7/site-packages/setuptools/dist.py", line 444, in __init__
        k: v for k, v in attrs.items()
      File "/opt/python-3.7.3/lib/python3.7/distutils/dist.py", line 292, in __init__
        self.finalize_options()
      File "/home/ajung/sandboxes/ugent-portaal-plone-4x/lib/python3.7/site-packages/setuptools/dist.py", line 732, in finalize_options
        ep.load()(self, ep.name, value)
      File "/tmp/pip-install-xzgp9es_/persistent/.eggs/cffi-1.12.3-py3.7-linux-x86_64.egg/cffi/setuptools_ext.py", line 217, in cffi_modules
        add_cffi_module(dist, cffi_module)
      File "/tmp/pip-install-xzgp9es_/persistent/.eggs/cffi-1.12.3-py3.7-linux-x86_64.egg/cffi/setuptools_ext.py", line 49, in add_cffi_module
        execfile(build_file_name, mod_vars)
      File "/tmp/pip-install-xzgp9es_/persistent/.eggs/cffi-1.12.3-py3.7-linux-x86_64.egg/cffi/setuptools_ext.py", line 25, in execfile
        exec(code, glob, glob)
      File "persistent/_ring_build.py", line 22, in <module>
        ffi = FFI()
      File "/tmp/pip-install-xzgp9es_/persistent/.eggs/cffi-1.12.3-py3.7-linux-x86_64.egg/cffi/api.py", line 48, in __init__
        import _cffi_backend as backend
    ImportError: /tmp/pip-install-xzgp9es_/persistent/.eggs/cffi-1.12.3-py3.7-linux-x86_64.egg/_cffi_backend.cpython-37m-x86_64-linux-gnu.so: failed to map segment from shared object
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-xzgp9es_/persistent/
You are using pip version 19.0.3, however version 19.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

persistent.tests.test_timestamp.PyAndCComparisonTests.test_strs_equal fails

[   15s] ======================================================================
[   15s] FAIL: test_strs_equal (persistent.tests.test_timestamp.PyAndCComparisonTests)
[   15s] ----------------------------------------------------------------------
[   15s] Traceback (most recent call last):
[   15s]   File "/home/abuild/rpmbuild/BUILD/persistent-4.2.4.2/persistent/tests/test_timestamp.py", line 253, in test_strs_equal
[   15s]     self.assertEqual(str(c), str(py))
[   15s] AssertionError: '2008-12-22 15:20:-11.700000' != '2008-12-22 15:20:48.300000'
[   15s] - 2008-12-22 15:20:-11.700000
[   15s] ?                  ^^^ ^
[   15s] + 2008-12-22 15:20:48.300000
[   15s] ?                  ^^ ^

Build log

This is running on completely normal run-of-the-mill Lenovo Thinkpad T480 with openSUSE Tumbleweed, so of course it is x86_64 architecture. Using persistent-4.2.4.2.tar.gz.

Replacing my comment in #41.

Type name varies in the repr() depending on the presence of the C extension

This manifests in breakage of BTrees tests when the C extensions are used (they pass when the pure-Python extensions are available).

This is a somewhat common problem. There's a python-dev thread and bpo-issue about this. In particular, in the issue, and one of the PRs that fixes it it's pointed out that getting the correct qualified name in a C type can be complex depending on things like Py_TPFLAGS_HEAPTYPE (and that's just on Python 3, I don't know if there's an equivalent of ht_qualname on Python 2).

Is this a bug here that we should fix, or should downstream consumers be prepared to deal with the difference? In one of my own downstream projects I dealt with the difference. But BTrees already deals with the difference in its own code to produce equivalent reprs in Python as we used to get from C. (See zopefoundation/BTrees#94)

RFC: Better default repr for persistent objects

Persistent objects inherit their default __repr__ from object. This shows the object's address:

<somemodule.SomeClass object at 0x102a3d460>

I propose that when a persistent object is stored in a database, the default __repr__ should be changed to:

<somemodule.SomeClass object OID_REPR in DB_REPR>

In this way, objects whose __reprs__ appear in logs or errors, can, at least potentially, looked up for inspection.

In addition, I propose defining a new method _p_repr.

If defined, it will be used instead of the default __repr__. If _p_repr__ fails because it's dabase connection is closes, then fall back to the default. Due to the desire to loosen the dependency of Persistent on ZODB, some additional abstract methods might be needed to determine if this is the source of the failure. An alternative would be to add something to the default __repr__ to indicate that it was called because the _p_repr failed.

How to get back repr of persistent < 4.4?

persistent 4.4 introduced a new __repr__ behaviour. It might be good for some use cases but in most of my use cases it is way to technical and fragile:

It requires to pimp year old doctests with ellipsis.
On application level I like talking about object paths (which where previously part of the repr) but not about OIDs and connections.

I tried to rebuild the previous repr behaviour in a Zope 4 application using:

class PortaPage(...):

    def _p_repr(self):
        path = '/'.join(self.getPhysicalPath())
        return '<{0.__class__.__name__} at {1}>'.format(self, path)

This returns a repr of e. g. <PortalPage at target> instead of <PortalPage at /portal/target>.
The reason seems to be that self inside _p_repr is not acquisition wrapped.

Is there a suggested way to get the previous behaviour back?

intern keys of object state

We might want to intern() the keys of the persistent object state as they are updated in the instance dict. i.e.

    def __setstate__(self, state):
        for k, v in state.items():
            self.__dict__[intern(k)] = v

cPickle normally does this via the BUILD opcode (https://github.com/python/cpython/blob/master/Modules/_pickle.c#L5958), but ZODB pickles the object type and state separately, so that opcode is not used.

Experimentation on one client project shows about 25% memory savings when loading all instances of a particular commonly used class, with negligible effect on performance. But more careful evaluation is warranted.

(Memory use could also be reduced by using a class with slots, but that requires more invasive changes to the application code.)

Can we require CFFI for PURE_PYTHON usage on CPython?

CFFI distributes binary wheels, so it's a lightweight dependency.

This would simplify our code by letting us remove one of the two implementations of picklecache.Ring, and it would simplify our test matrix and help us reach 100% coverage.

Use the modern way to build CFFI modules

We're using an old, semi-deprecated way that was required by ancient versions of CFFI as distributed in ancient versions of PyPy. Instead of building the module at install time, it builds it at import time. This means it doesn't get in the wheels (see #43).

Fix process of reference count during GC

When we subtract reference counter of an object, the destructor is automatically called if the value of counter is 0.
At the same time an error occurs, as the GC executed while the destructor is still in running phase.
To eliminate the error, we have added below program which exclude objects from GC in the destructor in line 4.

static void
Wrapper_dealloc(Wrapper *self)
{
PyObject_GC_UnTrack((PyObject )self); //Add this line to eliminate self from GC
Wrapper_clear(self);
self->ob_type->tp_free((PyObject)self);
}

Patch: ZODB3-3.10.7.zip

Error: unable to execute gcc: No such file or directory

My application is built (with zc.buildout) on a separate server with all necessary develop libraries. Then I transfer application (with all compiled eggs) to the production server where there is no developer packages (such as python-dev, gcc and other). When running application, I get an error:

unable to execute gcc: No such file or directory
Traceback (most recent call last):
  ...
  File "/home/user/myapp/src/myapp/myapp/users/resources.py", line 11, in <module>
    from persistent.persistence import Persistent
  File "/home/user/myapp/eggs/persistent-4.1.1-py2.7-linux-x86_64.egg/persistent/persistence.py", line 30, in <module>
    from . import ring
  File "/home/user/myapp/eggs/persistent-4.1.1-py2.7-linux-x86_64.egg/persistent/ring.py", line 161, in <module>
    """, include_dirs=[this_dir])
  File "/home/user/myapp/eggs/cffi-1.2.1-py2.7-linux-x86_64.egg/cffi/api.py", line 405, in verify
    lib = self.verifier.load_library()
  File "/home/user/myapp/eggs/cffi-1.2.1-py2.7-linux-x86_64.egg/cffi/verifier.py", line 100, in load_library
    self._compile_module()
  File "/home/user/myapp/eggs/cffi-1.2.1-py2.7-linux-x86_64.egg/cffi/verifier.py", line 196, in _compile_module
    outputfilename = ffiplatform.compile(tmpdir, self.get_extension())
  File "/home/user/myapp/eggs/cffi-1.2.1-py2.7-linux-x86_64.egg/cffi/ffiplatform.py", line 38, in compile
    outputfilename = _build(tmpdir, ext)
  File "/home/user/myapp/eggs/cffi-1.2.1-py2.7-linux-x86_64.egg/cffi/ffiplatform.py", line 65, in _build
    raise VerificationError('%s: %s' % (e.__class__.__name__, e))
cffi.ffiplatform.VerificationError: CompileError: command 'gcc' failed with exit status 1

Should `_p_oid` allow arbitrary types?

The docstring for IPersistent._p_oid claims that it must be bytes. Historically, the C implementation has allowed any type (although values which are not length 8 bytes blow up if used with a standard pickle cache).

See: #20 (diff) for pros and cons.

Regression since 4.0.9: Opening FileStorage horribly slow

I have a rather large database, about 50 GB of data. Before I updated my ZODB installation lately, opening it was rather instant, about a second. Since the upgrade from 4.0.9 to 4.1.1 it takes about 6 minutes to open it. Interrupting this process shows the following stack trace:

2016-01-11 09:42:31:ERROR:ZODB.FileStorage:loading index
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/ZODB/FileStorage/FileStorage.py", line 383, in _restore_index
    info = fsIndex.load(index_name)
  File "/usr/local/lib/python2.7/dist-packages/ZODB/fsIndex.py", line 138, in load
    data[ensure_bytes(k)] = fsBucket().fromString(ensure_bytes(v))
  File "/usr/local/lib/python2.7/dist-packages/BTrees/fsBTree.py", line 61, in fromString
    value, values = values[:6], values[6:]
KeyboardInterrupt

Traceback (most recent call last):
...
  File ".../utils/database/database.py", line 320, in open
    self._storage = FileStorage.FileStorage(self.path)
  File "/usr/local/lib/python2.7/dist-packages/ZODB/FileStorage/FileStorage.py", line 224, in __init__
    read_only=read_only,
  File "/usr/local/lib/python2.7/dist-packages/ZODB/FileStorage/FileStorage.py", line 1636, in read_index
    index.update(tindex)
  File "/usr/local/lib/python2.7/dist-packages/ZODB/fsIndex.py", line 179, in update
    self[ensure_bytes(k)] = v
  File "/usr/local/lib/python2.7/dist-packages/ZODB/fsIndex.py", line 159, in __setitem__
    tree[key[6:]] = value
  File "/usr/local/lib/python2.7/dist-packages/BTrees/_base.py", line 273, in __setitem__
    self._set(self._to_key(key), self._to_value(value))
  File "/usr/local/lib/python2.7/dist-packages/BTrees/_base.py", line 317, in _set
    self._values.insert(index, value)
  File "/usr/local/lib/python2.7/dist-packages/persistent/persistence.py", line 267, in __getattribute__
    oga(self, '_p_accessed')()
KeyboardInterrupt

Any ideas what went wrong here? (Linux, x86_64, cpython 2.7.10)

Appveyor failures for persistent 4.2.2 (test failures on 64-bit platforms)

AFAICT the latest persistent release has no Windows wheels on PyPI because https://ci.appveyor.com/project/mgedmin/persistent/build/1.0.31 is all red.

All 64-bit tests fail with

======================================================================
FAIL: test_hash_equal (persistent.tests.test_timestamp.PyAndCComparisonTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\projects\persistent\persistent\tests\test_timestamp.py", line 266, in test_hash_equal
    self.assertEqual(hash(c), hash(py))
AssertionError: -1419374591 != 1979033151
======================================================================
FAIL: test_hash_equal_constants (persistent.tests.test_timestamp.PyAndCComparisonTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\projects\persistent\persistent\tests\test_timestamp.py", line 334, in test_hash_equal_constants
    self.assertEqual(c.__hash__(), 1000006000001)
AssertionError: -721379967 != 1000006000001L
----------------------------------------------------------------------

(except on Python 3 the 1st error is AssertionError: -1419374591 != -3850693964765720575).

I can't figure out why the 32-bit tests failed (an error from twine upload?).

Troubles with PersistentMapping and copy.copy()

I've always had troubles with PersistentMapping.copy(), which caused connection._registered_objects to fill itself as if the PersistentMapping had been modified.

So so far, i used the "copy" module to do this job. But with 4.X releases, there has been a semantic change somewhere, and copy.copy() now creates a new mapping which however changes when the original mapping has an item deleted. Below is the test which PASSES with ZODB 4.3.1.
Is it a known behaviour ?

 def test_mapping_copy(self):

    a = PersistentMapping(dict(a=3, b=5))
    b = a.copy()
    c = copy.copy(a)
    d = copy.deepcopy(a)

    assert a == b
    assert a == c
    assert a == d
    assert a is not b
    assert a is not c
    assert b is not c
    assert a is not d

    del a["a"]
    assert b["a"]  # NOT impacted
    assert "a" not in c, c  # BUG, is impacted!
    assert d["a"]  # NOT impacted

C and Python TimeStamp objects have different `.raw()` and `repr()` values for same input and are not equal

This cropped up in testing ZODB (see zopefoundation/ZODB#33), which relies on the particular repr of known TimeStamp objects, but it's not hard to imagine this having implications in many other ways.

Here's a doctest-like example:

First, verify we have access to a C implementation:

>>> import sys
>>> sys.version
'2.7.9 (default, Dec 13 2014, 15:13:49) \n[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.56)]'
>>> import persistent.TimeStamp
>>> persistent.TimeStamp.TimeStamp
<built-in function TimeStamp>

Next, set up a constant time (this exact floating point value and timestamp come from ZODB tests):

>>> import time
>>> now = 1229959248.3
>>> val = (time.gmtime(now)[:5] + (now % 60,))
>>> val
(2008, 12, 22, 15, 20, 48.299999952316284)

Create a C and Python TimeStamp:

>>> C_ts = persistent.TimeStamp.TimeStamp(*val)
>>> Py_ts = persistent.TimeStamp.pyTimeStamp(*val)

Not only do they have different reprs and raw values (different in the last character):

>>> print(repr(C_ts))
'\x03z\xbd\xd8\xce\x14z\xdd'
>>> print(repr(Py_ts))
'\x03z\xbd\xd8\xce\x14z\xde'
>>> C_ts.raw()
'\x03z\xbd\xd8\xce\x14z\xdd'
>>> Py_ts.raw()
'\x03z\xbd\xd8\xce\x14z\xde'

They're not even equal to each other, despite having the same input:

>>> C_ts == Py_ts
False

To me this seems like a bug. I can try to put together a PR, but I'm not familiar with the code so it might take me awhile.

Option to prevent storage of non-persistent mutable objects

An important bit of ZODB fine print and, for some, a common pitfall is the persistence rule that if you store non-persistent subobjects, you need to take special case, possibly inviting bugs.

I think it would be useful to enforce a restriction to only storing persistent or immutable objects. Amongts the ways to do this:

White list known immutable objects, with a way to add objects to the white list. A possible way to whitelist custom objects might be to use an ABC. I suspect that a static white list would address 98% of cases. We might want to create some new immutable objects, like linked lists.
Inspect types for mutators (e.g. setattr, setitem). I can see this becomming a maintenance burden.

Persistent.setstate C SILENTLY fails on non-builtin dict

Key interning introduced in 7bd85c1 makes __setstate__ SILENTLY fail when a non-builtin dict is passed as state.

See agroszer@1c1ceba how to provoke that.

is there any constraint that state must be a builtin dict ?
the C and python implementation differs, because the python one does not fail
a major issue is that it silently fails, just not setting attributes and returning

Python3: AttributeError: 'RingNode' object has no attribute 'next'

When you run the tests under python3 I get the following error:

$ python3 setup.py test -q
running test
running egg_info
...
======================================================================
ERROR: test___delitem___w_ghost (persistent.tests.test_picklecache.PickleCacheTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/src/RPM/BUILD/python3-module-persistent-4.0.6/persistent/tests/test_picklecache.py", line 219, in test___delitem___w_ghost
    del cache[KEY]
  File "/usr/src/RPM/BUILD/python3-module-persistent-4.0.6/persistent/picklecache.py", line 90, in __delitem__
    node = self.ring.__next__
AttributeError: 'RingNode' object has no attribute '__next__'

======================================================================
ERROR: test___delitem___w_normal_object (persistent.tests.test_picklecache.PickleCacheTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/src/RPM/BUILD/python3-module-persistent-4.0.6/persistent/tests/test_picklecache.py", line 207, in test___delitem___w_normal_object
    del cache[KEY]
  File "/usr/src/RPM/BUILD/python3-module-persistent-4.0.6/persistent/picklecache.py", line 90, in __delitem__
    node = self.ring.__next__
AttributeError: 'RingNode' object has no attribute '__next__'

======================================================================
ERROR: test___delitem___w_persistent_class (persistent.tests.test_picklecache.PickleCacheTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/src/RPM/BUILD/python3-module-persistent-4.0.6/persistent/tests/test_picklecache.py", line 196, in test___delitem___w_persistent_class
    self.assertEqual(cache.ringlen(), 0)
  File "/usr/src/RPM/BUILD/python3-module-persistent-4.0.6/persistent/picklecache.py", line 130, in ringlen
    node = self.ring.__next__
AttributeError: 'RingNode' object has no attribute '__next__'

======================================================================
ERROR: test___delitem___w_remaining_object (persistent.tests.test_picklecache.PickleCacheTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/src/RPM/BUILD/python3-module-persistent-4.0.6/persistent/tests/test_picklecache.py", line 234, in test___delitem___w_remaining_object
    del cache[UPTODATE]
  File "/usr/src/RPM/BUILD/python3-module-persistent-4.0.6/persistent/picklecache.py", line 90, in __delitem__
    node = self.ring.__next__
AttributeError: 'RingNode' object has no attribute '__next__'

======================================================================
ERROR: test___setitem___ghost (persistent.tests.test_picklecache.PickleCacheTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/src/RPM/BUILD/python3-module-persistent-4.0.6/persistent/tests/test_picklecache.py", line 121, in test___setitem___ghost
    self.assertEqual(cache.ringlen(), 0)
  File "/usr/src/RPM/BUILD/python3-module-persistent-4.0.6/persistent/picklecache.py", line 130, in ringlen
    node = self.ring.__next__
AttributeError: 'RingNode' object has no attribute '__next__'

======================================================================
ERROR: test___setitem___non_ghost (persistent.tests.test_picklecache.PickleCacheTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/src/RPM/BUILD/python3-module-persistent-4.0.6/persistent/tests/test_picklecache.py", line 139, in test___setitem___non_ghost
    self.assertEqual(cache.ringlen(), 1)
  File "/usr/src/RPM/BUILD/python3-module-persistent-4.0.6/persistent/picklecache.py", line 130, in ringlen
    node = self.ring.__next__
AttributeError: 'RingNode' object has no attribute '__next__'

======================================================================
ERROR: test_empty (persistent.tests.test_picklecache.PickleCacheTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/src/RPM/BUILD/python3-module-persistent-4.0.6/persistent/tests/test_picklecache.py", line 58, in test_empty
    self.assertEqual(cache.ringlen(), 0)
  File "/usr/src/RPM/BUILD/python3-module-persistent-4.0.6/persistent/picklecache.py", line 130, in ringlen
    node = self.ring.__next__
AttributeError: 'RingNode' object has no attribute '__next__'

======================================================================
ERROR: test_full_sweep (persistent.tests.test_picklecache.PickleCacheTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/src/RPM/BUILD/python3-module-persistent-4.0.6/persistent/tests/test_picklecache.py", line 442, in test_full_sweep
    cache.full_sweep()
  File "/usr/src/RPM/BUILD/python3-module-persistent-4.0.6/persistent/picklecache.py", line 170, in full_sweep
    self._sweep(0)
  File "/usr/src/RPM/BUILD/python3-module-persistent-4.0.6/persistent/picklecache.py", line 243, in _sweep
    node = self.ring.__next__
AttributeError: 'RingNode' object has no attribute '__next__'

======================================================================
ERROR: test_incrgc_simple (persistent.tests.test_picklecache.PickleCacheTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/src/RPM/BUILD/python3-module-persistent-4.0.6/persistent/tests/test_picklecache.py", line 374, in test_incrgc_simple
    cache.incrgc()
  File "/usr/src/RPM/BUILD/python3-module-persistent-4.0.6/persistent/picklecache.py", line 165, in incrgc
    self._sweep(target)
  File "/usr/src/RPM/BUILD/python3-module-persistent-4.0.6/persistent/picklecache.py", line 243, in _sweep
    node = self.ring.__next__
AttributeError: 'RingNode' object has no attribute '__next__'

======================================================================
ERROR: test_incrgc_w_larger_drain_resistance (persistent.tests.test_picklecache.PickleCacheTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/src/RPM/BUILD/python3-module-persistent-4.0.6/persistent/tests/test_picklecache.py", line 426, in test_incrgc_w_larger_drain_resistance
    cache.incrgc()
  File "/usr/src/RPM/BUILD/python3-module-persistent-4.0.6/persistent/picklecache.py", line 165, in incrgc
    self._sweep(target)
  File "/usr/src/RPM/BUILD/python3-module-persistent-4.0.6/persistent/picklecache.py", line 243, in _sweep
    node = self.ring.__next__
AttributeError: 'RingNode' object has no attribute '__next__'

======================================================================
ERROR: test_incrgc_w_smaller_drain_resistance (persistent.tests.test_picklecache.PickleCacheTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/src/RPM/BUILD/python3-module-persistent-4.0.6/persistent/tests/test_picklecache.py", line 409, in test_incrgc_w_smaller_drain_resistance
    cache.incrgc()
  File "/usr/src/RPM/BUILD/python3-module-persistent-4.0.6/persistent/picklecache.py", line 165, in incrgc
    self._sweep(target)
  File "/usr/src/RPM/BUILD/python3-module-persistent-4.0.6/persistent/picklecache.py", line 243, in _sweep
    node = self.ring.__next__
AttributeError: 'RingNode' object has no attribute '__next__'

======================================================================
ERROR: test_invalidate_hit_multiple_mixed (persistent.tests.test_picklecache.PickleCacheTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/src/RPM/BUILD/python3-module-persistent-4.0.6/persistent/tests/test_picklecache.py", line 658, in test_invalidate_hit_multiple_mixed
    self.assertEqual(cache.ringlen(), 1)
  File "/usr/src/RPM/BUILD/python3-module-persistent-4.0.6/persistent/picklecache.py", line 130, in ringlen
    node = self.ring.__next__
AttributeError: 'RingNode' object has no attribute '__next__'

======================================================================
ERROR: test_invalidate_hit_single_ghost (persistent.tests.test_picklecache.PickleCacheTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/src/RPM/BUILD/python3-module-persistent-4.0.6/persistent/tests/test_picklecache.py", line 629, in test_invalidate_hit_single_ghost
    self.assertEqual(cache.ringlen(), 0)
  File "/usr/src/RPM/BUILD/python3-module-persistent-4.0.6/persistent/picklecache.py", line 130, in ringlen
    node = self.ring.__next__
AttributeError: 'RingNode' object has no attribute '__next__'

======================================================================
ERROR: test_invalidate_hit_single_non_ghost (persistent.tests.test_picklecache.PickleCacheTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/src/RPM/BUILD/python3-module-persistent-4.0.6/persistent/tests/test_picklecache.py", line 642, in test_invalidate_hit_single_non_ghost
    self.assertEqual(cache.ringlen(), 1)
  File "/usr/src/RPM/BUILD/python3-module-persistent-4.0.6/persistent/picklecache.py", line 130, in ringlen
    node = self.ring.__next__
AttributeError: 'RingNode' object has no attribute '__next__'

======================================================================
ERROR: test_lruitems (persistent.tests.test_picklecache.PickleCacheTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/src/RPM/BUILD/python3-module-persistent-4.0.6/persistent/tests/test_picklecache.py", line 249, in test_lruitems
    items = cache.lru_items()
  File "/usr/src/RPM/BUILD/python3-module-persistent-4.0.6/persistent/picklecache.py", line 145, in lru_items
    node = self.ring.__next__
AttributeError: 'RingNode' object has no attribute '__next__'

======================================================================
ERROR: test_minimize (persistent.tests.test_picklecache.PickleCacheTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/src/RPM/BUILD/python3-module-persistent-4.0.6/persistent/tests/test_picklecache.py", line 463, in test_minimize
    cache.minimize()
  File "/usr/src/RPM/BUILD/python3-module-persistent-4.0.6/persistent/picklecache.py", line 170, in full_sweep
    self._sweep(0)
  File "/usr/src/RPM/BUILD/python3-module-persistent-4.0.6/persistent/picklecache.py", line 243, in _sweep
    node = self.ring.__next__
AttributeError: 'RingNode' object has no attribute '__next__'

======================================================================
ERROR: test_mru_first (persistent.tests.test_picklecache.PickleCacheTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/src/RPM/BUILD/python3-module-persistent-4.0.6/persistent/tests/test_picklecache.py", line 333, in test_mru_first
    cache.mru(ONE)
  File "/usr/src/RPM/BUILD/python3-module-persistent-4.0.6/persistent/picklecache.py", line 109, in mru
    node = self.ring.__next__
AttributeError: 'RingNode' object has no attribute '__next__'

======================================================================
ERROR: test_mru_ghost (persistent.tests.test_picklecache.PickleCacheTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/src/RPM/BUILD/python3-module-persistent-4.0.6/persistent/tests/test_picklecache.py", line 292, in test_mru_ghost
    cache.mru(TWO)
  File "/usr/src/RPM/BUILD/python3-module-persistent-4.0.6/persistent/picklecache.py", line 109, in mru
    node = self.ring.__next__
AttributeError: 'RingNode' object has no attribute '__next__'

======================================================================
ERROR: test_mru_last (persistent.tests.test_picklecache.PickleCacheTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/src/RPM/BUILD/python3-module-persistent-4.0.6/persistent/tests/test_picklecache.py", line 353, in test_mru_last
    cache.mru(THREE)
  File "/usr/src/RPM/BUILD/python3-module-persistent-4.0.6/persistent/picklecache.py", line 109, in mru
    node = self.ring.__next__
AttributeError: 'RingNode' object has no attribute '__next__'

======================================================================
ERROR: test_mru_nonesuch_raises_KeyError (persistent.tests.test_picklecache.PickleCacheTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/src/RPM/BUILD/python3-module-persistent-4.0.6/persistent/tests/test_picklecache.py", line 258, in test_mru_nonesuch_raises_KeyError
    self.assertRaises(KeyError, cache.mru, _b('nonesuch'))
  File "/usr/lib64/python3.3/unittest/case.py", line 609, in assertRaises
    return context.handle('assertRaises', callableObj, args, kwargs)
  File "/usr/lib64/python3.3/unittest/case.py", line 173, in handle
    callable_obj(*args, **kwargs)
  File "/usr/src/RPM/BUILD/python3-module-persistent-4.0.6/persistent/picklecache.py", line 109, in mru
    node = self.ring.__next__
AttributeError: 'RingNode' object has no attribute '__next__'

======================================================================
ERROR: test_mru_normal (persistent.tests.test_picklecache.PickleCacheTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/src/RPM/BUILD/python3-module-persistent-4.0.6/persistent/tests/test_picklecache.py", line 271, in test_mru_normal
    cache.mru(TWO)
  File "/usr/src/RPM/BUILD/python3-module-persistent-4.0.6/persistent/picklecache.py", line 109, in mru
    node = self.ring.__next__
AttributeError: 'RingNode' object has no attribute '__next__'

======================================================================
ERROR: test_mru_was_ghost_now_active (persistent.tests.test_picklecache.PickleCacheTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/src/RPM/BUILD/python3-module-persistent-4.0.6/persistent/tests/test_picklecache.py", line 313, in test_mru_was_ghost_now_active
    cache.mru(TWO)
  File "/usr/src/RPM/BUILD/python3-module-persistent-4.0.6/persistent/picklecache.py", line 109, in mru
    node = self.ring.__next__
AttributeError: 'RingNode' object has no attribute '__next__'

======================================================================
ERROR: test_reify_hit_multiple_mixed (persistent.tests.test_picklecache.PickleCacheTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/src/RPM/BUILD/python3-module-persistent-4.0.6/persistent/tests/test_picklecache.py", line 603, in test_reify_hit_multiple_mixed
    self.assertEqual(cache.ringlen(), 1)
  File "/usr/src/RPM/BUILD/python3-module-persistent-4.0.6/persistent/picklecache.py", line 130, in ringlen
    node = self.ring.__next__
AttributeError: 'RingNode' object has no attribute '__next__'

======================================================================
ERROR: test_reify_hit_single_ghost (persistent.tests.test_picklecache.PickleCacheTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/src/RPM/BUILD/python3-module-persistent-4.0.6/persistent/tests/test_picklecache.py", line 572, in test_reify_hit_single_ghost
    self.assertEqual(cache.ringlen(), 0)
  File "/usr/src/RPM/BUILD/python3-module-persistent-4.0.6/persistent/picklecache.py", line 130, in ringlen
    node = self.ring.__next__
AttributeError: 'RingNode' object has no attribute '__next__'

======================================================================
ERROR: test_reify_hit_single_non_ghost (persistent.tests.test_picklecache.PickleCacheTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/src/RPM/BUILD/python3-module-persistent-4.0.6/persistent/tests/test_picklecache.py", line 587, in test_reify_hit_single_non_ghost
    self.assertEqual(cache.ringlen(), 1)
  File "/usr/src/RPM/BUILD/python3-module-persistent-4.0.6/persistent/picklecache.py", line 130, in ringlen
    node = self.ring.__next__
AttributeError: 'RingNode' object has no attribute '__next__'

These errors appear after the code conversion file conversion persistent/picklecache.py and persistent/tests/test_picklecache.py utility 2to3:

$ find ./ -name '*.py' -print0 | xargs -0i 2to3 -w '{}'
...
RefactoringTool: Refactored ./persistent/picklecache.py
--- ./persistent/picklecache.py (original)
+++ ./persistent/picklecache.py (refactored)
@@ -87,13 +87,13 @@
             del self.persistent_classes[oid]
         else:
             value = self.data.pop(oid)
-            node = self.ring.next
+            node = self.ring.__next__
             while node is not self.ring:
                 if node.object is value:
-                    node.prev.next, node.next.prev = node.next, node.prev
+                    node.prev.next, node.next.prev = node.__next__, node.prev
                     self.non_ghost_count -= 1
                     break
-                node = node.next
+                node = node.__next__

     def get(self, oid, default=None):
         """ See IPickleCache.
@@ -106,9 +106,9 @@
     def mru(self, oid):
         """ See IPickleCache.
         """
-        node = self.ring.next
+        node = self.ring.__next__
         while node is not self.ring and node.object._p_oid != oid:
-            node = node.next
+            node = node.__next__
         if node is self.ring:
             value = self.data[oid]
             if value._p_state != GHOST:
@@ -118,7 +118,7 @@
                 mru.next = node
         else:
             # remove from old location
-            node.prev.next, node.next.prev = node.next, node.prev
+            node.prev.next, node.next.prev = node.__next__, node.prev
             # splice into new
             self.ring.prev.next, node.prev = node, self.ring.prev
             self.ring.prev, node.next = node, self.ring
@@ -127,31 +127,31 @@
         """ See IPickleCache.
         """
         result = 0
-        node = self.ring.next
+        node = self.ring.__next__
         while node is not self.ring:
             result += 1
-            node = node.next
+            node = node.__next__
         return result

     def items(self):
         """ See IPickleCache.
         """
-        return self.data.items()
+        return list(self.data.items())

     def lru_items(self):
         """ See IPickleCache.
         """
         result = []
-        node = self.ring.next
+        node = self.ring.__next__
         while node is not self.ring:
             result.append((node.object._p_oid, node.object))
-            node = node.next
+            node = node.__next__
         return result

     def klass_items(self):
         """ See IPickleCache.
         """
-        return self.persistent_classes.items()
+        return list(self.persistent_classes.items())

     def incrgc(self, ignored=None):
         """ See IPickleCache.
@@ -212,13 +212,13 @@

     def debug_info(self):
         result = []
-        for oid, klass in self.persistent_classes.items():
+        for oid, klass in list(self.persistent_classes.items()):
             result.append((oid,
                             len(gc.get_referents(klass)),
                             type(klass).__name__,
                             klass._p_state,
                             ))
-        for oid, value in self.data.items():
+        for oid, value in list(self.data.items()):
             result.append((oid,
                             len(gc.get_referents(value)),
                             type(value).__name__,
@@ -234,28 +234,28 @@
     cache_size = property(lambda self: self.target_size)
     cache_drain_resistance = property(lambda self: self.drain_resistance)
     cache_non_ghost_count = property(lambda self: self.non_ghost_count)
-    cache_data = property(lambda self: dict(self.data.items()))
+    cache_data = property(lambda self: dict(list(self.data.items())))
     cache_klass_count = property(lambda self: len(self.persistent_classes))

     # Helpers
     def _sweep(self, target):
         # lock
-        node = self.ring.next
+        node = self.ring.__next__
         while node is not self.ring and self.non_ghost_count > target:
             if node.object._p_state not in (STICKY, CHANGED):
-                node.prev.next, node.next.prev = node.next, node.prev
+                node.prev.next, node.next.prev = node.__next__, node.prev
              RefactoringTool: Files that were modified:
RefactoringTool: ./persistent/picklecache.py
   node.object = None
                 self.non_ghost_count -= 1
-            node = node.next
+            node = node.__next__

     def _invalidate(self, oid):
         value = self.data.get(oid)
         if value is not None and value._p_state != GHOST:
             value._p_invalidate()
-            node = self.ring.next
+            node = self.ring.__next__
             while node is not self.ring:
                 if node.object is value:
-                    node.prev.next, node.next.prev = node.next, node.prev
+                    node.prev.next, node.next.prev = node.__next__, node.prev
                     break
         elif oid in self.persistent_classes:
             del self.persistent_classes[oid]
RefactoringTool: Skipping implicit fixer: buffer
...
RefactoringTool: Refactored ./persistent/tests/test_picklecache.py
--- ./persistent/tests/test_picklecache.py      (original)
+++ ./persistent/tests/test_picklecache.py      (refactored)
@@ -53,7 +53,7 @@
         cache = self._makeOne()

         self.assertEqual(len(cache), 0)
-        self.assertEqual(_len(cache.items()), 0)
+        self.assertEqual(_len(list(cache.items())), 0)
         self.assertEqual(_len(cache.klass_items()), 0)
         self.assertEqual(cache.ringlen(), 0)
         self.assertEqual(len(cache.lru_items()), 0)
@@ -152,7 +152,7 @@

         kitems = list(cache.klass_items())
         self.assertEqual(len(cache), 1)
-        self.assertEqual(_len(cache.items()), 0)
+        self.assertEqual(_len(list(cache.items())), 0)
         self.assertEqual(len(kitems), 1)
         self.assertEqual(kitems[0][0], KEY)
         self.assertTrue(kitems[0][1] is pclass)
@@ -443,7 +443,7 @@
         gc.collect() # banish the ghosts who are no longer in the ring

         self.assertEqual(cache.cache_non_ghost_count, 0)
-        self.assertTrue(cache.ring.next is cache.ring)
+ RefactoringTool: Files that were modified:
RefactoringTool: ./persistent/tests/test_picklecache.py
       self.assertTrue(cache.ring.__next__ is cache.ring)

         for oid in oids:
             self.assertTrue(cache.get(oid) is None)
RefactoringTool: Skipping implicit fixer: buffer
...

Workaround: Fix the conversion result with help of sed (see http://git.altlinux.org/people/solo/packages/python-module-persistent.git?p=python-module-persistent.git;a=commitdiff;h=2a6110d8f3c780dffd5ac17cf206df65191e9431):

sed -ri 's/\.__next__([^(]|$)/.next\1/g' \
    persistent/picklecache.py \
    persistent/tests/test_picklecache.py

pyTimeStamp and TimeStamp init accept different arguments

This is on master:

>>> from persistent.TimeStamp import TimeStamp
>>> TimeStamp(2014, 2, 3)
'\x03\xa4\x8a\xa0\x00\x00\x00\x00'
>>> from persistent.TimeStamp import pyTimeStamp
>>> pyTimeStamp(2014, 2, 3)
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/home/do3cc/dev/presistent/persistent/timestamp.py", line 86, in __init__
    raise TypeError('Pass either a single 8-octet arg '
TypeError: Pass either a single 8-octet arg or 5 integers and a float

Distribute the CFFI ring implementation in the wheels?

This is mostly a problem for PyPy users on Windows (who probably don't have compilers).

We should be able to do this with this snippet in setup.py (having made sure to install CFFI and zope.interface first; the zope.interface requirement could be dropped with some refactoring):

try:
    import cffi
    import persistent.ring
except ImportError:
    pass
else:
    ext_modules.append(persistent.ring.ffi.verifier.get_extension())

That should work back to PyPy 2.5.1 or so, which is the version on Travis.

Now the second half of this is that we don't distribute PyPy wheels, so to meet the stated purpose we'd also need to do that (at least on Windows)

Must `_p_invalidate` call `_p_deactivate`?

There are tests in ZODB which failed when the pure-Python Persistent._p_invalidate skipped explicitly calling _p_deactivate.

See: #20 (diff)

Enable coveralls and reach 100% coverage

`_p_repr` drops Acquisition wrappers, even when found on the instance, with the C extensions

As reported in zopefoundation/Zope#392 , among others. This has caused some pain for the Zope2/4 and Plone folks.

It would seem the obvious cause is that we look for _p_repr on the type, not the instance. Changing that makes the pure-Python versions work, but using the C extensions still fails (even when using Persistence.Persistent as a base class).

This test passes in pure-Python, fails in C when we look for _p_repr on the instance (it fails in both when we look on the type):

    def test__p_repr_acquisition(self):
        # This works on both PyPy/PURE_PYTHON and on CPython/c-extensions,
        # but it won't work when Base is defined in C and we use the
        # pure-python persistent class.
        if self._getTargetClass() != persistent.Persistent:
            raise unittest.SkipTest("Mixed C/pure")

        from ExtensionClass import Base
        from Acquisition import Implicit
        from Persistence import Persistent as Persistent
        class Foo(Implicit,
                  Persistent,
                  ):
            __slots__ = ()

            def _p_repr(self):
                print("Hi from", type(self))
                return "Parent is %s" % getattr(self, '__parent__')

        class MyContainer(Base):
            def __str__(self):
                return "Base"

        base = MyContainer()
        base.foo = Foo()
        foo = base.foo

        self.assertEqual(repr(foo), 'Parent is Base')

In pure-python, the print gives us Hi from <class 'ExtensionClass.ImplicitAcquisitionWrapper_Foo'>, while with C we get Hi from <class 'persistent.tests.test_persistence._Persistent_Base.test__p_repr_acquisition.<locals>.Foo'>

Here's what I see that goes wrong.

The acquisition wrapper's __repr__ slot asks for the __repr__ object of itself. The intent is to get a callable that's itself wrapped.
This gets to the "normal lookup" branch of Wrapper_findattr
An attribute is found. But instead of finding an object passing PyMethod_Check, which the wrapper knows how to re-wrap, we actually wind up with a "method-wrapper" (<method-wrapper '__repr__' of Foo object at 0x105273bd8>). This is the way that C slots are exposed as Python callables. Acquisition doesn't know how to deal with that, and so it gets returned as-is.
We invoke Persistent.__repr__ with the unwrapped object.

It's just not safe to call a C slot with a wrapped object, so it's not clear to me what, if anything, can be done about this.

Python 3.7 binary wheels on PyPI

Currently the last persistent release on PyPI lacks Windows binary wheels for Python 3.7.

This can be fixed one of two ways:

we make a new release and our appveyor.yml will build and upload all the wheels
OR we create a temporary appveyor branch based on the last release tag, edit appveyor.yml to add python: 37 and 37-64, remove all other pythons from the build matrix, and make the twine upload step unconditional, push it, wait for appveyor, delete the branch

Shared read-only state between objects with copy on write

There's a lot of interest in using ZODB with asynchronous frameworks, especially for applications that block on network requests to services. From a purely programming perspective, gevent makes this quite tractable, but the cost of maintaining many open ZODB connections with their own caches is a major challenge. The cost of maintaining many open connections could be mitigated if data could be shared among their caches.

One way to do this would be to have a shared state cache of read-only state objects. Consider the extremely common case of persistent objects that store their data in dictionaries (and leaving aside non-persistent subobjects, for the sake of discussion). Set-state for such objects could simply assign the instance dictionary to the state. First assigning an attribute to such an object could copy the state dict first. This would allow use of shared immutable state dicts, requiring no copying for read-only operations. Note that in this scenario, only state is shared, not persistent objects.

You could use slots, or secondary dictionaries for non-shared mutable state.

Similar schemes could be used for BTrees and Buckets, although we'd need to introduce new Python subobjects to represent shared state.

To make this work, we'd likely want to create persistent subobjects that disallowed storing non-persistent mutable subobjects, which would have other benefits.

Building Linux wheels seems to be broken

See https://travis-ci.org/zopefoundation/persistent/builds/530228105

TimeStamp repr returns a double quoted string

The test creates the string the same way so it does not fail.
Relevant commits:
db732a8
e692af8

and a traceback in the wild where it bit me:
ploneintranet/ploneintranet.todo@7f215f6#commitcomment-8752965

Docs: Document PersistentMapping and PersistentList

These aren't referenced from the local docs anywhere. (They're pulled into the ZODB docs on zodb.readthedocs.io for some reason.) Some narrative on why you'd want to use them might be appropriate here.

Change PickleCache from simple LRU to a more modern policy (like windowed LFU/SLRU)

Over in RelStorage we've been having a discussion based on the observation that strict LRU isn't necessarily a good cache policy for varying workloads.

Basically, if most queries/transactions/requests use a certain set of objects, those objects shouldn't be evicted from the cache just because an outlier request comes in that scans a different BTree. LRU is prone to that problem. More adaptive approaches aren't.

@ben-manes pointed us to a policy that can be near optimal for a variety of workloads.

I think that would be a good fit for the PickleCache.

I have a C and CFFI implementation that I'm using in RelStorage. We may or may not be able to directly share the code, I don't know (it'd be cool to split this off as its own library on PyPI and distribute binary wheels just for it so I don't have to do that for RelStorage), but we could at least start with it.

Other notes:

The code as is doesn't implement the probability frequency sketch. This feature cheaply allows the cache to have a "memory" of keys it has evicted so if they show up again soon it knows it made the wrong decision and is more biased towards keeping them. Given the way keys change in RelStorage simulations showed this feature wasn't particularly useful there, but since keys are unchanging OIDS in this cache, it might be more useful here. But that can always be added later.
Garbage collection (incrgc) and tuning become simpler, as the cache always stays at its preferred size, automatically, based on the frequency and LRU-ness of the data. We know exactly which items are good to evict from the cache. The drain_resistance goes away.
Of course, this is complicated by the fact that the pickle cache has the option to be based on estimated byte size in addition to object count. Ideally one of those would go away to be able to take full advantage of the existing code (plus the combination makes it difficult to reason about cache behaviour). My vote is for eliminating byte size at this level; let the lower levels worry about caching in terms of bytes.
Does anybody have some cache traces from an actual ZODB production instance, and info on where they came from? That could be plugged into simulations.

zopefoundation / persistent Goto Github PK

persistent's Introduction

persistent: automatic persistence for Python objects

persistent's People

Contributors

Stargazers

Watchers

Forkers

persistent's Issues

Recommend Projects

Recommend Topics

Recommend Org

`persistent`: automatic persistence for Python objects