Git Product home page Git Product logo

zlmdb's Introduction

Introduction to zLMDB

PyPI Build Documentation Deploy

Object-relational in-memory database layer based on LMDB:

  • High-performance (see below)
  • Supports multiple serializers (JSON, CBOR, Pickle, Flatbuffers)
  • Supports export/import from/to Apache Arrow
  • Support native Numpy arrays and Pandas data frames
  • Automatic indexes
  • Free software (MIT license)

zlmdb's People

Contributors

bgermann avatar meejah avatar oberstet avatar om26er avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

zlmdb's Issues

Please publish version 19.5.1 to pypi

crossbarfx sets its minimum requirement of zlmdb to 19.5.1 but it seems its not yet released. Also crossbarfx master uses the API MapTimestampStringFlatBuffers which is only available in zlmdb 19.5.1

Does not import on py3.8/Windows b/c `time.clock` is removed

  • ZLMDB version: * (?)
  • Python version: 3.8
  • Operating System: Windows

zlmdb will not import on Python 3.8/Windows since time.clock has been removed. (time.clock has been deprecated since 3.3.) As far as I can tell time.perf_counter should be used for Python >= 3.3 esp since it works for Windows and Unix.

The import error:

Python 3.8.1 (tags/v3.8.1:1b293b6, Dec 18 2019, 23:11:46) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import zlmdb
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "...\zlmdb\zlmdb\__init__.py", line 102, in <module>
    from ._transaction import Transaction, TransactionStats, walltime
  File "...\zlmdb\zlmdb\_transaction.py", line 39, in <module>
    walltime = time.clock
AttributeError: module 'time' has no attribute 'clock'

deleting from table with nullable index

In cfc-functests:

  • test_organizations_crud
  • test_webcluster_crud
2019-04-20T18:16:13+0200 2019-04-20T18:16:13+0200 [Container   28502] UserManager.create_organization(organization={}, details=CallDetails(registration=<autobahn.wamp.request.Registration object at 0x7f05f0631b38>, progress=None, caller=3159352971539072, caller_authid=superuser, caller_authrole=user, procedure=<crossbarfabriccenter.user.create_organization>, enc_algo=None, forward_for=None))
2019-04-20T18:16:13+0200 2019-04-20T18:16:13+0200 [Container   28502] new Organization object stored in database:
2019-04-20T18:16:13+0200 
2019-04-20T18:16:13+0200 {'name': None,
2019-04-20T18:16:13+0200  'oid': '73c0a037-3c43-4ff2-a9f5-26b2a16a254f',
2019-04-20T18:16:13+0200  'otype': None,
2019-04-20T18:16:13+0200  'registered': None}
2019-04-20T18:16:13+0200 
2019-04-20T18:16:13+0200 Event received on "cfc.user.on_organization_created" {'oid': '73c0a037-3c43-4ff2-a9f5-26b2a16a254f', 'name': None, 'otype': None, 'registered': None} EventDetails(subscription=Subscription(id=5731476024697015, is_active=True), publication=2344536254292897, publisher=None, publisher_authid=None, publisher_authrole=None, topic=<crossbarfabriccenter.user.on_organization_created>, retained=None, enc_algo=None, forward_for=None)
2019-04-20T18:16:13+0200 {'oid': '73c0a037-3c43-4ff2-a9f5-26b2a16a254f', 'name': None, 'otype': None, 'registered': None}
2019-04-20T18:16:13+0200 2019-04-20T18:16:13+0200 [Container   28502] Management API event <on_organization_created> published:
2019-04-20T18:16:13+0200 {'oid': '73c0a037-3c43-4ff2-a9f5-26b2a16a254f', 'name': None, 'otype': None, 'registered': None}

2019-04-20T18:16:13+0200 2019-04-20T18:16:13+0200 [Container   28502] UserManager.list_organizations(details=CallDetails(registration=<autobahn.wamp.request.Registration object at 0x7f05f0631c78>, progress=None, caller=3159352971539072, caller_authid=superuser, caller_authrole=user, procedure=<crossbarfabriccenter.user.list_organizations>, enc_algo=None, forward_for=None))
2019-04-20T18:16:13+0200 ['30928e6b-41d9-451a-840f-55edbc2aab60', '73c0a037-3c43-4ff2-a9f5-26b2a16a254f', '892f07f7-09d7-4ec9-8cec-731d6ad76206']
2019-04-20T18:16:13+0200 2019-04-20T18:16:13+0200 [Container   28502] UserManager.delete_organization(details=CallDetails(registration=<autobahn.wamp.request.Registration object at 0x7f05f0631ae8>, progress=None, caller=3159352971539072, caller_authid=superuser, caller_authrole=user, procedure=<crossbarfabriccenter.user.delete_organization>, enc_algo=None, forward_for=None))
2019-04-20T18:16:13+0200 2019-04-20T18:16:13+0200 [Container   28502] DomainController.onUserError(): "AssertionError: "
2019-04-20T18:16:13+0200 Traceback (most recent call last):
2019-04-20T18:16:13+0200   File "/home/oberstet/cpy373_1/lib/python3.7/site-packages/autobahn/wamp/protocol.py", line 1006, in onMessage
2019-04-20T18:16:13+0200     on_reply = txaio.as_future(endpoint.fn, *invoke_args, **invoke_kwargs)
2019-04-20T18:16:13+0200   File "/home/oberstet/cpy373_1/lib/python3.7/site-packages/txaio/tx.py", line 428, in as_future
2019-04-20T18:16:13+0200     return ensureDeferred(fun(*args, **kwargs))
2019-04-20T18:16:13+0200   File "/home/oberstet/cpy373_1/lib/python3.7/site-packages/twisted/internet/defer.py", line 911, in ensureDeferred
2019-04-20T18:16:13+0200     return _cancellableInlineCallbacks(coro)
2019-04-20T18:16:13+0200   File "/home/oberstet/cpy373_1/lib/python3.7/site-packages/twisted/internet/defer.py", line 1529, in _cancellableInlineCallbacks
2019-04-20T18:16:13+0200     _inlineCallbacks(None, g, status)
2019-04-20T18:16:13+0200 --- <exception caught here> ---
2019-04-20T18:16:13+0200   File "/home/oberstet/cpy373_1/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
2019-04-20T18:16:13+0200     result = g.send(result)
2019-04-20T18:16:13+0200   File "/home/oberstet/scm/crossbario/crossbarfx/crossbarfx/master/node/user.py", line 198, in delete_organization
2019-04-20T18:16:13+0200     del self.schema.organizations[txn, oid]
2019-04-20T18:16:13+0200   File "/home/oberstet/cpy373_1/lib/python3.7/site-packages/zlmdb/_pmap.py", line 463, in __delitem__
2019-04-20T18:16:13+0200     _idx_key = struct.pack('>H', index.pmap._slot) + index.pmap._serialize_key(index.fkey(value))
2019-04-20T18:16:13+0200   File "/home/oberstet/cpy373_1/lib/python3.7/site-packages/zlmdb/_types.py", line 272, in _serialize_key
2019-04-20T18:16:13+0200     assert type(key) == six.text_type
2019-04-20T18:16:13+0200 builtins.AssertionError: 
2019-04-20T18:16:13+0200 

or

2019-04-20T18:16:20+0200 2019-04-20T18:16:20+0200 [Container   28563] WebClusterManager.create_webcluster(webcluster={'tcp_port': 8080, 'tcp_shared': True, 'tcp_interface': '127.0.0.1', 'tcp_backlog': 100, 'http_display_tracebacks': True}, details=CallDetails(registration=<autobahn.wamp.request.Registration object at 0x7fb7e55ff5e8>, progress=None, caller=6236535363633262, caller_authid=superuser, caller_authrole=owner, procedure=<crossbarfabriccenter.mrealm.webcluster.create_webcluster>, enc_algo=None, forward_for=None))
2019-04-20T18:16:20+0200 2019-04-20T18:16:20+0200 [Container   28563] new WebCluster object stored in database:
2019-04-20T18:16:20+0200 
2019-04-20T18:16:20+0200 {'changed': None,
2019-04-20T18:16:20+0200  'http_access_log': None,
2019-04-20T18:16:20+0200  'http_client_timeout': None,
2019-04-20T18:16:20+0200  'http_display_tracebacks': True,
2019-04-20T18:16:20+0200  'http_hsts': None,
2019-04-20T18:16:20+0200  'http_hsts_max_age': None,
2019-04-20T18:16:20+0200  'name': None,
2019-04-20T18:16:20+0200  'oid': '8ae38960-f291-403a-a1dd-2f6b53dfc2c9',
2019-04-20T18:16:20+0200  'status': 'STOPPED',
2019-04-20T18:16:20+0200  'tcp_backlog': 100,
2019-04-20T18:16:20+0200  'tcp_interface': '127.0.0.1',
2019-04-20T18:16:20+0200  'tcp_port': 8080,
2019-04-20T18:16:20+0200  'tcp_shared': True,
2019-04-20T18:16:20+0200  'tcp_version': None,
2019-04-20T18:16:20+0200  'tls_ca_certificates': None,
2019-04-20T18:16:20+0200  'tls_certificate': None,
2019-04-20T18:16:20+0200  'tls_chain_certificates': None,
2019-04-20T18:16:20+0200  'tls_ciphers': None,
2019-04-20T18:16:20+0200  'tls_dhparam': None,
2019-04-20T18:16:20+0200  'tls_key': None}
2019-04-20T18:16:20+0200 
2019-04-20T18:16:20+0200 2019-04-20T18:16:20+0200 [Container   28563] Management API event <on_webcluster_created> published:
2019-04-20T18:16:20+0200 {'oid': '8ae38960-f291-403a-a1dd-2f6b53dfc2c9', 'name': None, 'status': 'STOPPED', 'changed': None, 'tcp_version': None, 'tcp_port': 8080, 'tcp_shared': True, 'tcp_interface': '127.0.0.1', 'tcp_backlog': 100, 'tls_key': None, 'tls_certificate': None, 'tls_chain_certificates': None, 'tls_ca_certificates': None, 'tls_dhparam': None, 'tls_ciphers': None, 'http_client_timeout': None, 'http_hsts': None, 'http_hsts_max_age': None, 'http_access_log': None, 'http_display_tracebacks': True}
2019-04-20T18:16:20+0200 {'oid': '8ae38960-f291-403a-a1dd-2f6b53dfc2c9', 'name': None, 'status': 'STOPPED', 'changed': None, 'tcp_version': None, 'tcp_port': 8080, 'tcp_shared': True, 'tcp_interface': '127.0.0.1', 'tcp_backlog': 100, 'tls_key': None, 'tls_certificate': None, 'tls_chain_certificates': None, 'tls_ca_certificates': None, 'tls_dhparam': None, 'tls_ciphers': None, 'http_client_timeout': None, 'http_hsts': None, 'http_hsts_max_age': None, 'http_access_log': None, 'http_display_tracebacks': True}
2019-04-20T18:16:20+0200 2019-04-20T18:16:20+0200 [Container   28563] WebClusterManager.list_webclusters(details=CallDetails(registration=<autobahn.wamp.request.Registration object at 0x7fb7e55ff8b8>, progress=None, caller=6236535363633262, caller_authid=superuser, caller_authrole=owner, procedure=<crossbarfabriccenter.mrealm.webcluster.list_webclusters>, enc_algo=None, forward_for=None))
2019-04-20T18:16:20+0200 ['20087354-66ae-4c99-ab63-469c30aa1e7f', '8ae38960-f291-403a-a1dd-2f6b53dfc2c9']
2019-04-20T18:16:20+0200 2019-04-20T18:16:20+0200 [Container   28563] WebClusterManager.add_webcluster_service(webcluster_oid=8ae38960-f291-403a-a1dd-2f6b53dfc2c9, path=info, webservice={'type': 'nodeinfo'}, details=CallDetails(registration=<autobahn.wamp.request.Registration object at 0x7fb7e55ff598>, progress=None, caller=6236535363633262, caller_authid=superuser, caller_authrole=owner, procedure=<crossbarfabriccenter.mrealm.webcluster.add_webcluster_service>, enc_algo=None, forward_for=None))
2019-04-20T18:16:20+0200 2019-04-20T18:16:20+0200 [Container   28563] New WebService object stored in database:
2019-04-20T18:16:20+0200 
2019-04-20T18:16:20+0200 {'oid': 'd424fa2a-2afb-4cba-b03e-8f8f56f6ac40',
2019-04-20T18:16:20+0200  'path': 'info',
2019-04-20T18:16:20+0200  'type': 'nodeinfo',
2019-04-20T18:16:20+0200  'webcluster_oid': '8ae38960-f291-403a-a1dd-2f6b53dfc2c9'}
2019-04-20T18:16:20+0200 
2019-04-20T18:16:20+0200 2019-04-20T18:16:20+0200 [Container   28563] Management API event <on_webservice_added> published:
2019-04-20T18:16:20+0200 {'oid': 'd424fa2a-2afb-4cba-b03e-8f8f56f6ac40', 'webcluster_oid': '8ae38960-f291-403a-a1dd-2f6b53dfc2c9', 'path': 'info', 'type': 'nodeinfo'}
2019-04-20T18:16:20+0200 {'oid': 'd424fa2a-2afb-4cba-b03e-8f8f56f6ac40', 'webcluster_oid': '8ae38960-f291-403a-a1dd-2f6b53dfc2c9', 'path': 'info', 'type': 'nodeinfo'}
2019-04-20T18:16:20+0200 2019-04-20T18:16:20+0200 [Container   28563] WebClusterManager.add_webcluster_service(webcluster_oid=8ae38960-f291-403a-a1dd-2f6b53dfc2c9, path=temp, webservice={'type': 'static', 'directory': '/tmp', 'options': {'enable_directory_listing': True}}, details=CallDetails(registration=<autobahn.wamp.request.Registration object at 0x7fb7e55ff598>, progress=None, caller=6236535363633262, caller_authid=superuser, caller_authrole=owner, procedure=<crossbarfabriccenter.mrealm.webcluster.add_webcluster_service>, enc_algo=None, forward_for=None))
2019-04-20T18:16:20+0200 2019-04-20T18:16:20+0200 [Container   28563] New WebService object stored in database:
2019-04-20T18:16:20+0200 
2019-04-20T18:16:20+0200 {'directory': '/tmp',
2019-04-20T18:16:20+0200  'oid': '925e08db-8906-4956-8a78-5b996a55fc52',
2019-04-20T18:16:20+0200  'options': {'enable_directory_listing': True},
2019-04-20T18:16:20+0200  'path': 'temp',
2019-04-20T18:16:20+0200  'type': 'static',
2019-04-20T18:16:20+0200  'webcluster_oid': '8ae38960-f291-403a-a1dd-2f6b53dfc2c9'}
2019-04-20T18:16:20+0200 
2019-04-20T18:16:20+0200 2019-04-20T18:16:20+0200 [Container   28563] Management API event <on_webservice_added> published:
2019-04-20T18:16:20+0200 {'oid': '925e08db-8906-4956-8a78-5b996a55fc52', 'directory': '/tmp', 'options': {'enable_directory_listing': True}, 'webcluster_oid': '8ae38960-f291-403a-a1dd-2f6b53dfc2c9', 'path': 'temp', 'type': 'static'}
2019-04-20T18:16:20+0200 {'oid': '925e08db-8906-4956-8a78-5b996a55fc52', 'directory': '/tmp', 'options': {'enable_directory_listing': True}, 'webcluster_oid': '8ae38960-f291-403a-a1dd-2f6b53dfc2c9', 'path': 'temp', 'type': 'static'}
2019-04-20T18:16:20+0200 2019-04-20T18:16:20+0200 [Container   28563] WebClusterManager.delete_webcluster(details=CallDetails(registration=<autobahn.wamp.request.Registration object at 0x7fb7e55ff638>, progress=None, caller=6236535363633262, caller_authid=superuser, caller_authrole=owner, procedure=<crossbarfabriccenter.mrealm.webcluster.delete_webcluster>, enc_algo=None, forward_for=None))
2019-04-20T18:16:20+0200 2019-04-20T18:16:20+0200 [Container   28563] MrealmController.onUserError(): "AssertionError: "
2019-04-20T18:16:20+0200 Traceback (most recent call last):
2019-04-20T18:16:20+0200   File "/home/oberstet/cpy373_1/lib/python3.7/site-packages/autobahn/wamp/protocol.py", line 1006, in onMessage
2019-04-20T18:16:20+0200     on_reply = txaio.as_future(endpoint.fn, *invoke_args, **invoke_kwargs)
2019-04-20T18:16:20+0200   File "/home/oberstet/cpy373_1/lib/python3.7/site-packages/txaio/tx.py", line 428, in as_future
2019-04-20T18:16:20+0200     return ensureDeferred(fun(*args, **kwargs))
2019-04-20T18:16:20+0200   File "/home/oberstet/cpy373_1/lib/python3.7/site-packages/twisted/internet/defer.py", line 911, in ensureDeferred
2019-04-20T18:16:20+0200     return _cancellableInlineCallbacks(coro)
2019-04-20T18:16:20+0200   File "/home/oberstet/cpy373_1/lib/python3.7/site-packages/twisted/internet/defer.py", line 1529, in _cancellableInlineCallbacks
2019-04-20T18:16:20+0200     _inlineCallbacks(None, g, status)
2019-04-20T18:16:20+0200 --- <exception caught here> ---
2019-04-20T18:16:20+0200   File "/home/oberstet/cpy373_1/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
2019-04-20T18:16:20+0200     result = g.send(result)
2019-04-20T18:16:20+0200   File "/home/oberstet/scm/crossbario/crossbarfx/crossbarfx/master/mrealm/webcluster.py", line 255, in delete_webcluster
2019-04-20T18:16:20+0200     del self.schema.webclusters[txn, oid]
2019-04-20T18:16:20+0200   File "/home/oberstet/cpy373_1/lib/python3.7/site-packages/zlmdb/_pmap.py", line 463, in __delitem__
2019-04-20T18:16:20+0200     _idx_key = struct.pack('>H', index.pmap._slot) + index.pmap._serialize_key(index.fkey(value))
2019-04-20T18:16:20+0200   File "/home/oberstet/cpy373_1/lib/python3.7/site-packages/zlmdb/_types.py", line 272, in _serialize_key
2019-04-20T18:16:20+0200     assert type(key) == six.text_type
2019-04-20T18:16:20+0200 builtins.AssertionError: 
2019-04-20T18:16:20+0200 

pytest-runner required for install

  • ZLMDB version:20.4.1
  • Python version:3.6.8
  • Operating System:Redhat 8 Linux

Description

I am trying to install crossbar.io (20.7.1) using an internal pypi server. zlmdb fails to install because of missing pytest-runner pkg, it should not fail as you can see in logs that internal pypi server has access to pytest-runner==5.2 so as a part of the requirements-min.txt, crossbar.io install should install zlmdb 20.4.1 as well. Linux server where I am trying to install crossbar.io is cutoff from the internet

What I Did

(venv) [user@redhat8 crossbar]$ pip3 install -r requirements-min.txt --index-url=http://internal_pypi_server:9700 --trusted-host internal_pypi_server
Ignoring twisted: markers 'sys_platform == "darwin"' don't match your environment
Ignoring twisted: markers 'sys_platform == "win32"' don't match your environment
Collecting pytest-runner==5.2 (from -r requirements-min.txt (line 1))
  Downloading http://internal_pypi_server:9700/packages/pytest_runner-5.2-py2.py3-none-any.whl
Collecting zlmdb==20.4.1 (from -r requirements-min.txt (line 2))
  Downloading http://internal_pypi_server:9700/packages/zlmdb-20.4.1.tar.gz (129kB)
    100% |████████████████████████████████| 133kB 33.7MB/s
    Complete output from command python setup.py egg_info:
    Download error on https://pypi.org/simple/pytest-runner/: [Errno 101] Network is unreachable -- Some packages may not be found!
    Couldn't find index page for 'pytest-runner' (maybe misspelled?)
    Download error on https://pypi.org/simple/: [Errno 101] Network is unreachable -- Some packages may not be found!
    No local packages or working download links found for pytest-runner
    ....SNIP....
    distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('pytest-runner')

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-5w4ezhz4/zlmdb/


requirements-min.txt

pytest-runner==5.2
zlmdb==20.4.1
setuptools==53.0.0
setuptools_scm==5.0.1
....SNIP...

Request: please add examples

  • ZLMDB version:
  • Python version:
  • Operating System:

Description

I can't find any demo examples for zlmdb

What I Did

Searched the github source

Paste the command(s) you ran and the output.
If there was a crash, please include the traceback here.

It might be good to create a /examples directory to the repo, and populate it with examples ranging from the simplest and most basic to examples of moderate complexity.

Do you really need `numpy==1.15.4`?

  • ZLMDB version: >=19,2,2
  • Python version: *
  • Operating System: *

The requirement numpy==1.15.4 seems a little narrow. Could this potentially be relaxed?

traceback with zero length keys

It should be possible to watch for all keys using

  • keys = [KeySet(b'm', prefix=True)}

when providing a filter for etcd.watch(keys, on_change).

However, this triggers a bug:

Traceback (most recent call last):
  File "/home/oberstet/cpy370_5/lib/python3.7/site-packages/twisted/internet/defer.py", line 460, in callback
    self._startRunCallbacks(result)
  File "/home/oberstet/cpy370_5/lib/python3.7/site-packages/twisted/internet/defer.py", line 568, in _startRunCallbacks
    self._runCallbacks()
  File "/home/oberstet/cpy370_5/lib/python3.7/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/home/oberstet/cpy370_5/lib/python3.7/site-packages/twisted/internet/defer.py", line 1475, in gotResult
    _inlineCallbacks(r, g, status)
--- <exception caught here> ---
  File "/home/oberstet/cpy370_5/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
  File "tests/test_etcd.py", line 18, in main
    d = etcd.watch(keys, on_change)
  File "/home/oberstet/scm/crossbario/txaio-etcd/txaioetcd/_client_tx.py", line 409, in watch
    d = self._start_watching(keys, on_watch, filters, start_revision, return_previous)
  File "/home/oberstet/scm/crossbario/txaio-etcd/txaioetcd/_client_tx.py", line 440, in _start_watching
    range_end = _increment_last_byte(key.key)
  File "/home/oberstet/scm/crossbario/txaio-etcd/txaioetcd/_types.py", line 47, in _increment_last_byte
    s[-1] = s[-1] + 1
builtins.IndexError: bytearray index out of range

Free space bug

The default DB mode in zLMDB at some point was changed to writemap=True - which is great! BUT: that also meant that map_size bytes are immediately used on disk - which is also cool - BUT: that in turn meant that the free space checker in the master node DB fails badly

2022-03-01T16:42:35+0100 [Container   36734] DomainController global database opened [dbpath=/home/oberstet/scm/typedefint/crossbar-cluster/stack/.recordevolution/master/.crossbar/.db-controller, maxsize=134217728]
2022-03-01T16:42:35+0100 [Container   36734] SUPERUSER already exists in database (oid=c22c8f64-b193-444e-a15d-0675d8b52990)
2022-03-01T16:42:35+0100 [Container   36734] Default management realm enabled
2022-03-01T16:42:35+0100 [Container   36734] ok, default management realm already exists (18cff99c-216d-40b4-92d7-bc8aa3844da9)
2022-03-01T16:42:35+0100 [Container   36734] Watch-to-pair enabled
2022-03-01T16:42:35+0100 [Container   36734] Environment variable "$CROSSBARFX_WATCH_TO_PAIR" not set - needed in configuration "auto_default_mrealm.watch_to_pair"
2022-03-01T16:42:35+0100 [Container   36734] skipping to watch "None" for node auto-pairing - not a directory!
2022-03-01T16:42:35+0100 [Container   36734] Global master database full! Initiating EMERGENCY SHUTDOWN .. [0.0% free]

def stats(self):

https://github.com/crossbario/crossbar/blob/cc862e53b67d0f8c5250be07fc7ac93bf3c90306/crossbar/master/node/controller.py#L512

Add tests for select() on tuple keyed pmaps

For example, see https://github.com/crossbario/crossbarfx/issues/308#issuecomment-495584520:

@table('256a071f-5aeb-47f3-8786-97cd8281bdb7', build=MNodeLog.build, cast=MNodeLog.cast)
class MNodeLogs(MapTimestampUuidFlatBuffers):
    """
    Persisted managed node heartbeat log records.
    Map :class:`zlmdb.MapTimestampUuidFlatBuffers` from ``(timestamp, node_id)`` to :class:`crossbarfx.cfxdb.logs.MNodeLog`
    """

https://github.com/crossbario/crossbarfx/blob/3f0dba95a2590f69d83166d2721b548fc0595d0b/cfxdb/cfxdb/log.py#L1140

This is the part to be fixed here (touching self.schema.mnode_logs):

with self.db.begin() as txn2:
        self.log.debug('RECORDS in mnode_logs: {cnt}', cnt=self.schema.mnode_logs.count(txn2))
        for ts, _node_id in sorted(
                self.schema.mnode_logs.select(
                    txn2,
                    # FIXME: using the following breaks stuff (*)
                    # from_key=(ts_prev_min, node_id),
                    # to_key=(ts_prev_min_to, node_id),
                    return_values=False)):
            # FIXME: see above (*)
            if ts >= ts_prev_min and ts < ts_prev_min_to:
                lrec = self.schema.mnode_logs[txn2, (ts, _node_id)]
                res['routers'] += lrec.routers
                res['containers'] += lrec.containers
                res['guests'] += lrec.guests
                res['proxies'] += lrec.proxies
                res['marketmakers'] += lrec.marketmakers

Remove bundled flatbuffers module

  • ZLMDB version: 20.8.1
  • Python version: 3.8.5
  • Operating System: macOS 10.15.6 (Catalina)

Description

Hello, it seems to be that zlmdb is exporting the flatbuffers package.

I say this because while packaging zlmdb, I see it installing a flatbuffers directory into place in .../site-packages, and also per this line here in 20.8.1's setup.py:

zlmdb/setup.py

Line 62 in 1eabd29

'flatbuffers',

This is causing issues when trying to install the official flatbuffers package separately.

I'm just trying to understand if this is indeed the case, and if so, why it would be necessary?

Implement etcd zlmdb replica

  • watch() etcd for key-value changes and mirror those in a local zlmdb.
  • when writing to the local zlmdb, the modified key-values are tracked and submitted as a single transaction first to etcd
  • when doing so, the compare() guard for the etcd transaction is checking that none of the data touched in etcd has been modified since the transaction began (using object versions)
  • the local zlmdb can have typed key-values, a notion of tables and indices etc
  • the etcd in the background serves as a logical single point of strongly consistent database backed truth (though etcd can of course be run in a HA multi node fashion)

note: the issue here might supersede crossbario/txaio-etcd#3

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.