newtdb / db Goto Github PK
View Code? Open in Web Editor NEWNewt DB is a Python object-oriented database with JSONB-based access and search in PostgreSQL
Home Page: http://www.newtdb.org
License: MIT License
Newt DB is a Python object-oriented database with JSONB-based access and search in PostgreSQL
Home Page: http://www.newtdb.org
License: MIT License
This can probably be accomplished in some SQL executed in making connections.
Hello. I am trying to get confident with newtdb and followed the "getting started" tutorial on the website
I installed postgresql locally (on a raspberry) and pip installed newt.db inside a virtual environment (python 3.7.3)
This is what happened:
(venv) pi@raspberry:/tmp $ python
Python 3.7.3 (default, Jan 22 2021, 20:04:44)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import newt.db
>>> connection = newt.db.connection('')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/opt/xpend/venv/lib/python3.7/site-packages/newt/db/_db.py", line 153, in connection
ZODB.connection(storage(dsn, **storage_options), **db_options)
File "/opt/xpend/venv/lib/python3.7/site-packages/newt/db/_db.py", line 124, in storage
return relstorage.storage.RelStorage(Adapter(dsn, options), options=options)
File "/opt/xpend/venv/lib/python3.7/site-packages/newt/db/_adapter.py", line 13, in __init__
driver = relstorage.adapters.postgresql.select_driver(self.options)
AttributeError: module 'relstorage.adapters.postgresql' has no attribute 'select_driver'
I modified line 13 of newt/db/_adapter.py to become
driver = relstorage.adapters.postgresql.drivers.select_driver(self.options)
but then another exception occurred:
>>> connection = newt.db.connection('')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/opt/xpend/venv/lib/python3.7/site-packages/newt/db/_db.py", line 153, in connection
ZODB.connection(storage(dsn, **storage_options), **db_options)
File "/opt/xpend/venv/lib/python3.7/site-packages/newt/db/_db.py", line 124, in storage
return relstorage.storage.RelStorage(Adapter(dsn, options), options=options)
File "/opt/xpend/venv/lib/python3.7/site-packages/newt/db/_adapter.py", line 13, in __init__
driver = relstorage.adapters.postgresql.drivers.select_driver(self.options)
File "/opt/xpend/venv/lib/python3.7/site-packages/relstorage/adapters/drivers.py", line 386, in <lambda>
sys.modules[name])
File "/opt/xpend/venv/lib/python3.7/site-packages/relstorage/adapters/drivers.py", line 50, in _select_driver_by_name
driver_name = casefold(driver_name)
TypeError: descriptor 'casefold' requires a 'str' object but received a 'Options'
(venv) pi@raspberry:/tmp $ pip freeze
BTrees==4.9.2
cffi==1.15.0
newt.db==0.9.0
perfmetrics==3.2.0.post0
persistent==4.7.0
pkg-resources==0.0.0
psycopg2==2.9.2
pycparser==2.21
RelStorage==3.4.5
six==1.16.0
transaction==3.0.1
zc.lockfile==2.0
ZConfig==3.6.0
ZODB==5.6.0
zodbpickle==2.2.0
zope.dottedname==4.3
zope.interface==5.4.0
Should I downgrade some package? To which version?
Thanks in advance
Enrico
Has newtdb dropped Python 2 support? I get a serialization error with a very simple test case under Python 2, but not Python 3:
import logging
logging.basicConfig()
import newt.db
from BTrees._OOBTree import OOBTree
uplinkdb_con = newt.db.connection('postgresql://[email protected]/svelte')
class Profile(newt.db.Persistent):
def __init__(self):
self.rank = 'test'
root = uplinkdb_con.root
root.profiles = OOBTree()
root.profiles['somekey'] = Profile()
uplinkdb_con.commit()
Under Python 2 (Python 2.7.14, PostgreSQL 9.6.6) this causes:
ERROR:newt.db.jsonpickle:Failed pickle load, oid: 0, pickle starts: '\x80\x03cpersistent.mapping\nPersistentMapping\nq\x01.\x80\x03}q\x02U\x04'
Traceback (most recent call last):
File "/home/svelte/venv/lib/python2.7/site-packages/newt/db/jsonpickle.py", line 509, in __call__
state = unpickler.load()
File "/home/svelte/venv/lib/python2.7/site-packages/newt/db/jsonpickle.py", line 227, in load
for op, arg, pos in self.ops:
File "/usr/lib64/python2.7/pickletools.py", line 1851, in genops
code))
ValueError: at position 28, opcode 'C' unknown
Under Python 3 (Python 2 (Python 3.6.3, PostgreSQL 9.6.6) I get no output as expected. Everything looks correct in psql:
svelte=> select * from newt;
zoid | class_name | ghost_pickle | state
------+--------------------------------------+------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------
0 | persistent.mapping.PersistentMapping | \x80036370657273697374656e742e6d617070696e670a50657273697374656e744d617070696e670a71002e | {"data": {"profiles": {"::": "persistent", "id": [1, "BTrees.OOBTree.OOBTree"], "::=>": 1}}}
2 | __main__.Profile | \x8003635f5f6d61696e5f5f0a50726f66696c650a71002e | {"rank": "test"}
The only virtualenv difference is Python 2 having trollius
and futures
.
Output of pip freeze
for Python 2:
BTrees==4.5.0
cffi==1.11.5
futures==3.2.0
newt.db==0.9.0
perfmetrics==2.0
persistent==4.2.4.2
psycopg2==2.7.4
pycparser==2.18
RelStorage==2.1.0
six==1.11.0
transaction==2.2.1
trollius==2.2
zc.lockfile==1.3.0
ZConfig==3.2.0
zdaemon==4.2.0
ZEO==5.2.0
ZODB==5.4.0
zodbpickle==1.0
zope.interface==4.5.0
Output of pip freeze
for Python 3:
BTrees==4.5.0
cffi==1.11.5
newt.db==0.9.0
perfmetrics==2.0
persistent==4.2.4.2
psycopg2==2.7.4
pycparser==2.18
RelStorage==2.1.0
six==1.11.0
transaction==2.2.1
zc.lockfile==1.3.0
ZConfig==3.2.0
zdaemon==4.2.0
ZEO==5.2.0
ZODB==5.4.0
zodbpickle==1.0
zope.interface==4.5.0
Hi, I'm trying to resolve below traceback:
Failed pickle load, oid: 1203, pickle starts: b'\x80\x03ctransistor.io.persistence.scraper_newt\nSplashCo'
Traceback (most recent call last):
File "C:\ProgramData\Miniconda3\envs\pythonocc\lib\site-packages\newt\db\jsonpickle.py", line 509, in __call__
state = unpickler.load()
File "C:\ProgramData\Miniconda3\envs\pythonocc\lib\site-packages\newt\db\jsonpickle.py", line 242, in load
getattr(self, op.name)(arg)
File "C:\ProgramData\Miniconda3\envs\pythonocc\lib\site-packages\newt\db\jsonpickle.py", line 320, in SETITEMS
for i in range(0, len(marked), 2)
File "C:\ProgramData\Miniconda3\envs\pythonocc\lib\site-packages\newt\db\jsonpickle.py", line 123, in update
self.v.update(v)
AttributeError: 'Instance' object has no attribute 'update'
Environment is Python 3.6.5 on Windows 10, at the point in time where the class is being saved to newt, at the final stage of a gevent routine.
The class object being saved is a fairly simple wrapper with class attributes containing basic data types, str(), dict(), and otherwise there is one JSON byte string which contains an html page and also a .png of the html page.
And finally, even though I get this error, I can't really find a side-effect of it, the object is still persisted to newt, and the data from the JSON byte string seems to be useable. For example, I can recreate the .png from the byte string, pulling it back out from the persisted newt.db object.
But after picking through it a few hours today and unable to resolve it, figured I'd report it here to see if it stands out. Just let me know if I can provide more info. Thank you.
Lean on PostgreSQL to implement sets.
Objects grow _newt__parents__
, which is an array of containers. When a container fetches its contents, it does so by querying objects which has it as one of their parents. Of course, the GC has to be aware of this. An object without references to it would still be non-garbage if it has any non-garbage parents.
Containers "store" their contents through these references. Containers have a length. Container conflicts are always resolvable, because they behave like lengths.
A variation on this is approximately ordered collections. Collections have min and max positions that behave like zope.min-max
. When an object is added at the front, the min position is decremented and used for the object position. Similar at the back. The position becomes part of the parent pointers. Positions are not unique.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.