dobatymo / lmdb-python-dbm Goto Github PK
View Code? Open in Web Editor NEWPython DBM style wrapper of LMDB
License: ISC License
lmdb-python-dbm's People
Stargazers
Watchers
lmdb-python-dbm's Issues
Add Shelve to the benchmark
Something like this:
import shelve
class ShelveBenchmark(BaseBenchmark):
def __init__(self, db_tpl):
super().__init__(db_tpl, "shelve")
def open(self):
return shelve.open(self.path)max_dbs
Is max_dbs > 1 supported?
Add support for custom max_readers
Hello,
(first of all, thanks for this library!)
I've been struggling with trying to use many concurrent workers reading from a database using this library. lmdb has a default parameter of 126 max_readers. Indeed, when I tried to use 128 concurrent workers, I got the expected error
mdb_txn_begin: MDB_READERS_FULL: Environment maxreaders limit reached
I saw that lmdbm doesn't expose the max_readers parameter, so I tried subclassing the Lmdb class to expose it in the open classmethod; however, when trying to set it to more than 126 and use more than 126 concurrent workers, I get the error:
mdb_txn_begin: Invalid argument
Remark that it's really an "and": if I set max_readers to whatever value > 126, but only use 126 concurrent workers or fewer, the error doesn't appear. So for practical purposes, setting max_readers flag to be > 126 is only changing the error I'm getting (and minor things like the lock.mdb file size), but not the practical results.
(to be double sure I didn't mess up with subclassing, I even tried hard-coding in the lmdbm code the line:
env = lmdb.open(file, map_size=map_size, max_dbs=1, readonly=True, create=False, mode=mode, max_readers=256)
but I got the exact same error)
Am I missing something in lmdb(m) implementation, or is there a bug somewhere? I wasn't able to track down where this error is coming from exactly...
Question: How does autogrow work?
Hi. I'm trying to understand how autogrow works from the code below. Could you please explain? Thanks
except lmdb.MapFullError:
if not self.autogrow:
raise
new_map_size = self.map_size * 2
self.map_size = new_map_size
logger.info(self.autogrow_msg, self.env.path(), new_map_size)
Add init keyword pass through?
Further to #6, it would be valuable to allow pass-through of keywords on the Lmdb.open() function to the Environment's constructor. Then we could do:
cxn = Lmdb.open('file.lmdb', readahead=False)
for example.
It's currently hard to override the open function on a subclass if you want to respect n as a flag, as the module doesn't export the remove_lmdbm() function.
mypy is not happy
Sorry, I'm not an expert on type annotations, I added this library and ran mypy, that was it.
> mypy my-package-src
lib/python3.10/site-packages/lmdbm/lmdbm.py:28: error: syntax error in type comment "(bool, ) -> None"
lib/python3.10/site-packages/lmdbm/lmdbm.py:98: error: syntax error in type comment "(int, ) -> None"
lib/python3.10/site-packages/lmdbm/lmdbm.py:103: error: syntax error in type comment "(object, ) -> bytes"
lib/python3.10/site-packages/lmdbm/lmdbm.py:113: error: syntax error in type comment "(bytes, ) -> KT"
lib/python3.10/site-packages/lmdbm/lmdbm.py:118: error: syntax error in type comment "(VT, ) -> bytes"
lib/python3.10/site-packages/lmdbm/lmdbm.py:128: error: syntax error in type comment "(bytes, ) -> VT"
lib/python3.10/site-packages/lmdbm/lmdbm.py:133: error: syntax error in type comment "(KT, ) -> VT"
lib/python3.10/site-packages/lmdbm/lmdbm.py:161: error: syntax error in type comment "(KT, ) -> None"
lib/python3.10/site-packages/lmdbm/lmdbm.py:188: error: syntax error in type comment "(object, ) -> bool"
lib/python3.10/site-packages/lmdbm/lmdbm.py:288: error: syntax error in type comment "(VT, ) -> bytes"
lib/python3.10/site-packages/lmdbm/lmdbm.py:294: error: syntax error in type comment "(bytes, ) -> VT"
Found 11 errors in 1 file (errors prevented further checking)
Version info:
lmdb @ file:/xxx/lmdb-1.3.0-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
lmdbm @ file:/xxx/lmdbm-0.0.5-py2.py3-none-any.whl
mypy @ file:/xxx/mypy-0.982-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
mypy-extensions @ file:/xxx/mypy_extensions-0.4.3-py2.py3-none-any.whl
> python
Python 3.10.6 (main, Aug 2 2022, 15:11:28) [GCC 9.4.0] on linux
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.