Comments (5)
@hyperxpro mmdbctl
uses the underlying mmdb-writer library for creating the actual MMDB, which doesn't support incremental writes. The MMDB format itself isn't very friendly to incremental writes, which is the root of the problem. It's probably not impossible, but it'd be a really complex and sensitive algorithm using partial files to help deal with the memory growth (similar to a large-scale merge-sort solution).
In practice, almost anyone producing a big enough MMDB file ends up having large RAM machines to do it, which is cheap these days, so little effort has been invested into optimizing that.
It could be something we do!
from mmdbctl.
Can we not write N bytes into the file and flush and increase offset and repeat the process? I am not sure if it is entirely possible like this but, is there any workaround we can do at the I/O level to fix this?
from mmdbctl.
@hyperxpro unfortunately, the memory and I/O model for MMDBs, and the relationship between them, is more complex than that.
See https://maxmind.github.io/MaxMind-DB for the actual MMDB spec
from mmdbctl.
For reference in my custom mmdb, it takes around 800G of ram and many hours to compile from an optimized ip_trie that takes a fraction of the time and memory to build and is threadsafe. From my shallow dive into it, a large part of it is the fact that the mmdb writer library uses reflection all over the place since it was written before real generics were part of the language.
from mmdbctl.
Hitting the same problem here, doing the same thing. MMDB->JSON->MMDB. JSON file has ~15m routes/prefixes. Un-possible thus far on any of the systems we have here (512G is max) to turn back into MMDB.
from mmdbctl.
Related Issues (16)
- Compatibility issue with IP2Location DB which uses int format for ip address range start and end HOT 2
- Support for merging mmdb or csv files? HOT 1
- MMDB with nested structures break read and export HOT 6
- Request support for installation through Scoop HOT 1
- file size difference HOT 8
- Low-level data about MMDB HOT 3
- Granular data section metadata
- File extensions not triggering default flags in import HOT 2
- is ipapi csv's conversion supported? HOT 7
- panic: runtime error on iplocation.mmdb
- err: nothing to import
- Corrupted Database
- 1.4.4 deb.sh script downloads 1.4.3 HOT 2
- err: failed to get record for next subnet: maxminddb: cannot unmarshal map into type string HOT 1
- Compatibility with maxmind datasets HOT 10
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mmdbctl.