Memory Issue about mmdbctl HOT 5 OPEN

@hyperxpro mmdbctl uses the underlying mmdb-writer library for creating the actual MMDB, which doesn't support incremental writes. The MMDB format itself isn't very friendly to incremental writes, which is the root of the problem. It's probably not impossible, but it'd be a really complex and sensitive algorithm using partial files to help deal with the memory growth (similar to a large-scale merge-sort solution).

In practice, almost anyone producing a big enough MMDB file ends up having large RAM machines to do it, which is cheap these days, so little effort has been invested into optimizing that.

It could be something we do!

from mmdbctl.

Can we not write N bytes into the file and flush and increase offset and repeat the process? I am not sure if it is entirely possible like this but, is there any workaround we can do at the I/O level to fix this?

from mmdbctl.

@hyperxpro unfortunately, the memory and I/O model for MMDBs, and the relationship between them, is more complex than that.

See https://maxmind.github.io/MaxMind-DB for the actual MMDB spec

from mmdbctl.

For reference in my custom mmdb, it takes around 800G of ram and many hours to compile from an optimized ip_trie that takes a fraction of the time and memory to build and is threadsafe. From my shallow dive into it, a large part of it is the fact that the mmdb writer library uses reflection all over the place since it was written before real generics were part of the language.

from mmdbctl.

Hitting the same problem here, doing the same thing. MMDB->JSON->MMDB. JSON file has ~15m routes/prefixes. Un-possible thus far on any of the systems we have here (512G is max) to turn back into MMDB.

from mmdbctl.