Hello I am finding it difficult to implement the Languages option in mmdbwriter.options

do we need the Descripton map option also?

Is there a specific document on the data structure and examples related to city

NOTE: This is a issue that does not happen if the developer does not work with netip or manual constructions of net.IP that does not match the maximum size of an IP of the given tree options. More details below.

I'm currently playing around with this library and I ran into the issue of mismatched IP Mask sizes when using a mixed IPV4/IPv6 compatible tree. This targets the following line within this library:

NOTE: The fix can be matching the size with the input ip, or matching the input ip with the mask size. More details below.

Example

Here is a minimal example code to produce the issue I had including "the correct way" for a lookup:

package main

import (
	"fmt"
	"log"
	"net"

	"github.com/maxmind/mmdbwriter"
	"github.com/maxmind/mmdbwriter/mmdbtype"
)

func main() {
	// Create a new writer
	writer, err := mmdbwriter.New(mmdbwriter.Options{DatabaseType: "Bug Report"})
	if err != nil {
		log.Fatal(err)
	}

	// IP we will use
	ip, network, _ := net.ParseCIDR("1.0.0.0/24")

	// Write a small sample
	err = writer.Insert(network, mmdbtype.Map{"country_code": mmdbtype.String("AU")})
	if err != nil {
		log.Fatal(err)
	}

	// "Incorrect" way for lookup
	{
		rNetwork, rData := writer.Get(net.IP{0x01, 0, 0, 0}) // This is a valid IP struct as referenced below
		fmt.Println(rNetwork, rData.(mmdbtype.Map)["country_code"]) // rNetwork should not print <nil>
	}

	// "Correct" way for lookup
	{
		rNetwork, rData := writer.Get(ip) // `net.IPv4(0x01, 0, 0, 0)` would also be possible as this is more explicit to represent a IPv4 with 16bytes
		fmt.Println(rNetwork, rData.(mmdbtype.Map)["country_code"])
	}
}

"Bug" in this library

The only issue is the manual construction of the *net.IPNet here:

Everything can be followed back to this specific sentence in the documentation for net.IP (here): "[...] accept either 4-byte (IPv4) or 16-byte (IPv6) slices as input. [...]". Internally they are not able to to automatically match the IP and Mask sizes, e.g. https://cs.opensource.google/go/go/+/refs/tags/go1.20.6:src/net/ip.go;l=497 , which results in an "incorrect" net.IPNet struct.

Thereby, matching the byte length of the input ip with the mask should fix this bug (or the other way around). A fix would be really helpful, as using netip is really useful to e.g. jump to the next ip. I'm also willing to submit a fix on my own if requested.

When I processed 250 million pieces of data, the memory usage reached about 60G, and the writeto method was called. The final mmdb file generated was 0kb in size, that is, no data was written. Does anyone know what happened?

I want the same CIDR, eg 1.0.0.1/24 to be added to different countries US and CN in my customized Country.mmdb file.

How can I do that?

I have created an mmdb file using the golang mmdbwriter, but had a hard time reading it geoip2 readers because of exceptions while reading the file (https://pypi.org/project/geoip2/).

I'm wondering if there is way to create a geoip2 compatible mmdb that could be read by geoip2 readers.

What is the actual difference between geoip2 and mmdbwriter?

It would be awesome if this repository could follow semantic versioning releases.

I am using the sample code as a reference to add specific ipv4 address to my mmdb file. For every ip i am using i keep getting a similar error to the following one

2021/03/26 06:33:41 attempt to insert ::a00:0/106, which is in a reserved network

sonmething particular about this error is that is showing an ipv6 format in the error

`package main

import (
"log"
"net"
"os"

"github.com/maxmind/mmdbwriter"
"github.com/maxmind/mmdbwriter/inserter"
"github.com/maxmind/mmdbwriter/mmdbtype"

)

func main() {
// Load the database we wish to enrich.
writer, err := mmdbwriter.Load("GeoIP2-Country.mmdb", mmdbwriter.Options{})
if err != nil {
log.Fatal(err)
}

// Define and insert the new data.
_, sreNet, err := net.ParseCIDR("10.0.0.0/10")
if err != nil {
	log.Fatal(err)
}
sreData := mmdbtype.Map{
	"AcmeCorp.DeptName": mmdbtype.String("SRE"),
	"AcmeCorp.Environments": mmdbtype.Slice{
		mmdbtype.String("development"),
		mmdbtype.String("staging"),
		mmdbtype.String("production"),
	},
}
if err := writer.InsertFunc(sreNet, inserter.TopLevelMergeWith(sreData)); err != nil {
	log.Fatal(err)
}

`

this is also happening with other ip address like
172.16.0.0 and so on

Hi,

I am going to use this repo to generate the self-defined ASN mmdb file. However, I have
met a problem that some ip ranges(such as: 10.100.64.0/24) can not be inserted because of error comes. The error follows:

/private/var/folders/sg/348d2cj950jd2989xn9wrmk40000gn/T/___go_build_github_com_maxmind_mmdbwriter_examples_asn_writer
2021/08/28 18:24:56 attempt to insert ::a64:4000/120, which is in a reserved network

I check this code and find the code lines of error comes. But,I cannot understand why this ip ranges
can not allowed to insert,can you give me some help. I am appreciated of you!

Mark.

A big.Int value has 32 bytes in size

• 1 byte for a bool
• 8 bytes for a slice pointer (one word)
• 8 bytes for the slice length (one word)
• 8 bytes for the slice capacity (one word)
• 7 padding bytes

The number that it is holding only has 16 bytes, so using a big.Int adds an extra allocation, indirection and triples the memory use.

I propose the type Uint128 to be changed to:

type Uint128 struct { High, Low uint64 }

There are libraries in the community that define a Uint128 with its operations. I don't think there is a need for any operations here, so we could have the serialization and deserialization hand-coded here (using the binary package). If we don't want to deal with those operations we could depend on one such library.

For consumers that already have a big.Int we could provide a Uint128FromBig(*big.Int) Uint128 function.

This would be a braking change on the API, but that is ok by the status on the README.

github.com/maxmind/mmdbwriter/mmdbtype

../go/src/github.com/maxmind/mmdbwriter/mmdbtype/types.go:290:23: syntax error: unexpected b00100000, expecting comma or )
../go/src/github.com/maxmind/mmdbwriter/mmdbtype/types.go:300:23: syntax error: unexpected b00101000, expecting comma or )
../go/src/github.com/maxmind/mmdbwriter/mmdbtype/types.go:314:23: syntax error: unexpected b00110000, expecting comma or )
../go/src/github.com/maxmind/mmdbwriter/mmdbtype/types.go:331:23: syntax error: unexpected b00111000, expecting comma or )

The Perl writer allows inserting IP ranges as well as IP networks. The Go writer only supports IP networks currently, so users requiring range insertions must convert ranges to networks manually first.

I rewrote my code like this, But it still doesn't work, you can try it:

writer, err := mmdbwriter.New(
	mmdbwriter.Options{
		DatabaseType: "GeoLite2-City",
			RecordSize: 24,
			IPVersion: 4,
			IncludeReservedNetworks: true,
			Languages: []string{"zh-CN"},
	},
)
if err != nil {
	return err
}

nameId := 0
for i:=0; i<len(customData); i++ {
	record := mmdbtype.Map{}
	record["continent"] = mmdbtype.Map{
		"geoname_id": mmdbtype.Uint64(nameId),
		"code": mmdbtype.String(""),
		"names":mmdbtype.Map{"zh-CN": mmdbtype.String("")},
	}
	nameId++

	record["country"] = mmdbtype.Map{
		"geoname_id": mmdbtype.Uint64(nameId),
		"iso_code": mmdbtype.String(""),
		"names":mmdbtype.Map{"zh-CN": mmdbtype.String(customData[i].Country)},
	}
	nameId++

	record["subdivisions"] = mmdbtype.Slice{
		mmdbtype.Map{
			"geoname_id": mmdbtype.Uint64(nameId),
			"iso_code": mmdbtype.String(""),
			"names":mmdbtype.Map{"zh-CN": mmdbtype.String(customData[i].Province)},
	  },
	}
	nameId++

	record["city"] = mmdbtype.Map{
		"geoname_id": mmdbtype.Uint64(nameId),
		"names":mmdbtype.Map{"zh-CN": mmdbtype.String(customData[i].City)},
	}
	nameId++

	record["county"] = mmdbtype.Map{
		"geoname_id": mmdbtype.Uint64(nameId),
		"names":mmdbtype.Map{"zh-CN": mmdbtype.String(customData[i].County)},
	}
	nameId++

	record["location"] = mmdbtype.Map{
		"accuracy_radius": mmdbtype.Uint16(0),
		"latitude": mmdbtype.Float64(0),
		"longitude": mmdbtype.Float64(0),
		"metro_code": mmdbtype.Uint64(0),
		"time_zone": mmdbtype.String(""),
	}
	nameId++

	record["postal"] = mmdbtype.Map{
		"code": mmdbtype.String(""),
	}
	nameId++

	record["registered_country"] = mmdbtype.Map{
		"geoname_id": mmdbtype.Uint64(nameId),
		"is_in_european_union": mmdbtype.Bool(false),
		"iso_code": mmdbtype.String(""),
		"names": mmdbtype.Map{"zh-CN": mmdbtype.String("")},
	}
	nameId++

	record["represented_country"] = mmdbtype.Map{
		"geoname_id": mmdbtype.Uint64(nameId),
		"is_in_european_union": mmdbtype.Bool(false),
		"iso_code": mmdbtype.String(""),
		"type": mmdbtype.String(""),
		"names": mmdbtype.Map{"zh-CN": mmdbtype.String("")},
	}
	nameId++

	tmpIp := net.ParseIP(customData[i].ipVal)
	writer.Insert(&net.IPNet{IP: tmpIp, Mask: tmpIp.DefaultMask()}, record)
}
fh, err := os.Create("out/out.mmdb")
if err != nil {
	log.Fatal(err)
}
_, err = dbWriter.WriteTo(fh)
if err != nil {
	log.Fatal(err)
}

func TestIPV6(t *testing.T) {

	fip, err := os.Open("./ipv6data.txt")
	if err != nil {
		fmt.Println("read file fail", err)
		return
	}
	defer fip.Close()
	buf := bufio.NewReader(fip)
	//一行一个
	//一行行读取
	scanner := bufio.NewScanner(buf)

	// Read each line until the end of the file
	count := 0
	for scanner.Scan() {
		line := scanner.Text()
		// Process the line as needed (e.g., print it)
		//去掉收尾空格
		line = strings.TrimSpace(line)

		count += 1
		if count == 1000 {
			break
		}

		// | 分割
		s := strings.Split(line, "|")
		//Insert(s, s[0], writer)
		ip1 := net.ParseIP(s[0])
		ip2 := net.ParseIP(s[1])

		startNetIP, ok := netipx.FromStdIP(ip1)
		if !ok {
			fmt.Println("start IP is invalid")
		}
		endNetIP, ok := netipx.FromStdIP(ip2)
		if !ok {
			fmt.Println("end IP is invalid")
		}

		r := netipx.IPRangeFrom(startNetIP, endNetIP)
		if !r.IsValid() {
			continue
		}
		//inputGeo = flag.String("i", "./GeoLite2-City.testmmdb", "Input GeoLite2-City.mmdb file path.")
		writer, _ := mmdbwriter.Load("./GeoLite2-City.testmmdb", mmdbwriter.Options{
			IncludeReservedNetworks: true,
			Description:             map[string]string{"en": fmt.Sprintf("Compiled with mmdb-editor (%v) https://github.com/iglov/mmdb-editor", Version)},
		})

		data := mmdbtype.Map{
			"country": mmdbtype.Map{
				"country":       mmdbtype.String("11"),
				"province":      mmdbtype.String("11"),
				"city":          mmdbtype.String("11"),
				"en_short_code": mmdbtype.String("11"),
			},
		}

		_, network, err := net.ParseCIDR(s[0] + "/64")
		if err != nil {
			log.Fatal(err)
		}

		err = writer.Insert(network, data)
		if err != nil {
			log.Fatal(err)
		}
		//err := writer.InsertRange(ip1, ip2, data)
		//if err != nil {
		//	fmt.Println("error1", err)
		//}
		fh, err := os.Create("./GeoLite2-ipv6.mmdb")
		if err != nil {
			fmt.Println("error2", err)
		}

		defer Check(fh.Close)

		_, err = writer.WriteTo(fh)
		if err != nil {
			fmt.Println("error3", err)
		}
	}

}

After importing 10000 rows of data, it was found that the exported database size is 2K

Hello! I'm curious if there is a max size or number of records in an mmdb database file?

Right now the Tree.Insert function defaults to the inserter.ReplaceWith strategy. You have to use Tree.InsertFunc to use any other strategy.

It would be nice if we could choose the default for Tree.Insert. Something close to what this diff achieves:

diff --git a/tree.go b/tree.go
index 4e436a8..ce0fca9 100644
--- a/tree.go
+++ b/tree.go
@@ -72,6 +72,11 @@ type Options struct {
        // implementations that do not correctly handle metadata pointers. Its
        // use should primarily be limited to existing database types.
        DisableMetadataPointers bool
+
+       // Inserter is the insert function used when calling `Insert`. It defaults
+       // to `inserter.ReplaceWith`, which replaces any conflicting old value
+       // entirely with the new.
+       Inserter func(value mmdbtype.DataType) (mmdbtype.DataType, error)
 }
 
 // Tree represents an MaxMind DB search tree.
@@ -88,6 +93,7 @@ type Tree struct {
        treeDepth               int
        // This is set when the tree is finalized
        nodeCount int
+       inserterFunc func(value mmdbtype.DataType) (mmdbtype.DataType, error)
 }
 
 // New creates a new Tree.
@@ -101,6 +107,7 @@ func New(opts Options) (*Tree, error) {
                ipVersion:               6,
                recordSize:              28,
                root:                    &node{},
+               inserterFunc:            inserter.ReplaceWith,
        }
 
        if opts.BuildEpoch != 0 {
@@ -123,6 +130,10 @@ func New(opts Options) (*Tree, error) {
                tree.recordSize = opts.RecordSize
        }
 
+       if opts.Inserter != nil {
+               tree.inserterFunc = opts.Inserter
+       }
+
        switch tree.ipVersion {
        case 6:
                tree.treeDepth = 128
@@ -214,7 +225,7 @@ func Load(path string, opts Options) (*Tree, error) {
 //
 // This is not safe to call from multiple threads.
 func (t *Tree) Insert(network *net.IPNet, value mmdbtype.DataType) error {
-       return t.InsertFunc(network, inserter.ReplaceWith(value))
+       return t.InsertFunc(network, t.inserterFunc(value))
 }
 
 // InsertFunc will insert the output of the function passed to it. The argument

I want to insert some ips into a new mmdb file.
The code for inserting custom data:

writer, err := mmdbwriter.New(
	mmdbwriter.Options{
		DatabaseType: "GeoLite2-City",
		RecordSize:   24,
		Languages: []string{"en"},
	},
)
if err != nil {
	return err
}

nameId := 0
for i:=0; i<len(customData); i++ {
		record := mmdbtype.Map{}
		record["country"] = mmdbtype.Map{"geoname_id": mmdbtype.Uint64(nameId),"names":mmdbtype.Map{"en": mmdbtype.String(customData[i].Country)}}
		nameId++

		record["province"] = mmdbtype.Map{"geoname_id": mmdbtype.Uint64(nameId),"names":mmdbtype.Map{"en": mmdbtype.String(customData[i].Province)}}
		nameId++

		record["city"] = mmdbtype.Map{"geoname_id": mmdbtype.Uint64(nameId),"names":mmdbtype.Map{"en": mmdbtype.String(customData[i].City)}}
		nameId++

		record["county"] = mmdbtype.Map{"geoname_id": mmdbtype.Uint64(nameId),"names":mmdbtype.Map{"en": mmdbtype.String(customData[i].County)}}
		nameId++

		tmpIp := net.ParseIP(customData[i].ipVal)
		writer.Insert(&net.IPNet{IP: tmpIp, Mask: tmpIp.DefaultMask()}, record)
}
fh, err := os.Create("out/out.mmdb")
if err != nil {
	log.Fatal(err)
}
_, err = writer.WriteTo(fh)
if err != nil {
	log.Fatal(err)
}

Then I search ip for testing:

city, err := geoip2Reader.City(net.ParseIP(ipStr))
	if err == nil {
		marshal, _ := json.Marshal(city)
		zap.S().Info(string(marshal))
	}

The result of searching is not correct.
Am I doing something wrong?

I am very new to both Go and the inner workings of MaxMind DB. My goal is to merge my private IP ranges into an existing City MMDB. I have been able to insert most data elements but I have not figured out how to add subdivisions as a JSON array in Go.

This does not produce an array:

		recordData := mmdbtype.Map{
			"city": mmdbtype.Map{
				"names": mmdbtype.Map{
					"en": mmdbtype.String(vlanLoc.City + ": " + vlanLoc.AdNet),
				},
			},
			"subdivisions": mmdbtype.Map{
				"iso_code": mmdbtype.String(vlanLoc.Province),
				"names": mmdbtype.Map{
					"en": mmdbtype.String("subdivison data here"),
				},
			},

Putting array brackets "[]" in from of the "mmdbtype.Map" triggers this error in the compiler.

cannot use []mmdbtype.Map literal (type []mmdbtype.Map) as type mmdbtype.DataType in map value:
	[]mmdbtype.Map does not implement mmdbtype.DataType (missing Copy method)

Example of the above:

		subdivisions := make([]mmdbtype.Map, 1)
		subdivisions[0] = mmdbtype.Map{
			"iso_code": mmdbtype.String(vlanLoc.Province),
			"names":    mmdbtype.Map{"en": mmdbtype.String("subdivison data here")},
		}

		recordData := mmdbtype.Map{
			"city": mmdbtype.Map{
				"names": mmdbtype.Map{
					"en": mmdbtype.String(vlanLoc.City + ": " + vlanLoc.AdNet),
				},
			},
			"subdivisions": subdivisions,

How can this be constructed so that an array like the following is added to the MMDB?

                    "subdivisions": [
                        {
                            "geoname_id": 5128638,
                            "iso_code": "NY",
                            "names": {
                                "de": "New York",
                                "en": "New York",
                            }
                        }
                    ]