maxmind / geoip2-csv-converter Goto Github PK

GeoIP2 CSV Format Converter

License: Apache License 2.0

Go 100.00%

geoip2-csv-converter's Introduction

GeoIP2 CSV Format Converter

This is a simple utility for converting the MaxMind GeoIP2 and GeoLite2 CSVs to different formats for representing IP addresses such as IP ranges or integer ranges.

Compiled binaries for Linux/x86_64, Windows, and macOS (darwin) can be downloaded from the GitHub releases page.

Usage

Required:

-block-file=[FILENAME] - The name of the block CSV file to use as input.
-output-file=[FILENAME] - The file name to the output CSV

In addition, at least one of these is required:

-include-cidr - Include the network in CIDR format
-include-range - Include the IP range of the network in string format
-include-integer-range - Include the IP range of the network in integer format
-include-hex-range - Include the IP range of the network in hexadecimal format

Output

CIDR (-include-cidr)

This will include the network in CIDR notation in the network column as it is in the original CSV.

Range (-include-range)

This adds network_start_ip and network_last_ip columns. These are string representations of the first and last IP address in the network.

Integer Range (-include-integer-range)

This adds network_start_integer and network_last_integer columns. These are integer representations of the first and last IP address in the network.

Hex Range (-include-hex-range)

This adds network_start_hex and network_last_hex columns. These are hexadecimal representations of the first and last IP address in the network.

Copyright and License

This is free software, licensed under the Apache License, Version 2.0.

geoip2-csv-converter's People

Contributors

Stargazers

Watchers

geoip2-csv-converter's Issues

error reading CSV header: EOF when -block-file is the same as -output-file

I have the converter in the same folder as the csv files and I am using

geoip2-csv-converter-v1.3.0-darwin-amd64.tar.gz
1.39 MB

I run with the following

./geoip2-csv-converter -block-file GeoLite2-Country-Blocks-IPv6.csv -include-hex-range -output-file GeoLite2-Country-Blocks-IPv6.csv

however I am getting a return of error reading CSV header: EOF whenever I run the above.

Stopped working on Mac High Sierra 10.13.6

This is the output I get:

failed MSpanList_Insert 0x253000 0x1f0956b5926a 0x0
fatal error: MSpanList_Insert

runtime stack:
runtime.throw(0x1fd80b)
	/usr/local/go/src/runtime/panic.go:491 +0xad fp=0x7ffeefbff030 sp=0x7ffeefbff000
runtime.MSpanList_Insert(0x21be08, 0x253000)
	/usr/local/go/src/runtime/mheap.c:692 +0x8f fp=0x7ffeefbff058 sp=0x7ffeefbff030
MHeap_FreeSpanLocked(0x218a00, 0x253000, 0x100)
	/usr/local/go/src/runtime/mheap.c:583 +0x163 fp=0x7ffeefbff098 sp=0x7ffeefbff058
MHeap_Grow(0x218a00, 0x8, 0x0)
	/usr/local/go/src/runtime/mheap.c:420 +0x1a8 fp=0x7ffeefbff0d8 sp=0x7ffeefbff098
MHeap_AllocSpanLocked(0x218a00, 0x1, 0x0)
	/usr/local/go/src/runtime/mheap.c:298 +0x365 fp=0x7ffeefbff118 sp=0x7ffeefbff0d8
mheap_alloc(0x218a00, 0x1, 0x12, 0x0)
	/usr/local/go/src/runtime/mheap.c:190 +0x121 fp=0x7ffeefbff140 sp=0x7ffeefbff118
runtime.MHeap_Alloc(0x218a00, 0x1, 0x10000000012, 0xc969)
	/usr/local/go/src/runtime/mheap.c:240 +0x66 fp=0x7ffeefbff178 sp=0x7ffeefbff140
MCentral_Grow(0x220778, 0x0)
	/usr/local/go/src/runtime/mcentral.c:197 +0x8b fp=0x7ffeefbff1e0 sp=0x7ffeefbff178
runtime.MCentral_CacheSpan(0x220778, 0x0)
	/usr/local/go/src/runtime/mcentral.c:85 +0x167 fp=0x7ffeefbff218 sp=0x7ffeefbff1e0
runtime.MCache_Refill(0x24f000, 0x12, 0x0)
	/usr/local/go/src/runtime/mcache.c:90 +0xa0 fp=0x7ffeefbff240 sp=0x7ffeefbff218
runtime.mcacheRefill_m()
	/usr/local/go/src/runtime/malloc.c:368 +0x57 fp=0x7ffeefbff260 sp=0x7ffeefbff240
runtime.onM(0x190848)
	/usr/local/go/src/runtime/asm_amd64.s:273 +0x9a fp=0x7ffeefbff268 sp=0x7ffeefbff260
runtime.mallocgc(0x120, 0x143300, 0x0, 0x0)
	/usr/local/go/src/runtime/malloc.go:178 +0x849 fp=0x7ffeefbff318 sp=0x7ffeefbff268
runtime.newobject(0x143300, 0x24f000)
	/usr/local/go/src/runtime/malloc.go:353 +0x49 fp=0x7ffeefbff340 sp=0x7ffeefbff318
runtime.newG(0x242aa)
	/usr/local/go/src/runtime/proc.go:233 +0x2a fp=0x7ffeefbff358 sp=0x7ffeefbff340
allocg(0x20ace0)
	/usr/local/go/src/runtime/proc.c:925 +0x1f fp=0x7ffeefbff368 sp=0x7ffeefbff358
runtime.malg(0x8000, 0x20ad80)
	/usr/local/go/src/runtime/proc.c:2106 +0x1f fp=0x7ffeefbff398 sp=0x7ffeefbff368
runtime.mpreinit(0x20b140)
	/usr/local/go/src/runtime/os_darwin.c:137 +0x27 fp=0x7ffeefbff3b0 sp=0x7ffeefbff398
mcommoninit(0x20b140)
	/usr/local/go/src/runtime/proc.c:201 +0xc9 fp=0x7ffeefbff3d8 sp=0x7ffeefbff3b0
runtime.schedinit()
	/usr/local/go/src/runtime/proc.c:138 +0x55 fp=0x7ffeefbff400 sp=0x7ffeefbff3d8
runtime.rt0_go(0x7ffeefbff430, 0x1, 0x7ffeefbff430, 0x0, 0x1, 0x7ffeefbff660, 0x0, 0x7ffeefbff6ba, 0x7ffeefbff6dc, 0x7ffeefbff6f8, ...)
	/usr/local/go/src/runtime/asm_amd64.s:95 +0x116 fp=0x7ffeefbff408 sp=0x7ffeefbff400

Can't run the executable file to change ip representation from cidr to range

As I wrote in Title i have a problem to change the network column format of GeoLite2-City-Blocks-IPv4.csv database from cidr to range type.
I'm using a macOS Big Sur version 11.0.1 and i downloaded the 'geoip2-csv-converter-v1.3.0-linux-amd64.tar.gz' zipped file from the releases page of github (https://github.com/maxmind/geoip2-csv-converter/releases). After that i unzipped the file and moved to the folder in the same directory where i keep the database file, then from Terminal i went inside geoip2-csv-converter-v1.3.0 directory and tried to run the following command:

./geoip2-csv-converter -block-file=GeoLite2-City-Blocks-IPv4.csv -output-file=new_GeoLite2-City-Blocks-IPv4.csv -include-integer-range

but i got the following error:
zsh: exec format error: ./geoip2-csv-converter

Does anyone know where i am wrong with this or do you know how can i solve my problem?

File Buffer only seems to be released after the end of programs execution.

I am making a program with java that uses this utility and currently I cannot time my tasks properly because the buffer seems to be release a while after the end of execution.

Build for ARM64?

Any plans to add arm64 build?

golang-runtime from 1.15.3 to 1.21.0

Can the golang runtime version be updated from 1.15.3 to 1.21.0 as the older version has this vulnerability
https://nvd.nist.gov/vuln/detail/CVE-2022-23806

Convert city database

I want to use City database here http://dev.maxmind.com/geoip/geoip2/geoip2-csv-databases/ as input file. Is that a valid input file for this converter or input file needs to have only one CIDR column and nothing else? If this is valid, does this convertermove other columns of City database to output file too?

Dependabot couldn't find a Gopkg.toml for this project

Dependabot couldn't find a Gopkg.toml for this project.

Dependabot requires a Gopkg.toml to evaluate your project's current Go dependencies. It had expected to find one at the path: /Gopkg.toml.

If this isn't a Go project, or if it is a library, you may wish to disable updates for it from within Dependabot.

View the update logs.

Example for csv to mmdb

Is there any example for csv to mmdb covert?

I need to write CD/CI dev/ops and need to change mmdb for local ips like 10.0.0.x or 192.168.1.x for special data just for testing act of our infrastructure.

"Error: invalid CIDR address: 16777216"

./geoip2-csv-converter -include-range -block-file=GeoLiteCity-Blocks.csv -output-file=blocks.csv

To be able to run it at all I had to remove your copyright line using:
sed -i '' 1d GeoLiteCity-Blocks.csv

What am I missing?

MySql query to convert the CIDR to an ipRange

Perhaps you can add this to the repo:

ALTER TABLE GeoIP2-Country-Blocks-IPv4 ADD `start_ip` VARCHAR(20) AFTER network;
ALTER TABLE GeoIP2-Country-Blocks-IPv4 ADD `end_ip` VARCHAR(20) AFTER start_ip;

CREATE INDEX idx_start_ip ON GeoIP2-Country-Blocks-IPv4 (start_ip);
CREATE INDEX idx_end_ip ON GeoIP2-Country-Blocks-IPv4 (end_ip);

UPDATE GeoIP2-Country-Blocks-IPv4 SET 
start_ip = INET_NTOA(INET_ATON( SUBSTRING_INDEX(network, '/', 1)) 
   & 0xffffffff ^ ((0x1 << ( 32 - SUBSTRING_INDEX(network, '/', -1))  ) -1 )),
end_ip = INET_NTOA(INET_ATON( SUBSTRING_INDEX(network, '/', 1)) 
   | ((0x100000000 >> SUBSTRING_INDEX(network, '/', -1) ) -1 ));

Error: invalid CIDR address

# go run main.go -include-cidr -block-file=/tmp/GeoIPASNum2.csv -output-file=/tmp/geoip.txt
Error: invalid CIDR address: 16778240
# go run main.go -include-range -block-file=/tmp/GeoIPASNum2.csv -output-file=/tmp/geoip.txt
Error: invalid CIDR address: 16778240
# grep ^16778240 /tmp/GeoIPASNum2.csv 
16778240,16779007,"AS56203 Big Red Group"
# head /tmp/GeoIPASNum2.csv 
16777216,16777471,"AS47872 SOFIA-CONNECT-AS"
16778240,16779007,"AS56203 Big Red Group"
16779008,16779263,"AS38803 Goldenit Pty ltd Australia"
16793600,16809983,"AS18144 Energia Communications,Inc."
16809984,16813055,"AS23969 TOT Public Company Limited"
16813056,16813567,"AS9737 TOT Public Company Limited"
16813568,16813823,"AS23969 TOT Public Company Limited"
16813824,16814079,"AS9737 TOT Public Company Limited"
16814080,16832255,"AS23969 TOT Public Company Limited"
16832256,16832511,"AS9737 TOT Public Company Limited"

Stopped working on MacOS Sierra

Used to work prior to upgrading to the latest MacOS. Now I get....

fatal error: unexpected signal during runtime execution
[signal 0xb code=0x1 addr=0xb01dfacedebac1e pc=0x2bec9]

runtime stack:
runtime.gothrow(0x182ff0, 0x2a)
/usr/local/go/src/runtime/panic.go:503 +0x8e fp=0xc20802dcd8 sp=0xc20802dcc0
runtime.sigpanic()
/usr/local/go/src/runtime/sigpanic_unix.go:14 +0x5e fp=0xc20802dd28 sp=0xc20802dcd8
runtime.MSpanList_Remove(0x253c30)
/usr/local/go/src/runtime/mheap.c:676 +0x29 fp=0xc20802dd30 sp=0xc20802dd28
runtime.MCentral_UncacheSpan(0x21f578, 0x253c30)
/usr/local/go/src/runtime/mcentral.c:121 +0x7f fp=0xc20802dd50 sp=0xc20802dd30
runtime.MCache_ReleaseAll(0x24f000)
/usr/local/go/src/runtime/mcache.c:111 +0x6a fp=0xc20802dd78 sp=0xc20802dd50
flushallmcaches()
/usr/local/go/src/runtime/mgc0.c:1221 +0x61 fp=0xc20802dda0 sp=0xc20802dd78
markroot(0xc208010000, 0xc200000004)
/usr/local/go/src/runtime/mgc0.c:533 +0x20a fp=0xc20802de00 sp=0xc20802dda0
runtime.parfordo(0xc208010000)
/usr/local/go/src/runtime/parfor.c:76 +0xb2 fp=0xc20802de80 sp=0xc20802de00
gc(0xc20802dfb8)
/usr/local/go/src/runtime/mgc0.c:1439 +0x25e fp=0xc20802df98 sp=0xc20802de80
runtime.gc_m()
/usr/local/go/src/runtime/mgc0.c:1368 +0xe0 fp=0xc20802dfd0 sp=0xc20802df98
runtime.onM(0xc208012000)
/usr/local/go/src/runtime/asm_amd64.s:257 +0x68 fp=0xc20802dfd8 sp=0xc20802dfd0
runtime.mstart()
/usr/local/go/src/runtime/proc.c:818 fp=0xc20802dfe0 sp=0xc20802dfd8

goroutine 1 [garbage collection]:
runtime.switchtoM()
/usr/local/go/src/runtime/asm_amd64.s:198 fp=0xc2080759f8 sp=0xc2080759f0
runtime.gogc(0x0)
/usr/local/go/src/runtime/malloc.go:469 +0x1cf fp=0xc208075a30 sp=0xc2080759f8
runtime.mallocgc(0x10, 0xf0480, 0x1, 0x24f010)
/usr/local/go/src/runtime/malloc.go:341 +0x391 fp=0xc208075ae0 sp=0xc208075a30
runtime.newobject(0xf0480, 0x24f000)
/usr/local/go/src/runtime/malloc.go:353 +0x49 fp=0xc208075b08 sp=0xc208075ae0
github.com/mikioh/ipaddr.ipv4Int.IP(0x4176dc00, 0x0, 0x0, 0x0)
/home/greg/MaxMind/go/src/github.com/mikioh/ipaddr/helper.go:60 +0x45 fp=0xc208075b20 sp=0xc208075b08
github.com/mikioh/ipaddr.(*IPv4).Addr(0xc208033ff8, 0x0, 0x0, 0x0)
/home/greg/MaxMind/go/src/github.com/mikioh/ipaddr/ipv4.go:122 +0x45 fp=0xc208075b48 sp=0xc208075b20
github.com/maxmind/geoip2-csv-converter/convert.intRangeLine(0x260c18, 0xc208033ff8, 0xc208079f90, 0x4, 0x7, 0x0, 0x0, 0x0)
/home/greg/MaxMind/go/src/github.com/maxmind/geoip2-csv-converter/convert/convert.go:113 +0x7a fp=0xc208075c08 sp=0xc208075b48
github.com/maxmind/geoip2-csv-converter/convert.func·004(0x260c18, 0xc208033ff8, 0xc208079f90, 0x4, 0x7, 0x0, 0x0, 0x0)
/home/greg/MaxMind/go/src/github.com/maxmind/geoip2-csv-converter/convert/convert.go:84 +0xd7 fp=0xc208075c70 sp=0xc208075c08
github.com/maxmind/geoip2-csv-converter/convert.func·004(0x260c18, 0xc208033ff8, 0xc208079f90, 0x4, 0x7, 0x0, 0x0, 0x0)
/home/greg/MaxMind/go/src/github.com/maxmind/geoip2-csv-converter/convert/convert.go:84 +0x8a fp=0xc208075cd8 sp=0xc208075c70
github.com/maxmind/geoip2-csv-converter/convert.convert(0x260bc8, 0xc208042028, 0x260bf0, 0xc208042020, 0xc20801e0a0, 0xc20801e0c0, 0x0, 0x0)
/home/greg/MaxMind/go/src/github.com/maxmind/geoip2-csv-converter/convert/convert.go:156 +0x3bf fp=0xc208075dc0 sp=0xc208075cd8
github.com/maxmind/geoip2-csv-converter/convert.Convert(0x260bc8, 0xc208042028, 0x260bf0, 0xc208042020, 0x10001, 0x0, 0x0)
/home/greg/MaxMind/go/src/github.com/maxmind/geoip2-csv-converter/convert/convert.go:73 +0x163 fp=0xc208075e18 sp=0xc208075dc0
github.com/maxmind/geoip2-csv-converter/convert.ConvertFile(0x7fff5fbffa7e, 0x1a, 0x7fff5fbffad5, 0x24, 0x10001, 0x0, 0x0)
/home/greg/MaxMind/go/src/github.com/maxmind/geoip2-csv-converter/convert/convert.go:40 +0x1d3 fp=0xc208075ec0 sp=0xc208075e18
main.main()
/home/greg/MaxMind/go/src/github.com/maxmind/geoip2-csv-converter/main.go:30 +0x2cf fp=0xc208075f98 sp=0xc208075ec0
runtime.main()
/usr/local/go/src/runtime/proc.go:63 +0xf3 fp=0xc208075fe0 sp=0xc208075f98
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc208075fe8 sp=0xc208075fe0

goroutine 2 [force gc (idle)]:
runtime.gopark(0x31640, 0x20a9e0, 0x15f310, 0xf)
/usr/local/go/src/runtime/proc.go:130 +0x105 fp=0xc20801a798 sp=0xc20801a768
runtime.goparkunlock(0x20a9e0, 0x15f310, 0xf)
/usr/local/go/src/runtime/proc.go:136 +0x48 fp=0xc20801a7c0 sp=0xc20801a798
runtime.forcegchelper()
/usr/local/go/src/runtime/proc.go:99 +0xce fp=0xc20801a7e0 sp=0xc20801a7c0
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc20801a7e8 sp=0xc20801a7e0
created by runtime.init·4
/usr/local/go/src/runtime/proc.go:87 +0x25

goroutine 3 [runnable]:
runtime.Gosched()
/usr/local/go/src/runtime/proc.go:113 +0x14 fp=0xc20801dfc0 sp=0xc20801dfb0
runtime.bgsweep()
/usr/local/go/src/runtime/mgc0.go:87 +0x59 fp=0xc20801dfe0 sp=0xc20801dfc0
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc20801dfe8 sp=0xc20801dfe0
created by gc
/usr/local/go/src/runtime/mgc0.c:1383

goroutine 4 [finalizer wait]:
runtime.gopark(0x31640, 0x212808, 0x15ed70, 0xe)
/usr/local/go/src/runtime/proc.go:130 +0x105 fp=0xc208018f30 sp=0xc208018f00
runtime.goparkunlock(0x212808, 0x15ed70, 0xe)
/usr/local/go/src/runtime/proc.go:136 +0x48 fp=0xc208018f58 sp=0xc208018f30
runtime.runfinq()
/usr/local/go/src/runtime/malloc.go:727 +0xba fp=0xc208018fe0 sp=0xc208018f58
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc208018fe8 sp=0xc208018fe0
created by runtime.createfing
/usr/local/go/src/runtime/malloc.go:707 +0x5e

utility is not functioning in Win 8.1

I am not able to get the utility to run in win 8.1- it just closes immediately. Apologies for not having any more specific information!

Invalid CIDR address

v1.1.0 MacOs

I am receiving the following error:
Error: error parsing network (5819): invalid CIDR address: 5819

Command:
./geoip2-csv-converter -block-file=GeoLite2-City-Locations-en.csv -output-file=blocks.csv -include-integer-range

I am using GeoLite2 City: CSV Format (updated 2020-06-30). Any ideas why this is happening?

maxmind / geoip2-csv-converter Goto Github PK

geoip2-csv-converter's Introduction

GeoIP2 CSV Format Converter

Usage

Output

CIDR (-include-cidr)

Range (-include-range)

Integer Range (-include-integer-range)

Hex Range (-include-hex-range)

Copyright and License

geoip2-csv-converter's People

Contributors

Stargazers

Watchers

Forkers

geoip2-csv-converter's Issues

Recommend Projects

Recommend Topics

Recommend Org