erbbysam / dnsgrep Goto Github PK
View Code? Open in Web Editor NEWQuickly Search Large DNS Datasets
License: MIT License
Quickly Search Large DNS Datasets
License: MIT License
hello, I try to use the project to grep the text from rapid7, the command is " go run dnsgrep.go -f fdns_a.sort.txt -i "erbbyam.com"", but the search failed for "Error: failed to find exact match via binary search". could you help to tell me what the problem ? thank you.
As it hardcoded, you have to fill a new dataset each time.
First, I did the following
root@pentest:~/internet/rapid# https://github.com/erbbysam/DNSGrep
-bash: https://github.com/erbbysam/DNSGrep: No such file or directory
root@pentest:~/internet/rapid# git clone https://github.com/erbbysam/DNSGrep
Cloning into 'DNSGrep'...
remote: Enumerating objects: 53, done.
remote: Counting objects: 100% (25/25), done.
remote: Compressing objects: 100% (18/18), done.
remote: Total 53 (delta 10), reused 19 (delta 6), pack-reused 28
Unpacking objects: 100% (53/53), 1.47 MiB | 3.18 MiB/s, done.
root@pentest:~/internet/rapid# go get "github.com/jessevdk/go-flags"
root@pentest:~/internet/rapid# go get "github.com/gorilla/mux"
root@pentest:~/internet/rapid# go get "github.com/golang/example/stringutil"
go get: github.com/golang/example@none updating to
github.com/golang/[email protected]: parsing go.mod:
module declares its path as: golang.org/x/example
but was required as: github.com/golang/example
root@pentest:~/internet/rapid# cd DNSGrep/
root@pentest:~/internet/rapid/DNSGrep# go run dnsgrep.go
dnsgrep.go:6:2: package dnsgrep/DNSBinarySearch is not in GOROOT (/usr/local/go/src/dnsgrep/DNSBinarySearch)
dnsgrep.go:11:2: no required module provides package github.com/jessevdk/go-flags: go.mod file not found in current directory or any parent directory; see 'go help modules'
root@pentest:~/internet/rapid/DNSGrep#
Second , I tried to clone dnsgrep to GOROOT src as mentioned here in the last comment.
#3
root@pentest:/usr/local/go/src# git clone https://github.com/erbbysam/DNSGrep
Cloning into 'DNSGrep'...
remote: Enumerating objects: 53, done.
remote: Counting objects: 100% (25/25), done.
remote: Compressing objects: 100% (18/18), done.
^[[A^[[A^[[Aremote: Total 53 (delta 10), reused 19 (delta 6), pack-reused 28
Unpacking objects: 100% (53/53), 1.47 MiB | 1.19 MiB/s, done.
root@pentest:/usr/local/go/src# lsmv DNSGrep dnsgrep^C
root@pentest:/usr/local/go/src# mv DNSGrep dnsgrep
root@pentest:/usr/local/go/src# cd dnsgrep/
root@pentest:/usr/local/go/src/dnsgrep# go run dnsgrep.go -f DNSBinarySearch/test_data.txt -i "amiccom.com.tw"
go: inconsistent vendoring in /usr/local/go/src:
github.com/gorilla/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
github.com/jessevdk/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
golang.org/x/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
golang.org/x/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
golang.org/x/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
golang.org/x/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
golang.org/x/[email protected]: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
golang.org/x/[email protected]: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
golang.org/x/[email protected]: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
golang.org/x/[email protected]: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
To ignore the vendor directory, use -mod=readonly or -mod=mod.
To sync the vendor directory, run:
go mod vendor
root@pentest:/usr/local/go/src/dnsgrep#
The problem is
go: inconsistent vendoring in /usr/local/go/src:
github.com/gorilla/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
github.com/jessevdk/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
golang.org/x/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
golang.org/x/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
golang.org/x/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
golang.org/x/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
golang.org/x/[email protected]: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
golang.org/x/[email protected]: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
golang.org/x/[email protected]: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
golang.org/x/[email protected]: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
To ignore the vendor directory, use -mod=readonly or -mod=mod.
To sync the vendor directory, run:
go mod vendor
I tried to run go mod tidy , go mod vendor as mentioned in the error and https://stackoverflow.com/questions/58511588/fixing-go-inconsistent-vendoring-in-c-go-src/58512507
Same error occur.
Any thoughts on how to solve this ?
When i type command: go run experimentalServer/experimentalServer.go
I have got the error:
experimentalServer/experimentalServer.go:4:2: cannot find package "dnsgrep/DNSBinarySearch".
The same as error result to type command line: go run dnsgrep.go -f DNSBinarySearch/test_data.txt -i "amiccom.com.tw"
What should i do and how to fix it?
Looking at the code, it doesn't appear to do regular expressions? It looks like it just does a binary search for something containing the string and finds other matches nearby.
The name seems to imply it should do something similar to the command 'grep', which if I recall was named from the g(lobal)/(regular expression)/p(rint) command sequence in ed. If it doesn't do grep, it probably shouldn't be called grep...
Hey man! Could you add into requirements the jq program?
Thanks you
Hi.
Change this.
from
split -b100M rdns.rev.lowercase.txt fileChunk
to
split -l2000000 rdns.rev.lowercase.txt fileChunk
in scripts/fdns_a.sh and scripts/rdns.sh
because lose some records when sorting.
/opt/DNSGrep# ./scripts/fdns_a.sh
--2019-08-14 15:38:21-- https://opendata.rapid7.com/sonar.fdns_v2/2019-01-25-1548417890-fdns_a.json.gz
Resolving opendata.rapid7.com (opendata.rapid7.com)... 52.200.143.45, 34.225.120.26
Connecting to opendata.rapid7.com (opendata.rapid7.com)|52.200.143.45|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: /sonar.fdns_v2/ [following]
--2019-08-14 15:38:21-- https://opendata.rapid7.com/sonar.fdns_v2/
Reusing existing connection to opendata.rapid7.com:443.
HTTP request sent, awaiting response... 200 OK
Length: 496866 (485K) [text/html]
Saving to: ‘fdns_a.gz’
fdns_a.gz 100%[===================>] 485.22K 2.24MB/s in 0.2s
2019-08-14 15:38:22 (2.24 MB/s) - ‘fdns_a.gz’ saved [496866/496866]
gzip: fdns_a.gz: not in gzip format
sort: cannot read: 'fileChunk*': No such file or directory
if search by ip address there is no need to reverse the search string,right?
Currently, Linux shell script is used to generate the intermediate cache files, which makes it impossible to run on Windows.
Can we rewrite this code to pure Golang, so it will be cross-platform ? Thanks.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.