rjohnsondev / go-safe-browsing-api Goto Github PK

Hi Richard,

Thank you for this really useful go binding! It is really nice.

I'm trying out the v3 adaption that github.com/kgilonne/go-safe-browsing-api provides.

One problem: I get a ton of false positives from MightBeListed(), and IsListed() is too slow. Is it possible to pull down the full hashes every 1800 minutes (or the update frequency that google requests), and to otherwise remain offline while having accurate (full) hash matching?

Thanks!
Jason

Using strings.Split on a data response where the hash contains a newline character causes the hashlen validation to fail. This results in a false negative for a hash that should match.

e.g. http://vibiraem-okna.ru/tmp/ji/ returns a false negative due to the hashlen being 31 instead of 32, because the hash has a 0xa character at position 32.

https://www.google.com/transparencyreport/safebrowsing/diagnostic/?#url=http://vibiraem-okna.ru/tmp/ji/

Sometimes after several days of use the package panics on "Update". Most likely there seems to be a kind of race condition.

github.com/rjohnsondev/go-safe-browsing-api.(*SafeBrowsingList).load(0xc499fadc20, 0xc46adf8000, 0x29f9, 0x3000, 0x0, 0x0)
	/go/src/github.com/rjohnsondev/go-safe-browsing-api/safebrowsinglist.go:125 +0x180
github.com/rjohnsondev/go-safe-browsing-api.(*SafeBrowsingList).loadDataFromRedirectLists(0xc499fadc20, 0x0, 0x0)
	/go/src/github.com/rjohnsondev/go-safe-browsing-api/safebrowsinglist.go:118 +0x474
github.com/rjohnsondev/go-safe-browsing-api.(*SafeBrowsing).update(0xc420168000, 0x759d80, 0xc47fc8e1e0, 0xc47fc8e1d0)
	/go/src/github.com/rjohnsondev/go-safe-browsing-api/safebrowsing.go:198 +0x17d
github.com/rjohnsondev/go-safe-browsing-api.(*SafeBrowsing).reloadLoop(0xc420168000)
	/go/src/github.com/rjohnsondev/go-safe-browsing-api/safebrowsing.go:351 +0x248
created by github.com/rjohnsondev/go-safe-browsing-api.(*SafeBrowsing).UpdateProcess
	/go/src/github.com/rjohnsondev/go-safe-browsing-api/safebrowsing.go:136 +0x159

Quite often the HTTP request when "Requesting updates" hangs. It might be worth setting a time out on the http.Client:

timeout := time.Duration(30 * time.Second)
client := &http.Client{Transport: Transport, Timeout: timeout}

I'd like to use this to check outbound links on some forums I run.

Could you add a licence to the project please so that I know that it is fine to do so.

I've actually done the work myself. It's not as memory efficient, yet, but I'm still working on it. I wrote a HTTP/WS server using this as a library and wanted to make compiling, trying, distributing painless so the lib had to go. Let me know if you're interested and we can work out something.

If not. Close and ignore :)

1. https://github.com/apokalyptik/gsbd
2. https://github.com/apokalyptik/go-safe-browsing-api
3. https://github.com/apokalyptik/quicktrie