40t / go-sniffer Goto Github PK

View Code? Open in Web Editor NEW

1.9K 59.0 355.0 6.37 MB

🔎Sniffing and parsing mysql,redis,http,mongodb etc protocol. 抓包截取项目中的数据库请求并解析成相应的语句。

License: MIT License

Go 100.00%

go sniffer capture mysql mongodb redis

go-sniffer's Introduction

go-sniffer

Capture mysql,redis,http,mongodb etc protocol... 抓包截取项目中的数据库请求并解析成相应的语句，如mysql协议会解析为sql语句,便于调试。不要修改代码，直接嗅探项目中的数据请求。

Support List:

mysql
Redis
Http
Mongodb
Kafka (developing)
...

Demo:

$ go-sniffer en0 mysql

Setup:

support : MacOS Linux Unix
not support : windows
If you encounter problems in the go get process, try upgrading the go version （如果go get 过程中遇到问题，请尝试升级go版本）

Centos

$ yum -y install libpcap-devel

Ubuntu

$ apt-get install libpcap-dev

MacOs

RUN

$ go get -v -u github.com/40t/go-sniffer
$ cp -rf $(go env GOPATH)/bin/go-sniffer /usr/local/bin
$ go-sniffer

Usage:

==================================================================================
[Usage]

    go-sniffer [device] [plug] [plug's params(optional)]

    [Example]
          go-sniffer en0 redis          Capture redis packet
          go-sniffer en0 mysql -p 3306  Capture mysql packet

    go-sniffer --[commend]
               --help "this page"
               --env  "environment variable"
               --list "Plug-in list"
               --ver  "version"
               --dev  "device"
    [Example]
          go-sniffer --list "show all plug-in"

==================================================================================
[device] : lo0 :   127.0.0.1
[device] : en0 : xx:xx:xx:xx:xx:xx  192.168.199.221
==================================================================================

中文使用说明

=======================================================================
[使用说明]

    go-sniffer [设备名] [插件名] [插件参数(可选)]

    [例子]
          go-sniffer en0 redis          抓取redis数据包
          go-sniffer en0 mysql -p 3306  抓取mysql数据包,端口3306

    go-sniffer --[命令]
               --help 帮助信息
               --env  环境变量
               --list 插件列表
               --ver  版本信息
               --dev  设备列表
    [例子]
          go-sniffer --list 查看可抓取的协议

=======================================================================
[设备名] : lo0 :   127.0.0.1
[设备名] : en0 : x:x:x:x:x5:x  192.168.1.3
[设备名] : utun2 :   1.1.11.1
=======================================================================

Example:

$ go-sniffer lo0 mysql 
$ go-sniffer en0 redis 
$ go-sniffer eth0 http -p 8080
$ go-sniffer eth1 mongodb

License:

MIT

go-sniffer's People

Contributors

Stargazers

Watchers

Forkers

nelsonking shanba hien fc13240 w7374520 phpip imagerec mfrank2016 imxyb bianjx zhangxianbo xjxyxgq chaossir minghuan2046 marshell88 gongxun0928 dikang123 ver007 qsdj futurebody 5up3rc bipabo1l freegit9527 gongcheng121 thy0108 shiliuxing devops-lio zerolugithub justintung hanhyu simman chennqqi xubingyue rustedlightning skymysky xiaosl zhanglei zhangwusheng weiboyiyou urans gpsbird yujinqiu forging2012 zoelov chyroc duzhanyuan xuxueyun-one oksbsb feiin bbotte jasonleemz wanyaoqi lzq123218 lushuozhi yuhuashangedward soosim u20024804 mid-summer dulumao dngiveu wuhua666 pythonsite hewei-github wkcaeser guanhui07 zhiwyan exfly open-party zhuomingliang it-xiaoge sanae10001 chisenzhang hhy5277 blueroutecn percyzhang lotapp carryon weibo1987 sultan8252 taek007 linpengstc lllby1993 gxw815lyl isgasho geolem stoensin cocktail18 xiaolongn100 lchjczw john123951 xuewindy darrynza hustjieke pepeshore 26597925 mooncreeks darkboy 1newstar chkch liujunandzhou

go-sniffer's Issues

[WSL]Socket type not supported

在Win10的WSL环境下成功安装了go-sniffer，运行go-sniffer报以下错误:

jclazz@j-dl-5570:~$ go-sniffer eth4 mysql -p 3306
2019/08/06 10:53:48 socket for SIOCETHTOOL(ETHTOOL_GET_TS_INFO): Socket type not supported

请问我应该如何解决这个问题？

mysql 5.7.23 抓不到包

只有下面两条，完了什么也没有了

 # go-sniffer lo mysql
 tcp and port 3306
 # Start new stream: 127.0.0.1->127.0.0.1 38994->3306
 # Start new stream: 127.0.0.1->127.0.0.1 3306->38994

loss data

When I use Sysbench to test the MySQL and go-sniffer, I am not able to capture all the packet.I print logs and find that the packet is in the queue.

在Mac下安装报错

之前一直都在Centos下使用，没遇到过问题。这次装到自己的Macbook pro上，在安装时候报错了。

go get -v -u github.com/40t/go-sniffer
github.com/40t/go-sniffer (download)
github.com/google/gopacket (download)
github.com/google/gopacket
github.com/40t/go-sniffer/plugSrc/mongodb/build/internal/json
github.com/40t/go-sniffer/plugSrc/http/build
github.com/40t/go-sniffer/plugSrc/redis/build
github.com/40t/go-sniffer/plugSrc/mysql/build
github.com/google/gopacket/layers
github.com/40t/go-sniffer/plugSrc/mongodb/build/bson
github.com/40t/go-sniffer/plugSrc/mongodb/build
github.com/google/gopacket/pcap
github.com/google/gopacket/tcpassembly
github.com/google/gopacket/tcpassembly/tcpreader
github.com/40t/go-sniffer/core
github.com/40t/go-sniffer
# github.com/40t/go-sniffer
/usr/local/Cellar/go/1.8.3/libexec/pkg/tool/darwin_amd64/link: /usr/local/Cellar/go/1.8.3/libexec/pkg/tool/darwin_amd64/link: combining dwarf failed: Unknown load command 0x32 (50)

这个是跟go的版本有关系吗？我的版本是：

go version go1.8.3 darwin/amd64

mac 抓取 mysql 能打印出数据流动方向，但是不打印sql语句

mysql的newPacket 函数 error 判断处理条件

centos install error

install error in centos, I think you could provide a binary file of multi system os.

OS Version

cat /etc/issue

CentOS release 6.8 (Final)
Kernel \r on an \m

install

yum -y install libcap
yum -y install libcap-devel

go get -v -u github.com/40t/go-sniffer
github.com/40t/go-sniffer (download)
github.com/google/gopacket (download)
# runtime/debug
gc 1 @0.005s 6%: 0.029+1.2+0.069 ms clock, 0.11+0.17/1.1/1.1+0.27 ms cpu, 4->4->3 MB, 5 MB goal, 4 P
gc 2 @0.011s 6%: 0.004+1.4+0.048 ms clock, 0.019+0.057/1.3/2.6+0.19 ms cpu, 5->6->5 MB, 6 MB goal, 4 P
github.com/google/gopacket/pcap
# github.com/google/gopacket/pcap
/devops/app/go/src/github.com/google/gopacket/pcap/pcap.go:22:18: fatal error: pcap.h: No such file or directory
 #include <pcap.h>

centos抓mysql报错

2018-11-01 16:21:39| cli -> ser |Stm id[10]: 'SELECT domain FROM xxx WHERE ticket = ? LIMIT 1 ';
set @p0 = 'xxxxx';
Execute stm id[10]: using @p0;
Drop stm id[10];

Start new stream: 127.0.0.1->127.0.0.1 33134->3306

2018/11/01 16:21:39 ERR : Not found stm id 19
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x7885cf]

goroutine 44 [running]:
github.com/40t/go-sniffer/plugSrc/mysql/build.(*stream).resolveClientPacket(0xc420401590, 0xc42051f200, 0x1e, 0x600, 0x0)
/Golang/src/github.com/40t/go-sniffer/plugSrc/mysql/build/entry.go:314 +0x4ef
github.com/40t/go-sniffer/plugSrc/mysql/build.(*stream).resolve(0xc420401590)
/Golang/src/github.com/40t/go-sniffer/plugSrc/mysql/build/entry.go:192 +0x98
created by github.com/40t/go-sniffer/plugSrc/mysql/build.(*Mysql).ResolveStream
/Golang/src/github.com/40t/go-sniffer/plugSrc/mysql/build/entry.go:71 +0x42e

Would you support the Database Postgresql

thanks.

会出现panic

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x7a8bf8]

goroutine 7 [running]:
github.com/40t/go-sniffer/plugSrc/mysql/build.(*stream).resolveClientPacket(0xc420056c80, 0xc42062e600, 0x1, 0x600, 0x1)
/data/home/xxx/Gosrc/src/github.com/40t/go-sniffer/plugSrc/mysql/build/entry.go:304 +0xd98
github.com/40t/go-sniffer/plugSrc/mysql/build.(*stream).resolve(0xc420056c80)
/data/home/xxx/Gosrc/src/github.com/40t/go-sniffer/plugSrc/mysql/build/entry.go:193 +0x95
created by github.com/40t/go-sniffer/plugSrc/mysql/build.(*Mysql).ResolveStream
/data/home/xxx/Gosrc/src/github.com/40t/go-sniffer/plugSrc/mysql/build/entry.go:71 +0x3ff

看代码，应该是没有判断从map取出来的是否为空指针，就直接给字段赋值，导致空指针错误了

mysql连接上去的无法被抓取

两个虚拟机
[root@geone ~]# go-sniffer ens33 mysql -p 3306
tcp and port 3306

Start new stream: 192.168.137.151->192.168.137.137 40944->3306

Start new stream: 192.168.137.151->192.168.137.137 41996->3306

Start new stream: 192.168.137.137->192.168.137.151 3306->41996

Start new stream: 192.168.137.151->192.168.137.137 41996->3306

Start new stream: 192.168.137.151->192.168.137.137 43058->3306

只能看到链接的信息，但是151操作137的相关操作，却看不到?为啥，

使用mysql客户端连接上去的无法被抓取，作者还维护吗？

环境：
系统：CentOS Linux release 7.6.1810
MySQL: 5.7.30-log
MySQL 服务端地址：192.168.66.200
端口：3306
测试连接客户端地址：192.168.66.101

监听方式：

go-sniffer ens32 mysql -p 3306

问题描述：
使用 navicat、Dbeaver 等工具连接的话是可以抓取到信息的
但使用mysql client 远程连接的话，执行的操作无法被抓取到

MySQL 客户端【101】使用连接命令

mysql -utestuser -p123456 -h192.168.66.200 -P 3306

mysql> show databases;
mysql> create database abc;

mysql> use abc
Database changed

mysql> create table a(id int);
Query OK, 0 rows affected (0.11 sec)

mysql> insert into a values(1);
Query OK, 1 row affected (0.02 sec)

MySQL服务端【200】go-sniffer展示内容如下，没有抓取到任何行为操作。

[root@initnode ~]# go-sniffer ens32 mysql -p 3306
tcp and port 3306
# Start new stream: 192.168.66.101->192.168.66.200 46632->3306
# Start new stream: 192.168.66.200->192.168.66.101 3306->46632

内存使用过大问题

抓mongo包，追加到文件中，二十分钟go-sniffer大概用了15G内存，生成的文件就83M左右，请问下这是什么情况？

异常退出，不给？

panic: runtime error: index out of range

goroutine 7 [running]:
github.com/40t/go-sniffer/plugSrc/mysql/build.(*stream).resolveServerPacket(0xc420056d80, 0xc4204cd000, 0x0, 0x200, 0xb)
/data/go/src/github.com/40t/go-sniffer/plugSrc/mysql/build/entry.go:219 +0x442
github.com/40t/go-sniffer/plugSrc/mysql/build.(*stream).resolve(0xc420056d80)
/data/go/src/github.com/40t/go-sniffer/plugSrc/mysql/build/entry.go:194 +0xc0
created by github.com/40t/go-sniffer/plugSrc/mysql/build.(*Mysql).ResolveStream
/data/go/src/github.com/40t/go-sniffer/plugSrc/mysql/build/entry.go:71 +0x3b8

Resolve Crash

panic: runtime error: index out of range [0] with length 0

goroutine 454 [running]:
github.com/40t/go-sniffer/plugSrc/mysql/build.(*Stmt).BindArgs(0xc1884aa380, {0xc188a3320a, 0x1, 0xc18848fe88?}, {0x0, 0x0, 0x0?}, {0x0, 0x0, 0x0})
/root/go/pkg/mod/github.com/40t/[email protected]/plugSrc/mysql/build/stmt.go:79 +0xbd4
github.com/40t/go-sniffer/plugSrc/mysql/build.(*stream).resolveClientPacket(0xc00006b3b0?, {0xc188a33200, 0x14, 0x600}, 0x0?)
/root/go/pkg/mod/github.com/40t/[email protected]/plugSrc/mysql/build/entry.go:344 +0x36d
github.com/40t/go-sniffer/plugSrc/mysql/build.(*stream).resolve(0xc00006b3b0)
/root/go/pkg/mod/github.com/40t/[email protected]/plugSrc/mysql/build/entry.go:193 +0x6b
created by github.com/40t/go-sniffer/plugSrc/mysql/build.(*Mysql).ResolveStream
/root/go/pkg/mod/github.com/40t/[email protected]/plugSrc/mysql/build/entry.go:71 +0x3f0

macOS下运行无任何输出，服务也没有启动

mysql plug panic

panic: runtime error: index out of range [3] with length 1

goroutine 14 [running]:
encoding/binary.littleEndian.Uint32(...)
	/Users/voidint/.g/go/src/encoding/binary/binary.go:63
github.com/40t/go-sniffer/plugSrc/mysql/build.LengthBinary(0xc000490001, 0x31, 0x5ff, 0x4031ce0, 0x44c3230)
	/Users/voidint/workspace/go/src/github.com/40t/go-sniffer/plugSrc/mysql/build/util.go:41 +0xd4
github.com/40t/go-sniffer/plugSrc/mysql/build.(*stream).resolveServerPacket(0xc000402600, 0xc000490000, 0x32, 0x600, 0x1)
	/Users/voidint/workspace/go/src/github.com/40t/go-sniffer/plugSrc/mysql/build/entry.go:235 +0x306
github.com/40t/go-sniffer/plugSrc/mysql/build.(*stream).resolve(0xc000402600)
	/Users/voidint/workspace/go/src/github.com/40t/go-sniffer/plugSrc/mysql/build/entry.go:195 +0xc5
created by github.com/40t/go-sniffer/plugSrc/mysql/build.(*Mysql).ResolveStream
	/Users/voidint/workspace/go/src/github.com/40t/go-sniffer/plugSrc/mysql/build/entry.go:71 +0x3b6

能否支持打印查询语句的输出

研究过一段时间的packetbeat，扩展过tns、drda协议的解析，希望可以加入你们的项目

有考虑支持Postgresql吗？

mac下面执行没有任何输出

mac系统 0.14 (18A391)
go 版本: go version go1.11.1 darwin/amd64

执行命令:
go-sniffer lo0 redis

当SQL语句太长的时候，sniffer会异常

例如，捕捉到一个有1000个参数的Insert的SQL语句的时候出错：

, @p906, @p907, @p908, @p909, @p910, @p911, @p912, @p913, @p914, @p915, @p916, @p917, @p918, @p919, @p920, @p921, @p922, @p923, @p924, @p925, @p926, @p927, @p928, @p929, @p930, @p931, @p932, @p933, @p934, @p935, @p936, @p937, @p938, @p939, @p940, @p941, @p942, @p943, @p944, @p945, @p946, @p947, @p948, @p949, @p950, @p951, @p952, @p953, @p954, @p955, @p956, @p957, @p958, @p959, @p960, @p961, @p962, @p963, @p964, @p965, @p966, @p967, @p968, @p969, @p970, @p971, @p972, @p973, @p974, @p975, @p976, @p977, @p978, @p979, @p980, @p981, @p982, @p983, @p984, @p985, @p986, @p987, @p988, @p989, @p990, @p991, @p992, @p993, @p994, @p995, @p996, @p997, @p998, @p999;
Drop stm id[4];

panic: runtime error: index out of range

goroutine 1411 [running]:
encoding/binary.binary.littleEndian.Uint32(...)
	/usr/local/go/src/encoding/binary/binary.go:63
github.com/40t/go-sniffer/plugSrc/mysql/build.LengthBinary(0xc4206d6c01, 0x32, 0x5ff, 0x9b47d8, 0xc420044c00)
	/root/go/src/github.com/40t/go-sniffer/plugSrc/mysql/build/util.go:41 +0xc2
github.com/40t/go-sniffer/plugSrc/mysql/build.(*stream).resolveServerPacket(0xc4200571c0, 0xc4206d6c00, 0x33, 0x600, 0x1)
	/root/go/src/github.com/40t/go-sniffer/plugSrc/mysql/build/entry.go:230 +0x2fc
github.com/40t/go-sniffer/plugSrc/mysql/build.(*stream).resolve(0xc4200571c0)
	/root/go/src/github.com/40t/go-sniffer/plugSrc/mysql/build/entry.go:194 +0xc0
created by github.com/40t/go-sniffer/plugSrc/mysql/build.(*Mysql).ResolveStream
	/root/go/src/github.com/40t/go-sniffer/plugSrc/mysql/build/entry.go:71 +0x3b8

执行命令，乱码严重，有些还不知道是啥

有考虑开一个程序，同时抓http、mysql、redis等驱动的包吗

mac上编译Linux可用的文件报错

多谢楼主的这个工具，很牛逼的样子。mac本地可以使用。
我想编译一个在Linux机器上使用，但是报错。

CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o go-sniffer main.go

../pkg/mod/github.com/google/[email protected]/pcap/pcap.go:30:22: undefined: pcapErrorNotActivated
../pkg/mod/github.com/google/[email protected]/pcap/pcap.go:52:17: undefined: pcapTPtr
../pkg/mod/github.com/google/[email protected]/pcap/pcap.go:64:10: undefined: pcapPkthdr
../pkg/mod/github.com/google/[email protected]/pcap/pcap.go:103:6: undefined: pcapBpfProgram
../pkg/mod/github.com/google/[email protected]/pcap/pcap.go:110:7: undefined: pcapPkthdr
../pkg/mod/github.com/google/[email protected]/pcap/pcap.go:268:33: undefined: pcapErrorActivated
../pkg/mod/github.com/google/[email protected]/pcap/pcap.go:269:33: undefined: pcapWarningPromisc
../pkg/mod/github.com/google/[email protected]/pcap/pcap.go:270:33: undefined: pcapErrorNoSuchDevice
../pkg/mod/github.com/google/[email protected]/pcap/pcap.go:271:33: undefined: pcapErrorDenied
../pkg/mod/github.com/google/[email protected]/pcap/pcap.go:748:14: undefined: pcapTPtr
../pkg/mod/github.com/google/[email protected]/pcap/pcap.go:271:33: too many errors

/usr/bin/go-sniffer: cannot execute binary file

root@xxxx-xxxxx-0:/usr/local/services# ll go-sniffer 
-rwxr-xr-x 1 root root 14563416 Dec 23 21:30 go-sniffer

root@xxxx-xxxxx-0:/usr/local/services# go-sniffer  
bash: /usr/bin/go-sniffer: cannot execute binary file

mac 10.15.6 上编译，上传到一台linux机器，执行go-sniffer时这种报错什么情况？
ps: mac本机执行看上去是正常的。

unrecognized import path "plugin"

[root❄anhk-pc:go-sniffer]☭ GOPATH=$(pwd)/vendor go get github.com/40t/go-sniffer
package plugin: unrecognized import path "plugin" (import path does not begin with hostname)
[root❄anhk-pc:go-sniffer]☭

Support SQLServer

Hello, will you support SQLServer in the future?

./bin/go-sniffer eth0 mysql

2018-12-05 14:41:24| ser -> cli |【Ok】 Effect Row:0
2018-12-05 14:41:24| ser -> cli |【Ok】 Effect Row:0
panic: runtime error: index out of range

goroutine 34 [running]:
github.com/40t/go-sniffer/plugSrc/mysql/build.(*stream).resolveServerPacket(0xc420418000, 0xc420456200, 0x0, 0x200, 0xb)
/gomeo2o/soft/go-sniffer/src/github.com/40t/go-sniffer/plugSrc/mysql/build/entry.go:219 +0x50d
github.com/40t/go-sniffer/plugSrc/mysql/build.(*stream).resolve(0xc420418000)
/gomeo2o/soft/go-sniffer/src/github.com/40t/go-sniffer/plugSrc/mysql/build/entry.go:194 +0xc0
created by github.com/40t/go-sniffer/plugSrc/mysql/build.(*Mysql).ResolveStream
/gomeo2o/soft/go-sniffer/src/github.com/40t/go-sniffer/plugSrc/mysql/build/entry.go:71 +0x3ff

github.com/google/gopacket/pcap

src/github.com/google/gopacket/pcap/pcap.go:22:18: fatal error: pcap.h: No such file or directory
#include <pcap.h>
^
compilation terminated.

40t / go-sniffer Goto Github PK

go-sniffer's Introduction

go-sniffer

Support List:

Demo:

Setup:

Centos

Ubuntu

MacOs

RUN

Usage:

中文使用说明

Example:

License:

go-sniffer's People

Contributors

Stargazers

Watchers

Forkers

go-sniffer's Issues

Start new stream: 127.0.0.1->127.0.0.1 33134->3306

Start new stream: 192.168.137.151->192.168.137.137 40944->3306

Start new stream: 192.168.137.151->192.168.137.137 41996->3306

Start new stream: 192.168.137.137->192.168.137.151 3306->41996

Start new stream: 192.168.137.151->192.168.137.137 41996->3306

Start new stream: 192.168.137.151->192.168.137.137 43058->3306

github.com/google/gopacket/pcap

Recommend Projects

Recommend Topics

Recommend Org