World-Wide-Web robot, also known as spiders and crawlers. The principle is to falsify network data by constructing appointed HTTP protocol data packet, then request resource to the specified host, goal is to access the data returned.
There are a large number of web information, human's hand movement such as copy-paste data
from web page is time-consuming
and laborious
, thus inspired the data acquisition industry.
Batch access to public network data does not break the law, but because there is no difference, no control, very violent means will lead to other services is not stable, therefore, most of the resources provider will filtering some data packets(falsify), in this context, batch small data acquisition has become a problem. Integrated with various requirements, such as various API development, automated software testing(all this have similar technical principle). So this project come into the world(very simple).
The Marmot
is very easy to understand, just like Python's library requests
(Not yet Smile~ --| ). By enhancing native Golang HTTP library, help you deal with some trivial logic (such as collecting information, checking parameters), and add some fault-tolerant mechanisms (such as add lock, close time flow, ensure the high concurrent run without accident).
It provides a human friendly API interface, you can reuse it often. Very convenient to support Cookie Persistence
, Crawler Proxy Settings
, as well as others general settings, such as HTTP request header settings, timeout/pause settings, data upload/post settings
.
It support all of the HTTP methods POST/PUT/GET/DELETE/...
and has built-in spider pool and browser UA pool, easy to develop UA+Cookie persistence distributed spider.
In addition, It also provides third party tool package. The library is simple and practical, just a few lines of code to replace the previous Spaghetti code
, has been applied in some large projects such as Full Golang Automatic Amazon Distributed crawler|spider
, has withstood the test of two thousand long acting proxy IP and high concurrency, single machine every day to get millions of data.
The main uses: WeChat development/ API docking / automated test / rush ticket scripting / site monitoring / vote plug-in / data crawling
You can get it by:
go get -v github.com/hunterhug/GoSpider
Or make your GOPATH sub dir: /src/github.com/hunterhug
, and
cd src/github.com/hunterhug
git clone https://github.com/hunterhug/GoSpider
all import package save in vendor
default, if some panic when run, please move the file in vendor
to your GOPATH
. you can choose use godep
(Optional)
godep restore
---example some example
---query html content parse, only two function
---spider core download module
---store store module
---myredis
---mysql
---myetcd
---mydb database Orm(use xorm)
---myhbase
---mycassandra
---util other tool
--- image picture cut
--- open open CAPTCHA picture
crypto.go encryption..
file.go file operation
time.go time...
---vendor some dependency package
---GOPATH some must move to your GOPATH
The most simple example such following, more see example
dir:
package main
// Example
import (
"fmt"
"github.com/hunterhug/GoSpider/spider"
)
func main() {
// 1. New a spider
sp, _ := spider.New(nil)
// 2. Set a URL
sp.SetUrl("http://www.lenggirl.com").SetUa(spider.RandomUa()).SetMethod(spider.GET)
// 3. Fetch
html, err := sp.Go()
if err != nil {
fmt.Println(err.Error())
}
// 4.Print content equal to fmt.Println(sp.ToString())
fmt.Println(string(html))
}
More detail Example is:
package main
import (
// 1:import package alias boss
boss "github.com/hunterhug/GoSpider/spider"
"github.com/hunterhug/GoSpider/util"
)
func init() {
// 2:Optional global setting
boss.SetLogLevel(boss.DEBUG) // optional, set log to debug
boss.SetGlobalTimeout(3) // optional, http request timeout time
}
func main() {
log := boss.Log() // optional, spider log you can choose to use
// 3: Must new a spider object, three ways
//sp, err := boss.NewSpider("http://smart:[email protected]:808") // proxy spider, format: protocol://user(optional):password(optional)@ip:port
//sp, err := boss.NewSpider(nil) // normal spider, default keep Cookie
//sp, err := boss.NewAPI() // API spider, not keep Cookie
sp, err := boss.New(nil) // NewSpider alias
if err != nil {
panic(err)
}
// 4: Set the request Method/URL and some others, can chain set, only SetUrl is required.
// SetUrl: required, the Url
// SetMethod: optional, HTTP method: POST/GET/..., default GET
// SetWaitTime: optional, HTTP request wait/pause time
sp.SetUrl("http://www.google.com").SetMethod(boss.GET).SetWaitTime(2)
sp.SetUa(boss.RandomUa()) // optional, browser user agent: IE/Firefox...
sp.SetRefer("http://www.google.com") // optional, url refer
sp.SetHeaderParm("diyheader", "lenggirl") // optional, some other diy http header
//sp.SetBData([]byte("file data")) // optional, if you want post JSON data or upload file
//sp.SetFormParm("username","jinhan") // optional: if you want post form
//sp.SetFormParm("password","123")
// 5: Start Run
//sp.Get() // default GET
//sp.Post() // POST form request data, data can fill by SetFormParm()
//sp.PostJSON() // POST JSON dara, use SetBData()
//sp.PostXML() // POST XML, use SetBData()
//sp.PostFILE() // POST to Upload File, data in SetBData() too
body, err := sp.Go() // if you use SetMethod(), otherwise equal to Get()
if err != nil {
log.Error(err.Error())
} else {
log.Infof("%s", string(body)) // Print return data
}
log.Debugf("%#v", sp) // if you not set log as debug, it will not appear
sp.Clear() // after get the return data by post data, you can clear the data you fill
//sp.ClearAll() // you can also want to clear all, include http header you set
// spider pool for concurrent, every Spider Object is serial as the browser. if you want collateral execution, use this.
boss.Pool.Set("myfirstspider", sp)
if poolspider, ok := boss.Pool.Get("myfirstspider"); ok {
go func() {
poolspider.SetUrl("http://www.baidu.com")
data, _ := poolspider.Get()
log.Info(string(data))
}()
util.Sleep(10)
}
}
Easy to use, you just need to New
one Spider
, and SetUrl
, then make some settings and sp.Go()
.
There are three kinds of spider:
sp, err := boss.NewSpider("http://smart:[email protected]:808")
// proxy spider, format:protocol://user(optional):password(optional)@ip:port
, alias toNew()
sp, err := boss.NewSpider(nil)
// normal spider, default keep Cookie, alias toNew()
sp, err := boss.NewAPI()
// API spider, not keep Cookie
Camouflage our spider:
sp.SetUrl("http://www.lenggirl.com")
// required: set url you want tosp.SetMethod(boss.GET)
// optional: set http methodPOST/GET/PUT/POSTJSON
and so onsp.SetWaitTime(2)
// optional: set timeout of http requestsp.SetUa(boss.RandomUa())
// optional: set http browser user agent, you can see spider/config/ua.txtsp.SetRefer("http://www.baidu.com")
// optional: set http request Refersp.SetHeaderParm("diyheader", "lenggirl")
// optional: set http diy headersp.SetBData([]byte("file data"))
// optional: set binary data for post or putsp.SetFormParm("username","jinhan")
// optional: set form data for post or putsp.SetCookie("xx=dddd")
// optional: you can set a init cookie, some website you can login and F12 copy the cookie
Run our spider:
body, err := sp.Go()
// if you use SetMethod(), auto use following ways, otherwise use Get()body, err := sp.Get()
// defaultbody, err := sp.Post()
// post form request, data fill by SetFormParm()body, err := sp.PostJSON()
// post JSON request, data fill by SetBData()body, err := sp.PostXML()
// post XML request, data fill by SetBData()body, err := sp.PostFILE()
// upload file, data fill by SetBData()body, err := sp.Delete()
// you know!body, err := sp.Put()
// ones http method...body, err := sp.PutJSON()
// put JSON requestbody, err := sp.PutXML()
body, err := sp.PutFILE()
body, err := sp.OtherGo("OPTIONS", "application/x-www-form-urlencoded")
// Other http method, Such as OPTIONS etcd.
Deal the return data, all data will be return as binary:
fmt.Println(string(html))
// type change directlyfmt.Println(sp.ToString())
// use spider method, after http response, data will keep in the field Raw, just use ToStringfmt.Println(sp.JsonToString())
// some json data will include chinese and other multibyte character, such as我爱你,我的小绵羊
,사랑해
Attention: after every request for a url, the next request you can cover your http request header, otherwise header you set still exist,
if just want clear post data, use Clear()
, and want clear header too please use ClearAll()
.
More see the code source.
It has already used in many project(although some is very simple) :
- Full Golang Automatic Amazon Distributed crawler|spider // Just see Picture
- Jiandan Distributed articles spider
- Zhihu API tool
- Picture helper
- Jiandan Picure helper
- Music Download // see example dir
- GoTaobao
- A lot closed source...
Project change you can see log
Install development environment you can refer(still chinese):
Copyright 2017 by GoSpider author.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License
Welcome Add PR/issues.
For questions, please email: [email protected].