hunterhug / marmot Goto Github PK
View Code? Open in Web Editor NEW💐Marmot A Golang HTTP Download
License: Apache License 2.0
marmot's People
Contributors
Stargazers
Watchers
Forkers
springswen qqiot thup yeyunxyz kotlin2018 frazierlei chenarno liangdabiao zhenyangze cchen-byte dmskys cci8go gofirestarmarmot's Issues
东西呢
东西呢?
go get err cannot find package
package github.com/hunterhug/GoSpider/util/xid: cannot find package "github.com/hunterhug/GoSpider/util/xid" in any of
How to use
See the Example https://github.com/hunterhug/GoSpiderExample
为什么使用 marmot 加无法访问 Google 翻译的链接？
访问的链接如下：
https://translate.google.cn/translate_a/single?client=webapp&sl=zh-CN&tl=zh-CN&hl=zh-CN&dt=at,bd,ex,ld,md,qca,rw,rm,sos,ss,t&ssel=0&tsel=3&kc=0&tk=470992.123882&q=This is an English sentence
Google 返回
Your client has issued a malformed or illegal request. That’s all we know
RandomUa()失效
RandomUa()需要读取该项目下txt文本中的浏览器标志,然而编译成exe文件后可能找不到该文件,需要将其写死在代码里,由于数据过大,建议增加一个GoSpider插件
非常好，希望继续添加其他平台的登陆支付等等功能以及持续维护
非常好，希望继续添加其他平台的登陆支付等等功能以及持续维护
标准http库只在目标HTTP服务器要求关闭时才会关闭网络连接。这意味着你的应用在某些条件下消耗完sockets/file的描述符
你可以通过设置请求变量中的Close域的值为true，来让http库在请求完成时关闭连接。
另一个选项是添加一个Connection的请求头，并设置为close。目标HTTP服务器应该也会响应一个Connection: close的头。当http库看到这个响应头时，它也将会关闭连接。
package main
import (
"fmt"
"net/http"
"io/ioutil"
)
func main() {
req, err := http.NewRequest("GET","http://golang.org",nil)
if err != nil {
fmt.Println(err)
return
}
req.Close = true
//or do this:
//req.Header.Add("Connection", "close")
resp, err := http.DefaultClient.Do(req)
if resp != nil {
defer resp.Body.Close()
}
if err != nil {
fmt.Println(err)
return
}
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
fmt.Println(err)
return
}
fmt.Println(len(string(body)))
}
你也可以取消http的全局连接复用。你将需要为此创建一个自定义的http传输配置。
package main
import (
"fmt"
"net/http"
"io/ioutil"
)
func main() {
tr := &http.Transport{DisableKeepAlives: true}
client := &http.Client{Transport: tr}
resp, err := client.Get("http://golang.org")
if resp != nil {
defer resp.Body.Close()
}
if err != nil {
fmt.Println(err)
return
}
fmt.Println(resp.StatusCode)
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
fmt.Println(err)
return
}
fmt.Println(len(string(body)))
}
如果你向同一个HTTP服务器发送大量的请求，那么把保持网络连接的打开是没问题的。然而，如果你的应用在短时间内向大量不同的HTTP服务器发送一两个请求，那么在引用收到响应后立刻关闭网络连接是一个好主意。增加打开文件的限制数可能也是个好主意。当然，正确的选择源自于应用。
我们一般不会向大量服务器请求,所以凑合着...
建议从glide切换到dep
RT
lack of delete/put/option method
wating for add
http响应失败时，resp变量将为nil，而err变量将是non-nil。然而，当你得到一个重定向的错误时，两个变量都将是non-nil。意味着最后依然会内存泄露
当你使用标准http库发起请求时，你得到一个http的响应变量。如果你不读取响应主体，你依旧需要关闭它。注意对于空的响应你也一定要这么做。
package main
import (
"fmt"
"net/http"
"io/ioutil"
)
func main() {
resp, err := http.Get("https://api.ipify.org?format=json")
if err != nil {
fmt.Println(err)
return
}
defer resp.Body.Close()//ok, most of the time :-)
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
fmt.Println(err)
return
}
fmt.Println(string(body))
}
大多数情况下，当你的http响应失败时，resp变量将为nil，而err变量将是non-nil。然而，当你得到一个重定向的错误时，两个变量都将是non-nil。这意味着你最后依然会内存泄露。
package main
import (
"fmt"
"net/http"
"io/ioutil"
)
func main() {
resp, err := http.Get("https://api.ipify.org?format=json")
if resp != nil {
defer resp.Body.Close()
}
if err != nil {
fmt.Println(err)
return
}
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
fmt.Println(err)
return
}
fmt.Println(string(body))
}
Exchange paste
Problem
Show here
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.