collector-m / supercrawler Goto Github PK

Zhijian get familiar with Python language.

1 week

Based some knowledge from some paper, I've written a draft design. Have a 
look to see what else can we do and is there any improvement.

Just found python urllib2 api can't support the timeout when reading from 
HTTPResponse. I dug into the code in python lib just found there's two 
solution for this:

1. Modify python lib code. This is the easist way but make changes to 
standard library. Currently the lib just read 8K each time with infinite 
blocking time. I can set a timeout for that, it's the easist way. For 
example, set a timouet as 2s, so each read operation must finish in 2s 
which means the download speed must be faster than 4k/s.

2. Implement another HTTP handler.
This is a little complex but doable.

Research & Design crawler architecture.
1 week