syhen / dynamicspider Goto Github PK

The user can provide page rule of the url if (GET URL) can download the full page without javascript.
Or, I may add some functions as:

• distinguish javascript or not.
• if is javascript, choose scrapy-splash or selenium.
• if there is a rule of url's next page. Identify it.
• xpath or css selector input.
• regular expression input.
• join rules

all in editor.

some pages is lazy load. (although we can get url from 查看更多, how can we deal with the same page as this.)
The page is modified by javascript, and we can't separate the page into n part. selenium can solve the problem.