nicohman / reimagined-wildbow-scraper Goto Github PK

View Code? Open in Web Editor NEW

11.0 7.0 6.0 6.12 MB

Scrapes wildbow's twig, pact, glow-worm, ward and worm into epubs with some extra ease-of-use features. Now deprecated.

License: GNU General Public License v3.0

JavaScript 100.00%

worm pact reimagined-wildbow-scraper twig wildbow glow-worm ward

reimagined-wildbow-scraper's Introduction

reimagined-wildbow-scraper

Now deprecated in favor of rust-wildbow-scraper. Go use that instead!

Scrapes wildbow's twig, pact and worm into epubs with some extra ease-of-use features. Run node app.js with -t, -w, -g, -r or -p to scrape twig, worm, glow-worm, ward or pact respectively. --help for more info. New: Now scrapes Ward(Worm 2)!

Installation

All you'll need is npm and node. Run:

git clone https://github.com/nicohman/reimagined-wildbow-scraper.git

cd reimagined-wildbow-scraper

npm install

node app.js [options]

You might want to add an alias such as alias riws = node ~/reimagined-wildbow-scraper/app.js to make it easier to use.

Usage

node app.js --help for list of commands:

    -V, --version         output the version number
    -r , --resume <json>  Resume from JSON file
    -a, --ward            Scrape Ward
    -w, --worm            Scrape Worm
    -t, --twig            Scrape Twig
    -p, --pact            Scrape Pact
    -g, --glow_worm       Scrape Worm 2 Prologue
    -h, --help            output usage information

It'll output an epub and a json file in your current directory. If you want to, you can save the json file and use to to resume later, saving you from having to download the whole rest of whatever book you're following along with.

reimagined-wildbow-scraper's People

Contributors

Stargazers

Watchers

Forkers

wassname hollowaytape jlr910168 tubbz-alt reaper2507 samirhosseini

reimagined-wildbow-scraper's Issues

Errors when inputing any command relating to scraping

For example, say I input "node app.js -a" to download Ward. This is what happens:

"C:\Users\rober\reimagined-wildbow-scraper>node app.js -a
Command Prompt - node app.js -a[6668]: src\node_contextify.cc:628: Assertion `args[1]->IsString()' failed.
1: node::DecodeWrite
2: node::DecodeWrite
3: uv_loop_fork
4: v8::internal::interpreter::BytecodeDecoder::Decode
5: v8::internal::RegExpImpl::Exec
6: v8::internal::RegExpImpl::Exec
7: v8::internal::RegExpImpl::Exec
8: 000003E4B2284281
"

But if I input "node app.js -h" to check the help menu, it comes out fine:

"C:\Users\rober\reimagined-wildbow-scraper>node app.js -h

Usage: app [options]

Options:

-V, --version         output the version number
-r , --resume <json>  Resume from JSON file
-a, --ward            Scrape Ward
-w, --worm            Scrape Worm
-t, --twig            Scrape Twig
-p, --pact            Scrape Pact
-g, --glow_worm       Scrape Worm 2 Prologue
-h, --help            output usage information"

I have double and triple re-cloned the source, and made sure I had all npm dependencies the program seemed to need. I've tried both GitBash and Windows Command prompt (all run as administrator). But I have no idea in heck what is going on. Do you have any input?

Getting "URLCon is not a constructor" error

$ node app.js -t
X:\...\reimagined-wildbow-scraper-master\app.js:57
        var nexturl = new URLCon(next);
                      ^

TypeError: URLCon is not a constructor
    at IncomingMessage.<anonymous> (X:\...\reimagined-wildbow-scraper-mast
er\app.js:57:23)
    at emitNone (events.js:91:20)
    at IncomingMessage.emit (events.js:185:7)
    at endReadableNT (_stream_readable.js:974:12)
    at _combinedTickCallback (internal/process/next_tick.js:80:11)
    at process._tickCallback (internal/process/next_tick.js:104:9)

I fixed it by replacing const URLCon = require('url').URL; with const URLCon = require('url'); and then replacing all instances of var nexturl = new URLCon(next); with var nexturl = URLCon.parse(next);.

Cannot find module 'follow-redirects'

This is my first time here, please correct me if I'm doing something wrong. Thanks for revising the Wildbow scraper. However, I can't get any of the books to scrape, I get the following error instead. I'm trying to run it while in the directory where app.js is located.

module.js:529
    throw err;
    ^

Error: Cannot find module 'follow-redirects'
    at Function.Module._resolveFilename (module.js:527:15)
    at Function.Module._load (module.js:476:23)
    at Module.require (module.js:568:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (/Users/mow/Downloads/reimagined-wildbow-scraper-master/reimagined-wildbow-scraper-master/app.js:2:13)
    at Module._compile (module.js:624:30)
    at Object.Module._extensions..js (module.js:635:10)
    at Module.load (module.js:545:32)
    at tryModuleLoad (module.js:508:12)
    at Function.Module._load (module.js:500:3)

(Also, Worm 2 is now known as Ward, perhaps -r?)

nicohman / reimagined-wildbow-scraper Goto Github PK

reimagined-wildbow-scraper's Introduction

reimagined-wildbow-scraper

Now deprecated in favor of rust-wildbow-scraper. Go use that instead!

Installation

Usage

reimagined-wildbow-scraper's People

Contributors

Stargazers

Watchers

Forkers

reimagined-wildbow-scraper's Issues

Errors when inputing any command relating to scraping

Getting "URLCon is not a constructor" error

Cannot find module 'follow-redirects'

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent