leoncvlt / blinkist-scraper Goto Github PK
View Code? Open in Web Editor NEW📚 Python tool to download book summaries and audio from Blinkist.com, and generate some pretty output
📚 Python tool to download book summaries and audio from Blinkist.com, and generate some pretty output
Hello and first of all thank you very much for your work!
It looks, like this is exactly the code that I was looking for, but unfortunately I'm not able to get it running because I get stuck in an endless Cloudflare hCaptcha loop on https://www.blinkist.com/en/nc/login when I'm trying to execute it the first time.
The "One more step - Please complete the security check to access - I am human" appears before entering the login information and no matter how often I solve it, I always end up at the next Captcha (tried it for at least 9 times in a row).
My system:
I've already tried:
Unfortunately I don't have any other ideas at the moment and feel pretty lost/stupid.
Did you encounter this problem before and have an idea how to solve it?
Or are there some logfiles or something I can collect that might help in this case?
Thank you very much in advance!
Peter
I will get this timeout every once in a while. But when using the --book flag to download the book with this error, it seems to work fine.
Is there a way to redownload books without restarting? It seems there isn't a way to do this at the moment. It will scrape everything over again.
Or alternatively, there is currently a way to not re-scrape json files and only download the books with json files. It is possible to scrape all json files first before downloading? Then I can manually take out the json files with books I have already downloaded?
======versions===========
chromedriver-autoinstaller 0.2.2
colorama 0.4.4
EbookLib 0.17.1
requests 2.25.1
selenium 3.141.0
selenium-wire 4.3.2
python 3.8.10
===============Error=================
Message: timeout: Timed out receiving message from renderer: 298.991
(Session info: chrome=91.0.4472.124)
(Driver info: chromedriver=91.0.4472.101 (af52a90bf87030dd1523486a1cd3ae25c5d76c9b-refs/branch-heads/4472@{#1462}),platform=Windows NT 10.0.18363 x86_64)
Traceback (most recent call last):
File "D:\hxh103\blinker2\blinkistscraper_main_.py", line 412, in
main()
File "D:\hxh103\blinker2\blinkistscraper_main_.py", line 368, in main
dump_exists = scrape_book(
File "D:\hxh103\blinker2\blinkistscraper_main_.py", line 257, in scrape_book
audio_files = scraper.scrape_book_audio(
File "D:\hxh103\blinker2\blinkistscraper\scraper.py", line 513, in scrape_book_audio
driver.get(book_reader_url)
File "C:\Users\hxh103\Anaconda3\envs\blink2\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 333, in get
self.execute(Command.GET, {'url': url})
File "C:\Users\hxh103\Anaconda3\envs\blink2\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "C:\Users\hxh103\Anaconda3\envs\blink2\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message: timeout: Timed out receiving message from renderer: 298.991
(Session info: chrome=91.0.4472.124)
(Driver info: chromedriver=91.0.4472.101 (af52a90bf87030dd1523486a1cd3ae25c5d76c9b-refs/branch-heads/4472@{#1462}),platform=Windows NT 10.0.18363 x86_64)
add the possibility to create-mobi file
I see that ebooklib does not supper mobi but I found this
https://github.com/jachinlin/kindle_maker
if interested it could be added to generate also mobi files
hi
i just run this repo and it does not work.
it just opens chrom and nothing else happens
[12:44:35] CRITICAL Uncaught Exception. Exiting...
this is my arguments.
python3 blinkistscraper [email protected] PASSWORD --concat-audio --language en --book https://www.blinkist.com/en/books/thinking-in-bets-en --create-epub --no-ublock
I noticed that a lot of Uncategorized books are in German and then are downloaded even if I select the flag --language en.
I tried to skip the "uncategorized" category altogether, but that didn't work. All the German books have a URL like www.blinkist.com/en/nc/reader/x-de. So should be able to add a case to skip if the last three characters of the url string are "-de" and language is "en."
Hello there, I hope all is well with you.
I have been using this script for a few months and I have to admit that it has significantly helped me in reading Blinkist on my kindle.
I have run into an annoying situation where I am unable to scrape some books.
For instance, I receive this error when the Chromedriver is started and the link to my book is opened.
Would you mind assisting me in completing this task?
best
Thank you first :)
i thought the book Homo Deus (https://www.blinkist.com/en/nc/reader/homo-deus-en) was in my dump & books folder, but i couldn't find it.
It occurs to other 2 books by Yuval Noah Harari: Sapiens, 21 Lessons for the 21st Century.
Would it be possible to be solved? ... Thank you so so much >_<
Homo Deus (https://www.blinkist.com/en/nc/reader/homo-deus-en)
Sapiens (https://www.blinkist.com/en/nc/reader/sapiens-en)
21 Lessons for the 21st Century (https://www.blinkist.com/en/nc/reader/21-lessons-for-the-21st-century-en)
I Used this library about a year ago and it worked like a charm. but there is a little problem. it uses Blinkist categories to download the Blinks and I noticed that not all the available books are listed in categories and around half of them are missing.
I suggest that instead of using categories to download all the books we use Blinkist sitemap. now you can conveniently find the Blinkist sitemap in https://www.blinkist.com/en/sitemap. I don't have the knowledge to solve this issue but I hope somebody can do it.
Program gets aborted at this particular book with FileNotFoundError
[.] Json dump for book https://www*blinkist*com/en/books/the-five-most-important-questions-you-will-ever-ask-about-your-organization-en already exixts, skipping scraping...
[.] Downloading audio file for blink 0...
[.] Downloading audio file for blink 1...
[.] Downloading audio file for blink 2...
[.] Downloading audio file for blink 3...
[.] Downloading audio file for blink 4...
[.] Downloading audio file for blink 5...
[.] Downloading audio file for blink 6...
[.] Combining audio files for the-five-most-important-questions-you-will-ever-ask-about-your-organization-en
[!] ffmpeg output file longer than 260 characters. Trying shorter filename...
[.] Generating .html for the-five-most-important-questions-you-will-ever-ask-about-your-organization-en
Traceback (most recent call last):
File "main.py", line 69, in
processed_books = process_book_json(book_json, processed_books)
File "main.py", line 29, in process_book_json
generator.generate_book_html(book_json)
File "C:\Users\Fang Yuan\Downloads\blinkist-scraper-master\generator.py", line 42, in generate_book_html
with open(html_file, 'w', encoding='utf-8') as outfile:
FileNotFoundError: [Errno 2] No such file or directory: 'books\Entrepreneurship\Peter F Drucker - The Five Most Important Questions You Will Ever Ask About Your Organization\Peter F Drucker - The Five Most Important Questions You Will Ever Ask About Your Organization.html'
Would it be possible to scrape my all library using an argument like "library" and eventually skip what has been already scraped? This way I could setup a crono scraping and every time I have a new book in my library it will be downloaded.
Hi! Thanks for amazing repo!
When I run python blinkistscraper email pass
chrome asks to solve the captcha, after solving the first captcha logs print INFO Logged into Blinkist. Loading Library...
but nothing happens, also chrome repeatedly asks to pass captchas again and again.
How to solve this issue?
I get this syntax error while running main.py
File "main.py", line 59 print(f"[#] Processed {processed_books} books in {formatted_time}") ^ SyntaxError: invalid syntax
If I try commenting that line I get other syntax errors in other files... I guess there is something wrong with my configuration
A lot of blinks download the audio, then when it comes to combining, it must error but all the audio files get deleted.
Audio downloads seem to be a bit iffy - sometimes the script just fails to capture the audio request to the first chapter, or a request just times out. Those are handled gracefully but it means that no audio for that book is downloaded. Look into adding a --retry-limit
flag so that in case of a failed audio download, the script waits a bit and retries as many times as defined by the flag.
Hi. Really interested to use this script but I have serious difficulties to follow instructions as per README file.
Basic usage calls for python main.py without underscores.... but there is no such file.
Only similar file is main.py with underscores before and after main word in blinkistscraper folder but I guess it is not the right one.
Salvatores-MacBook-Pro:blinkistscraper salva$ python main.py user pass https://www.blinkist.com/en/nc/reader/the-box-en
File "main.py", line 7
log = logger.get(f"blinkistscraper")
^
SyntaxError: invalid syntax
Salvatores-MacBook-Pro:blinkistscraper salva$
Can you pls give more clear instruction on how to execute first command?
Thanks
Is there any way to download German books and audios as well?
Could you add another flag to choose the language maybe?
I also ran into JSONDecodeErrors at some point in the program. I don't know when and why this happens exactly.
[.] Scraping book at https://www.blinkist.com/en/books/disrupted-en
Traceback (most recent call last):
File ".\main.py", line 62, in <module>
audio_files = scraper.scrape_book_audio(driver, book_json)
File "[...]\scraper.py", line 223, in scrape_book_audio
audio_url = audio_request.json()['url']
File "[...]\Anaconda3\lib\site-packages\requests\models.py", line 889, in json
self.content.decode(encoding), **kwargs
File "[...]\Anaconda3\lib\json\__init__.py", line 348, in loads
return _default_decoder.decode(s)
File "[...]\Anaconda3\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "[...]\Anaconda3\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
PS: you should add Chrome browser to dependencies, as chomedriver needs Chrome browser to be installed.
I'm getting constant errors when scraping even on the start, when it's connecting and downloading the booklist. The message is allways the same but it happens multiple times nearly once per second.
Error making request
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/seleniumwire/proxy/proxy2.py", line 91, in proxy_request
conn.request(self.command, path, req_body, dict(req.headers))
File "/usr/lib/python3.7/http/client.py", line 1260, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.7/http/client.py", line 1306, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.7/http/client.py", line 1255, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.7/http/client.py", line 1030, in _send_output
self.send(msg)
File "/usr/lib/python3.7/http/client.py", line 970, in send
self.connect()
File "/usr/local/lib/python3.7/dist-packages/seleniumwire/proxy/proxy2.py", line 368, in connect
super().connect()
File "/usr/lib/python3.7/http/client.py", line 1415, in connect
super().connect()
File "/usr/lib/python3.7/http/client.py", line 942, in connect
(self.host,self.port), self.timeout, self.source_address)
File "/usr/lib/python3.7/socket.py", line 707, in create_connection
for res in getaddrinfo(host, port, 0, SOCK_STREAM):
File "/usr/lib/python3.7/socket.py", line 748, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known
Scraping works as asprected but I can't find, what causes this error. It's not depending on any of the possible arguments.
the recaptcha stage carries on forever it keep asking to select images in a loop ... does recaptcha detect more than we know?
Referring to 502 errors, I'd suggest to add the possibility to ignore SSL errors by providing an optional command line parameter.
idk how to describe this, but from one day to the other I get this:
DevTools listening on ws://127.0.0.1:58651/devtools/browser/ef75920d-da4e-4d69-be34-8a83fa0e3391
←[2m[13:09:45]←[0m ←[34mINFO←[0m Please solve captcha to proceed!
←[2m[13:10:13]←[0m ←[34mINFO←[0m Logged into Blinkist. Loading Library...
←[2m[13:10:13]←[0m ←[31mERROR←[0m string indices must be integers
Traceback (most recent call last):
File "C:\Users\x8251\Blinkist Scraper\blinkistscraper_main_.py", line 412, in
main()
File "C:\Users\x8251\Blinkist Scraper\blinkistscraper_main_.py", line 336, in main
scrape_book(
File "C:\Users\x8251\Blinkist Scraper\blinkistscraper_main_.py", line 248, in scrape_book
book_json, dump_exists = scraper.scrape_book_data(
File "C:\Users\x8251\Blinkist Scraper\blinkistscraper\scraper.py", line 391, in scrape_book_data
if os.path.exists(get_book_dump_filename(book_url)) and not force:
File "C:\Users\x8251\Blinkist Scraper\blinkistscraper\utils.py", line 23, in get_book_dump_filename
return os.path.join("dump", book_json_or_url["slug"] + ".json")
TypeError: string indices must be integers
←[2m[13:10:13]←[0m ←[41mCRITICAL←[0m Uncaught Exception. Exiting...
Hello, I have just started using this library and all seems to be correctly set up. I ran python blinkistscraper email password with my credentials and Cloudflare unfortunately detects (I assume) an automated activity and blocks me from navigating to Blinkist.com on the browser instance that got opened by the script.
Any ideas?
when chromedriver opens up it gives following response - opening any site in it gives same response (my usual chrome works fine)
Error response
Error code: 502
Message: Bad Gateway.
Error code explanation: 502 - Invalid responses from another server/proxy.
Right now the script is hardcoded to use chromedriver.exe
which obviously only works on windows. Should make the chromedriver path into a proper argument so people can use the windows or linux executable as well.
Dear Leonardo, *
great script - used to backup some of my favorite books in January as epub and mp4. Worked like a charm.
Now it breaks shortly after initialising the (already updated) chromedriver:
C:\blinkistscraper>python main.py --books.txt --cooldown 5 --concat-audio --keep-noncat --save-cover --embed-cover-art
←[2m[13:10:21]←[0m ←[34mINFO←[0m Starting scrape run...
←[2m[13:10:22]←[0m ←[34mINFO←[0m Initialising chromedriver at ~\Python\Python38\lib\site-packages\chromedriver_autoinstaller\91\chromedriver.exe...DevTools listening on ws://~
←[2m[13:10:52]←[0m ←[34mINFO←[0m Logged into Blinkist. Loading Library...Traceback (most recent call last):
File "main.py", line 340, in
main()
File "main.py", line 303, in main
dump_exists = scrape_book(
File "main.py", line 217, in scrape_book
audio_files = scraper.scrape_book_audio(
File "~\scraper.py", line 504, in scrape_book_audio
del driver.requests
AttributeError: requests
Any(body) an idea what's the root cause resp. how to fix?
Thanks in advance!
Chromedriver is being installed earlier in proc from the autoinstaller dir then runs into another "missing" error when options.py is running. Unsure how to resolve.
Help is appreciated. Sorry if this is the wrong place to comment. This is my first time so any feedback is helpful.
$ python main.py --language en --categories market --create-html --save-cover -v @@@@@ XXXXX
[15:45:32] INFO Starting scrape run...
[15:45:32] INFO Initialising chromedriver at C:\Python38\lib\site-packages\chromedriver_autoinstaller\84\chromedriver.exe...
Traceback (most recent call last):
File "main.py", line 340, in <module>
main()
File "main.py", line 276, in main
driver = scraper.initialize_driver(
File "C:\Users\USER\Documents\GitHub\blinkist-scraper\blinkistscraper\scraper.py", line 67, in initialize_driver
chrome_options.add_extension(
File "C:\Python38\lib\site-packages\selenium\webdriver\chrome\options.py", line 131, in add_extension
raise IOError("Path to the extension doesn't exist")
OSError: Path to the extension doesn't exist
>python blinkistscraper @.* ********
←[2m[12:43:23]←[0m ←[34mINFO←[0m Starting scrape run...
←[2m[12:43:23]←[0m ←[34mINFO←[0m Initialising chromedriver at C:\Users***\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\chromedriver_autoinstaller\91\chromedriver.exe...
DevTools listening on ws://127.0.0.1:56424/devtools/browser/492b0c18-1e9f-431f-8080-49943064fb4b
←[2m[12:43:32]←[0m ←[34mINFO←[0m Logged into Blinkist. Loading Library...
←[2m[12:43:33]←[0m ←[31mERROR←[0m 'NoneType' object has no attribute 'split'
Traceback (most recent call last):
File "\blinkistscraper_main_.py", line 412, in
main()
File "\blinkistscraper_main_.py", line 358, in main
categories = scraper.get_categories(
File "**\blinkistscraper\scraper.py", line 322, in get_categories
"label": " ".join(label.split()).replace("&", "&"), "url": href
AttributeError: 'NoneType' object has no attribute 'split'
←[2m[12:43:33]←[0m ←[41mCRITICAL←[0m Uncaught Exception. Exiting...
Any idea?
is there a command/argument I can give to skip the audio scraping, if I m interested in the ebook only?
Hi all,
I have just tried the blinkist scraper with my free account. It does not work any more. It just pops up the captcha window. I insert them many times, then the captcha windows crashes and I get the following messages:
INFO Please solve captcha to proceed!
ERROR Error. Captcha needs to be solved within 1 minute
ERROR Unable to login into Blinkist
INFO Processed 0 books in 00:01:10
Any hints?
I m trying to run it on my raspberry pi4 but it seems that it does not like the chromedriver.
I had to arg the chromedriver location and I also tried with the latest version from https://github.com/electron/electron/releases but with same results.
Any idea pls?
pi@raspberrypi:~/blinkist-scraper $ python3 blinkistscraper/main.py user pass --book https://www.blinkist.com/en/nc/reader/on-saudi-arabia-en --chromedriver /usr/bin/chromedriver
[18:30:10] INFO Starting scrape run...
[18:30:10] INFO Initialising chromedriver at /usr/bin/chromedriver...
Traceback (most recent call last):
File "blinkistscraper/main.py", line 340, in
main()
File "blinkistscraper/main.py", line 279, in main
chromedriver_path=args.chromedriver,
File "/home/pi/blinkist-scraper/blinkistscraper/scraper.py", line 81, in initialize_driver
options=chrome_options,
File "/home/pi/.local/lib/python3.7/site-packages/seleniumwire/webdriver/browser.py", line 86, in init
super().init(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/selenium/webdriver/chrome/webdriver.py", line 81, in init
desired_capabilities=desired_capabilities)
File "/usr/local/lib/python3.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 157, in init
self.start_session(capabilities, browser_profile)
File "/usr/local/lib/python3.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "/usr/local/lib/python3.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python3.7/dist-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: Chrome failed to start: exited abnormally
(unknown error: DevToolsActivePort file doesn't exist)
(The process started from chrome location /usr/bin/chromium-browser is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
(Driver info: chromedriver=78.0.3904.108 (4b26898a39ee037623a72fcfb77279fce0e7d648-refs/branch-heads/3904@{#889}),platform=Linux 5.4.51-v7l+ armv7l)
First, thanks for this! This is great.
Wondering tho if it'd be possible to embed the album art into the .m4a file, or even (maybe both?) download the linked jpeg file in the HTML output file created into the containing folder?
When navigating the each category section, a list of books appears, which is used to gather all book urls and download them all afterwards. You would expect to be getting all books from Blinkist if you navigated the "Show all books" section from every existing category, right? Well... it's not the case.
Nonetheless, not all books seem to have a category, resulting in this (awesome) library to miss around 15% of all books (422 missing out of 2800). This problem is the cause of, for example, this issue.
I managed to craft a list of all currently existing books in Blinkist. I've done so through their own search engine. They use Algolia and from my previous experience, it can be queried using the client's x-algolia-application-id
and x-algolia-api-key
. Then simply call the /indexes/books-production/query
endpoint. Then, I did a bit of clean-up on the JSON response and constructed each book URL. Here's the list that you can easily use with the --books
argument.
Hi,
I keep getting this error after scraping few books (3 to 4 books). Please help
Traceback (most recent call last): File "main.py", line 62, in <module> audio_files = scraper.scrape_book_audio(driver, book_json) File "C:\Users\Fang Yuan\Downloads\blinkist-scraper-master\scraper.py", line 223, in scrape_book_audio audio_url = audio_request.json()['url'] File "C:\Python38\lib\site-packages\requests\models.py", line 888, in json return complexjson.loads( File "C:\Python38\lib\json\__init__.py", line 357, in loads return _default_decoder.decode(s) File "C:\Python38\lib\json\decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "C:\Python38\lib\json\decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Hello there,
I recently discovered this repository and started using. I'm trying to download the audio blinks, but I keep getting the following exception.
Is there something I'm missing here, or is this for everyone?
Please advise what I can do to prevent this error.
$ python blinkistscraper --language en --match-language --audio myEmail myPassword
[15:41:38] INFO Starting scrape run...
[15:41:39] INFO Initialising chromedriver at C:\Users\Girish\AppData\Local\Programs\Python\Python37\lib\site-packages\chromedriver_autoinstaller\90\chromedriver.exe...
[15:41:48] ERROR Timeout waiting for ublock config overwrite alert
[15:42:03] INFO Logged into Blinkist. Loading Library...
[15:42:07] INFO Scraping categories: Entrepreneurship, Politics, Marketing & Sales, Science, Health & Nutrition, Personal Development, Economics, History, Communication Skills, Corporate Culture, Management & Leadership, Motivation & Inspiration, Money & Investments, Psychology, Productivity, Sex & Relationships, Technology & the Future, Mindfulness & Happiness, Parenting, Society & Culture, Nature & the Environment, Biography & Memoir, Career & Success, Education, Religion & Spirituality, Creativity, Philosophy
[15:42:07] INFO Getting all books for category Entrepreneurship...
[15:42:13] INFO Found 198 books
[15:42:13] ERROR requests
Traceback (most recent call last):
File "blinkistscraper\__main__.py", line 412, in <module>
main()
File "blinkistscraper\__main__.py", line 373, in main
match_language=match_language,
File "blinkistscraper\__main__.py", line 258, in scrape_book
driver, book_json, args.language
File "blinkistscraper\scraper.py", line 504, in scrape_book_audio
del driver.requests
AttributeError: requests
[15:42:13] CRITICAL Uncaught Exception. Exiting...
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.