Amazon/Kindle AZW-format ebooks NOT stripping DRM with Calibre plugin about dedrm_tools HOT 67 CLOSED

Post a log of importing one of the books.

from dedrm_tools.

Could you tell me how to do that, exactly? Where do I find the log files in Calibre? What I pasted is the log file of the attempted conversion for three books (and they failed bc the DRM was still present), which is the only log I can find.

from dedrm_tools.

Remove the book from calibre. Then look for "Restart in debug mode" in the Preferences menu. Once it reopens, add the book again. Close calibre, and the log will open.

from dedrm_tools.

I have 497 books in my kindle library, all of which I downloaded today, none of which are listed by name by the Kindle app (so there's no way to find a book by its name that doesn't strip DRM properly). This is going to be a ridiculous amount of effort. Is there not any other way to give you what you want?

Also when I restart Calibre, aren't I going to lose my Job Log, which tells me which books didn't import/convert correctly? If so, that's going to really suck, since there's no way to export that so I can reference it later...

from dedrm_tools.

Pick a book you know has DRM. Remove and redownload it in Kindle for PC. Then sort your Kindle folder by date.

from dedrm_tools.

ok. thanks. I don't have a PC btw - I have a Mac (running latest version of OS X software as well).

Also, do you have an idea of how long Calibre should take to restart in debug mode? I'm at 13 minutes and counting, and it's still stuck on the load screen (says, "initializing user interface). The menus are all there in the background (but they're greyed-out), and I can see the library loaded behind the load screen, but obv nothing works). I'm running 2.47, updated today, no other problems that I've noticed.

Normally, it takes <1 minute to load Calibre.

from dedrm_tools.

And btw I already tried force-quitting, restarting Calibre normally (it was fine), and then restarting in debug mode again. Both times, same behavior (stuck on load screen).

from dedrm_tools.

No, it shouldn't take that long to open in debug mode. You could try another way to open in debug mode explained in the plugin Read Me and also on the FAQs page of the Alf blog:

On Macintosh, open the Terminal application (in your Utilities folder).
You should now have a text-based command-line window open. Make sure that calibre isn’t running.

Now type in ‘calibre-debug -g’ (without the ‘’) and press return key. Calibre will launch in debug mode, and various diagnostic information will be written to the terminal window.

Import the ebook into calibre. More information will be written to the terminal window.

Now copy the output from the terminal window.

from dedrm_tools.

I tried - this is what happened:
$ calibre-debug -g
-bash: calibre-debug: command not found

from dedrm_tools.

I tried running it from within the Applications directory as well, just in case. No joy. And yes I made sure that calibre was not running before I did any of this.

from dedrm_tools.

are you sure there shouldn't be a space between "calibre" and "-debug"? That's usually how such commands run, assuming that calibre-debug isn't the name of a separate application...

from dedrm_tools.

How's this for weird? From Terminal, I get this listing for the main Applications folder (the user-specific Apps folder is empty):

But when I take a screen shot of the SAME (blessed) Applications folder, I see calibre just fine (see attached). Not only that, but I have always installed Calibre the default method, i.e. by dragging and dropping the app into the Apps folder included in the installer. So it's not that I'm putting it in the wrong place. But how is it that calibre is hidden in Terminal, and visible in the Finder?

from dedrm_tools.

No space between calibre and -debug. Space before -g.

I'm stumped and headed off to bed. I'll send an email to a friend who's familiar with Mac to see if he can take a look at this and help you.

from dedrm_tools.

Yes well I copied and pasted your instructions into the CLI, so I know I got it right. I tried typing it manually both ways too, just to be sure. IDK what's going on, but this is very frustrating.

from dedrm_tools.

I take it back - calibre IS listed in the Apps folder, it's just NOT IN ALPHA ORDER, for some reason, and it's listed as calibre.app (even though I didn't change the name of the app or anything). Is this why I can't get calibre-debug to run? Is there some invisible character or is it the .app at the end that's messing up the CLI instructions? See attached, look at end of list.

from dedrm_tools.

well that didn't work either, so that's not it:
KBELL-MB-3:Applications kellybell$ calibre.app-debug -g
-bash: calibre.app-debug: command not found

And this didn't work either:
KBELL-MB-3:Applications kellybell$ calibre.app
-bash: calibre.app: command not found

I give up. Hopefully I'll hear back from someone tomorrow.

from dedrm_tools.

There are some variations as to where the applications are placed.
if you look in /Applications/calibre.app you should see a folder named Contents, and in the Contents folder there should be a MacOS folder, and the actual apps will be there.
$ls -al /Applications/calibre.app/Contents/MacOS
should produce a listing
the info at http://manual.calibre-ebook.com/cli/cli-index.html will be correct except for the initial suggestion that the applications live inside a calibre directory under MacOS

from dedrm_tools.

and it is not calibre.app calibre.app is essentially a container.
inside that container you will find all the actual applications.
so ...
$ /Applications/calibre.app/Contents/MacOS/calibre-debug --gui

from dedrm_tools.

to find the app you would potentially do
$ sudo find / -name calibre-debug -print
and that will confirm where the application in fact lives on your system

from dedrm_tools.

OK I got it running. I'm going to see if it will break out of the initialization routine, and will report back.

from dedrm_tools.

It's converting this time. I double-checked to make sure that this book DID NOT strip DRM or convert to another format without throwing the DRM error BEFORE I tried this. So why does it work in debug mode and not at all in regular mode? Here's the dump:

KBELL-MB-3:MacOS kellybell$ /Applications/calibre.app/Contents/calibre-debug.app/Contents/MacOS/calibre-debug -g
calibre 2.47 isfrozen: True is64bit: True
Darwin-15.2.0-x86_64-i386-64bit Darwin ('64bit', '')
('Darwin', '15.2.0', 'Darwin Kernel Version 15.2.0: Fri Nov 13 19:56:56 PST 2015; root:xnu-3248.20.55~2/RELEASE_X86_64')
Python 2.7.9
OSX: ('10.11.2', ('', '', ''), 'x86_64')
Successfully initialized third party plugins: DeDRM
Starting up...
Started up in 24.56 seconds with 11314 books
DeDRM v6.3.4: Trying to decrypt B000OIZSKA_EBOK.azw
Using Library AlfCrypto DLL/DYLIB/SO
MobiDeDrm v0.41.
Copyright © 2008-2012 The Dark Reverser et al.
MOBI header version 8, header length 264
Extra Data Flags: 3
Decrypting Kindle Format 8 ebook: Without Fail (Jack Reacher, Book 6)
Found 4 keys to try after 0.2 seconds
Crypto Type is: 2
File is encoded with PID DHJyxuCKQY.
Decrypting. Please wait . . . . . done
Decryption succeeded after 0.4 seconds
DeDRM v6.3.4: Finished after 0.4 seconds
Added Without Fail (Jack Reacher, Book 6) to db in: 0.6
Added 1 books in 7.4 seconds
Worker Launch took: 1.17990803719
2015-12-27 21:20:37.408 calibre-debug[10539:3259394] Communications error: <OS_xpc_error: <error: 0x7fff77303b90> { count = 1, contents =
"XPCErrorDescription" => <string: 0x7fff77303f40> { length = 22, contents = "Connection interrupted" }
}>
Job: 1 Convert book 1 of 1 (Without Fail (Jack Reacher, Book 6)) finished
Convert book 1 of 1 (Without Fail (Jack Reacher, Book 6))
Resolved conversion options
calibre version: 2.47.0
{'asciiize': False,
...
--- etc --- (removed lots of boring conversion data)
...
Looking for large trees in text/part0017.html...
No large trees found
EPUB output written to /var/folders/k6/rpx44f5j73s69qklkkf0b9wr0000gn/T/calibre_2.47.0_tmp_vjz0fg/toJ90L.epub

2015-12-27 21:21:38.329 Calibre[10581:3260762] deliverNotificationWithTitle()

• Notification delivered.

from dedrm_tools.

rofl, they all point to the same place, as they are placeholders for the gui apps you see if you view the contents from Finder (i.e. if you go to calibre in Finder, right click, and select "Show Package Contents" you will see an icon for calibre-debug, and that icon is linked to the actual command that you can find in /Applications/calibre.app/Contents/MacOS -- I once suggested that the calibre docs be fixed, but my suggestion was apparently not taken to heart :-(

Have you tried clicking on the calibre-debug icon?

from dedrm_tools.

I am no expert, but as I understand it, there is no reason for the import to work in debug mode and not in regular mode. So, delete these books, exit calibre, start calibre in regular mode, and add the book, then exit calibre. restart calibre and then try to convert. Beyond that you will have to get the apprentice's help

from dedrm_tools.

It's working fine now that I know how to run the app - that's not the problem. In fact, I just ran the whole process again, starting from scratch (deleting book in Calibre, archiving and re-downloading it in Kindle so I could tell which book it is, importing and converting) and it ALL WORKS FINE (strips DRM and converts w/out errors), AS LONG AS I RUN IN DEBUG MODE. But it DOES NOT work, for these same books, if I'm running in regular mode. So what do I do? Import all my Kindle books in debug mode from now on?

Is this a calibre bug or a plug-in bug? I would say plug-in, but IDK...

FWIW, I noticed that when I imported the test books in debug mode, they came in as AZW3 format, whereas in the regular-mode import (which failed), the books came in as AZW. Hmmm.

from dedrm_tools.

net-buoy, I already did all that. please read above. And no offense, but I kinda really DO want to talk to the experts on this one, bc basic troubleshooting I can do for myself. This is definitely a bug of some kind.

from dedrm_tools.

Oh also - when I ran debug mode from Terminal, it initialized VERY quickly. Same as regular mode. So something is definitely wrong with this version of Calibre in terms of starting debug mode from the GUI. So that's one calibre bug for sure that needs reporting, I assume.

from dedrm_tools.

but, net-buoy, thank you very much for demystifying WTF was going on with the file structure/app command in the CLI. I've never tun into that one before. Thank you.

from dedrm_tools.

Beats me - I suppose you could delete calibre and start from scratch (which is what is recommended in the FAQ if nothing else works)
And if that doesn't work ask Paul

from dedrm_tools.

well, hopefully the person who started helping me on this thread will come back tomorrow and tell me what they want me to do. Technically, I have a solution now (even if it's a workaround). But I need to know if I should report this to calibre or what...

Thanks again for your help with the package Contents silliness. Ugh.

from dedrm_tools.

yeah, I just confirmed that when in debug mode, calibre imports all these books as AZW3, not AZW (as it does in regular mode). IDK what that means in terms of allowing the DRM stripping to work, but maybe it will provide a clue. All the books that wouldn't convert before (65 or so) converted successfully in debug mode (proving that they have been successfully de-DRMd).

Please advise next steps as to reporting which pieces need to go to whomever needs to know?

Thank you,
Kelly

from dedrm_tools.

That all your files import as AZW implies that you don't have DeDRM installed. Since your debug log shows you have DeDRM installed, my guess is that you have more than one version of calibre on your computer, and that when you start up in normal mode, you're not starting the copy that's in your applications folder.

As for restarting in debug mode, I have Mac OS X 10.11.2 and calibre 2.47 and it works fine for me. (There is a small 'Debug mode' modal dialog that will need to be dismissed before you can start using the restarted calibre.) This is another hint that you might not be starting the right calibre - perhaps you're starting to copy on the disk image?

from dedrm_tools.

No. There's only one calibre. I can prove this easily by the location from
where I'm launching debug (which wouldn't work if I wasn't launching from
the same location). See new-buoy's instructions to see what I mean.

But furthermore I installed all the apps on this machine. In a programmer
for a living. I know where I installed it and how many times (once). I've
only had this machine for 7 months.

from dedrm_tools.

And, I always delete the installer images immediately after upgrading the
app. If you look at my screenshot's sidebars you can see that no installer
image is mounted on this machine. I would never run an app from a disk
image anyway; and in this case the proof is already right there in the
thread.

There are no other apps running on external or networked drives either.
Apps only run on local machines along with the OS.

from dedrm_tools.

I'm mystified, then. Do you still have problems restarting in debug mode without the DeDRM plugin installed? If so, go and get help in the calibre forum at MobileRead. I can't help feeling that that might be a symptom of some other problem that's also affecting DeDRM performance.

from dedrm_tools.

Yes. It just hangs up "initializing user interface" forever if I try to
start in debug mode from the GUI.

On Dec 28, 2015, at 01:45, Paul Durrant [email protected] wrote:

I'm mystified, then. Do you still have problems restarting in debug mode
without the DeDRM plugin installed? If so, go and get help in the calibre
forum at MobileRead. I can't help feeling that that might be a symptom of
some other problem that's also affecting DeDRM performance.

—
Reply to this email directly or view it on GitHub
#52 (comment)
.

from dedrm_tools.

Then I suggest you ask in the calibre support forum here:

http://www.mobileread.com/forums/forumdisplay.php?f=166

Once you have calibre reliably restarting in debug mode successfully, I suspect some of your other problems will go away.

from dedrm_tools.

As the OP notes that they are a "programmer" I am thinking that there is something in their GUI environment that is causing calibre to choke (most like having to do with python?), and the environment present when the CLI is employed being different, allows calibre to initiate without problems.

Has the OP looked at the calibre console entries to see what the OS is doing when calibre is hanging?

from dedrm_tools.

from dedrm_tools.

Something's definitely off with the calibre setup. You should be able to quickly open debug mode from within calibre or from the terminal window without switching to a specific directory. Also "regular" mode was acting as though DeDRM was not even installed.

Have you tried reinstalling calibre?

from dedrm_tools.

from dedrm_tools.

from dedrm_tools.

Ok, I hadn't heard any other Mac users not being able to follow the instructions as posted on the website and in the ReadMe file (which I quoted here.) And previously that was the only way to do it since it wasn't available from within calibre.

Yes, I'm aware you have the DeDRM installed and working in debug mode. But if it was working previously, none of the imported books would still have the AZW extension. Even those that didn't have DRM to begin with would now have AZW3, MOBI, or HTMLZ.

from dedrm_tools.

from dedrm_tools.

So are you saying that only books that import in AZW3 format can possibly
be de-DRM'd?

If so that's very confusing, bc I know for certain that most of the books
that were imported as AZW in regular mode and successfully deDRM'd
DEFINITELY had DRM in place to begin with.

A very small percentage were mobi-format books that I'd imported to read on
my kindle from other sources (they imported fine of course). But most were
books bought by me from Amazon.

from dedrm_tools.

Yes, the extension changes from AZW to AZW3 (or MOBI, etc.) if the DeDRM plugin is working. Some of them must have already been DRM-free.

from dedrm_tools.

Yes like I said, a small number were DRM-free. But out of the 50+ books
that were successfully converted, MOST of them were NOT DRM-free. I know
this for a fact bc I purchased them myself in Amazon.

from dedrm_tools.

Ok, in my experience, if everything is working correctly with the DeDRM plugin, the extension always changes from the generic AZW extension to the extension of the actual underlying format. If it has an AZW extension and it can be opened/converted, then it never had DRM to begin with. Again, this is how it should work or, at least, how it has worked.

from dedrm_tools.

At this point things might be a bit confused so it might help if you did a little matrix of what works and what doesn't. We have start-up, import, and convert versus calibre and calibre-debug via the finder, and calibre and calibre-debug via the CLI. I thought that I understood you to say that everything worked from the CLI, but nothing worked via the finder. If that is the case there are in my mind two explanations: 1) the contents necessary for the GUI is corrupted (in which case the best thing to do is delete, empty trash, and reinstall from a NEW download, or 2) there is something set differently between the GUI environment and the CLI environment, which is causing the GUI initialization to hang. As I mentioned above, there should be a calbre log accessible via console which should provide some possible guidance as to where calibre is hanging.

However, it seems pretty clear that it is a calbre issue, and the first thing I would do, is make it easy on yourself and delete calibre and empty trash, download a fresh copy, and start it, making sure that it opens properly without the plugin.

Then import a non-DRM book to make sure everything works.

Now, after you are sure that calibre is functioning correctly, download the latest plugin archive and install. Then exit calibre and restart. Now you have calibre working with the installed plugin.

Then, and only then, import and see what happens

from dedrm_tools.

Using the regular calibre GUI, roughly half the imports stripped DRM
and converted successfully. As best I can tell, the older books are
the only common denominator. The books that are over around 2-3 years
old converted just fine, whereas newer books did not.

• these older books are showing up as AZW format, yet they still strip
  and convert.
• The newer books ALSO import as AZW, but DRM is not stripped.
• The plugin manager says that the plugin is enabled and working fine.
• when I try running calibre-debug from the GUI, the app freezes at
  the load screen, saying "initializing UI".

Using the CLI to run calibre-debug, everything works as expected.

• ALL books import as AZW3, no matter their age.

Generally:

• This install of calibre is <6 months old, on a machine that was
  wiped clean with a brand new OS install at that same time.
• Calibre 2.47 upgrade was installed yesterday (before plugin)
• No other calibre plugins are being used except this one (de-DRM),
  and it was just installed for the first time yesterday, following to
  the exact directions for install
• this machine has not been modded extensively. Nothing else is
  breaking or acting weird either, and it's in use >12 hours/day.

Hope this helps. At this point, I'm planning to just continue using
the CLI, because the ginormous hassle of exporting, deleting,
re/installing etc is just way too much work that I don't really have
time for right now.

Net-buoy, I'll never forget your package contents trick. Muy bueno.
I'm not a Mac coder - my background is in OOP in Java, Ruby and PHP,
with some C++. But I've had a constant, steady stream of Macs since
1984, and I learned my lesson the hard way about modding the
Mac-native environment many years ago; that's why I use a VM. I've
never coded or packaged apps to run on a Mac either, so I've never
learned that trick. So - many thanks.

Thanks again to all, for your kind assistance.

from dedrm_tools.

Any books imported into calibre that retain their .azw extension have not been decrypted by the DeDRM plugin.

You mention older books and newer books. The decryption key for Kindle books depends on the device they were destined for. Books for eInk kindles require a serial number, books for Kindle for Mac depend on the particular installation of Kindle for Mac.

I suspect that if you wiped all your Kindle books, wiped Kindle for Mac, reinstalled Kindle for Mac and re-downloaded your books, all your problems (except restarting in Debug Mode) would go away.

But that is only a guess, since we haven't been able to see anything wrong in logs.

from dedrm_tools.

I'm thinking the DeDRM plugin just stopped working partway through the initial import. That can happen when you import a large batch at once.

For future reference, are the instructions on Alf's FAQs page for opening the calibre GUI in debug mode from the Terminal application missing something or just not working in this case?

from dedrm_tools.

Below

I'm thinking the DeDRM plugin just stopped working partway through the
initial import. That can happen when you import a large batch at once.

That was my initial thought too, though individual imports of books still
gave same results, both ways. Paul says there was no initial DRM on those
that succeeded bc they imported as AZW, though many in that list were
bestsellers, like some of the Jack Reacher series. And I've never known
Amazon not to vociferously protect its bestsellers. Not to mention, when I
import those same books in debug mode, they import in AZW3 format; whereas
GUI import brings all in as AZW, with DRM and without. So something IS
clearly wrong with the standard-mode de-DRM import process.

For future reference, are the instructions on Alf's FAQs page for opening
the calibre GUI in debug mode from the Terminal application missing
something or just not working in this case?

The faqs DO need to be edited. See net-buoy's comment in thread - he's
pointed it out to Alf before but no result.

—
Reply to this email directly or view it on GitHub
#52 (comment)
.

from dedrm_tools.

Yeah, a Jack Reacher book definitely had DRM since it's from one of the big publishers.

A reinstall of calibre might fix your problem. (Your library and plugins will remain.) But if you've already got your books DeDRMed, I understand not wanting to bother.

Net-buoy mentioned telling the calibre folks that their info was incorrect. But the Apprentice Alf blog and Apprentice Harper's GitHub here aren't connected to calibre. Alf's FAQs say to open the Terminal application and type calibre-debug -g. What should it say?

from dedrm_tools.

from dedrm_tools.

It should say just to use the restart in debug mode command on the preferences menu.

from dedrm_tools.

from dedrm_tools.

My feeling is that this shows a problem with your system that needs to be sorted out before doing any further troubleshooting.

from dedrm_tools.

Ah, in Windows, the calibre directory is in the system path so it doesn't matter what folder you have the command prompt open to.

Agree with Paul that the Alf blog should give instructions for the simpler GUI method (since it should work if there isn't a larger calibre issue). Those instructions were just written before that was available.

from dedrm_tools.

from dedrm_tools.

Your experience of using the GUI restart in debug mode does not match mine. We are both running the same system software and calibre version. Therefore there's a difference that we don't know about, but that difference is causing you problems. Since the problem persists when DeDRM isn't installed, I suggest, again, that you work with the calibre support at MobileRead to find the cause of the problem and fix it.

I suspect that that will fix your problem using DeDRM.

from dedrm_tools.

Remove DeDRM from your calibre installation. Does it now restart in debug mode from the menu? If not, get help at the calibre forum, as that means there's something wrong with your installation of calibre.that has nothing to do with DeDRM. Open a new issue once that's sorted.

from dedrm_tools.

Will do.

Kelly Bell
Founder and CTO
Gotham City Drupal
917.446.1555

On Jan 13, 2016, at 22:41, Apprentice Harper [email protected]
wrote:

Remove DeDRM from your calibre installation. Does it now restart in debug
mode from the menu? If not, get help at the calibre forum, as that means
there's something wrong with your installation of calibre.that has nothing
to do with DeDRM. Open a new issue once that's sorted.

—
Reply to this email directly or view it on GitHub
#52 (comment)
.

from dedrm_tools.

FWIW, I just had exactly the same experience as the reporter here: I bought four books from Amazon.com , tried to import all four to Calibre in 'regular' mode, and only one got decrypted (showed up as AZW3 and could be converted to epub), the other three did not. I tried removing those three and importing all four again (since I can't tell which of the four is the one that was already imported successfully), still didn't work for the other three. Then I restarted Calibre in debug mode and tried importing again, this time the three recalcitrant books got decrypted fine. Strange...

from dedrm_tools.

This was a calibre issue, not specific to the DeDRM plugin. Best place to look for the solution is over at the MobileRead calibre forum.

from dedrm_tools.

It is usually better to open a new issue rather than comment on an old issue (2015!) where the symptom might be the same, but the cause is almost certainly not.

I would be interested in working with you, AdamWIll, to work out why you had this experience. If you're willing to spend a little time on it, please open a new issue.

from dedrm_tools.

If I get a bit of time I will, but I'm not that invested, to be honest - I got it working and that's all I needed. I just figured it was worth dropping a note since I came across this report and it was pretty much an identical experience.

from dedrm_tools.

I'm happy to work with anyone on possible differences in operation between normal and debug calibre modes, but in a new issue, please.

from dedrm_tools.