Support kindle kfx format about dedrm_tools HOT 65 CLOSED

Hi,

I had a go at the mechanical Pascal to Python conversion myself. I've tested it with a KFX ebook and can successfully decrypt it (see main.py decrypt()). The various other print / debugging methods seem to work ok too.

Conversion from the output to EPUB may be more difficult, but it looks like Calibre already has some support there.

Thanks lulzkabulz for the original work.

from dedrm_tools.

"lulzkabulz" has reverse-engineered the KFX encryption from Kindle for PC and produced sample decryption code in Pascal. Their code removes the decryption only, leaving a document in the Ion format (see j-howell's comment above.)

I have added a copy of lulzkabulz's code to the repository.

The conversion of the decryption code to python is essentially mechanical, and conversion from Ion to other formats will probably be relatively straight-forward.

I'll be having a go at the Pascal->Python conversion myself, but I don't have much free time to spend on it. Contributions to this, or the Ion -> ePub conversion task are very welcome.

from dedrm_tools.

For those interested in the workings of KFX, Amazon has documentation for Ion, the data format that KFX is based on, online at https://amznlabs.github.io/ion-docs/

Reference implementations of Ion in Java, C and Python can be found at https://github.com/amznlabs

from dedrm_tools.

Someone asked on Apprentice Alf’s Blog about removing DRM from KFX books that were obtained via Kindle Unlimited and the response was that the DRM removal Tools are only intended for use on purchased books.

KFX voucher files have a license_type field that indicates whether that particular book is a rental, loan or purchase. I suggest that this field be checked if/when KFX DRM removal becomes part of the Tools.

from dedrm_tools.

Don't waste your money on Epubor. It's simply a repackaging of the tools freely provided here.

from dedrm_tools.

That's a good idea. It might be worth doing another version of the tools just to cut down on questions at the blog. And there is an Obok update that should be released in a full version. Hmm...

from dedrm_tools.

Kindle for Mac 1.17:
https://s3.amazonaws.com/kindleformac/44182/KindleForMac-44182.dmg

(Mentioned here in the FAQs.)

from dedrm_tools.

Decrypting new kfx format is possible, of course.

But firstly you need to sync your books via Kindle for PC/Mac, this sync will make these books downloaded as AZW format, then remove drm from AZW please.

As what I known, there is no good solution to decrypte azw6 format books. Books what we get will be KFX (new).

https://garydadic.medium.com/convert-kindle-kfx-to-epub-pdf-free-on-mac-10-15-catalina-included-7ea1d7d2b609

from dedrm_tools.

angelana,

For AZW6, if you have an e-ink kindle, then you can download an AZW3 for it from the website and then decrypt that.

from dedrm_tools.

I have looked at the Pascal program that was posted and it has multiple functions that do different things. This may be causing confusion.

One function implements the extraction of encryption keys from Kindle for PC. It duplicates existing functionality of the DRM tools.

Another function takes an encrypted KFX file (DRMION) and the extracted keys and removes the encryption, resulting in an unencrypted KFX container. That leaves a working KFX book without DRM. That function would be nice to have in the DRM removal toolkit.

A separate function decodes the KFX container format and the data structures it contains in Ion binary format and outputs a human readable version of the data. That is useful for delving into the workings of KFX and could provide a starting point for conversion of KFX to EPUB.

from dedrm_tools.

Does anyone know more about how these tools work?

The KFX support for these DRM Tools implemented by @tomthumb1997 requires a kfx-zip archive as its input, not a DRMION file.

A book in KFX format is composed of multiple files, including a DRM voucher with encryption keys. In order to remove DRM all of the files that make up the book are required, not just the encrypted DRMION file. In the Handling KFX files in Calibre thread on MobileRead it was decided that the best way to deal with this is to combine all of the required files into an archive and give it a .kfx-zip extension.

When I first created the KFX Input plugin for calibre it required the user to zip the component files of each KFX book to produce a kfx-zip archive. I later made a change to have that plugin gather all of the required files and produce the kfx-zip automatically to make it easier to use.

So if you install the KFX Input plugin for calibre then it should just work as expected. When you select the main KFX file for import, the plugin will combine it with the other required files into a kfx-zip, which will then be passed to the DeDRM plugin. After that completes the KFX Input plugin will take the de-DRMed result and combine it into a monolithic KFX file that will be added to your calibre library.

The resulting unencrypted KFX file can then be sideloaded to various Kindle apps and devices or it can be converted to other formats within calibre, again using the KFX Input plugin.

from dedrm_tools.

Thanks, everyone for all your work on this. I've merged the pull request, tweaked some readmes and version numbers, and v6.6.0 has now been released. It needs testing and no doubt some bug fixes. And the FAQs could do with an update. But that's all I have time to do today. Anything else will have to wait until I have some more free time.

from dedrm_tools.

It would be great to support the new format. And the new azw3/azw6 combo files too. But reverse engineering a new DRM scheme and format is a hard task, especially one that's only on physical kindles.

If anyone comes up with a method, I'll add it to the tools as soon as possible.

from dedrm_tools.

According to http://www.mobileread.com/forums/showthread.php?t=263902 it appears that DRM scheme in kfx format is not device-dependent. It's possible to simply move the files between different Kindle devices (even without associated Amazon account) and it still works. So basically it's just a reversible encryption with all required for decryption data stored in those files.

from dedrm_tools.

Would LOVE for this to happen.

from dedrm_tools.

@ElleKayEm

Thank you so much, this solution helped me.

from dedrm_tools.

Amazon is not allowing me to download the AZW3 for a book I've purchased in AZW6 format. It keeps giving me the following error message:

"You have selected content that is not compatible with any of your registered devices. Please remove incompatible content from selection for eligible devices to appear."

My registered devices include Kindle for Mac, an old Kindle Paperwhite (1st generation, I think), and Kindle for Android.

from dedrm_tools.

from dedrm_tools.

That's not the case for this book. This book simply uses a nice font with the AZW6 "advanced typesetting" feature. There's zero reason why it couldn't be offered in AZW3. This is a case of Amazon wanting to control distribution and/or make an extra buck by forcing you to upgrade their products.

from dedrm_tools.

I'm happy to discover that all my comics (AZW6 when downloaded directly to my Paperwhite) will now download to Kindle for PC as AZW3. No need to "download & transfer via USB" for my Paperwhite anymore. On the product pages they say they're also available for Kindle for Mac. Is that not the case for your book, gitasong?

from dedrm_tools.

It may be, but my machine is too old; Kindle for Mac is no longer supported (and the old versions no longer work, believe it or not).

from dedrm_tools.

The 1st generation Paperwhite is also listed. (They call it 5th gen.) Even Kindle Keyboard. Are you sure that Paperwhite is still registered to that account?

from dedrm_tools.

Maybe it's 2nd generation; I'm not sure. And yes—it's still registered. I was on the phone with Amazon Kindle support for like 45 minutes today. The issue seems to be that the publisher hasn't issued this book in AZW3 format yet. They're giving me a refund. Anyway, it would really nice to figure out how to circumvent this issue.

from dedrm_tools.

That sounds pretty flimsy to me! Blame the publisher! But at least they're good about refunds. I usually get comics from Google. They're often available in both epub and pdf (one purchase) and can be DeDRMed.

from dedrm_tools.

Now that KFX is being downloaded by Kindle for PC 1.19, I hope that some developers will work on reverse engineering it. I'd love to hear from anyone doing this.

from dedrm_tools.

In the mean time I have a suggestion. The latest version of K4PC uses the same extension, .azw, for all book formats. Encrypted KFX containers begin with an 8 byte signature that includes the string "DRMION". An explicit error message for this case would avoid some confusion in diagnosing DRM removal problems.

Edit: The exact string is: b'\xeaDRMION\xee'

from dedrm_tools.

Is there anyone that donations could be made to in order to increase the likelihood of this occuring?

from dedrm_tools.

I don't accept donations for these tools. I hope that someone is working on KFX decryption/decoding, but I don't know of anyone at the moment.

from dedrm_tools.

I notice that these KFx formats include a separate file with a .md extension. Is there a possibility that they are just encrypted markdown files? If so, is there a possibility of just decrypting the .md files? I understand this is a separate issue from the KFX one, but it's worth a shot to ask.

from dedrm_tools.

In this case "md" is short for metadata. It holds information about the book, such as author, title and cover image. This file is not encrypted, but it is in KFX binary container format, making it difficult to interpret.

from dedrm_tools.

Note that Epubor Ultmate does not remove DRM from KFX format files. This is unsurprising, since it's built on the scripts released here.

from dedrm_tools.

@apprenticenaomi, what does a decrypted e-book in Ion format look like?

from dedrm_tools.

@jcsalomon It's a CONT (container) file with a bunch of ENTYs (entities). The book contents are embedded in there as uncompressed UTF-8. I haven't looked at the structure too much.

You can "buy" a free book from Amazon, download it, and try it out yourself. Nothing in the file looks personally identifying, but I'm not about to upload a DRM-stripped book :-).

from dedrm_tools.

Many thanks for the python conversion! If anyone fancies having a go at convertingwhat I'm assuming is an ION format book (see the comment by j-howell), I suggest that conversion to epub (version 2) might make most sense and be easiest.

from dedrm_tools.

Everyone involved in this thread probably already has seen something like this, but for the benefit of folks reading this later: @j-howell has uploaded a snippet of a KFX file (from Kindle Previewer so no DRM involved) unpacked to Ion at https://mobileread.com/forums/showthread.php?t=288540#post3552191.

from dedrm_tools.

Terminology: A KFX book is split over multiple files each using KFX container format. A KFX container is conceptually similar to the PDB format used as the basis of MOBI and KF8. It holds a list of data records known as fragments or entities. The entities are data structures encoded in Amazon's Ion binary format. Amazon has an equivalent Ion text format, a superset of JSON, which can encode the same data in a human-readable fashion. Synonyms (more or less) for KFX (kindle format 10) are YJ (yellow jersey), KRF (?), and AZW8.

The process of converting a book from EPUB to KFX is handled by a very large Java program (KFXGenApp). It works much like a compiler, transforming the original EPUB into rendering instructions. In the process any information in the original EPUB that is not relevant to how the book is rendered into pages is discarded. The result bears little resemblance to the HTML/CSS within an EPUB. This is far different from KF8, which is just the original EPUB content packed into a MOBI container.

I believe that it will take a large effort to develop software that can convert KFX back into EPUB. (And even if successful the result will be inferior to that which can be obtained by just unpacking a KF8.) I propose that this effort is better left to an outside project, such as calibre, which is not hampered by the potential legal issues that come from being part of DRM removal software.

from dedrm_tools.

The converted Pascal program can parse and unpack the KFX container format, but it doesn't do much beyond that. Amazon's Python reference implementation might be the best way forward.

It does seem most reasonable for Calibre to do conversion, and leave DRM to this project.

from dedrm_tools.

aph3rson, do you have a link to the Kindle 1.17 Mac version download page? Thanks.

from dedrm_tools.

A very good suggestion. Thanks.

from dedrm_tools.

Two bits of KFX news:

1. I have released a calibre plugin that can convert from KFX to other formats. At present it is difficult to use due to the lack of KFX DRM removal support in a calibre plugin.

2. EPUBOR has announced that they now support DRM removal of KFX format. I assume that they incorporated the code that was posted here, but never integrated in the DRM TOOLS. It is probably only a matter of time before they add a KFX conversion function too. :(

from dedrm_tools.

Thank you again for all you work on this. My development time is limited at the moment, but I would welcome pull requests incorporating the new KFX decryption capability into the tools.

from dedrm_tools.

The download links for Kindle for PC/Mac 1.17 from Amazon listed under "A Recent Change to Kindle for PC/Kindle for Mac" in the FAQ are no longer operational.

from dedrm_tools.

@j-howell dmg backup: https://d.pr/f/94aJrl

from dedrm_tools.

I don't need the file myself, but others may find it useful.

I did a write-up that describes ways to work around KFX. It includes checksums to verify that you have found the correct older Kindle for PC or Mac file. (The one above looks good, for now.)

https://www.mobileread.com/forums/showthread.php?t=283371

from dedrm_tools.

Is this the place to watch for when KFX decryption capability becomes available?

from dedrm_tools.

This GitHub is the place to watch for when it gets integrated into the tools (and any other updates).

from dedrm_tools.

While waiting for this to happen, using the old K4PC-App to get an AZW-Version of the book should do the trick...

from dedrm_tools.

EPUBOR has announced that they now support DRM removal of KFX format.

That's right. I have tried it: conversion and decryption work, but the final output is a text file that loses all the formatting, chapters, etc. I have contacted them to understand if there is a way around it.

from dedrm_tools.

Apologies that I have had no time to work on this yet. Pull requests welcome.

from dedrm_tools.

I see a pull request for KFX support. I have been waiting for someone to integrate that capability into these tools. Hopefully it works. I will give it a try.

from dedrm_tools.

@chorichori

Start your own issue. You shouldn't have any trouble using DeDRM with an AZW3 file.

from dedrm_tools.

Re j-howell, the alluded to PR is #458 . Let us know how it goes!

from dedrm_tools.

@voltagex has combined the pull request into a plugin zip file that can be used for testing. (See the comments of #458.)

The KFX Input plugin for calibre also needs to be installed to allow creation of a KFX-ZIP archive when a single KFX file is selected for import into calibre and to allow conversion to more useful formats.

I gave it a try on an assortment of KFX books from both Kindle for PC 1.21 and a Kindle running 5.9.4 firmware. I had to select books using the "Add books from a single directory" option in calibre, since calibre would not allow import of books KFX format from the Device view.

All but a few were able to have their DRM removed successfully! I will try those that didn't work in debug mode and see if I can find what is going wrong with them. (I don't have any more time to spend on this at the moment.)

While testing I found a problem in the KFX Input plugin with the handling of e-book samples. I will work on a fix for that.

from dedrm_tools.

@j-howell For the few that were unsuccessful, perhaps their license_type != "Purchase" (DeDRM_plugin/kfxdedrm.py)? That could be expanded (or dropped). (I don't know what possible values that string can have.)

from dedrm_tools.

For the few that were unsuccessful, perhaps their license_type != "Purchase"

You were right, @cemeyer. It failed for books from Kindle Unlimited:

DeDRM v6.5.5: Failed to decrypt with error: This book is licensed as KindleUnlimited. These tools are intended for use on purchased books.

And for library books borrowed through OverDrive:

DeDRM v6.5.5: Failed to decrypt with error: This book is licensed as PublicLibraryLending. These tools are intended for use on purchased books.

I suppose @apprenticeharper will have to decide whether or not to keep that limitation when integrating the changes.

It also doesn't work for free Indic dictionaries in KFX format. Those use a different variant of KFX:

The .kfx-zip archive does not contain an encrypted DRMION file

It did work properly for all of the purchased books I tried.

from dedrm_tools.

Isn't this protection intended behavior? Why should it be changed for KFX files?

from dedrm_tools.

I've been testing the KFX branch written by @tomthumb1997. I see the following error with all the books sync'd down through the latest Kindle for Mac:

The .kfx DRMION file cannot be decrypted by itself. A .kfx-zip archive containing a DRM voucher is required.

The code which produces that message checks the first 8 bytes of file for the special chars "DRMION". I checked all my azw files and they all have this same first 8 bytes!

head -c 8 <azw_file>

Does anyone know more about how these tools work?

from dedrm_tools.

Isn't this protection intended behavior? Why should it be changed for KFX files?

The explicit license type used in KFX format means that the check for purchased books will be more reliably enforced than for other Kindle formats. Personally, I think that is a good thing. Others may not agree.

from dedrm_tools.

@mafrosis I think vouchers can be stored separately from kfx files by the Kindle software, but @
tomthumb1997 's plugin only handles kfx-zip with embedded voucher files. See https://gist.github.com/apprenticenaomi/b0103db2ac134cb51a3ead92f1d99f3a#file-main-py-L30 for example (it seems to use a weird callback scheme to locate appropriately named ".voucher" files in the same directory as the book files). Don't know when or how Kindle software decides to store zips or separate files.

from dedrm_tools.

@cemeyer Thanks for the info. Very useful

@j-howell I don't use Calibre, but again very useful info.

from dedrm_tools.

@j-howell Oh, KFX-zip is entirely something of your creation, not an Amazon thing? I definitely think we should support vanilla kfx files from Amazon. We can do both. :-)

The explicit license type used in KFX format means that the check for purchased books will be more reliably enforced than for other Kindle formats. Personally, I think that is a good thing. Others may not agree.

Seems harmless, but you know crapware like EPUBOR is going to remove the check.

from dedrm_tools.

Oh, KFX-zip is entirely something of your creation, not an Amazon thing?

It was decided that since books in KFX format are spilt into multiple files and calibre handles all books as single files that combining the KFX files that make up a book into a zip archive was the best way of making it work within that restriction of calibre.

The DRMION file contains only part of the content of the book. The KFX Input plugin will merge all of the component files in the KFX-ZIP into a single KFX file, but that can't be done until after DRM has been removed from the DRMION file.

from dedrm_tools.

I am using Devaun, and having the same problem.

Linux myserver 4.19.0-10-amd64 #1 SMP Debian 4.19.132-1 (2020-07-24) x86_64 GNU/Linux

I cannot seem to upgrade calibre beyond 3.9.1.

I have python 2.7 installed.

from dedrm_tools.

from dedrm_tools.

@wallyreport
I'd ask about calibre issues over in the calibre forum at MobileRead.

from dedrm_tools.