Comments (18)
archive-pdf-tools
relies on archive-hocr-tools
to parse hOCR files, so what I will do is:
- release new archive-hocr-tools
- release new archive-pdf-tools that depends on the newer archive-hocr-tools
For your other question - about keeping the text layer intact from an existing PDF is another matter, there are a few things I want to do there ultimately:
- I want to support just compressing images in a PDF and not touch anything else (preserve text layers) - this is not currently what
--from-pdf
does, it just reads one image per page and recompresses it (and it assumes you have hOCR for it). - Be able to create hOCR from text layers of existing PDFs and use that to either (re)generate the text layer, but also as input for the MRC algorithm.
For this request, #28 a better issue to comment/discuss I think.
from archive-pdf-tools.
Thanks for the report, I'll take a look as to why that hOCR is not accepted.
Somewhat related, I've been working on my own PDF -> hOCR based on PyMuPDF text extraction (so it can stay pure Python + mupdf): https://github.com/internetarchive/archive-hocr-tools/blob/master/bin/pdf-to-hocr - not production ready though.
from archive-pdf-tools.
It looks like it doesn't want to parse the hOCR file you shared because it does not contain the required namespace. I will see what I can do to work around this, since if I don't prefix the namespace, it will not parse documents with the namespace.
The other problem seems to be that this uses ocrx_block
as opposed to ocr_par
. I suppose I'll want to support both in the xpath
queries, then.
from archive-pdf-tools.
I think this is fixed in internetarchive/archive-hocr-tools@6cdb14d - I will do a bit more testing before I make a release, though.
from archive-pdf-tools.
Thanks for the report!
from archive-pdf-tools.
I think this is fixed in internetarchive/archive-hocr-tools@6cdb14d
@MerlijnWajer You point to a commit in the archive-hocr-tools repo for the solution of this issue in the archive-pdf-tools repo. Will archive-pdf-tools support hocr-files coming from different sources, or will you just make it read the text from an existing searchable pdf?
from archive-pdf-tools.
@MerlijnWajer I also made a hocr file via another route: djvu2hocr ~/Afbeeldingen/2022-01-08.djvu >220108.hocr
220108.zip
This one gives
recode_pdf --from-imagestack ~/Afbeeldingen/211115-000ga.tif --hocr-file ~/jwilk/ocrodjvu/220108.hocr --dpi 300 --bg-downsample 3 --mask-compression jbig2 -o 2022-01-08a.pdf
Traceback (most recent call last):
File "/usr/local/bin/recode_pdf", line 4, in
import('pkg_resources').run_script('archive-pdf-tools==1.4.11', 'recode_pdf')
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 667, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 1463, in run_script
exec(code, namespace, namespace)
File "/usr/local/lib/python3.8/dist-packages/archive_pdf_tools-1.4.11-py3.8-linux-x86_64.egg/EGG-INFO/scripts/recode_pdf", line 288, in
res = recode(args.from_pdf, args.from_imagestack, args.dpi, args.hocr_file,
File "/usr/local/lib/python3.8/dist-packages/archive_pdf_tools-1.4.11-py3.8-linux-x86_64.egg/internetarchivepdf/recode.py", line 628, in recode
create_tess_textonly_pdf(hocr_file, tess_tmp_path, in_pdf=in_pdf,
File "/usr/local/lib/python3.8/dist-packages/archive_pdf_tools-1.4.11-py3.8-linux-x86_64.egg/internetarchivepdf/recode.py", line 210, in create_tess_textonly_pdf
word_data = hocr_page_to_word_data(hocr_page, font_scaler)
File "/home/robert/.local/lib/python3.8/site-packages/hocr/parse.py", line 185, in hocr_page_to_word_data
conf = int(X_WCONF_REGEX.search(word.attrib['title']).group(1).split()[0])
AttributeError: 'NoneType' object has no attribute 'group'
from archive-pdf-tools.
@MerlijnWajer I also made a hocr file via another route: djvu2hocr ~/Afbeeldingen/2022-01-08.djvu >220108.hocr 220108.zip This one gives recode_pdf --from-imagestack ~/Afbeeldingen/211115-000ga.tif --hocr-file ~/jwilk/ocrodjvu/220108.hocr --dpi 300 --bg-downsample 3 --mask-compression jbig2 -o 2022-01-08a.pdf Traceback (most recent call last): File "/usr/local/bin/recode_pdf", line 4, in import('pkg_resources').run_script('archive-pdf-tools==1.4.11', 'recode_pdf') File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 667, in run_script self.require(requires)[0].run_script(script_name, ns) File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 1463, in run_script exec(code, namespace, namespace) File "/usr/local/lib/python3.8/dist-packages/archive_pdf_tools-1.4.11-py3.8-linux-x86_64.egg/EGG-INFO/scripts/recode_pdf", line 288, in res = recode(args.from_pdf, args.from_imagestack, args.dpi, args.hocr_file, File "/usr/local/lib/python3.8/dist-packages/archive_pdf_tools-1.4.11-py3.8-linux-x86_64.egg/internetarchivepdf/recode.py", line 628, in recode create_tess_textonly_pdf(hocr_file, tess_tmp_path, in_pdf=in_pdf, File "/usr/local/lib/python3.8/dist-packages/archive_pdf_tools-1.4.11-py3.8-linux-x86_64.egg/internetarchivepdf/recode.py", line 210, in create_tess_textonly_pdf word_data = hocr_page_to_word_data(hocr_page, font_scaler) File "/home/robert/.local/lib/python3.8/site-packages/hocr/parse.py", line 185, in hocr_page_to_word_data conf = int(X_WCONF_REGEX.search(word.attrib['title']).group(1).split()[0]) AttributeError: 'NoneType' object has no attribute 'group'
Ok, I'll fix that bug as well, thanks, reopening.
from archive-pdf-tools.
x_wconf gaat over de confidence. Die krijg je niet terug uit een searchable PDF.
from archive-pdf-tools.
When I comment out the confidence-line "archive-hocr-tools/hocr/parse.py" line 186
conf = 100 # int(X_WCONF_REGEX.search(word.attrib['title']).group(1).split()[0])
the conversion takes place.
The hocr-file coming from pdftotree has another scale than the hocr-file coming from djvu2hocr.
The route via djvu2hocr gives a MRC-pdf with the characters on the right positions.
The route via pdftotree (with dimensions that are only 400/72 = 5,556 the size of the dimensions of the other file) gives text behind only in the left uppercorner of the PDF. So the mapping of the HOCR on the images should take account of this 72dpi positional conversion in a PDF.
Both don't have the scan_res marker. pdftotree doesn't order the words right to readable lines, so the djvu-route seems more stable.
from archive-pdf-tools.
Your pdf-to-hocr does give a result that gives a readable text when copied from the PDF, but with too many line endings. During the selection the selectedtexts are a bit slanted.
from archive-pdf-tools.
The resulting pdf from recode_pdf with the options on the central readme is more than 3 times as big as the PDF resulting from DjVuSolo3.1/DjVuToy.
from archive-pdf-tools.
(Ik geef maar ff antwoord in Engels voor de andere mensen :-) )
@rmast - sorry for the delay, I did read your messages earlier and yes, the problem is indeed that it is expecting x_wconf to be there, even though it is optional, I will fix that.
Regarding the text selection, it is the same code that Tesseract uses (more or less), but it is possible there are too many line endings added in the conversation. Optimal would be to not have to re-create the text layer, as discussed earlier.
Regarding the compressed size, if you can share the files I can look to see if something can be improved.
from archive-pdf-tools.
from archive-pdf-tools.
I've released archive-hocr-tools 1.1.15 that should fix the word confidence problems as well. Please let me know if it works.
from archive-pdf-tools.
I've pulled the newest version and built and installed it. It comes to an end as it did with my dirty fix. However I get this warning for every page:
Deprecation: 'getImageList' removed from class 'Page' after v1.19.0 - use 'get_images'.
Deprecation: 'extractImage' removed from class 'Document' after v1.19.0 - use 'extract_image'.
from archive-pdf-tools.
Ah, that's likely in the --from-pdf
code path? I still need to give that some more attention. I'll fix that here and make sure it's fixed in the next release.
from archive-pdf-tools.
See 07ff850
from archive-pdf-tools.
Related Issues (20)
- pillow is not working properly HOT 27
- Need some inspiration? HOT 7
- Some scans become inverted HOT 7
- Detect if RGB images in pages are greyscale or even 1bit
- Define scope of tooling and work to improve for that scope
- Create better presets for users with quality-comparable options for openjpeg/grok/pillow and kakadu HOT 1
- Missing test suite? HOT 1
- pdfcomp: new tool, discussion, compression questions HOT 19
- Bug in foreground/background separator choosing massive block instead of character outline. HOT 14
- The choice for inverting, what's the use for perc_larger?
- pdfcomp: problems with inverted text that is often better in hocr. HOT 10
- Wrong resolution of mask image when foreground image is downsampled HOT 1
- First recode_pdf test: 'numpy' has no attribute 'int'. HOT 5
- IndexError: list index out of range (single TIFF file) HOT 5
- HOCR rendering compares unfavorably with tesseract PDF text layer HOT 11
- Installing on MacOS? HOT 29
- Q: accessible tagging/hints? HOT 4
- A certain PDF from Archive.org does not display all of its contents on Mac OS HOT 26
- A user-friendly example for a scanned multipage PDF needed HOT 3
- Recode does not merge hocr into pdf HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from archive-pdf-tools.