jonaswinkler / paperless-ng Goto Github PK

The lists get unmanageable otherwise with >100 entries.

Require the user to confirm closing a document / closing paperless when there are unsaved edits.

This is certainly nice to have. However, you usually don't do a whole lot of editing of the documents, so even if you should forget to save, you won't loose all that much time.

Hey Jonas, this absolutely looks amazing! I would have loved to see this half a year ago, especially the dashboard idea seems to be great!

I am kind of torn between Docspell and paperless-ng now, having just added all my files to Docspell, but really like your UI!

I was wondering what you think of the following things, which I think Docspell offers better:

1. Tags, Correspondent, etc shown in overview
   I would say this is a must for me to not only filter using them but also see the attached metadata in search results and so on. Having this is also makes sense to only offer edit and not show details. Right now I have to open edit mode to see metadata
   Okay, just figured out (see last point) that tags are shown really well and the correspondent is made a prefix of the title, which I like less (looks weird when the title also uses colon).
   Plus it would be nice if those properties were clickable
2. Show document in the browser
   right now there is just the ability to see the document in the edit mode. Clicking on the preview could for instance lead to the document itself (I think this is something, Paperless (original) offers as well)
3. Concerned person
   I really really like that I can add the information to whome a letter had been addressed: me, my wife, both of us, which of the children.
4. nested tags or tag categories
   This just helps a lot with organizing. Right now one has to workaround it by using some concerned-me, concerned-kid1, concerned-wife logic
5. Direction of item
   I guess most people mainly this type of app for letters in a broader sense - and sometimes I just find it really useful to also throw in the things I send out, e.g. some application for public funds or something like that
6. Syntax help
   Right now it only suggests the property, not the operator from what I saw. I loved the expert search from the original paperless!
7. Corresponding person (detail of corresponding organization)
   I have to admit that I use this not that often so far, but I like the idea :-)
8. User-specific filename
   Is this still possible like in original paperless?
9. MariaDB/MySQL support
   is this planned?
10. Multi-user support as already mentioned in #52
11. multiple files per item
    I like the idea to sometimes put things together, which belong together. An alternative could be to add relations between items!

• And please add a check if leaving edit mode with unsaved changes!

Don't have a limited selection of file types on the Document model. Don't check for file types when uploading new documents.

Rather, check the validity of a file type by checking for available parsers of that type.

When documents are exported with document_exporter, and two documents with the same generated filename exist in the database, the exporter will only export one of these files and references it in multiple documents in the manifest.

I was wondering if you decided to use redis as default broker because you had issues with the Django ORM broker?

I would have guessed that the ORM broker is good enough for the majority of paperless setups and it would remove the need to run redis.

Ability to have nested tags, for example

• bank accounts
  • bank 1
    • account 1
    • account 2
  • bank 2
    • account 3

This would allow easy filtering by multiple tags. Filter by bankaccount, and see documents of all accounts. Filter by account 2, and only documents from that account show up.

Any input on whether this is a good idea or what else to use it for is greatly appreciated!

As stated in the documentation, it provides no security at all, since:

• key is stored along with encrypted documents
• Paperless provides transparent access to encrypted documents
• Plain text information is stored in the database, including complete document contents, which contains all sensitive information this encryption was supposed to protect.

Removing this feature will decrease the code complexity in many places.

Since they are using the same paginated API.

Paperless-ng uses tesseract 4.0.0. It gets the job done, but a more recent version would be nice. Same goes for ghostscript, magick, etc.

The issue is as follows.

• Paperless-ng requires numpy.
• In order to install paperless on the Raspberry Pi within a reasonable time (I've heard people use this on the Pi), numpy needs to be pulled from https://www.piwheels.org/. For this to work, python-3.7 is required, since there is no python-3.8 wheel for numpy on ARM. Building the wheel on Pi requires several additional dependencies and many hours of compilation time.
• Therefore, a base image with python-3.7 has to be used, which will also imply that older versions of other binaries get installed.

Any pointers on how to fix this situation while still using python 3.7 are appreciated.

Essentially allowing consumption with any of these mobile scanner apps.

If this project gets a lot of attention, I'll have to update the documentation. Many things changed and are not valid anymore.

Inspired by what for example Grafana does (and due to the limited set of colors available in paperless), I patched it to auto-color tags and correspondents with a https://github.com/zenozeng/color-hash like implementation:

• jayme-github/paperless@db0390c
• jayme-github/paperless@9def166

I understand that people might like choosing the colors themselves, so feel free to close this. 😄 It's just that I don't care about the actual color of tags, I just want them to have different ones and I don't want to bothered with selecting them.

It just does not work with the new file handling logic that came into paperless earlier this year.

It will crash and leave your document archive in an intermediate state, from which no automatic recovery is possible.

I might be a little too nervous but I'm just too excited. I just tested direct scanning into the consumption directory, which leads to the following warning in the log:

`11/25/20, 10:02 PM WARNING Thumbnail generation with ImageMagick failed, falling back to ghostscript. Check your /etc/ImageMagick-x/policy.xml!

11/25/20, 10:02 PM INFO Consuming doc20201125222904.pdf`

The stdout log of the webserver container shows the following:
`22:02:09 [Q] INFO Enqueued 1
22:02:09 [Q] INFO Process-1:1 processing [doc20201125222904.pdf]
Consuming doc20201125222904.pdf
Parser: RasterisedDocumentParser based on mime type application/pdf
Generating thumbnail for doc20201125222904.pdf...
Execute: convert -density 300 -scale 500x5000> -alpha remove -strip -trim /usr/src/paperless/src/../consume/doc20201125222904.pdf[0] /tmp/paperless/paperless-3vtrk65p/convert.png
**** Error: Cannot find a 'startxref' anywhere in the file.
Output may be incorrect.
**** Error: An error occurred while reading an XREF table.
**** The file has been damaged. This may have been caused
**** by a problem while converting or transfering the file.
**** Ghostscript will attempt to recover the data.
**** However, the output may be incorrect.
**** Error: Trailer dictionary not found.
Output may be incorrect.

Requested FirstPage is greater than the number of pages in the file: 0
No pages will be processed (FirstPage > LastPage).
convert-im6.q16: no images defined /tmp/paperless/paperless-3vtrk65p/convert.png' @ error/convert.c/ConvertImageCommand/3258. Thumbnail generation with ImageMagick failed, falling back to ghostscript. Check your /etc/ImageMagick-x/policy.xml! **** Error: Cannot find a 'startxref' anywhere in the file. Output may be incorrect. **** Error: An error occurred while reading an XREF table. **** The file has been damaged. This may have been caused **** by a problem while converting or transfering the file. **** Ghostscript will attempt to recover the data. **** However, the output may be incorrect. **** Error: Trailer dictionary not found. Output may be incorrect. No pages will be processed (FirstPage > LastPage). Execute: convert -density 300 -scale 500x5000> -alpha remove -strip -trim /tmp/paperless/paperless-3vtrk65p/gs_out.png /tmp/paperless/paperless-3vtrk65p/convert.png convert-im6.q16: unable to open image /tmp/paperless/paperless-3vtrk65p/gs_out.png': No such file or directory @ error/blob.c/OpenBlob/2874.
convert-im6.q16: no images defined `/tmp/paperless/paperless-3vtrk65p/convert.png' @ error/convert.c/ConvertImageCommand/3258.
Deleting directory /tmp/paperless/paperless-3vtrk65p
22:02:09 [Q] ERROR Failed [doc20201125222904.pdf] - Convert failed at ['convert', '-density', '300', '-scale', '500x5000>', '-alpha', 'remove', '-strip', '-trim', '/tmp/paperless/paperless-3vtrk65p/gs_out.png', '/tmp/paperless/paperless-3vtrk65p/convert.png'] : Traceback (most recent call last):
File "/usr/src/paperless/src/paperless_tesseract/parsers.py", line 49, in get_thumbnail
logging_group=self.logging_group)
File "/usr/src/paperless/src/documents/parsers.py", line 107, in run_convert
raise ParseError("Convert failed at {}".format(args))
documents.parsers.ParseError: Convert failed at ['convert', '-density', '300', '-scale', '500x5000>', '-alpha', 'remove', '-strip', '-trim', '/usr/src/paperless/src/../consume/doc20201125222904.pdf[0]', '/tmp/paperless/paperless-3vtrk65p/convert.png']

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/src/paperless/src/documents/consumer.py", line 132, in try_consume_file
thumbnail = document_parser.get_optimised_thumbnail()
File "/usr/src/paperless/src/documents/parsers.py", line 168, in get_optimised_thumbnail
return self.optimise_thumbnail(self.get_thumbnail())
File "/usr/src/paperless/src/paperless_tesseract/parsers.py", line 73, in get_thumbnail
logging_group=self.logging_group)
File "/usr/src/paperless/src/documents/parsers.py", line 107, in run_convert
raise ParseError("Convert failed at {}".format(args))
documents.parsers.ParseError: Convert failed at ['convert', '-density', '300', '-scale', '500x5000>', '-alpha', 'remove', '-strip', '-trim', '/tmp/paperless/paperless-3vtrk65p/gs_out.png', '/tmp/paperless/paperless-3vtrk65p/convert.png']

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/django_q/cluster.py", line 436, in worker
res = f(*task["args"], **task["kwargs"])
File "/usr/src/paperless/src/documents/tasks.py", line 68, in consume_file
override_tag_ids=override_tag_ids)
File "/usr/src/paperless/src/documents/consumer.py", line 138, in try_consume_file
raise ConsumerError(e)
documents.consumer.ConsumerError: Convert failed at ['convert', '-density', '300', '-scale', '500x5000>', '-alpha', 'remove', '-strip', '-trim', '/tmp/paperless/paperless-3vtrk65p/gs_out.png', '/tmp/paperless/paperless-3vtrk65p/convert.png']

`

Is it possible to delay the consumption of new files while they're being written to avoid this?

Hi (again),

One cool feature would be to compare document contents and identify duplicates, maybe upon document consumption and/or in a separate workflow.

Once again, great kudos for the work, keep that up!

Some scanners like to write files as "something.pd~", which does not get picked up by the consumer, and then rename the file. The consumer ignores these files.

Also: The consumer should expect the files to have moved before consuming. Implement checks.

When opening a document, switching between any other page and the document detail page will revert any unsaved changes.

Close all seems really useful.
Maybe limit the documents to 10 and sort them by most recently edited.

Ability to click on tags and correspondents, which applies an instant filter to show only the selected tag/correspondent.

Several things to do:

• menu collapse doesn't work
• many layouts are too wide for mobile
• login does not scroll

Right now they are just gone.

such as PDF metadata (application, PDF version, author, creator, etc. Anything that's available.)

1393 documents, 11 tag(s), 1392 correspondent(s), 1392 document type(s).

investigate.

This is required to run Paperless on a sub path, i.e. localhost:8000/paperless/.

This is generally required if you wish to use a proxy server with many different services.

Right now, this does not work.

As for the front end, the src/documents/static/index.html contains a <base href="/">, which has to be adapted to href="/paperless" or similar.

This is easy since the index page is served as a django view and template context is available. See

The front end further has some static assets for logos, which are always pointing towards <host>/assets/image.png. This is unsolved right now.

Furthermore, login/logout paths have to respect this path, as well as the admin/ link on the front end.

I don't need this, but if anyone cares about this, feel free to look into it.

First of all: Thanks for the awesome work and all the effort you put into this!

I wanted to try the classifier with parts of my data and ran into a traceback on consumption

Traceback (most recent call last):
  File "/usr/src/paperless/src/documents/consumer.py", line 171, in try_consume_file
    document_consumption_finished.send(
  File "/usr/local/lib/python3.8/dist-packages/django/dispatch/dispatcher.py", line 177, in send
    return [
  File "/usr/local/lib/python3.8/dist-packages/django/dispatch/dispatcher.py", line 178, in <listcomp>
    (receiver, receiver(signal=self, sender=sender, **named))
  File "/usr/src/paperless/src/documents/signals/handlers.py", line 127, in set_tags
    matched_tags = matching.match_tags(document.content, classifier)
  File "/usr/src/paperless/src/documents/matching.py", line 36, in match_tags
    predicted_tag_ids = classifier.predict_tags(document_content)
  File "/usr/src/paperless/src/documents/classifier.py", line 224, in predict_tags
    tags_ids = self.tags_binarizer.inverse_transform(y)[0]
  File "/usr/local/lib/python3.8/dist-packages/sklearn/preprocessing/_label.py", line 1017, in inverse_transform
    if yt.shape[1] != len(self.classes_):
IndexError: tuple index out of range

I did also see a warning during "create_classifier" run, which I bluntly ignored ofc. 😄

270 documents, 1 tag(s), 1 correspondent(s), 0 document type(s).
Vectorizing data...
Training tags classifier...
/usr/local/lib/python3.8/dist-packages/sklearn/utils/validation.py:72: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().
  return f(**kwargs)
Training correspondent classifier...
There are no document types. Not training document type classifier.

I made a small patch to allow me to set tags based on sub directories of the consumer directory (jayme-github/paperless#2) and I would like to know if you would accept that feature to your fork.
Looking at the feature-ocrmypdf branch you switched back to using INotify again, so my patch would also bring back the ability to run a recursive consumer.

Let me know what you think and I'll rebase against whatever branch makes sense.

Ability to preview a document in the browser, full screen, without downloading it.

Each tesseract process launches multiple threads that do OCR on a single page in parallel. This marginally improves speed.

However: paperless processes multiple pages in parallel (up to cpu count). This results in a massive performance decrease.

Fix by settings OMP_THREAD_LIMIT=1.

• don't delete consumed emails.
• rather, mark consumed emails as consumed (there are IMAP labels for that.)
• allow multiple accounts.
• allow filters.
• allow documents in the inbox to have the same filename.

I.e., running / crashed.

This is easy since we're using supervisord, and there's an RCP api for getting the status.

Also see http://supervisord.org/configuration.html#program-x-section-settings, serverurl for figuring out if we're running with supervisord.

python-slim is based on debian buster, which contains the outdated 4.0.0 version of tesseract.

We cannot use alpine, because python-numpy takes ages to install on this image, since its being compiled from source due to incompatible libraries.

With one exception: Multiple tag selection should be allowed.

The document type selector is not shown

Hi there,

first of all - I'm really impressed by the work you did on paperless. I've noticed the issue you opened on the original paperless project and then quickly decided to give it a go. The resource optimizations and feature enhancements you made are quite impressive. Wow!

One thing I'm kind of missing though is - I've now imported a bunch of PDF documents. It all went fine but I went on creating correspondents, document types and tags on-the-fly. Now I've got the most common information in there and I'd like to re-run auto assignment on my existing documents. I didn't (yet) find it in the awesome documentation. Can this somehow be accomplished?

Thank you so much for the great work you're doing with this project!

When filtering for a specific correspondent, subsequent filters for tags or types should only display options which are still available.

Add the ocr'ed text as a text layer to the scanned documents so that text can be copied from them.

When editing documents, the "Next button" should always reflect the next document in the view that was previously visible (I.e. Search results, document list, saved view) and should reload data as necessary.

• There should also be a previous button

With response: "Enter a valid date/time."

Request URL:

/api/logs/?page=1&page_size=25&ordering=-created&created__lt=2020-11-11T22:13:57.749728+01:00&level__gte=20

Maybe I understand the idea wrong but I would be kind of stuck once I saved a view. When I open it again, I cannot change the filters, the sort order or anything else. This is bad from my point of view for two reasons:

1. You have to start from scratch in order to just adapt a small thing
2. you cannot use a dashboard for further drilling down into your documents

Idea would be to maybe

1. just use the normal controls for adding/changing filters and order
2. add some form of save / save as option

That

• checks if all originals are in place
• checks if the checksums match
• checks if there are unreferenced files
• checks if all thumbnails are there
• checks if the mime type of all documents matches

Basically a program that's executed in the background once in a while and tells you that everything is all right.

The code is not working for documents with special characters and the thumbnails look very bad.

That checks with GitHub if a new release is available.