Error during sync: entry about remarkablepocket HOT 13 CLOSED

I managed to reproduce this problem. Some articles are converted by the Remarkable as PDFs instead of epubs causing the program to fail. I am not sure what causes an article to be converted so I opened a support ticket to find out.

from remarkablepocket.

Thank you for all the efforts in addressing these issues.

I tried the new version and the initial sync and article update seems to work, sort of. It ran without the previous error messages, and it seems to properly identify and ignore "invalid articles":

Status: Downloaded newer image for ghcr.io/nov1n/remarkable-pocket:0.3.0
[2022-10-17 05:32:34] WARNING!!! Using the new 1.5 sync, this has not been fully tested yet!!! first sync can be slow
[2022-10-17 05:32:36] WARNING!!! Using the new 1.5 sync, this has not been fully tested yet!!! first sync can be slow
[2022-10-17 05:32:48] rmapi version
[2022-10-17 05:32:48] 0.0.22.1
[2022-10-17 05:32:48]
[2022-10-17 05:32:50] Starting sync...
[2022-10-17 05:33:47] Found 2 read article(s) on Remarkable.
[2022-10-17 05:33:47] (1/2) Marking '2 Companies Say Their Vaccines Are 95% Effective. What Does That Mean?' as read on Pocket...
[2022-10-17 05:33:50] (1/2) Deleting '2 Companies Say Their Vaccines Are 95% Effective. What Does That Mean?' from Remarkable...
[2022-10-17 05:33:52] (2/2) Marking 'How F.D.R.'s Death Changed the Vice-Presidency' as read on Pocket...
[2022-10-17 05:33:53] (2/2) Deleting 'How F.D.R.'s Death Changed the Vice-Presidency' from Remarkable...
[2022-10-17 05:34:04] Found 8 unread article(s) on Remarkable. Downloading 2 more from Pocket.
[2022-10-17 05:34:04] (1/2) Downloading: 'How to Clean Blood Stains'.
[2022-10-17 05:34:13] Download successful.
[2022-10-17 05:34:13] (2/2) Downloading: 'You Could Have Long COVID and Not Even Know It'.
[2022-10-17 05:34:19] Downloaded article is invalid. See https://github.com/nov1n/RemarkablePocket#limitations for possible causes.
[2022-10-17 05:34:19] (2/2) Downloading: 'Guy Fieri Is America's Most Misunderstood Chef'.
[2022-10-17 05:34:42] Failed to download article: Epub generation error..
[2022-10-17 05:34:42] (2/2) Downloading: 'When Elon Musk Tried to Destroy a Tesla Whistleblower'.
[2022-10-17 05:34:49] Downloaded article is invalid. See https://github.com/nov1n/RemarkablePocket#limitations for possible causes.
[2022-10-17 05:34:49] (2/2) Downloading: 'New Pixels—and new prices—are here'.
[2022-10-17 05:34:56] Downloaded article is invalid. See https://github.com/nov1n/RemarkablePocket#limitations for possible causes.
[2022-10-17 05:34:56] (2/2) Downloading: 'In Electric Car Market, It's Tesla and a Jumbled Field of Also-Rans'.
[2022-10-17 05:35:03] Download successful.
[2022-10-17 05:35:03] Uploading 2 article(s) to Remarkable.
[2022-10-17 05:35:03] (1/2) Uploading: 'How to Clean Blood Stains.epub'.
[2022-10-17 05:35:07] (2/2) Uploading: 'In Electric Car Market, It's Tesla and a Jumbled Field of Also-Rans.epub'.
[2022-10-17 05:35:13] Completed sync in 2m 23s.
[2022-10-17 05:35:13] Next sync in 8h.

I'll continue using the latest and report if I experience anything further.

from remarkablepocket.

Could you paste the output of the command with the -v flag appended for verbose output?

from remarkablepocket.

Here it is:
touch ~/.remarkable-pocket ~/.rmapi && mkdir -p ~/.rmapi-cache && docker run -it --env TZ=Europe/Amsterdam -p 65112:65112 -v ~/.remarkable-pocket:/root/.remarkable-pocket -v ~/.rmapi:/root/.rmapi -v ~/.rmapi-cache:/root/.cache/rmapi ghcr.io/nov1n/remarkable-pocket:0.2.1 -l 20 -v

[2022-09-12 14:19:12] Created temporary storage directory: /tmp/8531628116535244067.
[2022-09-12 14:19:13] Refreshing cache... This may take a while.
[2022-09-12 14:19:13] WARNING!!!
[2022-09-12 14:19:13] Using the new 1.5 sync, this has not been fully tested yet!!!
[2022-09-12 14:19:13] rmapi version
[2022-09-12 14:19:13] 0.0.20
[2022-09-12 14:19:13] Make sure you have a backup, in case there is a bug that could cause data loss!
[2022-09-12 14:19:13]
[2022-09-12 14:19:13] Created temporary working directory: /tmp/16092019688111950564.
[2022-09-12 14:19:13] Executing command: [/usr/local/bin/rmapi_amd64, -ni, mkdir, Pocket]
[2022-09-12 14:19:13] entry already exists
[2022-09-12 14:19:13] Starting sync...
[2022-09-12 14:19:13] Executing command: [/usr/local/bin/rmapi_amd64, -ni, ls, /Pocket/]
[2022-09-12 14:19:13] Executing command: [grep, ^[f]]
[2022-09-12 14:19:13] Executing command: [cut, -b5-]
[2022-09-12 14:19:14] Thor – L'opéra rock
[2022-09-12 14:19:14] Building (Easy-To-Adopt) Software while Doing Visualization Research
[2022-09-12 14:19:14] En pensant imiter Staline, Poutine a commis une grave erreur
[2022-09-12 14:19:14] Software is a process, not a series of products - the failure of traditional metrics
[2022-09-12 14:19:14] Top tips for making the most out of Binder
[2022-09-12 14:19:14] Légitimation et diversification du jeu de rôle
[2022-09-12 14:19:14] Paris Call – OSEC 2022
[2022-09-12 14:19:14] What is the cost of bioinformatics? A look at bioinformatics pricing and costs
[2022-09-12 14:19:14] Évolution du discours théorique
[2022-09-12 14:19:14] Why science needs more research software engineers
[2022-09-12 14:19:14] Le bon gros raciste
[2022-09-12 14:19:14] Getting metadata for document: Thor – L'opéra rock.
[2022-09-12 14:19:14] Executing command: [/usr/local/bin/rmapi_amd64, -ni, get, /Pocket/Thor – L'opéra rock]
[2022-09-12 14:19:14] downloading: [/Pocket/Thor – L'opéra rock]...
[2022-09-12 14:19:14] OK
[2022-09-12 14:19:14] Executing command: [mv, Thor – L'opéra rock.zip, /tmp/16092019688111950564]
[2022-09-12 14:19:15] Executing command: [/usr/local/bin/rmapi_amd64, -ni, stat, /Pocket/Thor – L'opéra rock]
[2022-09-12 14:19:15] Executing command: [sed, /{/,$!d]
[2022-09-12 14:19:15] {
[2022-09-12 14:19:15] "ID": "6fe474c6-0b70-4c1b-b4e9-d8b7063d7037",
[2022-09-12 14:19:15] "Version": 0,
[2022-09-12 14:19:15] "Message": "",
[2022-09-12 14:19:15] "Success": false,
[2022-09-12 14:19:15] "BlobURLGet": "",
[2022-09-12 14:19:15] "BlobURLGetExpires": "",
[2022-09-12 14:19:15] "ModifiedClient": "52689249-12-18T11:00:09Z",
[2022-09-12 14:19:15] "Type": "DocumentType",
[2022-09-12 14:19:15] "VissibleName": "Thor – L'opéra rock",
[2022-09-12 14:19:15] "CurrentPage": 0,
[2022-09-12 14:19:15] "Bookmarked": false,
[2022-09-12 14:19:15] "Parent": "470bd7fa-d78d-405e-972a-119e513263e0"
[2022-09-12 14:19:15] }
[2022-09-12 14:19:15] Thor – L'opéra rock: Current page: 1, page count: 30.
[2022-09-12 14:19:15] Getting metadata for document: Building (Easy-To-Adopt) Software while Doing Visualization Research.
[2022-09-12 14:19:15] Executing command: [/usr/local/bin/rmapi_amd64, -ni, get, /Pocket/Building (Easy-To-Adopt) Software while Doing Visualization Research]
[2022-09-12 14:19:16] downloading: [/Pocket/Building (Easy-To-Adopt) Software while Doing Visualization Research]...
[2022-09-12 14:19:16] OK
[2022-09-12 14:19:16] Executing command: [mv, Building (Easy-To-Adopt) Software while Doing Visualization Research.zip, /tmp/16092019688111950564]
[2022-09-12 14:19:16] Executing command: [/usr/local/bin/rmapi_amd64, -ni, stat, /Pocket/Building (Easy-To-Adopt) Software while Doing Visualization Research]
[2022-09-12 14:19:16] Executing command: [sed, /{/,$!d]
[2022-09-12 14:19:16] {
[2022-09-12 14:19:16] "ID": "bf91a6d4-b306-4a4e-b256-a6241101d76a",
[2022-09-12 14:19:16] "Version": 0,
[2022-09-12 14:19:16] "Message": "",
[2022-09-12 14:19:16] "Success": false,
[2022-09-12 14:19:16] "BlobURLGet": "",
[2022-09-12 14:19:16] "BlobURLGetExpires": "",
[2022-09-12 14:19:16] "ModifiedClient": "52683051-11-09T08:09:16Z",
[2022-09-12 14:19:16] "Type": "DocumentType",
[2022-09-12 14:19:16] "VissibleName": "Building (Easy-To-Adopt) Software while Doing Visualization Research",
[2022-09-12 14:19:16] "CurrentPage": 0,
[2022-09-12 14:19:16] "Bookmarked": false,
[2022-09-12 14:19:16] "Parent": "470bd7fa-d78d-405e-972a-119e513263e0"
[2022-09-12 14:19:16] }
[2022-09-12 14:19:16] Building (Easy-To-Adopt) Software while Doing Visualization Research: Current page: 1, page count: 9.
[2022-09-12 14:19:16] Getting metadata for document: En pensant imiter Staline, Poutine a commis une grave erreur.
[2022-09-12 14:19:16] Executing command: [/usr/local/bin/rmapi_amd64, -ni, get, /Pocket/En pensant imiter Staline, Poutine a commis une grave erreur]
[2022-09-12 14:19:17] downloading: [/Pocket/En pensant imiter Staline, Poutine a commis une grave erreur]...
[2022-09-12 14:19:17] OK
[2022-09-12 14:19:17] Executing command: [mv, En pensant imiter Staline, Poutine a commis une grave erreur.zip, /tmp/16092019688111950564]
[2022-09-12 14:19:17] Executing command: [/usr/local/bin/rmapi_amd64, -ni, stat, /Pocket/En pensant imiter Staline, Poutine a commis une grave erreur]
[2022-09-12 14:19:17] Executing command: [sed, /{/,$!d]
[2022-09-12 14:19:17] {
[2022-09-12 14:19:17] "ID": "e6f1317e-d2ce-4e54-8d2b-ec4bf47b18f7",
[2022-09-12 14:19:17] "Version": 0,
[2022-09-12 14:19:17] "Message": "",
[2022-09-12 14:19:17] "Success": false,
[2022-09-12 14:19:17] "BlobURLGet": "",
[2022-09-12 14:19:17] "BlobURLGetExpires": "",
[2022-09-12 14:19:17] "ModifiedClient": "52683052-05-18T02:06:02Z",
[2022-09-12 14:19:17] "Type": "DocumentType",
[2022-09-12 14:19:17] "VissibleName": "En pensant imiter Staline, Poutine a commis une grave erreur",
[2022-09-12 14:19:17] "CurrentPage": 0,
[2022-09-12 14:19:17] "Bookmarked": false,
[2022-09-12 14:19:17] "Parent": "470bd7fa-d78d-405e-972a-119e513263e0"
[2022-09-12 14:19:17] }
[2022-09-12 14:19:17] En pensant imiter Staline, Poutine a commis une grave erreur: Current page: 1, page count: 7.
[2022-09-12 14:19:17] Getting metadata for document: Software is a process, not a series of products - the failure of traditional metrics.
[2022-09-12 14:19:17] Executing command: [/usr/local/bin/rmapi_amd64, -ni, get, /Pocket/Software is a process, not a series of products - the failure of traditional metrics]
[2022-09-12 14:19:18] downloading: [/Pocket/Software is a process, not a series of products - the failure of traditional metrics]...
[2022-09-12 14:19:18] OK
[2022-09-12 14:19:18] Executing command: [mv, Software is a process, not a series of products - the failure of traditional metrics.zip, /tmp/16092019688111950564]
[2022-09-12 14:19:18] Executing command: [/usr/local/bin/rmapi_amd64, -ni, stat, /Pocket/Software is a process, not a series of products - the failure of traditional metrics]
[2022-09-12 14:19:18] Executing command: [sed, /{/,$!d]
[2022-09-12 14:19:18] {
[2022-09-12 14:19:18] "ID": "203321f3-d2d6-4ced-86b0-b9f643f4e462",
[2022-09-12 14:19:18] "Version": 0,
[2022-09-12 14:19:18] "Message": "",
[2022-09-12 14:19:18] "Success": false,
[2022-09-12 14:19:18] "BlobURLGet": "",
[2022-09-12 14:19:18] "BlobURLGetExpires": "",
[2022-09-12 14:19:18] "ModifiedClient": "52683052-03-11T11:32:11Z",
[2022-09-12 14:19:18] "Type": "DocumentType",
[2022-09-12 14:19:18] "VissibleName": "Software is a process, not a series of products - the failure of traditional metrics",
[2022-09-12 14:19:18] "CurrentPage": 0,
[2022-09-12 14:19:18] "Bookmarked": false,
[2022-09-12 14:19:18] "Parent": "470bd7fa-d78d-405e-972a-119e513263e0"
[2022-09-12 14:19:18] }
[2022-09-12 14:19:18] Software is a process, not a series of products - the failure of traditional metrics: Current page: 1, page count: 5.
[2022-09-12 14:19:18] Getting metadata for document: Top tips for making the most out of Binder.
[2022-09-12 14:19:18] Executing command: [/usr/local/bin/rmapi_amd64, -ni, get, /Pocket/Top tips for making the most out of Binder]
[2022-09-12 14:19:19] downloading: [/Pocket/Top tips for making the most out of Binder]...
[2022-09-12 14:19:19] OK
[2022-09-12 14:19:19] Executing command: [mv, Top tips for making the most out of Binder.zip, /tmp/16092019688111950564]
[2022-09-12 14:19:19] Error occurred during sync: entry
[2022-09-12 14:19:19] Stack trace:
java.lang.NullPointerException: entry
at java.base/java.util.Objects.requireNonNull(Unknown Source)
at java.base/java.util.zip.ZipFile.getInputStream(Unknown Source)
at nl.carosi.remarkablepocket.MetadataProvider.getMetadata(MetadataProvider.java:71)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(Unknown Source)
at java.base/java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(Unknown Source)
at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)
at java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source)
at java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(Unknown Source)
at java.base/java.util.stream.ReferencePipeline.toArray(Unknown Source)
at java.base/java.util.stream.ReferencePipeline.toArray(Unknown Source)
at java.base/java.util.stream.ReferencePipeline.toList(Unknown Source)
at nl.carosi.remarkablepocket.RemarkableService.listReadDocuments(RemarkableService.java:49)
at nl.carosi.remarkablepocket.SyncService.archiveReadArticles(SyncService.java:108)
at nl.carosi.remarkablepocket.SyncService.syncImpl(SyncService.java:83)
at nl.carosi.remarkablepocket.SyncService.sync(SyncService.java:65)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
[2022-09-12 14:19:19] Next sync in 1h.

from remarkablepocket.

Some changes... The article that causes the bug was no more in the Pocket list. When I re-put it, it works... bug of cache somehow ?

from remarkablepocket.

It seems like the Top tips for making the most out of Binder file on your remarkable was corrupted. Did you place this file in the Pocket folder manually?

The /Pocket/ folder on your remarkable is fully managed by this program and it does not expect any files to be added manually.

from remarkablepocket.

When I removed it and reput it in Pocket correctly, it finally worked. So yes it was a corrupted file.

from remarkablepocket.

Unfortunately, still running into this problem with my sync. I haven't found a consistency in which articles it seems to have as PDFs vs. epubs. Hopefully you can get a response to your support ticket soon!

[2022-09-26 20:56:27] Error occurred during sync: entry
[2022-09-26 20:56:28] Stack trace:
java.lang.NullPointerException: entry
at java.base/java.util.Objects.requireNonNull(Unknown Source)
at java.base/java.util.zip.ZipFile.getInputStream(Unknown Source)
at nl.carosi.remarkablepocket.MetadataProvider.getMetadata(MetadataProvider.java:71)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(Unknown Source)
at java.base/java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(Unknown Source)
at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)
at java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source)
at java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(Unknown Source)
at java.base/java.util.stream.ReferencePipeline.toArray(Unknown Source)
at java.base/java.util.stream.ReferencePipeline.toArray(Unknown Source)
at java.base/java.util.stream.ReferencePipeline.toList(Unknown Source)
at nl.carosi.remarkablepocket.RemarkableService.listReadDocuments(RemarkableService.java:49)
at nl.carosi.remarkablepocket.SyncService.archiveReadArticles(SyncService.java:108)
at nl.carosi.remarkablepocket.SyncService.syncImpl(SyncService.java:83)
at nl.carosi.remarkablepocket.SyncService.sync(SyncService.java:65)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)

from remarkablepocket.

I have created a fix in #15. It also contains a short description of the problem and the solution.

You can help me test it by running:

touch ~/.remarkable-pocket ~/.rmapi && mkdir -p ~/.rmapi-cache && docker run -it --env TZ=Europe/Amsterdam -p 65112:65112 -v ~/.remarkable-pocket:/root/.remarkable-pocket -v ~/.rmapi:/root/.rmapi -v ~/.rmapi-cache:/root/.cache/rmapi ghcr.io/nov1n/remarkable-pocket:0.2.3

Please let me know if you run into any problems.

from remarkablepocket.

Remarkable Cloud changed the API again, so rmapi which is used as the client, is currently investigating a fix. Until that is resolved I cannot guarantee it will work correctly. See juruen/rmapi#259.

from remarkablepocket.

I upgraded rmapi to v0.0.22.1 and fixed a few other issues (see #15) and pushed a new image: ghcr.io/nov1n/remarkable-pocket:0.3.0. Could you help test it @francoissabot @Raiders95 @andypillip ?

from remarkablepocket.

Hei hei.

I got around to testing version 0.3.0, and it seems to work well, with podman instead of docker!

First, I tried with the settings I had, but it gave me an error Caused by: java.lang.RuntimeException: Could not authenticate to Remarkable API.

I deleted the two files and the directory, to re-start the command.

It correctly identified 5 articles as read on remarkable, archived them successfully on Pocket, and deleted the files from the remarkable. It also tried to replace the 5 deleted ones with new articles, one failed due to an epub generation error.

So it all seems fine!

Thanks so much! Let me get you another coffee!

from remarkablepocket.

That's great to hear, I'm happy it works for you! I will wait one more week until I release it in case some unexpected problems arise.

And thanks for the coffees, you're too kind!

from remarkablepocket.