Corrupted feeds file about canto-next HOT 13 CLOSED

Well, corruption could happen if your daemon was killed unexpectedly, like if your machine lost power. I've also had reports of this happening under Upstart when it was failing to actually send SIGTERM on shutdown/restart so the daemon didn't know to stop running and probably died mid-write.

Not sure if there's anything I can do to stop this, but I guess I could make the daemon more resistant by keeping a fallback copy of the feeds file or something if this becomes a chronic issue but I've never seen the daemon corrupt the database during normal operation or a controlled shutdown.

If you eventually see a pattern that's reproducible, like it occurs after reboot, or something that would be interesting. Would also be interested to know if you're using any of the sync plugins.

from canto-next.

Well it is using systemd as init system and I'm using the user service file. But the machine didn't had lost power unexpectedly, else I wouldn't have opened this issue. Just power on and poweroff. Weird is also, that it happens on my desktop, but not on the laptop,with the same setup.
Will try to get more infos (And I hope a weird running clock cannot be the issue :D )

The only plugin I'm using is cleantitle, so no one of the sync plugins

(Edit: And I hope the IP-Forwarding (Laptop --> Desktop) cannot be a source of problems too.)

from canto-next.

Lovely. If something shall happen it won't.
But another question. What happens if I issued shutdown while canto is trying to retrieve feeds?

from canto-next.

If you shutdown, the daemon should receive a signal (SIGTERM) from your init system which invokes a signal handler that calls cleanup() and then sys.exit(). The very first thing cleanup() does is stop_feeds() which sets marks every Feed object as "stopped".

Any threads currently fetching data will finish grabbing data from the internet but see that the feed is stopped and discard the new data before dying without touching the disk (top of Feed.index).

Any threads working on indexing that data (i.e. already got data and started to parse it while the feed was live) will finish parsing but see that they're stopped without touching the disk (bottom of Feed.index)

Back in the main thread, cleanup() waits for all of the feed threads to finish by locking all of them, this ensures that no threads are still in that small window where it was currently writing to the disk when the feed was stopped because the fetch threads hold these locks until they're 100% complete. After the main threads holds all of these locks, is syncs() and closes() the database and anything else trying to use it will error.

I'm pretty sure that this part of the daemon is solid and a controlled shutdown isn't the source of the corruption.

from canto-next.

Good to hear. But I somehow achieve this. It happened again, no unusual poweroff and totally clueless.

If internet connection goes down before the daemon is also covered? (in the shutdown process?)

from canto-next.

That should be handled in the same way losing internet is always handled, the fetch thread gets an exception and exits.

Some questions:

1. Since you're asking about shutdown, is it only on the first run on your desktop? i.e. have you determined it is shutdown that's causing it?
2. Are you running an experimental FS?
3. Is your disk ancient?
4. Before you remove the feeds file because it's corrupted, what's actually in it? Is it empty? If not, try to cat and zcat it and see if you get anything interesting.

Another idea:

1. Stop using the systemd init file so that canto-curses will start the daemon itself if it's not running.

2. Then use your shell rc to do something like

   alias canto-curses="rm -r ~/.config/canto.bak; cp -r ~/.config/canto{,.bak}; canto-curses"

   (assuming your config is in .config and not .canto-ng, change if necessary)

3. Then, next time this problem occurs you'll have the old daemon-log and be able to see if it has errors, or if it wasn't shutdown correctly, etc.

from canto-next.

Shutdown could also be a wrong assumption. (And I got a corrupted feed file on another system too)
Just guessing/pointing to shutdown because Before-Shutdown canto is working and after-boot, when the daemon should be running (even if there is no connection to the internet) and start canto-curses it hangs, and daemon was in fact failing because of the corrupted file.
It doesn't happen regulary. Just now an then (like 3 times now in dunno. 10-20 boot ups?). So I thought about some race condition that could eventually happen on shutdown.

Both systems are using ext4, but aren't the youngest one. 6-8 years old. Didn't look at the SMART values for quite a while, so maybe there is a problem arising with the disks.

For the last corrupted one I tried to unpack it, but not using zcat and the decompression errored out and left me with an empty file (whereas the compressed one had a size >0 IIRC). Dunno if the archiver got something wrong.

from canto-next.

Hi !

Same problem here. It was working good and then, the feed file get empty, resulting in an canto-curses hanging.

from canto-next.

Okay guys, I still can't reproduce this but please try on git. I've made three changes.

First, the daemon will march on even if it fails to load the feeds. Not great behavior, but probably better than requiring you to break out rm to get the same effect. This should keep canto-curses from hanging, but if you see a bunch of items come back, the daemon log will say something like

19:49:45 : SHELF -> Failed to JSON load, old shelf?
19:49:45 : SHELF -> Failed to migrate old shelf: db type could not be determined
19:49:45 : SHELF -> BAD DATA: []
19:49:45 : SHELF -> Carrying on with empty shelf

If you see this, I'm curious what it spits out for BAD DATA (here it's an empty file I made on purpose of course).

Second, I made the daemon dump the file to a tempfile in the same directory and then move it over on sync. This should mean that even if the daemon does get killed in the middle of a sync (for some reason) your old copy of the data should be there (marginally better than corruption). If the machine dies while the move is happening, I'd expect the FS journal to catch that.

Third, I made SIGINT and SIGTERM identical. I did find some bad behavior with getting TERM while threads are running, but even in this case the db was synchronized (some threads were just left hung). I'm not convinced this is the cause of the issue, but it could be somehow.

Anyway, grab git and try again.

from canto-next.

Ok, will do

from canto-next.

Before trying the git version, I had one more time this bug.
Then, I post the content of the daemon's log file :
File "/usr/lib/python3.5/dbm/__init__.py", line 88, in open raise error[0]("db type could not be determined") dbm.error: db type could not be determined
I'll try the git version, and report here :)

from canto-next.

Hi !

Everything fine since git's installation. Didn't have any corrupted file, or didn't notice.

from canto-next.

Okay, I'll close this. Feel free to reopen if this crops up again.

from canto-next.