Hot backup and fault tolerance limitations about libbitcoin-database HOT 5 CLOSED

I don't know about tolerance of hard shutdown during write. In fact, one of the initial design goals was sacrificing some level of tolerance to gain performance - but - only if that tolerance resulted in an unusable database (explicit, not implicit). There is some level of tolerance (write transactions, write block header, write references to map) that if earlier writes fail, it may be recoverable (but not really a goal). No, the database should always shut down cleanly, if not, you should have to re-download the blockchain.

The hot backup is interesting. Because if you can get a snapshot of the database memory mapped files, then you should be able to copy them because technically data is only added. Unlike a LevelDB store, no previous existing data is modified in theory (I'd need to check this assumption).

However I don't know whether this desirable property is worth spending time over, since the blockchain is widely available and duplicatable. If I wanted a backup, I'd run another instance.

from libbitcoin-database.

I took these design goals from some early documentation. This issue is just to caution people who may consider these features of the system.

The hot backup problem is the same problem as hard shutdown. The snapshot may occur in the middle of a write, or after an unflushed write, resulting in inconsistent data.

I have looked at mitigation and I agree that this is not worth trying to "fix". This is a reasonable trade-off for other significant benefits. I do have plans to provide incomplete write detection across restarts. This will mitigate the uncertainty associated with managing the server.

We have had a material number of reported issues from production server hosts (including Airbitz and OpenBazaar) on this issue. Preventing hard shutdown prevents the failures, but it is tedious from a management standpoint. The errors can be delayed for long periods of time, making it currently necessary to rebuild the blockchain database after any uncontrolled shutdown. Practically writes are rare, so this can be significantly improved with detection.

from libbitcoin-database.

Agreed.

from libbitcoin-database.

Master (v3) now includes robust hard shutdown detection, including forced flush at configurable points. This allows a hard shutdown to succeed as long as the write sentinel file is not detected. Similarly a hot backup will succeed as long as a write sentinel does not intervene, although one would have to monitor for the creation of the sentinel during the copy.

from libbitcoin-database.

Version4 splits headers and bodies, and eliminates updates in bodies, so that files are truly append-only. A transactor object is used to isolate writes during hot flush (backup) of headers. Corruption is detected (as in v3) using a flush lock file. Backup and recovery are atomic based on a file system directory rename in each direction. A headers backup can be restored against a set of body files, logically truncating the bodies at the size indicated by the headers. As such a process termination is recoverable to the most recent backup. Backups constitute about 2% of the store size and can be performed while running (which pauses writes).

from libbitcoin-database.