Comments (11)
That's an interesting approach. Yeah, I'm not opposed to supporting restic if it's just an HTTP proxy or STDIN/STDOUT API.
from litefs.
Is this still being pursued?
A lot of the backup functionality has been implemented in LiteFS Cloud which we'll likely open source in the near future. We may also do a simplified version that's built into LiteFS which doesn't try to do multi-level compactions. That would trade restore performance for simplicity.
In theory I could still use litestream in parallel to LiteFS by ensuring it only runs on the primary, right?
Yes, you can still use Litestream on a LiteFS primary. It works best if you have a static primary, however, you could also separately backup multiple candidate nodes to different Litestream replication paths.
from litefs.
Is this still being pursued?
In theory I could still use litestream in parallel to LiteFS by ensuring it only runs on the primary, right?
from litefs.
Any updates on the progress of this?
I currently use litestream for backing up one of my databases - could I just run litestream on /litefs/database.db (given a static lease) if I'd like to use litefs?
from litefs.
Any updates on the progress of this?
I don't have an ETA but streaming backups is the next thing I'm working on.
Could I just run litestream on /litefs/database.db (given a static lease) if I'd like to use litefs?
Yes, Litestream should work fine on the LiteFS primary as long as you're using a static lease.
from litefs.
👀
from litefs.
Rather than keeping it limited to S3 only, would it be possible to support other remotes too?
Preferably via something like rclone (https://rclone.org/) similar to how restic did it for backups (https://restic.net/blog/2018-04-01/rclone-backend/).
from litefs.
@darthShadow I'm not opposed to supporting other endpoints. In my experience with Litestream, though, the vast majority of people used S3 or S3-compatible storage.
I get a bit nervous pulling in a dependency like rclone which does "everything" since it makes it difficult to support all the providers well. Someone will inevitably open an issue for something like Alibaba Cloud (which I've never used before) and I won't have any ability to support it without signing up for an account.
from litefs.
It won't be a direct dependency like a library. The way restic does it is that it uses it as a separate binary on the system and all the communication is done over the stdio interface.
From the above article:
It turned out that it is also possible to run an HTTP2 connection over stdin/stdout of a newly started process. We’ve implemented this in restic and rclone. Internally, restic runs rclone serve restic --stdio, and it will serve HTTP requests via HTTP2 on stdin/stdout
support all the providers well
That's the beauty of doing it this way in that you only need to support the standard rclone interface and any provider-specific issues can be redirected to the rclone tracker to be resolved. I believe most (if not all) of the functionality is already tested automatically on every commit/release but you can also send any PRs for any missing tests so there's very little chance for any breakage unless the provider changes something.
In the worst case, rclone can be used to mount any provider as a local filesystem and then configure that as the local backup/restore point.
Sorry for the delay in the response, missed the comment notification.
from litefs.
Alright, thanks for the update!
from litefs.
Any updates on the progress of this?
I don't have an ETA but streaming backups is the next thing I'm working on.
Could I just run litestream on /litefs/database.db (given a static lease) if I'd like to use litefs?
Yes, Litestream should work fine on the LiteFS primary as long as you're using a static lease.
Is there anyone who has managed to set this up successfully?
When I tried, it seems like liteFS is supposed to capture changes in the target database, but no changes occurred in the target database, so litestream did not replicate any changes.
I'm not sure if this is the correct approach.
fuse:
dir: "/data"
allow-other: true
debug: true
data:
dir: "/var/lib/litefs"
http:
addr: ":20202"
lease:
type: "static"
advertise-url: "http://0.0.0.0:20202"
candidate: true
dbs:
- path: /data/data.db
replicas:
- name: backup
url: abs://datafolder@data/data
from litefs.
Related Issues (20)
- Manual editing of data and forcing sync HOT 1
- Windows Support HOT 2
- Working multiple clusters/hiding databases HOT 2
- Retry primary lookup in proxy if not found
- Support for per-database primary on different nodes (regions)? HOT 3
- Continuously flush proxy response HOT 1
- Websocket connection on a replica node HOT 3
- Primary should check backup server before marking ready HOT 1
- Sync and Client API security
- Proxy Server not starting up? HOT 9
- Replication Not Working HOT 8
- Writing to the WAL while using EXCLUSIVE locking mode should be prevented HOT 11
- Support WebSockets in LiteFS Proxy HOT 3
- SQLITE_CONSTRAINT_CHECK error and rollback stops syncing with other nodes HOT 2
- How to test synchorization lag? HOT 1
- Proxy timeout when auto-scaling machines HOT 2
- Consul Key Question, Request for Documentation HOT 1
- Creating new databases HOT 1
- Add the option to act as a real proxy instead of using fly-replay header
- Object Store Mediated Replication HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from litefs.