Comments (19)
I pushed up a PR for it so it's available at pr-137
in Docker now. That'll keep litefs
running even if it hits some kind of error on startup so the fly instance will be accessible via ssh
.
from litefs.
@kentcdodds Thanks for writing this up. I realized I had an old version (pr-109
) still on the litefs-example
which it looks like you have in your Dockerfile as well. Sorry about that. Can you try changing this line here to:
FROM flyio/litefs:0.2 AS litefs
I'm surprised to see that error on a brand new volume as it happens when LiteFS is validating the existing database state. Can you retry with the new litefs
version and let me know if you still have the same issue?
from litefs.
Thanks! I've still go the same issue: https://github.com/kentcdodds/kentcdodds.com/actions/runs/3322212924/jobs/5491067040
from litefs.
I'm a bit stuck on deploying LiteFS until this is resolved. Any ideas?
from litefs.
@kentcdodds The error is strange because it is essentially saying that the database state on disk exists (checksum e3d3906d74cc0273
) but there's no associated replication data (checksum 0000000000000000
). However, you're seeing that error even when you deploy with a clean volume so there shouldn't be any database state.
2022-10-25T15:35:44Z [info]cannot open store: open databases: open database("sqlite.db"): verify database file: database checksum (e3d3906d74cc0273) does not match latest LTX checksum (0000000000000000)
Can you try removing the volumes on your staging set up and re-deploying and seeing if you still have the same error?
from litefs.
I think I've figured out what's going on. When I create the new volume, my old (pre litefs) app restarts and applies migrations to the new db in the volume which is what causes this issue.
What I'm trying now is to deploy a version of my app that does not do anything to the database so then I can have that one running when I recreate the volume, and then deploy the litefs version. Will let you know what happens.
from litefs.
Ok, cool. Thanks for digging into it more. I also created an issue for keeping litefs
running on error so it's easier to ssh in and debug the state. #136
from litefs.
Good news! It's running now.
Now I'm going to try to create more regions. It just occurred to me that I'll need to create volumes for the regions first right? If I try to deploy my app to a region without a persistent volume things will break right?
from litefs.
Interestingly, I added a volume to maa
, and then tried to add a region there and got this error message:
Error App 'kcd-staging' uses volumes to control regions. Add or remove volumes to change region placement.
So I just scaled up to a count of 2
and maa
started right up! Wahoo! Thanks a ton for the help!
Now I just need to figure out how to determine the primary region via that .primary
file and then I think I should be ready to go with this to prod!
from litefs.
I got this again on a new deploy of the app:
2022-10-25T22:33:59.223 app[18d0f7c2] den [info] ERROR: cannot open store: open databases: open database("sqlite.db"): verify database file: database checksum (f13013272ddb586c) does not match latest LTX checksum (da9624ecbb43ad42)
I'm not sure what I'm doing wrong :(
from litefs.
Here's the failed build, not sure how useful it'll be: https://github.com/kentcdodds/kentcdodds.com/actions/runs/3324709637/jobs/5496681848
from litefs.
@kentcdodds This is a known bug that can occur on restart with the rollback journal. I have a fix for this one. We should have a v0.3.0 release coming early next week that will have WAL support and stability fixes in it.
from litefs.
In case, it's helpful, I tried the SHA release of litefs just now and got the same error:
https://github.com/kentcdodds/kentcdodds.com/actions/runs/3351908722/jobs/5553715931#step:6:78
from litefs.
@kentcdodds Thanks for trying it. Is this running on a clean volume or the existing one?
from litefs.
Existing one
from litefs.
I added a possible fix for this with #157. Although, depending on the exact nature of the issue, #158 could help too. It's hard to say without looking at the data files in the LiteFS directory.
This may resolve the issue on the existing volume but if it's a bug that was resolved by #158 then you'll need to wipe the volume and start with a clean database.
I'm going to close this for now but please reopen if you hit the issue again. Thanks, @kentcdodds!
from litefs.
well, my production instance is now dead, been up for 7 months
seems like due to that issue
I have 1 container and 1 volume
from litefs.
neither I'm able to connect to the instance, it's in pending
state
from litefs.
@AlexBlokh I'm sorry to hear that. Do you know what version of LiteFS you were running?
Also, you may be able to recover your underlying database. If you copy out the database
file and wal
file to a different directory with SQLite standard names, you can open in SQLite and then do an integrity check to ensure it's valid:
# Replace LITEFS_DATA_DIR & #DBNAME with your appropriate values.
# You only need to copy the "wal" file if you're using WAL mode and if the file exists.
$ cp $LITEFS_DATA_DIR/dbs/$DBNAME/database /tmp/db
$ cp $LITEFS_DATA_DIR/dbs/$DBNAME/wal /tmp/db-wal
# Open using SQLite & run an integrity check.
$ sqlite3 /tmp/db
sqlite> PRAGMA integrity_check
If it returns ok
then the underlying database is valid. If it returns errors then you'll need to recover from a backup.
from litefs.
Related Issues (20)
- Manual editing of data and forcing sync HOT 1
- Windows Support HOT 2
- Working multiple clusters/hiding databases HOT 2
- Retry primary lookup in proxy if not found
- Support for per-database primary on different nodes (regions)? HOT 3
- Continuously flush proxy response HOT 1
- Websocket connection on a replica node HOT 3
- Primary should check backup server before marking ready HOT 1
- Sync and Client API security
- Proxy Server not starting up? HOT 9
- Replication Not Working HOT 8
- Writing to the WAL while using EXCLUSIVE locking mode should be prevented HOT 11
- Support WebSockets in LiteFS Proxy HOT 3
- SQLITE_CONSTRAINT_CHECK error and rollback stops syncing with other nodes HOT 2
- How to test synchorization lag? HOT 1
- Proxy timeout when auto-scaling machines HOT 2
- Consul Key Question, Request for Documentation HOT 1
- Creating new databases HOT 1
- Add the option to act as a real proxy instead of using fly-replay header
- Object Store Mediated Replication HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from litefs.