Git Product home page Git Product logo

Comments (6)

zanieb avatar zanieb commented on August 16, 2024

Hi! I'm interested in looking into this. I read your contribution guidelines and am curious if you're interested in someone poking at this / you're actively working on it / a rough sketch of what you were thinking this would look like?

from litefs.

benbjohnson avatar benbjohnson commented on August 16, 2024

hey @madkinsz. Yeah, I'd be up for someone taking a look at this and trying to get it to work. Or at least get the test suite running—even if it is failing.

My lack of TCL knowledge is my biggest hurdle right now. The SQLite test suite writes a bunch of files into the same directory right now, even ones that aren't SQLite databases. That's an issue with LiteFS as it expects everything to be a SQLite database (or a supporting file like a journal or WAL file). We'll have to figure out how to either accept writes of non-SQLite databases or move those files out of the directory.

I think the first step is just to get the suite running against a LiteFS directory and seeing what failures we get. Most of the SQLite test suite is not even for the storage layer itself so once we get some basic tests running then I would expect a lot of them to pass. Most of the suite tests the parser/lexer, execution engine, etc.

Does that help?

from litefs.

dylanlingelbach avatar dylanlingelbach commented on August 16, 2024

I've been poking around here and have been able to run a single TCL test suite (select1.test) against litefs (will try to post the docker file and test command I've been using later this week) that mostly passes.

The biggest blocker I've found so far is not supporting removing litefs.FileTypeDatabase file types here:

litefs/fuse/root_node.go

Lines 258 to 283 in dec5a73

func (n *RootNode) Remove(ctx context.Context, req *fuse.RemoveRequest) (err error) {
dbName, fileType := ParseFilename(req.Name)
db := n.fsys.store.DB(dbName)
if db == nil {
return fuse.ToErrno(syscall.ENOENT)
}
switch fileType {
case litefs.FileTypeJournal:
if err := db.RemoveJournal(ctx); err != nil {
log.Printf("fuse: commit error: %s", err)
return err
}
return nil
case litefs.FileTypeWAL:
return db.RemoveWAL(ctx)
case litefs.FileTypeSHM:
return db.RemoveSHM(ctx)
default:
return fuse.ToErrno(syscall.ENOSYS)
}
}

A lot of the TCL tests delete and recreate the DB during the test suite which blows up.

from litefs.

benbjohnson avatar benbjohnson commented on August 16, 2024

@dylanlingelbach Thanks for trying that out. Deleting the database isn't too difficult. I've just been pushing it off. :)

I added an issue (#257) and I'll get it implemented for the v0.4.0 release.

from litefs.

dylanlingelbach avatar dylanlingelbach commented on August 16, 2024

Put up #258 - really meant for it to be draft but hopefully will aid in testing. I was testing against select1.test which runs until the DB is deleted:

Example output from extraquick.test (I have debug logging turned on when running tests)

litefs % docker run --device /dev/fuse --cap-add SYS_ADMIN -it <image> extraquick.test
config file read from /etc/litefs.yml
trace log enabled: /home/build/log/lifefs/trace.log
LiteFS development build
Using static primary: primary=true hostname=myhost advertise-url=http://myhost:20202
3499EDE34CE7010091554496: primary lease acquired, advertising as http://myhost:20202
LiteFS mounted to: /home/build/litefs
http server listening on: http://localhost:20202
waiting to connect to cluster
connected to cluster, ready
starting subprocess: /home/build/sqlite-src-3400100/testfixture [/home/build/sqlite-src-3400100/test/extraquick.test --testdir=/home/build/litefs]
waiting for signal or subprocess to exit
3499EDE34CE7010091554496 [p]: <- Getattr [ID=0x4 Node=0x1 Uid=1000 Gid=1000 Pid=18] 0x0 fl=0
3499EDE34CE7010091554496 [p]: -> [ID=0x4] Getattr valid=1m0s ino=1 size=0 mode=drwxrwxrwx
3499EDE34CE7010091554496 [p]: <- Access [ID=0x6 Node=0x1 Uid=1000 Gid=1000 Pid=18] mask=0x1
3499EDE34CE7010091554496 [p]: -> [ID=0x6] Access
3499EDE34CE7010091554496 [p]: <- Lookup [ID=0x8 Node=0x1 Uid=1000 Gid=1000 Pid=18] "test.db"
3499EDE34CE7010091554496 [p]: -> [ID=0x8] Lookup error=ENOENT
3499EDE34CE7010091554496 [p]: <- Lookup [ID=0xa Node=0x1 Uid=1000 Gid=1000 Pid=18] "test.db-journal"
3499EDE34CE7010091554496 [p]: -> [ID=0xa] Lookup error=ENOENT
3499EDE34CE7010091554496 [p]: <- Lookup [ID=0xc Node=0x1 Uid=1000 Gid=1000 Pid=18] "test.db-wal"
3499EDE34CE7010091554496 [p]: -> [ID=0xc] Lookup error=ENOENT
3499EDE34CE7010091554496 [p]: <- Lookup [ID=0xe Node=0x1 Uid=1000 Gid=1000 Pid=18] "test.db"
3499EDE34CE7010091554496 [p]: -> [ID=0xe] Lookup error=ENOENT
3499EDE34CE7010091554496 [p]: <- Lookup [ID=0x10 Node=0x1 Uid=1000 Gid=1000 Pid=18] "test.db"
3499EDE34CE7010091554496 [p]: -> [ID=0x10] Lookup error=ENOENT
3499EDE34CE7010091554496 [p]: <- Create [ID=0x12 Node=0x1 Uid=1000 Gid=1000 Pid=18] "test.db" fl=OpenReadWrite+OpenCreate+0x20000 mode=-rw-r--r-- umask=-----w--w-
database file is zero length on initialization: /home/build/lib/litefs/dbs/test.db/database
3499EDE34CE7010091554496 [p]: -> [ID=0x12] Create {0x2 gen=0 valid=1m0s attr={valid=0s ino=12591727185171964952 size=0 mode=-rw-rw-rw-}} {0x1 fl=OpenKeepCache}
3499EDE34CE7010091554496 [p]: <- Getattr [ID=0x14 Node=0x2 Uid=1000 Gid=1000 Pid=18] 0x0 fl=0
3499EDE34CE7010091554496 [p]: -> [ID=0x14] Getattr valid=0s ino=12591727185171964952 size=0 mode=-rw-rw-rw-
3499EDE34CE7010091554496 [p]: <- Getattr [ID=0x16 Node=0x2 Uid=1000 Gid=1000 Pid=18] 0x0 fl=0
3499EDE34CE7010091554496 [p]: -> [ID=0x16] Getattr valid=0s ino=12591727185171964952 size=0 mode=-rw-rw-rw-
3499EDE34CE7010091554496 [p]: <- Getattr [ID=0x18 Node=0x2 Uid=1000 Gid=1000 Pid=18] 0x0 fl=0
3499EDE34CE7010091554496 [p]: -> [ID=0x18] Getattr valid=0s ino=12591727185171964952 size=0 mode=-rw-rw-rw-
3499EDE34CE7010091554496 [p]: <- Getattr [ID=0x1a Node=0x2 Uid=1000 Gid=1000 Pid=18] 0x1 fl=GetattrFh
3499EDE34CE7010091554496 [p]: -> [ID=0x1a] Getattr valid=0s ino=12591727185171964952 size=0 mode=-rw-rw-rw-
3499EDE34CE7010091554496 [p]: <- Read [ID=0x1c Node=0x2 Uid=1000 Gid=1000 Pid=18] 0x1 4096 @0x0 dir=false fl=0 owner=0 ffl=OpenReadWrite+0x20000
3499EDE34CE7010091554496 [p]: -> [ID=0x1c] Read 0
3499EDE34CE7010091554496 [p]: <- Getattr [ID=0x1e Node=0x2 Uid=1000 Gid=1000 Pid=18] 0x0 fl=0
3499EDE34CE7010091554496 [p]: -> [ID=0x1e] Getattr valid=0s ino=12591727185171964952 size=0 mode=-rw-rw-rw-
3499EDE34CE7010091554496 [p]: <- Getattr [ID=0x20 Node=0x2 Uid=1000 Gid=1000 Pid=18] 0x0 fl=0
3499EDE34CE7010091554496 [p]: -> [ID=0x20] Getattr valid=0s ino=12591727185171964952 size=0 mode=-rw-rw-rw-
3499EDE34CE7010091554496 [p]: <- Flush [ID=0x22 Node=0x2 Uid=1000 Gid=1000 Pid=18] 0x1 fl=0x0 owner=5a8e1fbd0e9640ef
3499EDE34CE7010091554496 [p]: -> [ID=0x22] Flush
3499EDE34CE7010091554496 [p]: <- Release [ID=0x24 Node=0x2 Uid=0 Gid=0 Pid=0] 0x1 fl=OpenReadWrite+0x20000 rfl=0 owner=0
3499EDE34CE7010091554496 [p]: -> [ID=0x24] Release
3499EDE34CE7010091554496 [p]: <- Getattr [ID=0x26 Node=0x2 Uid=1000 Gid=1000 Pid=18] 0x0 fl=0
3499EDE34CE7010091554496 [p]: -> [ID=0x26] Getattr valid=0s ino=12591727185171964952 size=0 mode=-rw-rw-rw-
3499EDE34CE7010091554496 [p]: <- Remove [ID=0x28 Node=0x1 Uid=1000 Gid=1000 Pid=18] "test.db" dir=false
3499EDE34CE7010091554496 [p]: -> [ID=0x28] Remove error=ENOSYS
/home/build/sqlite-src-3400100/testfixture: error deleting "test.db": function not implemented
    while executing
"file delete -force $filename"
    (procedure "do_delete_file" line 30)
    invoked from within
"do_delete_file true {*}$args"
    (procedure "forcedelete" line 2)
    invoked from within
"forcedelete test.db"
    (procedure "reset_db" line 3)
    invoked from within
"reset_db"
    (file "/home/build/sqlite-src-3400100/test/../ext/expert/../../test/tester.tcl" line 621)
    invoked from within
"source $testdir/tester.tcl"
    (file "/home/build/sqlite-src-3400100/test/../ext/expert/expert1.test" line 23)
    invoked from within
"source /home/build/sqlite-src-3400100/test/../ext/expert/expert1.test"
    invoked from within
"interp eval tinterp $script"
    (procedure "slave_test_script" line 30)
    invoked from within
"slave_test_script [list source $zFile] "
    invoked from within
"time { slave_test_script [list source $zFile] }"
    (procedure "slave_test_file" line 23)
    invoked from within
"slave_test_file $file"
    (procedure "run_tests" line 36)
    invoked from within
"run_tests extraquick -presql {} -files {shared3.test /home/build/sqlite-src-3400100/test/../ext/session/sessionA.test func7.test upfrom4.test /home/bu..."
    ("uplevel" body line 1)
    invoked from within
"uplevel run_tests $name $::testspec($name)"
    (procedure "run_test_suite" line 5)
    invoked from within
"run_test_suite extraquick"
    (file "/home/build/sqlite-src-3400100/test/extraquick.test" line 14)
    invoked from within
"source $argv0"
    invoked from within
"if {[llength $argv]>=1} {
set argv0 [lindex $argv 0]
set argv [lrange $argv 1 end]
source $argv0
} else {
set line {}
while {![eof stdin]} {
if {$line..."
subprocess exited, litefs shutting down
3499EDE34CE7010091554496: exiting primary, destroying lease
litefs shut down complete

from litefs.

benbjohnson avatar benbjohnson commented on August 16, 2024

I just merged in #257 to support removing databases. Sorry for the delay. I got stuck on some write forwarding functionality that took longer than I expected.

from litefs.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.