Git Product home page Git Product logo

go-whosonfirst-sqlite's Introduction

go-whosonfirst-sqlite

Go package for working with SQLite databases.

Important

This package has been deprecated. You should use aaronland/go-sqlite instead.

Example

Simple

import (
	"github.com/whosonfirst/go-whosonfirst-geojson-v2/feature"
	"github.com/whosonfirst/go-whosonfirst-sqlite/database"
	"github.com/whosonfirst/go-whosonfirst-sqlite-features/tables"
)

func main (){

	db, _ := database.NewDB("wof.db")
	defer db.Close()

	# Or you could just invoke these two calls with the handy:
	# st, _ := tables.NewSPRTableWithDatabase(db)

	st, _ := features.NewSPRTable()
	st.InitializeTable(db)

	f, _ := feature.LoadWOFFeatureFromFile("123.geojson")
	st.IndexFeature(db, f)
}

Error handling has been removed for the sake of brevity.

Tables

If you're looking for all the tables related to Who's On First documents they've been moved in to the go-whosonfirst-sqlite-features package.

Custom tables

Sure. You just need to write a per-table package that implements the Table interface, described below. For examples, consult the tables directories in the go-whosonfirst-sqlite-features or go-whosonfirst-sqlite-brands packages.

DSN strings

:memory:

To account for this issue DSN strings that are :memory: will be rewritten as:

file::memory:?mode=memory&cache=shared

things that don't start with file:

To account for this issue DSN strings that are not :memory: and don't start with :file: will be rewritten as:

file:{DSN}?cache=shared&mode=rwc

Interfaces

Database

type Database interface {
     Conn() (*sql.DB, error)
     DSN() string
     Close() error
}

Table

type Table interface {
     Name() string
     Schema() string
     InitializeTable(Database) error
     IndexRecord(Database, interface{}) error
}

It is left up to people implementing the Table interface to figure out what to do with the second value passed to the IndexRecord method. For example:

func (t *BrandsTable) IndexRecord(db sqlite.Database, i interface{}) error {
	return t.IndexBrand(db, i.(brands.Brand))
}

func (t *BrandsTable) IndexBrand(db sqlite.Database, b brands.Brand) error {
	// code to index brands.Brands here
}

Dependencies and relationships

If you look around the whosonfirst organization you'll notice there are a bunch of go-whosonfirst-sqlite-* packages. Specifically:

The first two are meant to be generic and broadly applicable to any SQLite database. The last two are specific to Who's On First documents.

And then there's this which is relevant because it needs to index databases that have been created using the packages above:

The relationship / dependency-chain for these five packages looks like this:

See also

go-whosonfirst-sqlite's People

Contributors

botsonfirst avatar thisisaaronland avatar

Stargazers

 avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

go-whosonfirst-sqlite's Issues

hangup without -timings

This hangs up after a few minutes. After some hustle CPU usage becomes 0% and seems doing nothing. 100% reproducible.

% ./bin/wof-sqlite-index -live-hard-die-fast -all -dsn ~/whosonfirst-data-sqlite/coutry-20171219.db -mode meta ~/whosonfirst-data/meta/wof-country-latest.csv 

This finishes.

% ./bin/wof-sqlite-index -timings -live-hard-die-fast -all -dsn ~/whosonfirst-data-sqlite/coutry-20171219.db -mode meta ~/whosonfirst-data/meta/wof-country-latest.csv 
15:12:09.986057 [wof-sqlite-index] STATUS time to index names (212) : 4.871253559s
15:12:09.987858 [wof-sqlite-index] STATUS time to index ancestors (212) : 90.853579ms
15:12:09.987866 [wof-sqlite-index] STATUS time to index concordances (212) : 231.325858ms
15:12:09.987870 [wof-sqlite-index] STATUS time to index geojson (212) : 1.717307853s
15:12:09.987874 [wof-sqlite-index] STATUS time to index spr (212) : 19.577323157s
15:12:09.987878 [wof-sqlite-index] STATUS time to index all (212) : 1m0.006167516s
15:12:13.443246 [wof-sqlite-index] STATUS time to index geojson (229) : 1.781150908s
15:12:13.443263 [wof-sqlite-index] STATUS time to index spr (229) : 31.566210978s
15:12:13.443268 [wof-sqlite-index] STATUS time to index names (229) : 5.139434872s
15:12:13.443271 [wof-sqlite-index] STATUS time to index ancestors (229) : 96.603249ms
15:12:13.443275 [wof-sqlite-index] STATUS time to index concordances (229) : 243.57579ms
15:12:13.443279 [wof-sqlite-index] STATUS time to index all (229) : 1m3.464910322s
% 

Errors causing throttles to block / contexts to not cancel

This might be a bug in go-whosonfirst-index but it's manifesting itself here:

./bin/wof-sqlite-index -geometry -driver spatialite -mode repo -dsn test.db /usr/local/data/whosonfirst-data-constituency-us/
2018/01/22 08:21:09 WHAT WHAT WHAT table geometry_whosonfirst has no column named placeype
08:21:09.126098 [wof-sqlite-index] WARNING failed to index feature (/usr/local/data/whosonfirst-data-constituency-us/data/110/876/474/1/1108764741.geojson) in 'geometry_whosonfirst' table because table geometry_whosonfirst has no column named placeype
2018/01/22 08:21:09 WHAT WHAT WHAT table geometry_whosonfirst has no column named placeype
08:21:09.140836 [wof-sqlite-index] WARNING failed to index feature (/usr/local/data/whosonfirst-data-constituency-us/data/110/877/466/1/1108774661.geojson) in 'geometry_whosonfirst' table because table geometry_whosonfirst has no column named placeype
2018/01/22 08:21:09 WHAT WHAT WHAT table geometry_whosonfirst has no column named placeype
08:21:09.150417 [wof-sqlite-index] WARNING failed to index feature (/usr/local/data/whosonfirst-data-constituency-us/data/110/874/665/3/1108746653.geojson) in 'geometry_whosonfirst' table because table geometry_whosonfirst has no column named placeype
2018/01/22 08:21:09 WHAT WHAT WHAT table geometry_whosonfirst has no column named placeype
08:21:09.158476 [wof-sqlite-index] WARNING failed to index feature (/usr/local/data/whosonfirst-data-constituency-us/data/110/874/665/1/1108746651.geojson) in 'geometry_whosonfirst' table because table geometry_whosonfirst has no column named placeype
2018/01/22 08:21:09 WHAT WHAT WHAT table geometry_whosonfirst has no column named placeype
08:21:09.180635 [wof-sqlite-index] WARNING failed to index feature (/usr/local/data/whosonfirst-data-constituency-us/data/110/874/665/9/1108746659.geojson) in 'geometry_whosonfirst' table because table geometry_whosonfirst has no column named placeype
2018/01/22 08:21:09 WHAT WHAT WHAT table geometry_whosonfirst has no column named placeype
08:21:09.208988 [wof-sqlite-index] WARNING failed to index feature (/usr/local/data/whosonfirst-data-constituency-us/data/110/874/666/1/1108746661.geojson) in 'geometry_whosonfirst' table because table geometry_whosonfirst has no column named placeype
2018/01/22 08:21:09 WHAT WHAT WHAT table geometry_whosonfirst has no column named placeype
08:21:09.222473 [wof-sqlite-index] WARNING failed to index feature (/usr/local/data/whosonfirst-data-constituency-us/data/110/874/665/5/1108746655.geojson) in 'geometry_whosonfirst' table because table geometry_whosonfirst has no column named placeype
2018/01/22 08:21:09 WHAT WHAT WHAT table geometry_whosonfirst has no column named placeype
08:21:09.234489 [wof-sqlite-index] WARNING failed to index feature (/usr/local/data/whosonfirst-data-constituency-us/data/110/874/664/1/1108746641.geojson) in 'geometry_whosonfirst' table because table geometry_whosonfirst has no column named placeype

(Note the problem is a typo in the column name placetype)

Remove the has_table cache lookup

The local cache is more trouble than it's worth and we just assume to the cost (insignificant) of asking the database every time if a table exists, because this:

./bin/wof-dist-build -workdir /usr/local/data/dist -timings -verbose -build-meta -build-bundle whosonfirst-data-constituency-us
2018/07/13 15:04:36 git lfs clone --depth 1 https://github.com/whosonfirst-data/whosonfirst-data-constituency-us.git /usr/local/data/dist/whosonfirst-data-constituency-us
15:04:55.496655 [wof-dist-build] STATUS time to clone whosonfirst-data-constituency-us 18.649286384s
15:04:55.496679 [wof-dist-build] STATUS local_repo is /usr/local/data/dist/whosonfirst-data-constituency-us
2018/07/13 15:04:55 HAS TABLE LOOKUP file:/usr/local/data/dist/whosonfirst-data-constituency-us-latest.db?cache=shared&mode=rwc#geojson false
2018/07/13 15:04:55 HAS TABLE (FOR REALZ) false
2018/07/13 15:04:55 HAS TABLE LOOKUP file:/usr/local/data/dist/whosonfirst-data-constituency-us-latest.db?cache=shared&mode=rwc#spr false
2018/07/13 15:04:55 HAS TABLE (FOR REALZ) false
2018/07/13 15:04:55 HAS TABLE LOOKUP file:/usr/local/data/dist/whosonfirst-data-constituency-us-latest.db?cache=shared&mode=rwc#names false
2018/07/13 15:04:55 HAS TABLE (FOR REALZ) false
2018/07/13 15:04:55 HAS TABLE LOOKUP file:/usr/local/data/dist/whosonfirst-data-constituency-us-latest.db?cache=shared&mode=rwc#ancestors false
2018/07/13 15:04:55 HAS TABLE (FOR REALZ) false
2018/07/13 15:04:55 HAS TABLE LOOKUP file:/usr/local/data/dist/whosonfirst-data-constituency-us-latest.db?cache=shared&mode=rwc#concordances false
2018/07/13 15:04:55 HAS TABLE (FOR REALZ) false
2018/07/13 15:04:55 RETURN NEW SQLITE INDEXER
2018/07/13 15:04:55 INDEXER YEAH
15:05:55.501207 [wof-dist-build] STATUS time to index ancestors (2762) : 752.638576ms
15:05:55.501320 [wof-dist-build] STATUS time to index concordances (2762) : 1.011299s
15:05:55.501364 [wof-dist-build] STATUS time to index geojson (2762) : 1.857307407s
15:05:55.501378 [wof-dist-build] STATUS time to index spr (2762) : 53.954843056s
15:05:55.501418 [wof-dist-build] STATUS time to index names (2762) : 609.332689ms
15:05:55.501431 [wof-dist-build] STATUS time to index all (2762) : 1m0.001008141s
15:06:55.501616 [wof-dist-build] STATUS time to index geojson (5861) : 4.046417458s
15:06:55.501642 [wof-dist-build] STATUS time to index spr (5861) : 1m47.583352703s
15:06:55.501661 [wof-dist-build] STATUS time to index names (5861) : 1.123377339s
15:06:55.501673 [wof-dist-build] STATUS time to index ancestors (5861) : 1.423058191s
15:06:55.501685 [wof-dist-build] STATUS time to index concordances (5861) : 2.086135602s
15:06:55.501697 [wof-dist-build] STATUS time to index all (5861) : 2m0.001427178s
2018/07/13 15:07:27 ALL DONE SQLITE
2018/07/13 15:07:27 CLOSING DATABASE
2018/07/13 15:07:27 ACTUALLY CLOSING...
15:07:27.096550 [wof-dist-build] STATUS local sqlite is /usr/local/data/dist/whosonfirst-data-constituency-us-latest.db
15:07:27.096579 [wof-dist-build] STATUS build metafile from sqlite (/usr/local/data/dist/whosonfirst-data-constituency-us-latest.db)
2018/07/13 15:07:27 WORKDIR /usr/local/data/dist
2018/07/13 15:07:27 MODE sqlite
2018/07/13 15:07:27 SOURCE /usr/local/data/dist/whosonfirst-data-constituency-us-latest.db
2018/07/13 15:07:27 META INDEXER sqlite &{sqlite 0x5ba4a0 0xc42031dc80 0 0}
2018/07/13 15:07:27 INDEX sqlite /usr/local/data/dist/whosonfirst-data-constituency-us-latest.db
2018/07/13 15:07:27 INDEX FROM SQLITE /usr/local/data/dist/whosonfirst-data-constituency-us-latest.db
2018/07/13 15:07:27 HAS TABLE LOOKUP file:/usr/local/data/dist/whosonfirst-data-constituency-us-latest.db?cache=shared&mode=rwc#geojson false
2018/07/13 15:07:27 HAS TABLE false

Which is... wrong.

Indicdate which database is being indexed in status messages

For example:

15:58:16.456812 [wof-sqlite-index] STATUS time to index ancestors (253822) : 1m13.585465655s
15:58:16.456819 [wof-sqlite-index] STATUS time to index all (253822) : 9m0.002456066s
15:59:16.456902 [wof-sqlite-index] STATUS time to index geojson (271066) : 41.113838624s
15:59:16.456933 [wof-sqlite-index] STATUS time to index spr (271066) : 1m28.652087017s
15:59:16.456944 [wof-sqlite-index] STATUS time to index names (271066) : 29.273664333s
15:59:16.456953 [wof-sqlite-index] STATUS time to index ancestors (271066) : 1m19.230992032s
15:59:16.456964 [wof-sqlite-index] STATUS time to index concordances (271066) : 34.995738162s
15:59:16.456973 [wof-sqlite-index] STATUS time to index all (271066) : 10m0.00264286s

"Giving up. Parser stack overflow" errors

What are these?

02:18:05.788139 [wof-sqlite-index] STATUS time to index names (313282) : 1m57.069806443s
02:18:05.788142 [wof-sqlite-index] STATUS time to index ancestors (313282) : 54.143448786s
02:18:05.788144 [wof-sqlite-index] STATUS time to index concordances (313282) : 53.159837976s
02:18:05.788147 [wof-sqlite-index] STATUS time to index all (313282) : 47m0.006320424s
Giving up.  Parser stack overflow
02:19:05.788236 [wof-sqlite-index] STATUS time to index concordances (317594) : 53.713790284s
02:19:05.788262 [wof-sqlite-index] STATUS time to index geometries (317594) : 11m3.017227421s
02:19:05.788277 [wof-sqlite-index] STATUS time to index geojson (317594) : 2m33.117795329s

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.