whosonfirst / go-whosonfirst-index Goto Github PK
View Code? Open in Web Editor NEWGo package for indexing Who's On First documents
License: BSD 3-Clause "New" or "Revised" License
Go package for indexing Who's On First documents
License: BSD 3-Clause "New" or "Revised" License
Things like:
#
~
For example this uses go-whosonfirst-index
to build an in-memory PIP spatial database:
> go run -mod vendor cmd/point-in-polygon/main.go -query 'properties.sfomuseum:placetype=gallery' -query 'properties.mz:is_current=1' -spatial-database-uri 'sqlite://?dsn=:memory:' \
-spatial-source /usr/local/data/sfomuseum-data-architecture /usr/local/data/sfomuseum-data-architecture/
2021/02/16 11:26:49 PATHS [/usr/local/data/sfomuseum-data-architecture]
2021/02/16 11:26:50 INDEX 1729802937
2021/02/16 11:26:50 INDEX 1729802937
2021/02/16 11:26:51 -122.38771606591274 37.6143874695422 1
2021/02/16 11:26:51 Update /usr/local/data/sfomuseum-data-architecture/data/172/980/301/9/1729803019.geojson
2021/02/16 11:26:51 -122.38769587933967 37.61435000323429 1
2021/02/16 11:26:51 -122.38796986270393 37.61431349822708 2
2021/02/16 11:26:51 San Francisco Airport Commission Aviation Library and Louis A. Turpen Aviation Museum 1729802937
2021/02/16 11:26:51 San Francisco Airport Commission Aviation Library and Louis A. Turpen Aviation Museum 1729802937
2021/02/16 11:26:51 Failed crawl callback for /usr/local/data/sfomuseum-data-architecture/data/147/785/595/5/1477855955.geojson: Multiple results (2), after filtering
exit status 1
And fails because there is a #{WOFID}.geojson
file:
> git grep 1729802937 | grep wof:id
data/172/980/293/7/#1729802937.geojson#: "wof:id": 1729802937,
data/172/980/293/7/1729802937.geojson: "wof:id": 1729802937,
Some generic mechanism to track and record things processed, things skipped, failures etc.
https://github.com/murphy214/ld-geojson
Might just be "one more dependency"...
Specifically the SQLite and Git indexers. These should be imported using user-declared sql.DB
or go-cloud
dependency injection but I haven't had the time to figure out the details yet.
See also:
Related go-whosonfirst-timer
should write to the logger
Use it here (and replace unnecessary code):
https://github.com/whosonfirst/go-whosonfirst-index/blob/master/index.go
This will introduce a backwards incompatible change.
one row per process
For example, take this which is in the go-whosonfirst-pip-v2
package and make it a high-level shared thingy for use in other packages...
type ApplicationIndexerOptions struct {
IndexMode string
IsWOF bool
IncludeDeprecated bool
IncludeSuperseded bool
IncludeCeased bool
IncludeNotCurrent bool
}
func NewApplicationIndexer(appindex pip.Index, opts ApplicationIndexerOptions) (*index.Indexer, error) {
cb := func(fh io.Reader, ctx context.Context, args ...interface{}) error {
var f geojson.Feature
if opts.IsWOF {
ok, err := pip_utils.IsValidRecord(fh, ctx)
if err != nil {
return err
}
if !ok {
return err
}
tmp, err := feature.LoadWOFFeatureFromReader(fh)
if err != nil {
return err
}
if !opts.IncludeNotCurrent {
fl, err := whosonfirst.IsCurrent(f)
if err != nil {
return err
}
if fl.IsTrue() && fl.IsKnown() {
return nil
}
}
// and so on...
This will cause some amount of pain for certain tools but it will mean that non-SQLite-things that use this package will be able to cross-compile again. Currently this is not possible because of the SQLite stuff depends on go-sqlite3
.
Using this:
Which is not properly documented yet. Usage is demonstrated here:
I am not sure yet whether there is a way to do this, or a way to do this that isn't overly complicated, that doesn't introduce backwards incompatible changes. The package still hasn't even been blessed as v1
but maybe it's worth just saying this is v2
?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.