Git Product home page Git Product logo

go-whosonfirst-spatial's Introduction

go-whosonfirst-spatial

Go package defining interfaces for Who's On First specific spatial operations.

Documentation

Documentation, particularly proper Go documentation, is incomplete at this time.

Motivation

The goal of the go-whosonfirst-spatial package is to de-couple the various components that made up the now-deprecated go-whosonfirst-pip-v2 package โ€“ indexing, storage, querying and serving โ€“ in to separate packages in order to allow for more flexibility.

It is the "base" package that defines provider-agnostic, but WOF-specific, interfaces for a limited set of spatial queries and reading properties.

These interfaces are then implemented in full or in part by provider-specific classes. For example, an in-memory RTree index or a SQLite database or even a Protomaps database:

You may have noticed the absence of an equivalent go-whosonfirst-spatial-postgis or even go-whosonfirst-spatial-mysql implementation. That's only because I've been focusing on implementations with fewer requirements, dependencies and less overhead to set up and maintain. There is no reason there couldn't be implementations for either database and some day I hope there will be.

Building on that there are equivalent base packages for "server" implementations, like:

The idea is that all of these pieces can be easily combined in to purpose-fit applications. As a practical matter it's mostly about trying to identify and package the common pieces in to as few lines of code as possible so that they might be combined with an application-specific import statement. For example:

import (
         _ "github.com/whosonfirst/go-whosonfirst-spatial-MY-SPECIFIC-REQUIREMENTS"
)

Here is a concrete example, implementing a point-in-polygon service over HTTP using a SQLite backend:

It is part of the overall goal of:

  • Staying out people's database or delivery choices (or needs)
  • Supporting as many databases (and delivery (and indexing) choices) as possible
  • Not making database B a dependency (in the Go code) in order to use database A, as in not bundling everything in a single mono-repo that becomes bigger and has more requirements over time.

Importantly this package does not implement any actual spatial functionality. It defines the interfaces that are implemented by other packages which allows code to function without the need to consider the underlying mechanics of how spatial operations are being performed.

The layout of this package remains in flux and is likely to change. Things have almost settled but not quite yet.

Interfaces

SpatialIndex

type SpatialIndex interface {
	IndexFeature(context.Context, []byte) error
	RemoveFeature(context.Context, string) error
	PointInPolygon(context.Context, *orb.Point, ...Filter) (spr.StandardPlacesResults, error)
	PointInPolygonCandidates(context.Context, *orb.Point, ...Filter) ([]*PointInPolygonCandidate, error)
	PointInPolygonWithChannels(context.Context, chan spr.StandardPlacesResult, chan error, chan bool, *orb.Point, ...Filter)
	PointInPolygonCandidatesWithChannels(context.Context, chan *PointInPolygonCandidate, chan error, chan bool, *orb.Point, ...Filter)
	Disconnect(context.Context) error
}

Where orb.* and spr.* refer to the paulmach/orb and whosonfirst/go-whosonfirst-flags packages respectively.

SpatialDatabase

type SpatialDatabase interface {
	reader.Reader
	writer.Writer
	spatial.SpatialIndex
}

Where reader.Reader and writer.Writer are the whosonfirst/go-reader and whosonfirst/go-writer interfaces, respectively.

Filter

type Filter interface {
	HasPlacetypes(flags.PlacetypeFlag) bool
	MatchesInception(flags.DateFlag) bool
	MatchesCessation(flags.DateFlag) bool
	IsCurrent(flags.ExistentialFlag) bool
	IsDeprecated(flags.ExistentialFlag) bool
	IsCeased(flags.ExistentialFlag) bool
	IsSuperseded(flags.ExistentialFlag) bool
	IsSuperseding(flags.ExistentialFlag) bool
	IsAlternateGeometry(flags.AlternateGeometryFlag) bool
	HasAlternateGeometry(flags.AlternateGeometryFlag) bool
}

Where flags.* refers to the whosonfirst/go-whosonfirst-flags package.

Implementations

Servers and clients

WWW

gRPC

Services and Operations

See also

go-whosonfirst-spatial's People

Contributors

straup avatar thisisaaronland avatar

Stargazers

 avatar  avatar

Watchers

 avatar  avatar

go-whosonfirst-spatial's Issues

Speed and memory concern

Thanks for README. I understand it's for structural reorganization.

go-whosonfirst-pip-v2 consumes days for Indexing. Is it possible to speed up indexing? hopefully ten or hundred times.

Another concern is memory usage. I'm not sure that memory thrashing is the main cause of the slow indexing or not. Of course we could use a larger machine, but we have some restriction.

Remove go-whosonfirst-geojson-v2.Polygon stuff from caching layer

It is causing things to be slow, as documented here:

The reasons are outlined here:

This points to a wholesale refactoring of how geometries are handled in go-whosonfirst-geojson-v2 but in the meantime we should just update the caching layer, here, to be more efficient. If that work is successful perhaps it can be applied to the geojson package.

Allow empty go-whosonfirst-index -mode flag

Principally for use with the SQLite-related package where the assumption is the -spatial-database-uri flag may reference a database that has already been indexed.

At present the work-around is do this:

go run -mod vendor cmd/server/main.go -... -mode directory://

Where we set the -mode flag but don't pass any files to index.

Ensure indexing alternate files in `app/walker.go` works

> make debug APIKEY=*** REPO=/usr/local/data/whosonfirst-data-admin-us

go run -mod vendor cmd/server/main.go -enable-www -enable-properties -spatial-database-uri 'rtree:///?strict=false&index_alt_files=1' -properties-reader-uri 'whosonfirst://?reader=fs:////usr/local/data/whosonfirst-data-admin-us/data&cache=gocache://' -geojson-reader-uri 'fs:///usr/local/data/whosonfirst-data-admin-us/data' -nextzen-apikey *** -mode repo:// /usr/local/data/whosonfirst-data-admin-us

2020/12/17 12:45:33 -enable-www flag is true causing the following flags to also be true: -enable-data -enable-candidates -enable-properties
12:45:33.755384 [main] FATAL failed to index paths because Failed crawl callback for /usr/local/data/whosonfirst-data-admin-us/data/101/713/493/101713493-alt-quattroshapes_pg.geojson: Feature is missing a properties.wof:name property
12:45:33.755407 [main] FATAL failed to index paths because Failed crawl callback for /usr/local/data/whosonfirst-data-admin-us/data/101/713/493/101713493-alt-quattroshapes_pg.geojson: Feature is missing a properties.wof:name property
exit status 1
make: *** [debug] Error 1

Update filter.Filter to have IsAlt method

Something like:

IsAlt(flags.ExistentialFlag) bool

But that begs the question of how you filter on a specific alternate geometry or geometries. If we did something like we did for placetypes:

HasPlacetypes(flags.PlacetypeFlag) bool
HasAlternates(flags.AlternateFlag) bool

We will need to change the go-whosonfirst-spr interface.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.