Git Product home page Git Product logo

go-whosonfirst-index's Issues

Ignore Emacs-style temporary files

Things like:

  • Files that start with #
  • Files that end with ~

For example this uses go-whosonfirst-index to build an in-memory PIP spatial database:

> go run -mod vendor cmd/point-in-polygon/main.go -query 'properties.sfomuseum:placetype=gallery' -query 'properties.mz:is_current=1' -spatial-database-uri 'sqlite://?dsn=:memory:' \
-spatial-source /usr/local/data/sfomuseum-data-architecture /usr/local/data/sfomuseum-data-architecture/
2021/02/16 11:26:49 PATHS [/usr/local/data/sfomuseum-data-architecture]
2021/02/16 11:26:50 INDEX 1729802937
2021/02/16 11:26:50 INDEX 1729802937
2021/02/16 11:26:51 -122.38771606591274 37.6143874695422 1
2021/02/16 11:26:51 Update /usr/local/data/sfomuseum-data-architecture/data/172/980/301/9/1729803019.geojson
2021/02/16 11:26:51 -122.38769587933967 37.61435000323429 1
2021/02/16 11:26:51 -122.38796986270393 37.61431349822708 2
2021/02/16 11:26:51 San Francisco Airport Commission Aviation Library and Louis A. Turpen Aviation Museum 1729802937
2021/02/16 11:26:51 San Francisco Airport Commission Aviation Library and Louis A. Turpen Aviation Museum 1729802937
2021/02/16 11:26:51 Failed crawl callback for /usr/local/data/sfomuseum-data-architecture/data/147/785/595/5/1477855955.geojson: Multiple results (2), after filtering
exit status 1

And fails because there is a #{WOFID}.geojson file:

> git grep 1729802937 | grep wof:id
data/172/980/293/7/#1729802937.geojson#:    "wof:id": 1729802937,
data/172/980/293/7/1729802937.geojson:    "wof:id": 1729802937,

Add utility callback for filtering records by existential flags

For example, take this which is in the go-whosonfirst-pip-v2 package and make it a high-level shared thingy for use in other packages...

type ApplicationIndexerOptions struct {
        IndexMode         string
        IsWOF             bool
        IncludeDeprecated bool
        IncludeSuperseded bool
        IncludeCeased     bool
        IncludeNotCurrent bool
}

func NewApplicationIndexer(appindex pip.Index, opts ApplicationIndexerOptions) (*index.Indexer, error) {

        cb := func(fh io.Reader, ctx context.Context, args ...interface{}) error {

                var f geojson.Feature

                if opts.IsWOF {

                        ok, err := pip_utils.IsValidRecord(fh, ctx)

                        if err != nil {
                                return err
                        }

                        if !ok {
                                return err
                        }

                        tmp, err := feature.LoadWOFFeatureFromReader(fh)

                        if err != nil {
                                return err
                        }

                        if !opts.IncludeNotCurrent {

                                fl, err := whosonfirst.IsCurrent(f)

                                if err != nil {
                                        return err
                                }

                                if fl.IsTrue() && fl.IsKnown() {
                                        return nil
                                }
                        }
 
                         // and so  on...

Move SQLite indexer in to a separate package

This will cause some amount of pain for certain tools but it will mean that non-SQLite-things that use this package will be able to cross-compile again. Currently this is not possible because of the SQLite stuff depends on go-sqlite3.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.