Git Product home page Git Product logo

go-whosonfirst-spatial's Issues

Ensure indexing alternate files in `app/walker.go` works

> make debug APIKEY=*** REPO=/usr/local/data/whosonfirst-data-admin-us

go run -mod vendor cmd/server/main.go -enable-www -enable-properties -spatial-database-uri 'rtree:///?strict=false&index_alt_files=1' -properties-reader-uri 'whosonfirst://?reader=fs:////usr/local/data/whosonfirst-data-admin-us/data&cache=gocache://' -geojson-reader-uri 'fs:///usr/local/data/whosonfirst-data-admin-us/data' -nextzen-apikey *** -mode repo:// /usr/local/data/whosonfirst-data-admin-us

2020/12/17 12:45:33 -enable-www flag is true causing the following flags to also be true: -enable-data -enable-candidates -enable-properties
12:45:33.755384 [main] FATAL failed to index paths because Failed crawl callback for /usr/local/data/whosonfirst-data-admin-us/data/101/713/493/101713493-alt-quattroshapes_pg.geojson: Feature is missing a properties.wof:name property
12:45:33.755407 [main] FATAL failed to index paths because Failed crawl callback for /usr/local/data/whosonfirst-data-admin-us/data/101/713/493/101713493-alt-quattroshapes_pg.geojson: Feature is missing a properties.wof:name property
exit status 1
make: *** [debug] Error 1

Update filter.Filter to have IsAlt method

Something like:

IsAlt(flags.ExistentialFlag) bool

But that begs the question of how you filter on a specific alternate geometry or geometries. If we did something like we did for placetypes:

HasPlacetypes(flags.PlacetypeFlag) bool
HasAlternates(flags.AlternateFlag) bool

We will need to change the go-whosonfirst-spr interface.

Speed and memory concern

Thanks for README. I understand it's for structural reorganization.

go-whosonfirst-pip-v2 consumes days for Indexing. Is it possible to speed up indexing? hopefully ten or hundred times.

Another concern is memory usage. I'm not sure that memory thrashing is the main cause of the slow indexing or not. Of course we could use a larger machine, but we have some restriction.

Allow empty go-whosonfirst-index -mode flag

Principally for use with the SQLite-related package where the assumption is the -spatial-database-uri flag may reference a database that has already been indexed.

At present the work-around is do this:

go run -mod vendor cmd/server/main.go -... -mode directory://

Where we set the -mode flag but don't pass any files to index.

Remove go-whosonfirst-geojson-v2.Polygon stuff from caching layer

It is causing things to be slow, as documented here:

The reasons are outlined here:

This points to a wholesale refactoring of how geometries are handled in go-whosonfirst-geojson-v2 but in the meantime we should just update the caching layer, here, to be more efficient. If that work is successful perhaps it can be applied to the geojson package.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.