whosonfirst / go-whosonfirst-s3 Goto Github PK

View Code? Open in Web Editor NEW

0.0 7.0 1.0 23.85 MB

Go package for working with Who's On First data and S3 buckets

License: BSD 3-Clause "New" or "Revised" License

Makefile 3.03% Go 96.97%

golang whosonfirst aws s3

go-whosonfirst-s3's Introduction

go-whosonfirst-s3

Go package for working with Who's On First data and S3 buckets

Important

This package has been superseded by whosonfirst/go-whosonfirst-blob and is no longer maintained.
There is only local -> remote (S3) synchronization at the moment. See above.
There is no retry (for failed uploads) code yet.

Install

You will need to have both Go (specifically version 1.12 or higher) and the make programs installed on your computer. Assuming you do just type:

make tools

All of this package's dependencies are bundled with the code in the vendor directory.

Usage

Please write me

Tools

wof-s3-delete

Given an ID (say 1159324849) this will recursively delete everything in PREFIX/115/932/484/9.

./bin/wof-s3-delete -h
Usage of ./bin/wof-s3-delete:
  -dryrun
    	Go through the motions but don't actually delete anything.
  -lambda-clients int
    	The number of concurrent Lambda functions to invoke. (default 10)
  -lambda-dsn string
    	A valid go-whosonfirst-aws DSN string for talking to Lambda.
  -lambda-func string
    	The name of the Lambda function to invoke.
  -lambda-invoke
    	Invoke this code as a Lambda function.
  -lambda-type string
    	A valid go-aws-sdk lambda.InvocationType string (default "RequestResponse")
  -s3-dsn string
    	A valid go-whosonfirst-aws DSN string for talking to S3.
  -stdin
    	Read IDs to delete from STDIN.

For example:

$> cat /usr/local/data/to-delete.csv | ./bin/wof-s3-delete -lambda-invoke -lambda-dsn 'region=us-west-2 credentials=session' -lambda-func DeleteMedia -dryrun -stdin

wof-s3-sync

./bin/wof-s3-sync -h
Usage of ./bin/wof-s3-sync:
  -acl string
       A valid AWS S3 ACL string for permissions. (default "public-read")
  -bucket string
    	  The name of your S3 bucket. (default "data.whosonfirst.org")
  -credentials string
    	       What kind of AWS credentials to use for syncing data. (default "iam:")
  -dryrun
	Go through the motions but don't actually sync anything.
  -dsn string
       A valid go-whosonfirst-aws DSN string.
  -force
	Sync local files even if they haven't changed remotely.
  -mode string
    	The mode to use for reading local data. Valid modes are: directory,feature,feature-collection,files,geojson-ls,meta,path,repo,sqlite. (default "repo")
  -prefix string
    	  The prefix (or subdirectory) for syncing data (default "data")
  -rate-limit int
    	      The maximum number or concurrent processes. (default 100000)
  -region string
    	  The region your S3 bucket lives in. (default "us-east-1")
  -verbose
	Be chatty.

For example:

./bin/wof-s3-sync -rate-limit 100000 -dsn 'bucket=data.whosonfirst.org region=us-east-1 prefix=data credentials=iam:' -mode repo /usr/local/data/whosonfirst-data
2017/12/12 14:12:02 109820 indexed
2017/12/12 14:13:02 209831 indexed
2017/12/12 14:14:02 309822 indexed
2017/12/12 14:15:02 409789 indexed
2017/12/12 14:16:02 509838 indexed
2017/12/12 14:17:02 609817 indexed
2017/12/12 14:18:02 709817 indexed
2017/12/12 14:19:02 809843 indexed
2017/12/12 14:20:02 909810 indexed
2017/12/12 14:20:23 time to index 9m20.532420899s
2017/12/12 14:20:23 time to index 936153 documents : 9m20.532461673s

go-whosonfirst-s3's People

Contributors

Watchers

Forkers

straup

go-whosonfirst-s3's Issues

Use rcrowley's metrics package

For recording all the stuff

Optionally write errors to a file...

... that can be consumed by s3-sync-files for retries after the fact. For example stuff like this:

wof-sync-dirs] 23:10:23.662676 [error] failed to HEAD data/320/365/209/320365209.geojson because 500 Internal Server Error
[wof-sync-dirs] 23:10:23.662706 [warning] failed to determine whether data/320/365/209/320365209.geojson had changed, because '500 Internal Server Error'

Is maxprocs being set / honoured correctly?

[wof-sync-dirs] 18:21:45.952059 [info] Scheduled 20506 Completed 20490 Success 0 Error 5 Skipped 20485 Retried 0 Goroutines 191 Time 9m10.013456885s
[wof-sync-dirs] 18:21:55.952308 [info] Scheduled 20923 Completed 20907 Success 0 Error 5 Skipped 20902 Retried 0 Goroutines 196 Time 9m20.013705908s
[wof-sync-dirs] 18:22:05.952564 [info] Scheduled 21373 Completed 21357 Success 0 Error 5 Skipped 21352 Retried 0 Goroutines 194 Time 9m30.013961686s
[wof-sync-dirs] 18:22:15.952803 [info] Scheduled 21806 Completed 21790 Success 0 Error 5 Skipped 21785 Retried 0 Goroutines 191 Time 9m40.014204681s
[wof-sync-dirs] 18:22:25.953034 [info] Scheduled 22242 Completed 22226 Success 0 Error 5 Skipped 22221 Retried 0 Goroutines 193 Time 9m50.014433021s
[wof-sync-dirs] 18:22:35.953301 [info] Scheduled 22633 Completed 22617 Success 0 Error 5 Skipped 22612 Retried 0 Goroutines 190 Time 10m0.014697154s
[wof-sync-dirs] 18:22:45.953545 [info] Scheduled 23070 Completed 23054 Success 0 Error 5 Skipped 23049 Retried 0 Goroutines 192 Time 10m10.01494475s
[wof-sync-dirs] 18:22:55.953782 [info] Scheduled 23514 Completed 23498 Success 0 Error 5 Skipped 23493 Retried 0 Goroutines 194 Time 10m20.01518172s
[wof-sync-dirs] 18:23:05.954045 [info] Scheduled 23953 Completed 23937 Success 0 Error 5 Skipped 23932 Retried 0 Goroutines 191 Time 10m30.015443746s
[wof-sync-dirs] 18:23:15.954275 [info] Scheduled 24407 Completed 24391 Success 0 Error 5 Skipped 24386 Retried 0 Goroutines 191 Time 10m40.015675005s
[wof-sync-dirs] 18:23:25.954507 [info] Scheduled 24860 Completed 24844 Success 0 Error 5 Skipped 24839 Retried 0 Goroutines 189 Time 10m50.01590512s
[wof-sync-dirs] 18:23:35.954759 [info] Scheduled 25240 Completed 25224 Success 0 Error 5 Skipped 25219 Retried 0 Goroutines 194 Time 11m0.016156573s
[wof-sync-dirs] 18:23:45.955000 [info] Scheduled 25636 Completed 25620 Success 0 Error 5 Skipped 25615 Retried 0 Goroutines 194 Time 11m10.016399384s
[wof-sync-dirs] 18:23:55.955236 [info] Scheduled 26018 Completed 26002 Success 0 Error 5 Skipped 25997 Retried 0 Goroutines 191 Time 11m20.016633045s
[wof-sync-dirs] 18:24:05.955493 [info] Scheduled 26382 Completed 26366 Success 0 Error 5 Skipped 26361 Retried 0 Goroutines 190 Time 11m30.016890929s
[wof-sync-dirs] 18:24:15.955738 [info] Scheduled 26728 Completed 26712 Success 0 Error 5 Skipped 26707 Retried 0 Goroutines 188 Time 11m40.017137164s
[wof-sync-dirs] 18:24:25.955986 [info] Scheduled 27085 Completed 27069 Success 0 Error 5 Skipped 27064 Retried 0 Goroutines 190 Time 11m50.017384205s
[wof-sync-dirs] 18:24:35.956225 [info] Scheduled 27424 Completed 27408 Success 0 Error 5 Skipped 27403 Retried 0 Goroutines 190 Time 12m0.017624912s
[wof-sync-dirs] 18:24:45.956454 [info] Scheduled 27777 Completed 27761 Success 0 Error 5 Skipped 27756 Retried 0 Goroutines 192 Time 12m10.017853468s
[wof-sync-dirs] 18:24:55.956731 [info] Scheduled 28137 Completed 28121 Success 0 Error 5 Skipped 28116 Retried 0 Goroutines 191 Time 12m20.01812667s
[wof-sync-dirs] 18:25:05.956949 [info] Scheduled 28480 Completed 28464 Success 0 Error 5 Skipped 28459 Retried 0 Goroutines 191 Time 12m30.01834978s
[wof-sync-dirs] 18:25:15.957179 [info] Scheduled 28838 Completed 28822 Success 0 Error 5 Skipped 28817 Retried 0 Goroutines 192 Time 12m40.018579453s
[wof-sync-dirs] 18:25:25.957436 [info] Scheduled 29160 Completed 29144 Success 0 Error 5 Skipped 29139 Retried 0 Goroutines 190 Time 12m50.018833891s

Note the consistency of Scheduled - Completed = 16 of the samples. Unclear if this is a thing with the tunny workpool or the walk directory crawler. Unclear how much effort this should be given right now - aka it's probably fast enough.

status monitor is not being updated correctly

Unsure what the root cause is but basically after retries the code seems to go in to a loop where it imagines that anything ever completes. See below.

Increasingly thinking that the easiest thing is to remove retry code and simply write failed updates to disk per issue #13 and establish the convention of just re-running wof-sync-files...

[wof-sync-dirs] 00:33:34.923751 [info] Scheduled 1516740 Completed 1516466 Success 713429 Error 246 Skipped 802755 Retried 100 Goroutines 277 Time 1h19m42.459685325s
[wof-sync-dirs] 00:33:44.923945 [info] Scheduled 1516740 Completed 1516466 Success 713429 Error 246 Skipped 802755 Retried 100 Goroutines 277 Time 1h19m52.459878857s

14:09:26.289929 [wof-s3-sync] STATUS PUT 1209126037 (120/912/603/7/1209126037.geojson) AS '120/912/603/7/1209126037.geojson#ACL=public-read' 
14:09:26.298728 [wof-s3-sync] STATUS Has 120/912/603/7/1209126037.geojson changed: false

Which makes it look like the "has changed" test failed, when it hasn't

$> ./sync -bucket com.mapzen.debug -credentials creds -root /usr/local/mapzen/whosonfirst-data/data/101
SKIP /716/257/101716257.geojson because it is unchanged
SKIP /721/173/101721173.geojson because it is unchanged
SKIP /724/103/101724103.geojson because it is unchanged
SKIP /723/001/101723001.geojson because it is unchanged
SKIP /715/191/101715191.geojson because it is unchanged
SKIP /727/001/101727001.geojson because it is unchanged
SKIP /725/099/101725099.geojson because it is unchanged
SKIP /719/001/101719001.geojson because it is unchanged
SKIP /713/063/101713063.geojson because it is unchanged
SKIP /720/001/101720001.geojson because it is unchanged
SKIP /726/387/101726387.geojson because it is unchanged
SKIP /722/057/101722057.geojson because it is unchanged
SKIP /712/265/101712265.geojson because it is unchanged
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x38 pc=0x43f573]

goroutine 7 [running]:
com.mapzen/whosonfirst/s3.func·002()
    /usr/local/mapzen/go-mapzen-whosonfirst-s3/src/com.mapzen/whosonfirst/s3/sync.go:124 +0x143
github.com/jeffail/tunny.func·001(0x67d620, 0xc208296000, 0x0, 0x0)
    /usr/local/mapzen/go-mapzen-whosonfirst-s3/src/github.com/jeffail/tunny/tunny.go:196 +0x67
github.com/jeffail/tunny.(*tunnyDefaultWorker).TunnyJob(0xc208038058, 0x67d620, 0xc208296000, 0x0, 0x0)
    /usr/local/mapzen/go-mapzen-whosonfirst-s3/src/github.com/jeffail/tunny/tunny.go:90 +0x50
github.com/jeffail/tunny.(*workerWrapper).Loop(0xc20803af90)
    /usr/local/mapzen/go-mapzen-whosonfirst-s3/src/github.com/jeffail/tunny/worker.go:54 +0x162
created by github.com/jeffail/tunny.(*workerWrapper).Open
    /usr/local/mapzen/go-mapzen-whosonfirst-s3/src/github.com/jeffail/tunny/worker.go:80 +0x165

goroutine 1 [semacquire]:
sync.(*WaitGroup).Wait(0xc2080403d0)
    /usr/local/mapzen/golang/go/src/sync/waitgroup.go:132 +0x169
github.com/MichaelTJones/walk.Walk(0x7ffff897e83d, 0x2b, 0xc208028990, 0x0, 0x0)
    /usr/local/mapzen/go-mapzen-whosonfirst-s3/src/github.com/MichaelTJones/walk/walk.go:159 +0x2cb
github.com/whosonfirst/go-mapzen-whosonfirst-crawl/src/com.mapzen/whosonfirst.Crawler.Crawl(0x7ffff897e83d, 0x2b, 0xc20803acc0, 0x0, 0x0)
    /usr/local/mapzen/go-mapzen-whosonfirst-s3/src/github.com/whosonfirst/go-mapzen-whosonfirst-crawl/src/com.mapzen/whosonfirst/crawl.go:45 +0xcb
com.mapzen/whosonfirst/s3.Sync.SyncDirectory(0x782cd0, 0xb, 0xc2080e0000, 0x7ffff897e7ff, 0x10, 0xc208064400, 0x20, 0x20, 0xc2080c6000, 0x20, ...)
    /usr/local/mapzen/go-mapzen-whosonfirst-s3/src/com.mapzen/whosonfirst/s3/sync.go:91 +0x2e9
main.main()
    /usr/local/mapzen/go-mapzen-whosonfirst-s3/bin/sync.go:58 +0x3d9

goroutine 5 [chan receive]:
main.func·001(0xc208058120)
    /usr/local/mapzen/go-mapzen-whosonfirst-s3/bin/sync.go:51 +0x5b
created by main.main
    /usr/local/mapzen/go-mapzen-whosonfirst-s3/bin/sync.go:55 +0x357

goroutine 6 [chan send]:
github.com/jeffail/tunny.(*workerWrapper).Loop(0xc20803af60)
    /usr/local/mapzen/go-mapzen-whosonfirst-s3/src/github.com/jeffail/tunny/worker.go:51 +0xbf
created by github.com/jeffail/tunny.(*workerWrapper).Open
    /usr/local/mapzen/go-mapzen-whosonfirst-s3/src/github.com/jeffail/tunny/worker.go:80 +0x165

goroutine 8 [chan send]:
github.com/jeffail/tunny.(*workerWrapper).Loop(0xc20803afc0)
    /usr/local/mapzen/go-mapzen-whosonfirst-s3/src/github.com/jeffail/tunny/worker.go:61 +0x232
created by github.com/jeffail/tunny.(*workerWrapper).Open
    /usr/local/mapzen/go-mapzen-whosonfirst-s3/src/github.com/jeffail/tunny/worker.go:80 +0x165

goroutine 9 [select]:
net/http.(*persistConn).roundTrip(0xc20835e2c0, 0xc208029700, 0x0, 0x0, 0x0)
    /usr/local/mapzen/golang/go/src/net/http/transport.go:1082 +0x7ad
net/http.(*Transport).RoundTrip(0xc208186630, 0xc2083ea0d0, 0xc2082294a0, 0x0, 0x0)
    /usr/local/mapzen/golang/go/src/net/http/transport.go:235 +0x558
net/http.send(0xc2083ea0d0, 0x7fa2f16a7e38, 0xc208186630, 0x43, 0x0, 0x0)
    /usr/local/mapzen/golang/go/src/net/http/client.go:219 +0x4fc
net/http.(*Client).send(0xc208230360, 0xc2083ea0d0, 0x43, 0x0, 0x0)
    /usr/local/mapzen/golang/go/src/net/http/client.go:142 +0x15b
net/http.(*Client).doFollowingRedirects(0xc208230360, 0xc2083ea0d0, 0x806a08, 0x0, 0x0, 0x0)
    /usr/local/mapzen/golang/go/src/net/http/client.go:367 +0xb25
net/http.(*Client).Do(0xc208230360, 0xc2083ea0d0, 0x773b50, 0x0, 0x0)
    /usr/local/mapzen/golang/go/src/net/http/client.go:174 +0xa4
github.com/goamz/goamz/s3.(*S3).run(0xc2080e0000, 0xc208382100, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/mapzen/go-mapzen-whosonfirst-s3/src/github.com/goamz/goamz/s3/s3.go:1015 +0x4eb
github.com/goamz/goamz/s3.(*Bucket).Head(0xc208382090, 0xc2083420c0, 0x1a, 0xc2080f80c0, 0x20, 0x0, 0x0)
    /usr/local/mapzen/go-mapzen-whosonfirst-s3/src/github.com/goamz/goamz/s3/s3.go:322 +0x1f3
com.mapzen/whosonfirst/s3.func·002()
    /usr/local/mapzen/go-mapzen-whosonfirst-s3/src/com.mapzen/whosonfirst/s3/sync.go:122 +0x13e
github.com/jeffail/tunny.func·001(0x67d620, 0xc208342100, 0x0, 0x0)
    /usr/local/mapzen/go-mapzen-whosonfirst-s3/src/github.com/jeffail/tunny/tunny.go:196 +0x67
github.com/jeffail/tunny.(*tunnyDefaultWorker).TunnyJob(0xc208038068, 0x67d620, 0xc208342100, 0x0, 0x0)
    /usr/local/mapzen/go-mapzen-whosonfirst-s3/src/github.com/jeffail/tunny/tunny.go:90 +0x50
github.com/jeffail/tunny.(*workerWrapper).Loop(0xc20803aff0)
    /usr/local/mapzen/go-mapzen-whosonfirst-s3/src/github.com/jeffail/tunny/worker.go:54 +0x162
created by github.com/jeffail/tunny.(*workerWrapper).Open
    /usr/local/mapzen/go-mapzen-whosonfirst-s3/src/github.com/jeffail/tunny/worker.go:80 +0x165

...

goroutine 229 [IO wait]:
net.runtime_pollWait(0x7f1c5a26bc18, 0x77, 0xc8207e8ab0)
    /usr/local/mapzen/golang/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc8203e5b10, 0x77, 0x0, 0x0)
    /usr/local/mapzen/golang/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitWrite(0xc8203e5b10, 0x0, 0x0)
    /usr/local/mapzen/golang/go/src/net/fd_poll_runtime.go:82 +0x36
net.(*netFD).connect(0xc8203e5ab0, 0x0, 0x0, 0x7f1c5a26b8f0, 0xc8207d9c60, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/mapzen/golang/go/src/net/fd_unix.go:114 +0x1f6
net.(*netFD).dial(0xc8203e5ab0, 0x7f1c583d7130, 0x0, 0x7f1c583d7130, 0xc8202acd20, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/mapzen/golang/go/src/net/sock_posix.go:137 +0x351
net.socket(0x7cf818, 0x3, 0x2, 0x1, 0x0, 0xc8202acd00, 0x7f1c583d7130, 0x0, 0x7f1c583d7130, 0xc8202acd20, ...)
    /usr/local/mapzen/golang/go/src/net/sock_posix.go:89 +0x411
net.internetSocket(0x7cf818, 0x3, 0x7f1c583d7130, 0x0, 0x7f1c583d7130, 0xc8202acd20, 0x0, 0x0, 0x0, 0x1, ...)
    /usr/local/mapzen/golang/go/src/net/ipsock_posix.go:160 +0x141
net.dialTCP(0x7cf818, 0x3, 0x0, 0xc8202acd20, 0x0, 0xc800000000, 0x0, 0x1, 0x0, 0x0)
    /usr/local/mapzen/golang/go/src/net/tcpsock_posix.go:171 +0x11e
net.dialSingle(0xc82076e200, 0x7f1c583d70a0, 0xc8202acd20, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/mapzen/golang/go/src/net/dial.go:364 +0x3f5
net.dialSerial.func1(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/mapzen/golang/go/src/net/dial.go:336 +0x75
net.dial(0x7cf818, 0x3, 0x7f1c583d70a0, 0xc8202acd20, 0xc8204735f8, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /usr/local/mapzen/golang/go/src/net/fd_unix.go:40 +0x60
net.dialSerial(0xc82076e200, 0xc8207e8aa0, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/mapzen/golang/go/src/net/dial.go:338 +0x760
net.(*Dialer).Dial(0xc820473870, 0x7cf818, 0x3, 0xc8207d93c0, 0x14, 0x0, 0x0, 0x0, 0x0)
    /usr/local/mapzen/golang/go/src/net/dial.go:232 +0x50f
net.DialTimeout(0x7cf818, 0x3, 0xc8207d93c0, 0x14, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/mapzen/golang/go/src/net/dial.go:193 +0xa2
github.com/goamz/goamz/s3.(*S3).run.func1(0x7cf818, 0x3, 0xc8207d93c0, 0x14, 0x0, 0x0, 0x0, 0x0)
    /usr/local/mapzen/go-mapzen-whosonfirst-s3/src/github.com/goamz/goamz/s3/s3.go:990 +0x94
net/http.(*Transport).dial(0xc820331ef0, 0x7cf818, 0x3, 0xc8207d93c0, 0x14, 0x0, 0x0, 0x0, 0x0)
    /usr/local/mapzen/golang/go/src/net/http/transport.go:499 +0x79
net/http.(*Transport).dialConn(0xc820331ef0, 0x0, 0x81e1c0, 0x5, 0xc8207d93c0, 0x14, 0xc82042e240, 0x0, 0x0)
    /usr/local/mapzen/golang/go/src/net/http/transport.go:596 +0x19a9
net/http.(*Transport).getConn.func4(0xc820331ef0, 0x0, 0x81e1c0, 0x5, 0xc8207d93c0, 0x14, 0xc820540540)
    /usr/local/mapzen/golang/go/src/net/http/transport.go:549 +0x66
created by net/http.(*Transport).getConn
    /usr/local/mapzen/golang/go/src/net/http/transport.go:551 +0x265

Add support for -prefix

Paths inside buckets...