histograph / data Goto Github PK

Blariacum zit er bijvoorbeeld wel in maar de koppeling met het actuele Blerick (geonames) is verdwenen.... rara?

See for instance http://erfgeo.nl/thesaurus/#search=Alkmaar%20type=hg:Municipality

@mmmenno @bertspaan Can you two have a look, together with Niene tomorrow?

(PDOK) Kadaster heeft nu shapefiles van 2013, 2014 en 2015 gepubliceerd, met shapefiles zie:
https://www.pdok.nl/nl/producten/pdok-downloads/basis-registratie-kadaster/bestuurlijke-grenzen-historie

Is het de moeite waard om de wijzigingen sinds 2007 ook te gaan verwerken???
Lijkt wellicht de moeite waard... ?

Add missing / in cli help message, currently display /Users/bertspaan/code/nypl-spacetimedata-*

[ ] zijn die rechtenvrij?
[ ] hebben we een recente versie?
[ ] moet die nog gemapped?

From histograph/api#59

Currently, TGN module does not properly download gvp:PhysPlaceConcept objects (i.e. http://vocab.getty.edu/tgn/7271174)

this is the config (graphmalizer branch)

histograph@vps662:~/config$ node test.js --config ../api-erfgeo-nl.config.yml
{
  "api": {
    "host": "localhost",
    "port": 3000,
    "baseUrl": "https://api.erfgeo.nl/",
    "dataDir": "/home/histograph/uploads",
    "admin": {
      "name": "histograph",
      "password": "***********"
    }
  },
  "redis": {
    "host": "localhost",
    "port": 6379,
    "queue": "histograph"
  },
  "elasticsearch": {
    "host": "localhost",
    "port": 9200
  },
  "neo4j": {
    "host": "localhost",
    "port": 7474
  },
  "core": {
    "batchSize": 100,
    "batchTimeout": 250
  },
  "viewer": {
    "language": "en"
  },
  "data": {
    "geonames": {
      "countries": [
        "NL"
      ],
      "extraUris": "./extra-uris.json"
    },
    "tgn": {
      "parents": [
        "tgn:7016845"
      ]
    },
    "bag": {
      "db": {
        "host": "localhost",
        "port": 5432,
        "user": "postgres",
        "password": "postgres",
        "database": "bag"
      }
    }
  },
  "import": {
    "dirs": [
      "/home/histograph/data"
    ]
  },
  "logo": [
    "   ●───────●    ",
    "  /║       ║\\  ",
    " / ║       ║ \\ ",
    "●  ║═══════║  ● ",
    " \\ ║       ║ / ",
    "  \\║       ║/  ",
    "   ●───────●    "
  ]
}

this is the error (graphmalizer branch)

histograph@vps662:~/data$ node index.js --config ../api-erfgeo-nl.config.yml
Processing source geonames:
  Executing step download...
    Done!
  Executing step convert...

assert.js:86
  throw new assert.AssertionError({
        ^
AssertionError: missing path
    at Module.require (module.js:363:3)
    at require (module.js:384:17)
    at Object.<anonymous> (/home/histograph/data/pits-and-relations.js:3:14)
    at Module._compile (module.js:460:26)
    at Object.Module._extensions..js (module.js:478:10)
    at Module.load (module.js:355:32)
    at Function.Module._load (module.js:310:12)
    at Module.require (module.js:365:17)
    at require (module.js:384:17)
    at Object.exports.convert (/home/histograph/data/geonames/geonames.js:223:22)

Set die door Gerard Kuys eerder tijdens de Pilot-fase gemaakt is.

[ ] status van deze set?
[ ] moeite waard? (vanwege meertaligheid?)

At some point, we should include dynamic reasoning/inferencing in Histograph Core. For now, we can do it statically while converting data sources to NDJSON.

For examples of comparable technology, see Silk: https://www.assembla.com/spaces/silk/wiki/Linkage_Rule_Editor

Remove hg namespace prefixes in all NDJSON generation scripts!!!!!!

I would expect all of them to build, but it only runs geonames.

histograph@vps662:~/data$ node index.js --config ../api-erfgeo-nl.config.yml
Processing source geonames:
  Executing step download...
    Done!
  Executing step convert...

works when you explicitly specify what to build

histograph@vps662:~/data$ node index.js --config ../api-erfgeo-nl.config.yml tgnProcessing source tgn:  Executing step download...^C

Ze hebben nu alleen namen, geen uri's omdat er nog geen source is die ze heeft.

Optie 1:
Op wikipedia zijn ze terug te vinden dus wellicht dat DBpedia ze inderdaad heeft?
Zie: https://nl.wikipedia.org/wiki/Departementen_in_de_Nederlanden

• uitzoeken of ze daar gevonden kunnen worden

Optie 2:
Gemeentegeschiedenis uri's voor departementen gaan gebruiken?
http://www.gemeentegeschiedenis.nl/departement/NM

• Is dat handig? Hoe persistent zijn die?

In ieder geval de Deense (en Duitse?) die al gemapped zijn naar geonames

[ ] moeite insteken om andere buitenlandse plaatsen te mappen?

reinv@lingui:~/data/Github/histograph/data$ node index.js nwb
module.js:328
throw err;
^

Error: Cannot find module 'adm-zip'
at Function.Module._resolveFilename (module.js:326:15)
at Function.Module._load (module.js:277:25)
at Module.require (module.js:354:17)
at require (internal/module.js:12:17)
at Object. (/var/data/Github/histograph/data/nwb/nwb.js:6:14)
at Module._compile (module.js:398:26)
at Object.Module._extensions..js (module.js:405:10)
at Module.load (module.js:344:32)
at Function.Module._load (module.js:301:12)
at Module.require (module.js:354:17)
reinv@lingui:~/data/Github/histograph/data$ npm install adm-zip
[email protected] /var/data/Github/histograph/data
└── [email protected] extraneous

npm WARN EPACKAGEJSON [email protected] No repository field.
reinv@lingui:~/data/Github/histograph/data$ node index.js nwb
Processing dataset nwb:
Executing step download...
Done!
Executing step convert...
Done!
Executing step done...
Done!

All datasets done!
Done!

All datasets done!

With npm -v 3.3.12 and node -v 5.3.0 npm install fails with

npm ERR! Darwin 15.2.0
npm ERR! argv "/usr/local/Cellar/node/5.3.0/bin/node" "/usr/local/bin/npm" "install"
npm ERR! node v5.3.0
npm ERR! npm  v3.3.12
npm ERR! code ELIFECYCLE

npm ERR! [email protected] install: `node-gyp rebuild`
npm ERR! Exit status 1

Upgrading to the latest release fixes the problem for me, but might break with other versions of node/npm?

Stefanos-MacBook-Pro:data SB$ git diff
diff --git a/package.json b/package.json
index afb2766..62b1d46 100644
--- a/package.json
+++ b/package.json
@@ -21,7 +21,7 @@
     "geojsonhint": "^1.0.0",
     "highland": "^2.5.1",
     "is-my-json-valid": "^2.10.1",
-    "level": "^0.19.1",
+    "level": "^1.4.0",
     "minimist": "^1.1.1",
     "ndjson": "^1.3.0",
     "pg": "^4.3.0",

Import list of separate GeoNames URIs, for example cities or countries.

Unclear on how this should work, but at the moment it isn't. Switched to histograph-config data-modules branch, with a freshly pulled data-geonames:

data:                                       # Data module options (http://github.com/histograph/data)
  modulePrefix: data-
  baseDir: /home/reinv/data/Github/histograph
  generatedDir: /home/reinv/data/Github/histograph/generated-data
  modules:
    geonames:
      countries:
        - NL
      extraUris: ./extra-uris.json

But on:
reinv@lingui:~/data/Github/histograph/data$ node index.js --all

Using data modules in /home/reinv/data/Github/histograph/data-*
  Saving data to /home/reinv/data/Github/histograph/generated-data

No data modules found...

Usage: node index.js [--all] [--steps [step1,step2,...]] [--config /path/to/config.yml] [module ...]

I've successfully installed histograph core and am trying to run data import scripts but get the error:

$node index.js geonames

/home/dev/build/data/geonames/geonames.js:49
    var countryFilenames = config.countries.map(function(country) {
                                           ^
    TypeError: Cannot read property 'countries' of undefined

. README says to configure data sources in histograph configuration file. Do I follow correctly that geonames importer is expecting a countries mapping in the histograph configuration file? If so, can someone give an example of such a config?

Idea:

Allow subsets per data source, for example bag.streets, bag.buildings, tgn.places, tgn.terms, et cetera. An owner can make as many sub sources, as long as he owns the parent source (or maybe subsources are created implicitly when adding PITs or relations to them). Users can query on parent sources (source=bag or source=bag.*), or on a single subsource (source=bag.streets).

This is a good idea, because:

• Some data sources consist of different types of data, with different meta data (in data field) per PIT
• Using subsources, Core itself can add a subsource for inferred relations (e.g. tgn.places.inferred)
• It's easy to update/delete a subset of your data, without touching the rest

@mmmenno, @phpetra, @wires what do you think?

This should be implemented in API, IO, Core and Viewer as well!

Polder de Beemster http://www.geonames.org/2757799/de-beemster.html mist in onze data. Terwijl ik net wilde aangeven dat de Beemstermeer 'originated' polder de Beemster!

Wel is er een TGN 'region' De Beemster http://geothesaurus.nl/hgconcept/tgn/1111895 , maar ik weet niet of dat hetzelfde is.

Vgl Amsterdam. In de pits.ndjson staan geometrieën van 1812, 1841, 1871, etc. opgenomen, maar ik zie alleen die van 1812 terug.

Gaat 't mis omdat ze dezelfde source-uri hebben?

• pleiades
• ilvb
• poorterboeken
• atlas-verstedelijking
• graafschappen
• simon-hart
• verdwenen-dorpen
• voc-opvarenden
• carnaval
• departmenten1812

... andere?

http://www.correlatesofwar.org/data-sets/folder_listing

country identifiers back to 1816. I think hierarchical and provenance relations can be deduced from this set. Since the Cshapes dataset which we imported earlier has these codes, the cshapes geometries might be extendable backwards in time with these listings. Or at least the PiTs without geometries can be created.

To start with I'm working on parsing this internally, but I thought I'd mention this here already. Also because with all these relations, both temporal/provenance and hierarchical, this might be a nice test case for automatic inferencing

node index.js geonames
Processing source geonames:
  Executing step download...
    Done!
  Executing step convert...
    Error: [{"field":"data","message":"no (or more than one) schemas match"}]
Error: [object Object]

All sources done!

?

[ ] relevante user story voor kloeke codes?
[ ] hebben we de meest actuele set?
[ ] rechten?

• de vorige versie van DBpedia was nergens aan gemapped en zorgde daarvoor voor dubbele PiTs. Dus een nieuwe import heeft alleen zin als de set gemapped wordt naar geonames of een andere bestaande source.

[ ] uitzoeken wat er in oorspronkelijke set zit (alleen NL?)
[ ] mappen naar andere source, en eventueel aanvullen?