histograph / data Goto Github PK
View Code? Open in Web Editor NEWHome Page: http://histograph.io
License: MIT License
Home Page: http://histograph.io
License: MIT License
Blariacum zit er bijvoorbeeld wel in maar de koppeling met het actuele Blerick (geonames) is verdwenen.... rara?
See for instance http://erfgeo.nl/thesaurus/#search=Alkmaar%20type=hg:Municipality
@mmmenno @bertspaan Can you two have a look, together with Niene tomorrow?
(PDOK) Kadaster heeft nu shapefiles van 2013, 2014 en 2015 gepubliceerd, met shapefiles zie:
https://www.pdok.nl/nl/producten/pdok-downloads/basis-registratie-kadaster/bestuurlijke-grenzen-historie
Is het de moeite waard om de wijzigingen sinds 2007 ook te gaan verwerken???
Lijkt wellicht de moeite waard... ?
Add missing /
in cli help message, currently display /Users/bertspaan/code/nypl-spacetimedata-*
[ ] zijn die rechtenvrij?
[ ] hebben we een recente versie?
[ ] moet die nog gemapped?
From histograph/api#59
Currently, TGN module does not properly download gvp:PhysPlaceConcept
objects (i.e. http://vocab.getty.edu/tgn/7271174)
this is the config (graphmalizer branch)
histograph@vps662:~/config$ node test.js --config ../api-erfgeo-nl.config.yml
{
"api": {
"host": "localhost",
"port": 3000,
"baseUrl": "https://api.erfgeo.nl/",
"dataDir": "/home/histograph/uploads",
"admin": {
"name": "histograph",
"password": "***********"
}
},
"redis": {
"host": "localhost",
"port": 6379,
"queue": "histograph"
},
"elasticsearch": {
"host": "localhost",
"port": 9200
},
"neo4j": {
"host": "localhost",
"port": 7474
},
"core": {
"batchSize": 100,
"batchTimeout": 250
},
"viewer": {
"language": "en"
},
"data": {
"geonames": {
"countries": [
"NL"
],
"extraUris": "./extra-uris.json"
},
"tgn": {
"parents": [
"tgn:7016845"
]
},
"bag": {
"db": {
"host": "localhost",
"port": 5432,
"user": "postgres",
"password": "postgres",
"database": "bag"
}
}
},
"import": {
"dirs": [
"/home/histograph/data"
]
},
"logo": [
" ●───────● ",
" /║ ║\\ ",
" / ║ ║ \\ ",
"● ║═══════║ ● ",
" \\ ║ ║ / ",
" \\║ ║/ ",
" ●───────● "
]
}
this is the error (graphmalizer branch)
histograph@vps662:~/data$ node index.js --config ../api-erfgeo-nl.config.yml
Processing source geonames:
Executing step download...
Done!
Executing step convert...
assert.js:86
throw new assert.AssertionError({
^
AssertionError: missing path
at Module.require (module.js:363:3)
at require (module.js:384:17)
at Object.<anonymous> (/home/histograph/data/pits-and-relations.js:3:14)
at Module._compile (module.js:460:26)
at Object.Module._extensions..js (module.js:478:10)
at Module.load (module.js:355:32)
at Function.Module._load (module.js:310:12)
at Module.require (module.js:365:17)
at require (module.js:384:17)
at Object.exports.convert (/home/histograph/data/geonames/geonames.js:223:22)
Set die door Gerard Kuys eerder tijdens de Pilot-fase gemaakt is.
[ ] status van deze set?
[ ] moeite waard? (vanwege meertaligheid?)
At some point, we should include dynamic reasoning/inferencing in Histograph Core. For now, we can do it statically while converting data sources to NDJSON.
For examples of comparable technology, see Silk: https://www.assembla.com/spaces/silk/wiki/Linkage_Rule_Editor
Remove hg namespace prefixes in all NDJSON generation scripts!!!!!!
I would expect all of them to build, but it only runs geonames
.
histograph@vps662:~/data$ node index.js --config ../api-erfgeo-nl.config.yml
Processing source geonames:
Executing step download...
Done!
Executing step convert...
works when you explicitly specify what to build
histograph@vps662:~/data$ node index.js --config ../api-erfgeo-nl.config.yml tgnProcessing source tgn: Executing step download...^C
Ze hebben nu alleen namen, geen uri's omdat er nog geen source is die ze heeft.
Optie 1:
Op wikipedia zijn ze terug te vinden dus wellicht dat DBpedia ze inderdaad heeft?
Zie: https://nl.wikipedia.org/wiki/Departementen_in_de_Nederlanden
Optie 2:
Gemeentegeschiedenis uri's voor departementen gaan gebruiken?
http://www.gemeentegeschiedenis.nl/departement/NM
In ieder geval de Deense (en Duitse?) die al gemapped zijn naar geonames
[ ] moeite insteken om andere buitenlandse plaatsen te mappen?
reinv@lingui:~/data/Github/histograph/data$ node index.js nwb
module.js:328
throw err;
^
Error: Cannot find module 'adm-zip'
at Function.Module._resolveFilename (module.js:326:15)
at Function.Module._load (module.js:277:25)
at Module.require (module.js:354:17)
at require (internal/module.js:12:17)
at Object. (/var/data/Github/histograph/data/nwb/nwb.js:6:14)
at Module._compile (module.js:398:26)
at Object.Module._extensions..js (module.js:405:10)
at Module.load (module.js:344:32)
at Function.Module._load (module.js:301:12)
at Module.require (module.js:354:17)
reinv@lingui:~/data/Github/histograph/data$ npm install adm-zip
[email protected] /var/data/Github/histograph/data
└── [email protected] extraneous
npm WARN EPACKAGEJSON [email protected] No repository field.
reinv@lingui:~/data/Github/histograph/data$ node index.js nwb
Processing dataset nwb:
Executing step download...
Done!
Executing step convert...
Done!
Executing step done...
Done!
All datasets done!
Done!
All datasets done!
With npm -v 3.3.12 and node -v 5.3.0 npm install fails with
npm ERR! Darwin 15.2.0
npm ERR! argv "/usr/local/Cellar/node/5.3.0/bin/node" "/usr/local/bin/npm" "install"
npm ERR! node v5.3.0
npm ERR! npm v3.3.12
npm ERR! code ELIFECYCLE
npm ERR! [email protected] install: `node-gyp rebuild`
npm ERR! Exit status 1
Upgrading to the latest release fixes the problem for me, but might break with other versions of node/npm?
Stefanos-MacBook-Pro:data SB$ git diff
diff --git a/package.json b/package.json
index afb2766..62b1d46 100644
--- a/package.json
+++ b/package.json
@@ -21,7 +21,7 @@
"geojsonhint": "^1.0.0",
"highland": "^2.5.1",
"is-my-json-valid": "^2.10.1",
- "level": "^0.19.1",
+ "level": "^1.4.0",
"minimist": "^1.1.1",
"ndjson": "^1.3.0",
"pg": "^4.3.0",
Import list of separate GeoNames URIs, for example cities or countries.
Unclear on how this should work, but at the moment it isn't. Switched to histograph-config data-modules branch, with a freshly pulled data-geonames:
data: # Data module options (http://github.com/histograph/data)
modulePrefix: data-
baseDir: /home/reinv/data/Github/histograph
generatedDir: /home/reinv/data/Github/histograph/generated-data
modules:
geonames:
countries:
- NL
extraUris: ./extra-uris.json
But on:
reinv@lingui:~/data/Github/histograph/data$ node index.js --all
Using data modules in /home/reinv/data/Github/histograph/data-*
Saving data to /home/reinv/data/Github/histograph/generated-data
No data modules found...
Usage: node index.js [--all] [--steps [step1,step2,...]] [--config /path/to/config.yml] [module ...]
I've successfully installed histograph core and am trying to run data import scripts but get the error:
$node index.js geonames
/home/dev/build/data/geonames/geonames.js:49
var countryFilenames = config.countries.map(function(country) {
^
TypeError: Cannot read property 'countries' of undefined
. README says to configure data sources in histograph configuration file. Do I follow correctly that geonames importer is expecting a countries mapping in the histograph configuration file? If so, can someone give an example of such a config?
Idea:
Allow subsets per data source, for example bag.streets
, bag.buildings
, tgn.places
, tgn.terms
, et cetera. An owner can make as many sub sources, as long as he owns the parent source (or maybe subsources are created implicitly when adding PITs or relations to them). Users can query on parent sources (source=bag
or source=bag.*
), or on a single subsource (source=bag.streets
).
This is a good idea, because:
data
field) per PITtgn.places.inferred
)@mmmenno, @phpetra, @wires what do you think?
This should be implemented in API, IO, Core and Viewer as well!
Polder de Beemster http://www.geonames.org/2757799/de-beemster.html mist in onze data. Terwijl ik net wilde aangeven dat de Beemstermeer 'originated' polder de Beemster!
Wel is er een TGN 'region' De Beemster http://geothesaurus.nl/hgconcept/tgn/1111895 , maar ik weet niet of dat hetzelfde is.
Vgl Amsterdam. In de pits.ndjson staan geometrieën van 1812, 1841, 1871, etc. opgenomen, maar ik zie alleen die van 1812 terug.
Gaat 't mis omdat ze dezelfde source-uri hebben?
... andere?
http://www.correlatesofwar.org/data-sets/folder_listing
country identifiers back to 1816. I think hierarchical and provenance relations can be deduced from this set. Since the Cshapes dataset which we imported earlier has these codes, the cshapes geometries might be extendable backwards in time with these listings. Or at least the PiTs without geometries can be created.
To start with I'm working on parsing this internally, but I thought I'd mention this here already. Also because with all these relations, both temporal/provenance and hierarchical, this might be a nice test case for automatic inferencing
node index.js geonames
Processing source geonames:
Executing step download...
Done!
Executing step convert...
Error: [{"field":"data","message":"no (or more than one) schemas match"}]
Error: [object Object]
All sources done!
?
[ ] relevante user story voor kloeke codes?
[ ] hebben we de meest actuele set?
[ ] rechten?
[ ] uitzoeken wat er in oorspronkelijke set zit (alleen NL?)
[ ] mappen naar andere source, en eventueel aanvullen?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.