Git Product home page Git Product logo

csv2geojson's Introduction

Build Status

csv2geojson

Converts CSV and TSV files into GeoJSON data suitable for maps..

Using as a binary:

npm install -g csv2geojson
csv2geojson geodata.csv > geodata.geojson
➟ csv2geojson
Usage: csv2geojson --lat [string] --lon [string] --line [boolean] --delimiter [string] FILE

Options:
  --lat        the name of the latitude column
  --lon        the name of the longitude column
  --line       whether or not to output points as a LineString  [default: false]
  --delimiter  the type of delimiter                            [default: ","]
  --numeric-fields comma separated list of fields to convert to numbers

Using in nodejs

npm install --save csv2geojson
var csv2geojson = require('csv2geojson');

var geoJson = csv2geojson.csv2geojson(csvString, function(err, data) {
    // err has any parsing errors
    // data is the data.
});

api

csv2geojson.csv2geojson(csvString, {
    latfield: 'LATFIELDNAME',
    lonfield: 'LONFIELDNAME',
    delimiter: ','
}, function(err, data) {
});

Parse a CSV file and derive a GeoJSON FeatureCollection object from it. Err is non-falsy if latitude and longitude values cannot be detected or if there are invalid rows in the file. Delimiter can be ',' for CSV or '\t' for TSV or '|' and other delimiters.

Delimiter can also be auto, and it will try , \t | ; and choose the 'best'.

csv2geojson.dsv(delimiter).parse(dsvString);

The dsv library for barebones DSV parsing.

csv2geojson.auto(dsvString);

Automatically choose a delimiter to parse a dsv string with, and do it.

csv2geojson.toPolygon(gj);
csv2geojson.toLine(gj);

Given a GeoJSON file consisting of points, derive one consisting of a polygon or line that has the coordinates of those points, in the order given.

Using in webpages

The latest build will be at

https://npmcdn.com/csv2geojson@latest/csv2geojson.js

Open that path in a browser to be redirected to the lastest pinned version.

Looks for fields like /^Lat/i.

See Also

  • topojson supports joining data in CSV
  • gdal supports specific CSV structures to and from other data formats

This is what powers the CSV/TSV import of geojson.io.

csv2geojson's People

Contributors

aaronlidman avatar alexanderbelokon avatar andrewharvey avatar fil avatar jaumesala avatar jfirebaugh avatar manabu avatar max-mapper avatar mbostock avatar mikemorris avatar muellerma avatar muellermatthew avatar stefanw avatar stevage avatar themarex avatar tmcw avatar yohanboniface avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

csv2geojson's Issues

JavaScript heap out of memory

I'm trying to convert a csv that is 86 mb to geojson using the command line. I installed csv2geojson as a binary. Does anyone know how to fix the following error I'm getting?

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
1: node::Abort() [node]
2: 0x13647ec [node]
3: v8::Utils::ReportOOMFailure(char const*, bool) [node]
4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [node]
5: v8::internal::Factory::NewRawTwoByteString(int, v8::internal::PretenureFlag) [node]
6: v8::internal::IncrementalStringBuilder::Extend() [node]
7: v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_(v8::internal::Handlev8::internal::Object, bool, v8::internal::Handlev8::internal::Object) [node]
8: v8::internal::JsonStringifier::SerializeJSReceiverSlow(v8::internal::Handlev8::internal::JSReceiver) [node]
9: v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_(v8::internal::Handlev8::internal::Object, bool, v8::internal::Handlev8::internal::Object) [node]
10: v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_(v8::internal::Handlev8::internal::Object, bool, v8::internal::Handlev8::internal::Object) [node]
11: v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_(v8::internal::Handlev8::internal::Object, bool, v8::internal::Handlev8::internal::Object) [node]
12: v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_(v8::internal::Handlev8::internal::Object, bool, v8::internal::Handlev8::internal::Object) [node]
13: v8::internal::JsonStringifier::Stringify(v8::internal::Handlev8::internal::Object, v8::internal::Handlev8::internal::Object, v8::internal::Handlev8::internal::Object) [node]
14: v8::internal::Builtin_JsonStringify(int, v8::internal::Object**, v8::internal::Isolate*) [node]
15: 0xe3c41a8415d

bugs in numeric-fields feature

Hi team,

I am wondering how we can use the numeric-fields option? currently when running csv2geojson on command line with out any option, all of the fields have converted into String Type. I am wondering how to convert certain fields into numeric type? like Int?

Is it something like this
csv2geojson --numeric-fields [“Full_Value”,”Year”]

Thank you,
Heng

missing files?

When I download a zip or use your demo the buttons do not do anything. It seems like there are some files missing.
One possible missing file is: csv2geojson.js

I got it to work partially in the terminal but it only converted points, no lines and polygons.

Numeric fields examples

I struggled to utilize the numeric fields option in JavaScript until I realized it needed to be entered as “numericFields”. Posting this here in case someone else finds themselves in my situation. Could be clarified in the documentation, or with an HTML example where a numericFields is identified.

Thanks this is a great bit of code 👍👍

optional altitude field

The GeoJSON spec says "Altitude or elevation MAY be included as an optional third element.". We should provide an option to specify an altitude field for use as the third element in the position.

This was alluded to in #60.

longitude with exponents are misinterpreted

I have a CSV such as the following

label,latitude,longitude
House,4.7161357378478E1,1.27381324768066E1
...

The output after conversion is:

    {
        "type": "Feature",
        "properties": {
            "label": "House"
        },
        "geometry": {
            "type": "Point",
            "coordinates": [
                1.27381324768066,
                47.161357378478
            ]
        }
    }

As you can see the latitude is correctly interpreted, while the longitude sees its exponent dropped.

I traced down the issue being in lines 101-102 of csv2geojson.js:

        a = sexagesimal(lonk, 'EW');
        if (a) lonk = a;

lonk is turned from 1.27381324768066E1 to 1.27381324768066.

<3<3

Love this library, you're awesome!

Close this issue when you've read this comment 👍

working with tsvs

I'm using the npm install method and i'm having a terrible time working with tsv files. none of the commandline options seem to work as expected. the header line wasnt even recognized until i swapped it out for a csv one. now looking at the resulting file the translation is all skewed for some reason. Can someone show me the correct syntax for this?

Header-less CSV support?

Is there any enthusiasm for supporting header-less CSV for which lat and lon columns would be given by number instead of name?

error running the script

csv2geojson osmose-planet-latest.csv > osmose_geodata.geojson
buffer.js:11
super(arg1, arg2, arg3);
^

RangeError: Invalid typed array length
at Buffer.Uint8Array (native)
at FastBuffer (buffer.js:11:5)
at createUnsafeBuffer (buffer.js:38:12)
at allocate (buffer.js:181:12)
at Function.Buffer.allocUnsafe (buffer.js:141:10)
at Function.Buffer.concat (buffer.js:309:23)
at bufferConcat (/usr/lib/node_modules/csv2geojson/node_modules/concat-stream/index.js:116:17)
at ConcatStream.getBody (/usr/lib/node_modules/csv2geojson/node_modules/concat-stream/index.js:63:42)
at ConcatStream. (/usr/lib/node_modules/csv2geojson/node_modules/concat-stream/index.js:36:51)
at emitNone (events.js:91:20)

fails to convert 1 mill rows of CSV data

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory

running from within OSX Terminal session on a 16GB RAM machine resulted in above error. The CSV rows are simple with 4 columns: Latitude,Longitude,GUID,16 char string

csv2geojson is not in the npm registry

Seems something wrong with npm.

root@n3b:~# cat /root/npm-debug.log 
0 info it worked if it ends with ok
1 verbose cli [ '/usr/bin/nodejs', '/usr/bin/npm', 'install', 'csv2geosjon' ]
2 info using [email protected]
3 info using [email protected]
4 verbose cache add [ 'csv2geosjon', null ]
5 verbose cache add name=undefined spec="csv2geosjon" args=["csv2geosjon",null]
6 verbose parsed url { protocol: null,
6 verbose parsed url   slashes: null,
6 verbose parsed url   auth: null,
6 verbose parsed url   host: null,
6 verbose parsed url   port: null,
6 verbose parsed url   hostname: null,
6 verbose parsed url   hash: null,
6 verbose parsed url   search: null,
6 verbose parsed url   query: null,
6 verbose parsed url   pathname: 'csv2geosjon',
6 verbose parsed url   path: 'csv2geosjon',
6 verbose parsed url   href: 'csv2geosjon' }
7 silly makeCacheDir cache dir uid, gid [ 0, 0 ]
8 silly lockFile f853376e-csv2geosjon csv2geosjon
9 verbose lock csv2geosjon /root/.npm/f853376e-csv2geosjon.lock
10 silly lockFile f853376e-csv2geosjon csv2geosjon
11 silly lockFile f853376e-csv2geosjon csv2geosjon
12 verbose addNamed [ 'csv2geosjon', '' ]
13 verbose addNamed [ null, '*' ]
14 silly lockFile b6ce4cd6-csv2geosjon csv2geosjon@
15 verbose lock csv2geosjon@ /root/.npm/b6ce4cd6-csv2geosjon.lock
16 silly addNameRange { name: 'csv2geosjon', range: '*', hasData: false }
17 verbose url raw csv2geosjon
18 verbose url resolving [ 'https://registry.npmjs.org/', './csv2geosjon' ]
19 verbose url resolved https://registry.npmjs.org/csv2geosjon
20 info trying registry request attempt 1 at 18:53:07
21 http GET https://registry.npmjs.org/csv2geosjon
22 http 404 https://registry.npmjs.org/csv2geosjon
23 silly registry.get cb [ 404,
23 silly registry.get   { server: 'CouchDB/1.3.1 (Erlang OTP/R15B03)',
23 silly registry.get     date: 'Mon, 14 Oct 2013 14:53:08 GMT',
23 silly registry.get     'content-type': 'application/json',
23 silly registry.get     'content-length': '52',
23 silly registry.get     'cache-control': 'must-revalidate' } ]
24 silly lockFile b6ce4cd6-csv2geosjon csv2geosjon@
25 silly lockFile b6ce4cd6-csv2geosjon csv2geosjon@
26 error 404 'csv2geosjon' is not in the npm registry.
26 error 404 You should bug the author to publish it
26 error 404
26 error 404 Note that you can also install from a
26 error 404 tarball, folder, or http url, or git url.
27 error System Linux 2.6.32-274.7.1.el5.028stab095.1
28 error command "/usr/bin/nodejs" "/usr/bin/npm" "install" "csv2geosjon"
29 error cwd /root
30 error node -v v0.10.20
31 error npm -v 1.3.10
32 error code E404
33 verbose exit [ 1, true ]

sexagesimal dependency changed namespace

Installing this module yields

npm WARN deprecated [email protected]: This module is now under the @mapbox namespace: install @mapbox/sexagesimal instead

This could be a very simple PR introducing the 8 characters they are asking for in front of the dependency into package.json

lat/lon field regex finder are too broad

For example, they will match this field as latitude: Isolation des murs et planchers bas, even if there is a latitude header some column later.

I was about to work on a PR, but I'd prefer have your opinion on the best option before:

  1. change regex to only match the beginning of the header (including spaces) /^( )*(Lat)(itude)?/gi
  2. loop over all headers and only retain the longest match (that would need switch for isLat/isLon to matchLat/matchLon that would return the matched string; given that isLat/isLon are exported from the module, this may cause trouble)
  3. allow to override the regex from the options

My preference would be the option 1, even if it's a breaking change, because I feel matching lat or lon in the middle of the the header name does not really make sense (but I may be missing something).
Option 2 is nice, but we should decide what to do for isLat/isLon: keep them even if we don't use them anymore? Or break the API?

Thoughts?

Clarify output format

@tmcw Does this always output a FeatureCollection object? If so, that might be worth adding to the docs, right? Happy to PR if you agree.

Cannot create a string longer than 0x1fffffe8 characters

So... it would seem that the code tries to slurp the whole file into memory in one go, and convert it to one big string. That's not a good idea....

Error: Cannot create a string longer than 0x1fffffe8 characters
    at Buffer.toString (node:buffer:784:17)
    at convert (C:\Users\chris\GITW\nsldist\client\node_modules\csv2geojson\csv2geojson:23:34)
    at ConcatStream.<anonymous> (C:\Users\chris\GITW\nsldist\client\node_modules\concat-stream\index.js:36:43)
    at ConcatStream.emit (node:events:402:35)
    at finishMaybe (C:\Users\chris\GITW\nsldist\client\node_modules\readable-stream\lib\_stream_writable.js:475:14)
    at endWritable (C:\Users\chris\GITW\nsldist\client\node_modules\readable-stream\lib\_stream_writable.js:485:3)
    at ConcatStream.Writable.end (C:\Users\chris\GITW\nsldist\client\node_modules\readable-stream\lib\_stream_writable.js:455:41)
    at ReadStream.onend (node:internal/streams/readable:693:10)
    at Object.onceWrapper (node:events:509:28)
    at ReadStream.emit (node:events:390:28) {
  code: 'ERR_STRING_TOO_LONG'
}```

Longitude Values Getting Messed Up

Tried using the npm module and geojson.io to convert csv to geojson.

This is how the geojson ends up:

{
"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"properties": {
"rank": "1",
"name": "Mount Elbert",
"range": "Sawatch Range",
"id": "1",
"elevation": "14,440 ft\n4401 m",
"prominence": "9,093 ft\n2772 m",
"latitude": "39.1178",
"image": "/img/Elbert2006.jpg"
},
"geometry": {
"type": "Point",
"coordinates": [
-106.4454,
671
]
}
},

The coordinates get messed up.

The same csv ends up fine using http://togeojson.com/

Error or null geometry when lng,lat not detected?

In previous versions, this module threw an error when lng,lat were not detected. The documentation suggests this still should happen:

Err is non-falsy if latitude and longitude values cannot be detected or if there are invalid rows in the file.

However, I think commit ee8d854 changed this behavior. The module now returns a null geometry. However, that commit did not update the documentation, and it left the relevant test in a strange place, where its description no longer matches its outcome.

So @tmcw I'm wondering which route you think would be best for this module. Should we document and clearly test the actual current behavior — not erroring, instead returning null geometry — or go back to the way things were — erroring?

replace dsv with d3-dsv

the former is discontinued. d3-dsv changes the function signature enough that this isn't a drop-in replacement.

grouping geojson to create polygon

Hi Tom,

Thanks for the code.

I would like to format the geojson into a specific format. As you can see from the sample csv, Alcona is a polygon that is composed of the 3 points. Is there a way to output Alcona as 1 polygon? Currently its only outputting as 3 points. Thanks in advance!

  • Gar

This is a sample of the csv:
LABEL,Longitude,Latitude
Alcona,-83.31528,44.59827
Alcona,-83.3164,44.59826
Alcona,-83.31735,44.59826

This is what its outputting:
{
"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"properties": {
"LABEL": "Alcona",
"Longitude": "-83.31528",
"Latitude": "44.59827"
},
"geometry": {
"type": "Point",
"coordinates": [
-83.31528,
44.59827
]
}
},
{
"type": "Feature",
"properties": {
"LABEL": "Alcona",
"Longitude": "-83.3164",
"Latitude": "44.59826"
},
"geometry": {
"type": "Point",
"coordinates": [
-83.3164,
44.59826
]
}
},
{
"type": "Feature",
"properties": {
"LABEL": "Alcona",
"Longitude": "-83.31735",
"Latitude": "44.59826"
},
"geometry": {
"type": "Point",
"coordinates": [
-83.31735,
44.59826
]
}
}
]
}

This is how I'd like the geojson to look like:
{
"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"properties": {
"LABEL": "Alcona"
},
"geometry": {
"type": "Polygon",
"coordinates": [[[-83.31528,44.59827],[-83.3164,44.59826],[-83.31735,44.59826]]]
}
}
]
}

Support for single coordinates field

I'm trying to use csv2geojson with a standard CSV export from the Memento Database Android app. The export stores coordinates in a single field with the lat and long separated by a colon (42.719765710136365:-73.79929259419441). Is there any way to currently handle this? Thanks!

Option to parse a column value as int rather than string while conversion?

Hey, I have a dataset that has additional data besides lat,long (let's call it "count") that holds a number. When I use the csv2geojson tool, "count" is treated as string while generating the geojson.

Is there way/option/argument that I can pass suggesting the datatype for a particular column?

Numeric fields being enclosed in double quotes

I have some CSV input that looks like this:

latitude,longitude,downloadThroughput
47.48910140991211,-122.29080200195312,16.585082892477303
47.48910140991211,-122.29080200195312,11.732173309099407
47.606201171875,-122.33209991455078,13.501345759285433

The resulting GeoJSON output adds downloadThroughput as a property, but enclosed in double quotes. This in itself isn't necessarily a problem, except that it seems that Turf.js refuses to do math operations on what it thinks is a string. It seems that csv2geojson should not make strings out of purely numeric data. When I convert my CSV file using, for example, the following tool, downloadThroughput is not wrapped in quotes and turf.average() works as expected.

http://www.convertcsv.com/csv-to-geojson.htm

Am I perhaps doing something wrong?

Working with GeometryCollections

I have a CSV which I would like to transform into a GeometryCollection.

country_dest country_dest_lng country_dest_lat country_org country_org_lat country_org_lng value
Albania 20.168331 41.153332 Iran (Islamic Rep. of) 38.4703541 47.0571193 42
Albania 20.168331 41.153332 Iraq 33.223191 43.679291 1

Using the web interface this is the output of my CSV:

{
    "type": "FeatureCollection",
    "features": [
        {
            "type": "Feature",
            "properties": {
                "country_dest": "Albania",
                "country_org": "Iran (Islamic Rep. of)",
                "country_org_lat": "38.4703541",
                "country_org_lng": "47.0571193",
                "value": "42"
            },
            "geometry": {
                "type": "Point",
                "coordinates": [
                    20.168331,
                    41.153332
                ]
            }
        },
        {
            "type": "Feature",
            "properties": {
                "country_dest": "Albania",
                "country_org": "Iraq",
                "country_org_lat": "33.223191",
                "country_org_lng": "43.679291",
                "value": "1"
            },
            "geometry": {
                "type": "Point",
                "coordinates": [
                    20.168331,
                    41.153332
                ]
            }
        }
    ]
}

And here is the desired output:

{
    type: "GeometryCollection",
    geometries: [
        {   "type": "LineString",
            "coordinates": [[-77.05, 38.91], [116.35, 39.91]],
            "properties": {"value": 10}
        },
        {   "type": "LineString",
            "coordinates": [[-39.927962, -5.058878], [46.587640, -18.909864]],
            "properties": {"value": 180}
        }
    ]
}

Do you know if this is possible with csv2geojson?

Thanks!

Faulty conversion of csv to geojson

Hi,

I am getting a faulty conversion whne I convert csv file to geojson. I have got a csv file from .pcd file which contains lat, long, alt. I want to convert the csv file to geojson. the geometry field in geojson file is NULL. I am including a small portion of the geojson file.

Please help

{
  "type": "FeatureCollection",
  "features": [
    {
      "type": "Feature",
      "properties": {
        "   ": "0.488573 0.18476 -0.594612 3819609"
      },
      "geometry": null
    },
    {
      "type": "Feature",
      "properties": {
        "   ": "0.482611 0.18281 -0.588335 3884889"
      },
      "geometry": null
    },
    {
      "type": "Feature",
      "properties": {
        "   ": "0.480358 0.18226 -0.586565 3884629"
      },
      "geometry": null
    },

npm install broken?

Tried just now both on my laptop and on a server:

ybon@ks3267459:~/tmp$ npm install csv2geojson
npm ERR! Linux 3.8.13-xxxx-grs-ipv6-64
npm ERR! argv "/usr/bin/node" "/usr/bin/npm" "install" "csv2geojson"
npm ERR! node v0.10.44
npm ERR! npm  v2.15.0
npm ERR! path /home/ybon/node_modules/csv2geojson/csv2geojson
npm ERR! code ENOENT
npm ERR! errno 34

npm ERR! enoent ENOENT, chmod '/home/ybon/node_modules/csv2geojson/csv2geojson'
npm ERR! enoent This is most likely not a problem with npm itself
npm ERR! enoent and is related to npm not being able to find a file.
npm ERR! enoent 

npm ERR! Please include the following file with any support request:
npm ERR!     /home/ybon/tmp/npm-debug.log

Missing something on the published package?

npm install [email protected] works OK.

Time parsed as NaN

Hi,

I have a CSV with a time field in the 24h format, for example 01:00. On the resulting GeoJSON the corresponding property value is NaN, instead of the expected "01:00".

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.