Comments (3)
The latter (alternate download) is definitely easier, and is also what I'd suggest should be done by environmentaldata.org (and I'm 99% sure can be done totally client side). When a user requests a data export on environmentaldata.org, in addition to providing the new single link to download data from multiple feeds, environmentaldata.org could also simply call ESDR's feed metadata API, for example like this:
http://esdr.cmucreatelab.org/api/v1/feeds/?fields=id,name,latitude,longitude&whereOr=id=26,id=28
And get something like this:
{
"code": 200,
"status": "success",
"data": {
"totalCount": 2,
"rows": [
{
"id": 26,
"name": "Lawrenceville ACHD",
"latitude": 40.46542,
"longitude": -79.960757
},
{
"id": 28,
"name": "Liberty ACHD",
"latitude": 40.323768,
"longitude": -79.868062
}
],
"offset": 0,
"limit": 1000
}
}
And it could then present the above data to the user as CSV or JSON, as text-on-screen or as a click-this-button-to-copy-to-your-clipboard interface, or as a file download.
Your former suggestion, of including the data in the export file itself, it what I'm having a harder time with. Exporting of multiple feeds is live now, so we can work with real examples. Calling this
https://esdr.cmucreatelab.org/api/v1/feeds/export/26.OUT_T_DEGC,28.SO2_PPM?from=1609563600&to=1609576200&format=csv
Gets you this:
EpochTime,3.feed_26.OUT_T_DEGC,3.feed_28.SO2_PPM
1609565400,4.7,0.003
1609569000,6.7,0.004
1609572600,10.5,0.001
1609576200,10.1,0
So what I'm not understanding is where you want the name/lat/long to go? None of the following three options--the only ones I can think of--seem ideal to me. In order from best-to-worst:
Option 1: Some sort of metadata header
Actual format doesn't matter to me at all. The point is that it's some sort of "commented-out" header that we either hope CSV parsers ignore, or that users need to manually tell their parser to ignore:
# Feed 26: Lawrenceville ACHD (40.46542, -79.960757)
# Feed 28: Liberty ACHD (40.323768, -79.868062)
EpochTime,3.feed_26.OUT_T_DEGC,3.feed_28.SO2_PPM
1609565400,4.7,0.003
1609569000,6.7,0.004
1609572600,10.5,0.001
1609576200,10.1,0
Option 2: Essentially two CSVs in one
FeedId,FeedName,Latitude,Longitude
26,Lawrenceville ACHD,40.46542,-79.960757
28,Liberty ACHD,40.323768,-79.868062
EpochTime,3.feed_26.OUT_T_DEGC,3.feed_28.SO2_PPM
1609565400,4.7,0.003
1609569000,6.7,0.004
1609572600,10.5,0.001
1609576200,10.1,0
Option 3: Include as extra fields
Maybe the values only actually appear on line 1 to reduce data size, but that's just an implementation detail. This one is worst partially because of file size bloat, but mostly due to the need for ESDR to parse and modify the datastore's response rather than just piping the output to the browser.
EpochTime,3.feed_26.OUT_T_DEGC,3.feed_28.SO2_PPM,26.name,26.lat,26.lng,28.name,28.lat,28.lng
1609565400,4.7,0.003,Lawrenceville ACHD,40.46542,-79.960757,Liberty ACHD,40.323768,-79.868062
1609569000,6.7,0.004,,,,,,
1609572600,10.5,0.001,,,,,,
1609576200,10.1,0,,,,,,
Summary
There are similar but different issues for JSON output. Regardless, none seems like a perfect solution, when environmentaldata.org could simply provide the info with no further changes to ESDR required.
All that said, I'm not saying no-always-and-forever, I'm just unsure how high to prioritize this when there are other, faster-to-production viable options. And it's not clear to me how users will typically use this. If they're regularly exporting the same(ish) set of feeds, then they only need the metadata once since feed ID, name, lat/long will never change.
from esdr.
Resolution: add a format
query string option to the feeds metadata API. Defaults to JSON, but will accept csv
(case insensitive) for CSV output of the metadata. Similar in spirit to the format
option for export.
from esdr.
Closing here, replacing with issue #61
from esdr.
Related Issues (20)
- Improve search for /browse HOT 3
- Can we minimize the impact of paging search results on /browse? HOT 1
- Can't see numeric values in ESDR chart at peak HOT 1
- Auto-update ESDR chart when showing real-time data HOT 1
- Can't see numeric values on Y axis for Avalon SO2 and H2S
- View Histogram broken on /browse with Safari HOT 1
- /browse "feedback" covers bottom of search results
- /browse search result checkmarks not set when linked to channels HOT 1
- Export link goes to foo.bar HOT 6
- [improvement] Place into a Docker container
- ESDR explorer search clears when you hit enter HOT 1
- /explore: Fork /explore to a new repo HOT 3
- /explore and main api: Multi-channel download HOT 1
- PM2.5 Visualization Broken due to Syntax Error HOT 1
- Add option to get feed metadata as CSV HOT 1
- Add process which produces a JSON cache of all public feeds HOT 2
- Click or double-click in browse often pans thousands of miles HOT 1
- On browse, filtering on SO2 makes charts empty HOT 1
- Filter to show only locations with recent data on /browse HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from esdr.