Git Product home page Git Product logo

opensensemap-api's Introduction

openSenseMap API

This repository contains the code of the openSenseMap API, which is the HTTP REST API used by https://opensensemap.org running at https://api.opensensemap.org. To get more information about openSenseMap and senseBox visit the before mentioned links or have a look at this video, read the API docs or the openSenseMap chapter in our documentation. openSenseMap is part of the senseBox project.

Originally, this API has been built as part of the bachelor thesis of @mpfeil at the ifgi (Institute for Geoinformatics, WWU Münster). Developers and previous maintainer include @umut0, @felixerdy, @noerw, @chk1 and @ubergesundheit.

You'll find that the repostiory uses yarn workspaces to separate the API and the database models for reuse in other projects. While the API is not published on npm, the package @sensebox/opensensemap-api-models is published from packages/models folder.

Configuration

Configuration of both the api and the models is done using mechanisms provided by lorenwest/node-config. You can find an annotated example configuration with all keys in config/config.example.json.

Development

  • Have Node.js v16, yarn, Docker and docker-compose installed
  • Start your development database (docker-compose up -d db)
  • Create branch for your feature (git checkout my-awesome-feature)
  • Run yarn install
  • Commit your changes to your branch and push it to your fork
  • Create a pull request against the master branch

See also: CONTRIBUTING

Running Tests

You can run the tests in containers using Docker and docker-compose.

# Run this the first time or every time you change dependencies in package.json
yarn build-test-env

yarn test

Related projects

Services

Libraries

Deployment

Technologies

Organization

Branches

  • master (runs on testing server)
    • Is used for container build tags
  • gh-pages

Tags and Versions

Git Tags are used for Github Container Registry builds (like v1). Version number is increased by following semantic versioning. Docker images are built automatically by Github Actions for all tags starting with v all pushes to master and all pull requests against master.

Versioned container images

  • Check out master branch
  • Go to root directory
  • Run tests
  • Optional: Build docker image locally
  • Commit everything needed for the container image
  • Run yarn tag-container
  • Run git push origin master

License

MIT - Matthias Pfeil 2015 - now

opensensemap-api's People

Contributors

adorfer avatar chk1 avatar dependabot-preview[bot] avatar dependabot[bot] avatar felixerdy avatar gitter-badger avatar janwirwahn avatar mpfeil avatar noerw avatar paulreichmuth avatar robarto avatar snyk-bot avatar thiemann96 avatar ubergesundheit avatar umut0 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

opensensemap-api's Issues

#105 sensebox/openSenseMap: Filter Function not working given a Time Interval

@SpeckiJ on sensebox/openSenseMap#105:

The official Filter Function located in the "Filter Submenu" does not work given a Time Interval.

Steps to Reproduce the Issue:
1. Select Random Interval in Filter-Function on the official OpenSenseMap Site
2. apply Filter
3. Wait :)

After a considerable amount of Time the GET-Request fails and responds with HTTP Response Code 502.
There is **no** visual Feedback (besides inside the console log) indicating that the Request has failed. In the UI the User is still shown the loading Message of "Bitte warten, die Daten werden geladen...", albeit no Data being loaded anymore after the initial GET Request failed.

The Process can only be stopped by reloading the Site as the "reset Filter" Button is greyed out while a filter is being applied.

--------------------------------------------------------------------------------------------------------------
Auf der openSenseMap im Untermenü "Filter" funktioniert das filtern unter Angabe eines Zeitraumes nicht. 

Schritte zum Reproduzieren:

1. In der Filter Funktion der OpenSenseMap einen Zeitraum angeben
2. "Filter anwenden" drücken
3. Warten :)

Nach erheblicher (!) Wartezeit antwortet die API mit einem Http Response Code 502 und der GET-Request schlägt fehlt.
Es gibt **kein** visuelles Feedback (ausserhalb der Konsole) das anzeigt das der Request abgelehnt wurde und dem Nutzer wird weiter "Bitte warten, die Daten werden geladen..." angezeigt, obwohl keine Daten mehr geladen werden.
Der Prozess kann nur durch ein komplettes Neuladen der Seite gestoppt werden, da der "Filter zurücksetzen"-Button ausgegraut ist während gerade ein Filter angewendet wird.

include measurement unit in CSV download

As a phenomenon may be measured in different units, the sensors unit of measurement should be included in the CSV download (/boxes/data?...) to allow proper data evaluation

Add height over sea level to the API and box configuration dialog

Currently, we can configure the location of a senseBox.

Having information about the height over sea level would be very useful in statistical analysis of pressure, temperature and lighting data.

If a box has no height information, or the user does not want to find and provide any information, a checkbox in the GUI could tell the API to use the height from openDEM ( http://opendem.info/download_srtm.html ). For the already existing boxes, this checkbox should be initialized to "true" so all boxes immediately have valid height information.

Also, this information should be returned in the "loc" item of the API call.

Missing Data every 13 Minutes for 2-3 Minutes

Someone just pointed out to me that every 13 minutes we have a gap in nearly every series.

I'll investigate!

  • The problem seems to be the call /boxes/:boxId/data/:sensorId
  • In the date parsing

Error in boxesController.js:366 Can't set headers after they are sent.

Error: Can't set headers after they are sent.
    at ServerResponse.OutgoingMessage.setHeader (_http_outgoing.js:356:11)
    at ServerResponse.format (/usr/src/app/node_modules/restify/lib/response.js:147:10)
    at ServerResponse.send (/usr/src/app/node_modules/restify/lib/response.js:340:14)
    at /usr/src/app/lib/controllers/boxesController.js:366:11
    at process._tickDomainCallback (internal/process/next_tick.js:129:7)

MQTT(TypeError: Cannot read property 'forEach' of undefined)

TypeError: Cannot read property 'forEach' of undefined
    at outgoing.(anonymous function) (/usr/src/app/node_modules/mqtt/lib/client.js:465:11)
    at that.outgoing.(anonymous function) (/usr/src/app/node_modules/mqtt/lib/client.js:140:15)
    at /usr/src/app/node_modules/mqtt/lib/client.js:881:9
    at Store.del (/usr/src/app/node_modules/mqtt/lib/store.js:77:5)
    at MqttClient._handleAck (/usr/src/app/node_modules/mqtt/lib/client.js:868:26)
    at MqttClient._handlePacket (/usr/src/app/node_modules/mqtt/lib/client.js:293:12)
    at process (/usr/src/app/node_modules/mqtt/lib/client.js:242:12)
    at Writable.writable._write (/usr/src/app/node_modules/mqtt/lib/client.js:252:5)
    at doWrite (/usr/src/app/node_modules/readable-stream/lib/_stream_writable.js:319:64)
    at writeOrBuffer (/usr/src/app/node_modules/readable-stream/lib/_stream_writable.js:308:5)
    at Writable.write (/usr/src/app/node_modules/readable-stream/lib/_stream_writable.js:246:11)
    at Socket.ondata (_stream_readable.js:555:20)
    at emitOne (events.js:96:13)
    at Socket.emit (events.js:188:7)
    at readableAddChunk (_stream_readable.js:176:18)
    at Socket.Readable.push (_stream_readable.js:134:10)
    at TCP.onread (net.js:548:20)

Webhooks

Let users configure webhooks for several conditions like value of sensor x is above n

specific validation errors should be returned to user

currently the user recieves { code: 422, message: 'validation failed' }on invalid requests to PUT /boxes.

it would be helpful to return the actual specific validation requirement, which is provided by mongoose

DELETE senseBox request

If you delete a senseBox over the api and send the request a second time the response is "Box deleted" but it should be instead NotFoundError

Feature wish: Email notifaction in case of abnormal data

The registering user should be notified via email if their sensebox transmits unlikely data:

  • very high or very low values
    • e.g. <-20 or >45°C temperature
    • possibly region depending
    • possibly season depending
  • constant values for days
    • e.g. 100% humidity
    • good threshold IMO: three days?
  • no transmission for days
    • i.e. when the status changes to inactive
    • or even earlier (again three days?)

That could help discover (and respond to) issues with ones sensebox earlier - when the two senseboxes I'm operating have issues I usually don't notice that for days or weeks as I don't manually check the OSeM very often.

Unify response messages

Unify response message format to:

{
    code: "", // e.g. NotAuthorized or NotFound (restify errors) , Ok or Created for 2xx responses!
    message: "" // e.g. ApiKey valid or box data
}

integrate senseBox:basic template

It would be good if we add the old senseBox:basic template in the osem again. A lot of users have the old setup and would like to setup up the box without adding all sensors manually and edit the code.

Sensor names in OSM box .ino creation script

When setting up a manually configured sense box with more than one temperature sensor, both sensors are assigned the same name (TEMPSENSOR_ID) in the mailed sensebox.ino file. Numbering (for example TEMPSENSOR2_ID) would be desirable.

Also, (at least) humidity and light (digital) sensors are named SENSOR1_ID (here numbering works) with their phenomenon only mentioned in a comment. Naming as done with temperature sensors would be desirable.

URGENT: API broken/server down/no data submits possible

I just discovered that my box stopped working at 02:44 this morning due to a failing data transmission to the sensebox API. After restarting it's still the same. I checked several other boxes, they too don't have any data since last night, so the error must be on the API side.

"sensor_id" on measurements is wrong

It appears that measurement object's sensor_id property is wrong, at least in newer measurements.

How to reproduce:

Query some recent data

db.measurements.find({
    createdAt: { 
            $gt: ISODate("2016-02-13 12:00:00.000Z"),
            $lt: ISODate("2016-02-13 13:30:00.000Z") 
    }
})

Copy the "sensor_id" and query sensors:

db.sensors.find({
    _id: ObjectId("stuff") 
})

No results on new measurement sensor_ids. Works fine on old measurements.sensor_ids.

Moveable SenseBox

Hi,

is it possible to integrate a moveable SenseBox? For a example a SenseBox which is mounted on a bicycle? At the moment every SenseBox is fixed to one location and the measurements don't store an individual location.

Kind regards,
Marius

Add feeds

For example for

  • boxes
  • measurements of boxes
  • last measurements of single boxes

(RSS, atom)

Real time

Nice project!
What do you think about an real time API (WebSocket)?
For example: A client opens a WebSocket connection and tells the server on which sensor it (the client) pays attention on. If the sensor pushes data, the API server looks, if there is a client which is paying attention on this event and if yes, the client(s) is/are receiving the change.
I don't know if you consider this as a good or just complicated feature.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.