projecthorus / sondehub-infra Goto Github PK
View Code? Open in Web Editor NEWLicense: GNU General Public License v3.0
License: GNU General Public License v3.0
We can probably just use the sondehub pypi package and lambda to grab it from the s3 bucket at the moment.
The API URL is: https://api.v2.sondehub.org/sites
Response type is gzipped JSON:
{
"-1": {
"descent_std": 2.5,
"rs_types": [
"41"
],
"@version": "1",
"burst_samples": 52,
"station": "-1",
"position": [
144.947375,
-37.689883
],
"descent_rate": 5.7,
"@timestamp": "2021-11-23T22:08:57.372Z",
"station_name": "Melbourne BoM Training Annex (Training and Ozonesondes) (Australia)",
"alt": 119,
"burst_std": 5382,
"descent_samples": 48,
"burst_altitude": 28068
},
etc...
}
There have been a few cases of either mistaken, or malicious (probably not) recovery reporting of sondes before they have actually landed.
At the moment it is not possible to re-mark an already recovered sonde as recovered, so this means the real recovery report cannot be submitted.
I would suggest removing any limits on marking sondes as recovered/not-recovered - the client for the /recovered API should be responsible for picking out the latest entry.
Need to redirect URLs
Suggest an optional frequency_range field, containing [min_freq, max_freq] in MHz.
This can be used on the SondeHub map to get some idea of what frequency coverage range each station can support. (e.g. 400 MHz, 1680 MHz).
We should be able to use function_score to better calculate which payload to show. Ideally this is the latest payload, followed by one with xdata, followed by pressure, then humidity
Our websocket service doesn't scale with demand. We should be able to have writer server -> reader server(s)
it gives an immediate indication to people looking at the map that a sonde was recovered, and that they shouldn't go after something that might have landed a few hours ago
as a user
I want to know if a sonde has been recovered
so that
I stop chasing the radiosonde
GET /recovered
GET /recovered/{serial}
PUT /recovered/{serial
{
"serial": "S1234567",
"lat": -34.0,
"lon": 138.0,
"alt": 100.0,
"recovered": true,
"recovered_by": "VK5QI",
"description": "In a gigantic tree. But I had a pole."
},
{
"serial": "S1112234",
"lat": -34.1,
"lon": 138.1,
"alt": 100.0,
"recovered": false,
"recovered_by": "VK5FAIL",
"description": "In a gigantic tree. But I didn't have a pole."
},
Only permit uploads if the serial has been seen and the recovered postition is within 5km of the last seen data point
Actively run predictions and store in ElasticSearch so that we can display them on historic radiosonde data.
Can we get total unique callsign and total telemetry count fields added to the API so that I can make badges for them.
We're seeing a lot of chase car position uploads from callsign 'CHANGEME_RDZTTGO', which are all from misconfigured rdz_ttgo_sonde devices. These jump around the map and just cause annoyance.
It would be useful if we had a block-list somewhere for chase-car callsigns. Something similar for station position and telemetry uploads would probably not be a bad idea either.
We can generate graphs showing the number and breakdown of receivers using the Visualise tool in ElasticSearch however it can be hard to make these publically accessible. The solution to this would be to create an API that returns the necessary data to generate these graphs.
The following ElasticSearch requests gets the last 7 days of telemetry data (~44 million packets) and returns the software names and versions according to the unique number of callsigns for each. The response also includes the raw number of packets which match each software and version.
{
"aggs": {
"2": {
"terms": {
"field": "software_name.keyword",
"order": {
"1": "desc"
},
"size": 10
},
"aggs": {
"1": {
"cardinality": {
"field": "uploader_callsign.keyword"
}
},
"3": {
"terms": {
"field": "software_version.keyword",
"order": {
"1": "desc"
},
"size": 10
},
"aggs": {
"1": {
"cardinality": {
"field": "uploader_callsign.keyword"
}
}
}
}
}
}
},
"size": 0,
"stored_fields": [
"*"
],
"script_fields": {},
"docvalue_fields": [
{
"field": "@timestamp",
"format": "date_time"
},
{
"field": "datetime",
"format": "date_time"
},
{
"field": "time_received",
"format": "date_time"
},
{
"field": "time_server",
"format": "date_time"
}
],
"_source": {
"excludes": []
},
"query": {
"bool": {
"must": [],
"filter": [
{
"match_all": {}
},
{
"range": {
"datetime": {
"gte": "2022-01-05T07:50:52.795Z",
"lte": "2022-01-12T07:50:52.795Z",
"format": "strict_date_optional_time"
}
}
}
],
"should": [],
"must_not": []
}
}
}
I'm not sure if this can easily be made into an API, it shouldn't require any input fields.
This is the ElasticSearch response from the above request:
{
"took": 7063,
"timed_out": false,
"_shards": {
"total": 3,
"successful": 3,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 44772779,
"max_score": null,
"hits": []
},
"aggregations": {
"2": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"1": {
"value": 541
},
"3": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 1053782,
"buckets": [
{
"1": {
"value": 312
},
"key": "1.5.8",
"doc_count": 23576665
},
{
"1": {
"value": 91
},
"key": "1.5.7",
"doc_count": 4688112
},
{
"1": {
"value": 53
},
"key": "1.5.6",
"doc_count": 3088206
},
{
"1": {
"value": 28
},
"key": "1.5.5",
"doc_count": 1642882
},
{
"1": {
"value": 22
},
"key": "1.5.3",
"doc_count": 1510429
},
{
"1": {
"value": 13
},
"key": "1.5.1",
"doc_count": 909441
},
{
"1": {
"value": 13
},
"key": "1.5.4",
"doc_count": 435886
},
{
"1": {
"value": 11
},
"key": "1.5.2",
"doc_count": 608260
},
{
"1": {
"value": 5
},
"key": "1.5.0",
"doc_count": 390165
},
{
"1": {
"value": 3
},
"key": "1.5.8-beta2",
"doc_count": 273166
}
]
},
"key": "radiosonde_auto_rx",
"doc_count": 38176994
},
{
"1": {
"value": 147
},
"3": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"1": {
"value": 98
},
"key": "devel20211101",
"doc_count": 4381623
},
{
"1": {
"value": 36
},
"key": "master_v0.9.0",
"doc_count": 1211087
},
{
"1": {
"value": 4
},
"key": "devel20211003",
"doc_count": 101988
},
{
"1": {
"value": 3
},
"key": "devel20211005",
"doc_count": 188027
},
{
"1": {
"value": 3
},
"key": "devel20211010",
"doc_count": 91851
},
{
"1": {
"value": 2
},
"key": "devel20211013",
"doc_count": 137199
},
{
"1": {
"value": 2
},
"key": "devel20211016",
"doc_count": 110658
},
{
"1": {
"value": 2
},
"key": "devel20211023",
"doc_count": 55280
},
{
"1": {
"value": 1
},
"key": "devel20210913",
"doc_count": 137183
},
{
"1": {
"value": 1
},
"key": "devel20210914",
"doc_count": 68819
}
]
},
"key": "rdzTTGOsonde",
"doc_count": 6483715
},
{
"1": {
"value": 8
},
"3": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"1": {
"value": 6
},
"key": "6.2.5.8",
"doc_count": 44004
},
{
"1": {
"value": 1
},
"key": "6.2.4.7",
"doc_count": 66646
},
{
"1": {
"value": 1
},
"key": "6.2.5.3",
"doc_count": 1420
}
]
},
"key": "SondeMonitor",
"doc_count": 112070
}
]
}
}
}
The API would need to return the following information:
{
"time_generated": UTC,
"total_packets": 44772779,
"data": {
"radiosonde_auto_rx": {"count": 38176994, "unique_count": 541, "versions": {"1.5.8":{"count": 23576665, "unique_count": 312}, etc}},
"rdzTTGOsonde": {},
"SondeMonitor": {}
}
}
The ElasticSearch query takes about 7000ms so I would suggest caching it heavily using Cloudfront.
under linux, trying to retrieve sounding telemetry using wget or curl. It creates a file,
but its all hex. Doing it in a browser with the wget on the test page is fine. What wget or curl options do I need for command line in Ubuntu?
Ray
Add to the datanew endpoint as normal sondes but with _chase after it
Hi,
I'm quite sure it was discussed in the past, but could not find the answer:
What's the recommended way to convert the sonde "ID" value used in radiosonde_auto_rx packet summary to the "serial" key expceted by the sondehub API?
It seems that for some types they are the same (e.g. Vaisala), but for IMET, the ID also includes the type while the serial key shouldn't (so "IMET-A6C9113D" needs to be looked up as "A6C9113D").
Is there a generic conversion or is it actually type-specific? Maybe I can simply use whatever-after-the-last dash (if any) in the sonde ID as the API's serial?
Thanks.
Stop stations with a lat/lon of 0 from being returned when using the GET /listeners API endpoint.
Allow the PUT /listener API to accept a payload without the uploader_position
field.
Process and show packets in the backend where uploader_position
is set to null
.
need a build pipeline for backend and frontend. github actions might be a good option for us, alternativly codepipeline in the sondehub aws account could work
Hi,
In the previous HubHab API (https://legacy-snus.habhub.org/tracker/get_predictions.php?vehicles=), the prediction data included an extremely useful "descending" field (obviously, the predication is much more accurate after the balloon bursts).
I don't see that field in the new API.
Will it be possible to add it? Or maybe it's already there and I've missed it somehow...?
Thanks.
This could be useful for ttgo devices
This needs to be added both to the API and the S3 exporter. We'll have to add it to the telementry format and meta data tags. Also considering a new S3 folder for specific launch sites.
Notes:
A lot of the lambda functions have a main handler to test functionality, we should convert these to pytests
Hi, I have encountered a very weird bug recently.
Some frames uploaded from an rdz_ttgo_sonde station, are shown on sondehub only if at the time of upload you are in the site (which means it's sent by the websocket). but after reloading sondehub (requesting /telemetry), it's missing and not showing the recent frames uploaded by the rdz_ttgo_sonde.
I assume its a sondehub-infra issue and not a rdz_ttgo_sonde issue since the data is ending up at the API.
I saw this today after recovering a sonde that was last received by an auto_rx station at a high altitude, and after I came to the landing zone and used rdz_ttgo_sonde which uploaded frames of the sonde in the ground, it's missing after refreshing sondehub.
screenshot of sondehub that was opened when the frames were uploaded (showing the recent data by a rdz_ttgo_sonde station):
screenshot after refreshing sondehub (recent rdz_ttgo_sonde data missing):
I hope I was clear in describing the issue. Thanks.
just copy to s3
Allow multiple serials as a list for input
Recovery Statistics Endpoint
Block old versions of autorx for imets due to imet-1 issue
something like this - wait for autorx version from @darksidelemm
if imet:
if autorx == software name:
if version.parse(software_version) < version.parse(1.5.9):
#add error message
# drop payload
need to check if from packaging import version
available on lambda
We currently have a limited but growing number of users reporting recovery information directly to SondeHub via the tracker or Chasemapper. The tracker has recently been updated to handle increased amounts of recovery data so can easily scale. The Radiosondy.info database currently has about 40x the daily reported recoveries so incorporating this data could be highly beneficial.
I cannot find any public APIs to access the Radiosondy.info data so screen scraping will be required as demonstrated in this proof of concept Python script that can be set to run automatically: https://gist.github.com/LukePrior/e02b62f35eccddea096b141ffd871e4f
The two main barriers moving forwards are getting permission from SQ6KXY and adding checks to the tracker to limit the number of displayed recovery icons.
if "heading" in telemetry and telemetry["heading"] > 360:
return (False,f"Heading {telemetry['heading']} is above 360")
It would be nice to be able to show old predictions for historic sondes
Some stations are sending bad data for their listener update which is causing Lambda errors. We should catch parsing errors and return that back to the client
Probably needs
a) summary like we have for normal sondes
b) decimated history like normal sondes
c) raw data like history
Easiest way I think is to scale up both clusters, then use logstash with dead letter queues to migrate the data over night
it's possible that 6hour, and 1 day queries to ES during peak time are creating too many buckets and causing an internal server error to return on both the legacy endpoints and the new endpoints
Everything else is Python, it'll make deploying easier
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.