rcgsheffield / urban_flows Goto Github PK

View Code? Open in Web Editor NEW

2.0 2.0 2.0 3.88 MB

Research Computing work for the Urban Flows project

Python 57.93% Shell 0.41% Jupyter Notebook 41.66%

urban_flows's People

Contributors

Stargazers

Watchers

Forkers

dambem christopher-wild

urban_flows's Issues

Awesome: schedule - need at least 8 hours of data for AQI calculation

if the code runs hourly then there won't be enough data to calculate accurately

https://uk-air.defra.gov.uk/air-pollution/daqi?view=more-info&pollutant=ozone#pollutant

Awesome: sync metadata during data transfer?

What happens if a new sensor/site is created during the data transfer?

Arrow parse warnings

Arrow ArrowParseWarning: The .get() parsing method without a format string will parse more strictly in version 0.15.0
arrow-py/arrow#612

Awesome: repeat run on schedule

sync every 5 minutes

Awesome: Sync AQI readings using remote latest timestamp

Is this possible?

Something like locations/{id}/aqi-readings where results are sorted desc. by created. This doesn't seem to exist currently.

Awesome: reading min/max values

https://github.com/rcgsheffield/urban_flows/blob/master/awesome/maps.py#L73

Awesome: Map family to sensor_type_id

https://github.com/rcgsheffield/urban_flows/blob/master/awesome/maps.py#L65

Awesome: AQI UF data units

Get units and convert to the required units

get all required conversion funcs

Awesome: AQI sync not working

https://urbanflowsawesome.slack.com/archives/C01DWG08GTY/p1620913070002200

Awesome: Validate AQI readings

check units are correct
manually recalculate
online AQI calculator
online AQI database
compare bonfire night
compare night vs. daytime (expect lower at night)
airmonitors have the most metrics

Awesome: Fix sync_readings with new ufdex code

it now works on the basis of readings, not rows

All harvesters: log uncaught exceptions

Currently, only caught errors are recorded to the error file. All uncaught errors should be logged by default for all harvesters (by default to error.log in cwd)
https://stackoverflow.com/a/16993115/8634200

Awesome: point to production UFO instance

URL = 'http://ufdev.shef.ac.uk/uflobin/ufdexF1'
URL = 'http://uffront01.shef.ac.uk/uflobin/ufdex'

Harvester progress info

how record? info logging to file?

Parallel processing

Enable multi-threading to sync in parallel and increase throughput?

For Awesome data bridge: run AQI and data in separate processes? so they both sync in near real-time

Refactor code using DEFRA project as an example

Set TIME_START to beginning of UFO project

Aeroqual: convert timestamp to UTC

test before/after march-october

Old code:
a749d6f#diff-0bb5446cd5a42e1244ace667db79b27f4bddb9b042705e278b9c381a31c5eeb2

Awesome: Use F1 for metadata and data endpoints

Stat-Xplore query problem

https://stackoverflow.com/a/65341265/8634200

Awesome: grab data by family rather than sensor

more efficient due to partitioning of netcdf files (by family and date)

Secure password storage

find a suitable solution for encrypted secret storage
make /home/uflo/configs not world-readable

All harvesters: Remove output file if 0 rows retreived

after completing harvester, use row count to determine if output file should be deleted

Awesome: sync more site/location info

description
tags

https://docs.google.com/spreadsheets/d/1_-YtQWzjNtP9n8XiYh9lQCrwIEHOYIr4vngjYaTVgWs/edit?usp=sharing

DEFRA SOS get stations client error

I'm on master 39032ef

When I run > python pipeline.py -d 2020-01-01 -od test.csv I get the output below.

ERROR:http_session:400 Client Error: Bad Request for url: https://uk-air.defra.gov.uk/sos-ukair/api/v1/stations?near=%7B'near':%20'%7B%22center%22%3A%7B%22type%22%3A%22Point%22%2C%22coordinates%22%3A%5B53.379699%2C-1.469815%5D%7D%2C%22radius%22%3A50%7D'%7D ERROR:http_session:<html><head><title>Apache Tomcat/7.0.82 - Error report</title><style></style> </head><body><h1>HTTP Status 400 - </h1><HR size="1" noshade="noshade"><p><b>type</b> Status report</p><p><b>message</b> <u></u></p><p><b>description</b> <u>The request sent by the client was syntactically incorrect.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.82</h3></body></html> Traceback (most recent call last): File "pipeline.py", line 54, in <module> main() File "pipeline.py", line 32, in main stations = list(fh.get_stations()) File "C:\Users\joesh\Documents\University of Sheffield\Urban Flows Observatory\Data Pipelines\urban_flows\defra_sos\download.py", line 54, in get_stations data = self.session.call(self.base_url, 'stations?near={}'.format(query)) File "C:\Users\joesh\Documents\University of Sheffield\Urban Flows Observatory\Data Pipelines\urban_flows\defra_sos\http_session.py", line 37, in call response = self._call(base_url, endpoint, **kwargs) File "C:\Users\joesh\Documents\University of Sheffield\Urban Flows Observatory\Data Pipelines\urban_flows\defra_sos\http_session.py", line 26, in _call response.raise_for_status() File "C:\Users\joesh\Miniconda3\envs\uflo\lib\site-packages\requests\models.py", line 939, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://uk-air.defra.gov.uk/sos-ukair/api/v1/stations?near=%7B'near':%20'%7B%22center%22%3A%7B%22type%22%3A%22Point%22%2C%22coordinates%22%3A%5B53.379699%2C-1.469815%5D%7D%2C%22radius%22%3A50%7D'%7D

Awesome: filter nulls

Make the data bridge remove null or -32768 values (don't upload them to remote server)