jdemaeyer / brightsky Goto Github PK

Let's assume that no processor takes longer than 30 minutes. Then no lock should be held for longer than 30 minutes.

Seems like it's not releasing everything when done parsing a record source?

Add a timezone parameter to the API which will be used

• to parse the date and last_date parameter if they have no timezone information, and
• as timezone for the returned record timestamps.

We could possibly include a local special value where we determine the timezone from the requested location (e.g. via https://github.com/pegler/pytzwhere).

Note: Deal with #52 first to see if this is actually worth it.

E.g. take three sources per observation type, then query weather with a source_id IN clause. In a second step, try caching the sources by lat/lon (rounded to .01 precision)

If one of the meteorological elements is not available from our preferred station, get it from the next one.

I.e. instead of

SELECT DISTINCT ON (timestamp)
    temperature,
    ...

do

SELECT
    timestamp,
    FIRST(temperature),
    ...
GROUP BY timestamp

This implies that we cannot return source information (station_id, location, etc.), since a single timestamp could be comprised of elements from multiple sources.

We can probably get by with the much more lightweight earthdistance: https://blog.rebased.pl/2020/04/07/why-you-probably-dont-need-postgis.html

Either SYNOP reports or BEOB files

Can we throw out stations not providing all elements for simplicity?

Should be real, considering https://opendata.dwd.de/climate_environment/CDC/observations_germany/climate/hourly/air_temperature/historical/stundenwerte_TU_00656_19480101_20191231_hist.zip

Writing MOSMIX updates to the database takes about ten minutes on my machine. Is there a way we can speed it up?

Can we remove lat/lon from the weather table but accomodate station location changes?