dsmrreader / dsmr-reader Goto Github PK

Because sometimes there is simply no gas meter attached to the dsmr device.

Possibly affects:

• Mandatory database fields
• Compactor services
• Temperature readings hook.
• Frontend interface

Originally reported by: Dennis Siemensma (Bitbucket: dennissiemensma, GitHub: dennissiemensma)

Graphs' data are reversed now, starting with the most recent data on the left. It should be a setting.

• Bitbucket: https://bitbucket.org/dennissiemensma/dsmr-reader/issue/20

Originally reported by: Dennis Siemensma (Bitbucket: dennissiemensma, GitHub: dennissiemensma)

https://docs.djangoproject.com/en/1.8/howto/custom-management-commands/#module-django.core.management

"Note

When you are using management commands and wish to provide console output, you should write to self.stdout and self.stderr, instead of printing to stdout and stderr directly. By using these proxies, it becomes much easier to test your custom command. Note also that you don’t need to end messages with a newline character, it will be added automatically, unless you specify the ending parameter"

• Bitbucket: https://bitbucket.org/dennissiemensma/dsmr-reader/issue/23

Originally reported by: Dennis Siemensma (Bitbucket: dennissiemensma, GitHub: dennissiemensma)

There is quite some terminology used and over the past months I mixed some terms or used different words to describe the same thing.

Time to anaylze all GUI text used and sort it out

• Bitbucket: https://bitbucket.org/dennissiemensma/dsmr-reader/issue/41

Originally reported by: Dennis Siemensma (Bitbucket: dennissiemensma, GitHub: dennissiemensma)

We should check whether ChartJS is still the best choice in this project.

If decision is made to keep it, we should remove django-chartjs and replace it with hardcoded statics, as the plugin currently only provides the ChartJS statics.

• Bitbucket: https://bitbucket.org/dennissiemensma/dsmr-reader/issue/37

Originally reported by: Dennis Siemensma (Bitbucket: dennissiemensma, GitHub: dennissiemensma)

Notes are temporarily not rendered anymore in GUI. We should restore that later.

• Bitbucket: https://bitbucket.org/dennissiemensma/dsmr-reader/issue/39

Originally reported by: Dennis Siemensma (Bitbucket: dennissiemensma, GitHub: dennissiemensma)

We should have coverage tested as well. Services should be covered 100%.

• Bitbucket: https://bitbucket.org/dennissiemensma/dsmr-reader/issue/38

Originally reported by: Dennis Siemensma (Bitbucket: dennissiemensma, GitHub: dennissiemensma)

After 'flush' command the dashboard &statistics pages won't load anymore:

IndexError at /
list index out of range

/home/dsmr/dsmr-reader/dsmr_frontend/views/dashboard.py in get_context_data
                    latest_electricity = ElectricityConsumption.objects.all().order_by('-read_at')[0] ...

• Bitbucket: https://bitbucket.org/dennissiemensma/dsmr-reader/issue/26

Originally reported by: Dennis Siemensma (Bitbucket: dennissiemensma, GitHub: dennissiemensma)

The management command for cleaning up data should be configurable as a setting. We should also look for an app to schedule tasks, as we also need to export data periodically for external API's.

• Bitbucket: https://bitbucket.org/dennissiemensma/dsmr-reader/issue/13

Some features can be disabled in the admin interface. It would be nice if the GUI would show a message or have a status page (extend statistics page?) to also display whether features are enabled.

For example, it would be devestating if a user for some reason disabled P1 tracking, missing data.

Originally reported by: Dennis Siemensma (Bitbucket: dennissiemensma, GitHub: dennissiemensma)

The current compactor only compact gas each hour and electricity each reading (~ 10 seconds).

In order to support statistics and graphs on daily, weekly, monthly and yearly basis we should make another compactor.
To keep things separated, we might have to rename the existing compactor to something similar as 'extractor'.

• Bitbucket: https://bitbucket.org/dennissiemensma/dsmr-reader/issue/3

Originally reported by: Dennis Siemensma (Bitbucket: dennissiemensma, GitHub: dennissiemensma)

• Extract using: http://xml.buienradar.nl/
• Log outside temperature.
• Manage in Django admin?
• Separate app?

• Bitbucket: https://bitbucket.org/dennissiemensma/dsmr-reader/issue/18

A friend of mine showed me Weather Underground, which is much more specific than the current source Buienradar. Almost every town has a use which shares his weather meter via this source.

I heard the API takes 150 free hits a day. I'm not sure that this is enough.

Originally reported by: Jeroen Peters (Bitbucket: jeroenpeters, GitHub: jeroenpeters)

Every time I use cu/minicom I run into the fact that I don't know how to properly exit. That might be getting users stuck on install.

• Bitbucket: https://bitbucket.org/dennissiemensma/dsmr-reader/issue/27

Originally reported by: Dennis Siemensma (Bitbucket: dennissiemensma, GitHub: dennissiemensma)

As a follow up of #17 we should look into any possibilities to create a pre-installed image for SD card, skipping the entire installation for end users.

• Bitbucket: https://bitbucket.org/dennissiemensma/dsmr-reader/issue/33

Originally reported by: Dennis Siemensma (Bitbucket: dennissiemensma, GitHub: dennissiemensma)

https://docs.djangoproject.com/es/1.9/topics/migrations/#squashing-migrations

Can only be executed (remove old migrations) when every user has deployed all migrations squeezed.

• ./manage.py squashmigrations dsmr_stats 0012_meta
• ./manage.py squashmigrations dsmr_weather 0002_meta

• Bitbucket: https://bitbucket.org/dennissiemensma/dsmr-reader/issue/31

Originally reported by: Dennis Siemensma (Bitbucket: dennissiemensma, GitHub: dennissiemensma)

Follow-up of #21.

• Remove deprecation of dsmr_stats_datalogger by dropping command forwarding introduced by #21.
• The same applies for dsmr_stats_compactor

• Bitbucket: https://bitbucket.org/dennissiemensma/dsmr-reader/issue/22

Originally reported by: Dennis Siemensma (Bitbucket: dennissiemensma, GitHub: dennissiemensma)

The README should be quite clear, but I think it should have a better flow regarding chapters.

• Bitbucket: https://bitbucket.org/dennissiemensma/dsmr-reader/issue/25

Originally reported by: Dennis Siemensma (Bitbucket: dennissiemensma, GitHub: dennissiemensma)

Currently GasConsumption.read_at represents the passed hour read, so the end timestamp of the hour. We should make that the start of the hour instead.

Now

read_at = 22:00 = reading was between 21:00 and 22:00

Aligned

read_at = 21:00 = reading was between 21:00 and 22:00

Should not be very hard to alter, as migrations will be easy in the database (-1 hour).

• Bitbucket: https://bitbucket.org/dennissiemensma/dsmr-reader/issue/40

Originally reported by: Dennis Siemensma (Bitbucket: dennissiemensma, GitHub: dennissiemensma)

Implement PVOutput API. Docs @ http://pvoutput.org/help.html#api-spec

• Bitbucket: https://bitbucket.org/dennissiemensma/dsmr-reader/issue/9

Originally reported by: Dennis Siemensma (Bitbucket: dennissiemensma, GitHub: dennissiemensma)

Django mentions somewhere mysql_tzinfo_to_sql — Load the Time Zone Tables. We should check whether MySQL data is stored timezone aware.

If not, we should mention the required steps in the installation guide of this project.

• Bitbucket: https://bitbucket.org/dennissiemensma/dsmr-reader/issue/35

Originally reported by: Jeroen Peters (Bitbucket: jeroenpeters, GitHub: jeroenpeters)

Now that the EnergySupplierPrice model is being used in the front end. It would be nice to use the description in the frontend and the Django Admin too

• Bitbucket: https://bitbucket.org/dennissiemensma/dsmr-reader/issue/4

Would be nice to have every commit in master checked by runnings tests.

https://github.com/integrations/travis-ci

Originally reported by: Dennis Siemensma (Bitbucket: dennissiemensma, GitHub: dennissiemensma)

Some graphs display day names, but they are not translated properly. This happens because the labels are injected into javascript using json, wihout translation tags.

• Bitbucket: https://bitbucket.org/dennissiemensma/dsmr-reader/issue/42

Originally reported by: Dennis Siemensma (Bitbucket: dennissiemensma, GitHub: dennissiemensma)

Test dsmr_weather.test_weather_tracking does NOT mock its external call to Buienradar. We should mock it and also verify XML parsing using a dummy XML response.

• Bitbucket: https://bitbucket.org/dennissiemensma/dsmr-reader/issue/32

Originally reported by: Dennis Siemensma (Bitbucket: dennissiemensma, GitHub: dennissiemensma)

Hook Django's clearsessions management command to one of the backend signal hooks. The session storage won't grow rapidly, but we should still purge it once in a while.

• Bitbucket: https://bitbucket.org/dennissiemensma/dsmr-reader/issue/30

Originally reported by: Dennis Siemensma (Bitbucket: dennissiemensma, GitHub: dennissiemensma)

It would be great to allow end-users to place notes on a daily basis. For example, when experimenting with the thermostat or replacing home appliances, which might result in different gas or electricity usage.

• Date + description
• Display in daily graphs/pages
• Management via Django admin.

• Bitbucket: https://bitbucket.org/dennissiemensma/dsmr-reader/issue/11

Originally reported by: Dennis Siemensma (Bitbucket: dennissiemensma, GitHub: dennissiemensma)

DoesNotExist at /
EnergySupplierPrice matching query does not exist.

#!python

/home/dsmr/dsmr-reader/dsmr_stats/mixins.py in get_context_data
                            day=electricity[0].read_at.astimezone(settings.LOCAL_TIME_ZONE) ...
▶ Local vars
/home/dsmr/dsmr-reader/dsmr_stats/services.py in day_consumption
                    target_date=consumption['day'] ...
▶ Local vars
/home/dsmr/dsmr-reader/dsmr_stats/models/energysupplier.py in by_date
                        start__lte=target_date, 

• Bitbucket: https://bitbucket.org/dennissiemensma/dsmr-reader/issue/5

Originally reported by: Dennis Siemensma (Bitbucket: dennissiemensma, GitHub: dennissiemensma)

Currently the recent history displays the last 28 or 30 days or so. It should be converted to a setting, having several predefined options:

• last week
• last two weeks
• last three weeks
• last four weeks

Reason for this setting is to accelerate the page for slower Pi's.

• Bitbucket: https://bitbucket.org/dennissiemensma/dsmr-reader/issue/29

While exploring options for #42, I stumbled across Django's timezone.localtime(x), which:
Converts an aware datetime.datetime to local time.

Local time is defined by the current time zone, unless another time zone
is specified.

I'm used to datetime.astimezone(settings.LOCAL_TIME_ZONE), which seemed to work, but is not very neat anyway. So I checked the difference between the two:

localtime()
2016-02-14 06:25:00+00:00

astimezone()
2016-02-14 07:25:00+01:00

localtime relies on settings.TIME_ZONE, which is set hardcoded to UTC in this project to avoid DST bugs. However, I manualy convert lookups using an hardcoded settings.LOCAL_TIME_ZONE, which targets CET. The database also stores either UTC (mysql) or CET + 1 (postgres).

I think it would be best to have all data stored as UTC. It would also drop the need for having the 'hack' LOCAL_TIME_ZONE.

Originally reported by: Dennis Siemensma (Bitbucket: dennissiemensma, GitHub: dennissiemensma)

It should be possible for non-technical users to export certain data points in CSV format.

Data points may be:

• Electricity (1 or 2) delivered or returned.
• Gas delivered.
• Electricity / gas cost
• Total cost

As we should only allow export of pre-calculated date ranges, we might have to keep track of daily, weekly, monthly and even yearly statistics.

This option should be built as a django admin feature, using a custom method or something.

• Bitbucket: https://bitbucket.org/dennissiemensma/dsmr-reader/issue/2

Originally reported by: Dennis Siemensma (Bitbucket: dennissiemensma, GitHub: dennissiemensma)

For the weather app (#18) I am about to use signals to keep the apps seperated. I just realized that it should be the de facto way of compacting readings.

Currently the compactor runs in a sperate command/supervisor process, doing nothing most of the time.
Django signals will make sure to both dispose the command and also only trigger when there is actually something to do.

• Bitbucket: https://bitbucket.org/dennissiemensma/dsmr-reader/issue/19

I like to default to English to keep the documentation in sync with the code base. However, most users will be Dutch so I should make the README available in Dutch as well.

Might worth checking out:

• https://www.transifex.com/signup/
• https://poeditor.com/

Originally reported by: Dennis Siemensma (Bitbucket: dennissiemensma, GitHub: dennissiemensma)

• Bitbucket: https://bitbucket.org/dennissiemensma/dsmr-reader/issue/1

Originally reported by: Dennis Siemensma (Bitbucket: dennissiemensma, GitHub: dennissiemensma)

These migrations in dsmr_stats are unnamed:

• 0002_auto_20151113_1936
• 0003_auto_20151113_2012
• 0004_auto_20151118_2120

• Bitbucket: https://bitbucket.org/dennissiemensma/dsmr-reader/issue/15

Originally reported by: Dennis Siemensma (Bitbucket: dennissiemensma, GitHub: dennissiemensma)

Locales are already supported but we should:

• Add trans-blocks in templates.
• Run makemessages.
• Translate PO-file.

• Bitbucket: https://bitbucket.org/dennissiemensma/dsmr-reader/issue/14

The following data is stored for every base DSMR reading:

        "0-0:96.14.0": "electricity_tariff",
        "0-0:96.7.21": "power_failure_count",
        "0-0:96.7.9": "long_power_failure_count",
        "1-0:32.32.0": "voltage_sag_count_l1",
        "1-0:52.32.0": "voltage_sag_count_l2",
        "1-0:72.32.0": "voltage_sag_count_l3",
        "1-0:32.36.0": "voltage_swell_count_l1",
        "1-0:52.36.0": "voltage_swell_count_l2",
        "1-0:72.36.0": "voltage_swell_count_l3",

The data itself might be interesting, but should not get stored every time. Therefor we should wipe it from the data structure and have dsmr_datalogger.services.telegram_to_reading() ignore it.

The best solution I can come up with is to have dsmr_datalogger.services.read_telegram(), which is the data source for telegram_to_reading(), send a signal every time a telegram is read. Some other app (probably dsms_stats) should act upon it, storing it to a daily table (or Solo-model) once a day.

This will:

• Improve reliability for parsing telegrams.
• Save lots of data (I currently have 500.000+ readings over just 60 days, times the number of fields above)
• Separate extra data not required for core functions.

Github currently analyzes this project as:
JavaScript 64.9% CSS 25.1% Python 7.5% HTML 2.3% Shell 0.2%

Around 65% of javascript seems a bit to much. I probably added *-dev packages or versions as well. Should get checked out and cleaned up, if possible.

Originally reported by: Dennis Siemensma (Bitbucket: dennissiemensma, GitHub: dennissiemensma)

Currently the stats-app consists of both the datalogger and consumption. The datalogger should have its own app, settings and trigger signals to consumption app.

• Bitbucket: https://bitbucket.org/dennissiemensma/dsmr-reader/issue/21

Originally reported by: Dennis Siemensma (Bitbucket: dennissiemensma, GitHub: dennissiemensma)

My Pi just wouldn't install due to broken or laggy mirrors. I fixed it by using some german university as mirror (http://www.raspbian.org/RaspbianMirrors).

We might want to have a read-to-use installer-config.txt anyway, which might shorten the installation guide a lot (for the headless users). More info @ https://github.com/debian-pi/raspbian-ua-netinst#installer-customization

• Bitbucket: https://bitbucket.org/dennissiemensma/dsmr-reader/issue/17

Originally reported by: Dennis Siemensma (Bitbucket: dennissiemensma, GitHub: dennissiemensma)

• ^/nginx
• ^/postgresql
• ^/requirements
• ^/supervisor
• ^/dsmrreader/config/mysql.py
• ^/dsmrreader/config/postgresql.py

• Bitbucket: https://bitbucket.org/dennissiemensma/dsmr-reader/issue/6

Originally reported by: Dennis Siemensma (Bitbucket: dennissiemensma, GitHub: dennissiemensma)

Create test for generating and checking 100% translation.

• Bitbucket: https://bitbucket.org/dennissiemensma/dsmr-reader/issue/24

Originally reported by: Dennis Siemensma (Bitbucket: dennissiemensma, GitHub: dennissiemensma)

I do not like GIT a lot, but GitHub seems to be the preferred place for open source projects.

Import should be easy using:
https://import.github.com/new/

Clone URL:
https://[email protected]/dennissiemensma/dsmr-reader

• Bitbucket: https://bitbucket.org/dennissiemensma/dsmr-reader/issue/28

http://choosealicense.com/licenses/

"Non-Commercial: Licensee may not use Software for commercial purposes. for the purpose of this license, commercial purposes means that a 3rd party has to pay in order to access Software or that the Website that runs Software is behind a paywall."

Backups should be made easy. We should check for any free online storage we can use and integrate.

For example, it would be nice when users can create or link their Dropbox, Onedrive, etc to the application, which will ensure backups.

Note sure if possible, but worth investigating.

Dropbox

https://www.dropbox.com/developers-v1/core/docs/python

OneDrive

https://github.com/onedrive/onedrive-sdk-python

Google Drive

https://developers.google.com/drive/v3/web/quickstart/python

Originally reported by: Dennis Siemensma (Bitbucket: dennissiemensma, GitHub: dennissiemensma)

But also make sure it's formatted nice, as the current version of INSTALL is just a draft. Not yet important, but it should be completed when the repo is stable enough for more publicity.

• Bitbucket: https://bitbucket.org/dennissiemensma/dsmr-reader/issue/7

Originally reported by: Dennis Siemensma (Bitbucket: dennissiemensma, GitHub: dennissiemensma)

The grouping of consumption is currently a management command argument. We should move that to the database and have some other settings as well.

Consumption grouping:

• Is now management command argument
• Is now either 10 seconds or a minute.
• Allow: 1, 2, 3, 5, 10, 15, 30 or 60 minutes.

• Bitbucket: https://bitbucket.org/dennissiemensma/dsmr-reader/issue/12

Originally reported by: Dennis Siemensma (Bitbucket: dennissiemensma, GitHub: dennissiemensma)

I recently added baud_rate settings, but it seems that older versions also use different settings for byte size and parity

DSMR 2/3

    serial_handle.baudrate = 9600
    serial_handle.bytesize = serial.SEVENBITS
    serial_handle.parity = serial.PARITY_EVEN

    serial_handle.baudrate = 115200
    serial_handle.bytesize = serial.EIGHTBITS
    serial_handle.parity = serial.PARITY_NONE

The buad rate setting should be converted to a predefined DSMR-version choice.

• Bitbucket: https://bitbucket.org/dennissiemensma/dsmr-reader/issue/36

Originally reported by: Dennis Siemensma (Bitbucket: dennissiemensma, GitHub: dennissiemensma)

Implement external API. Docs @ https://www.mindergas.nl/member/api

Note. Terms of use:

#!python

Gebruiksvoorwaarden

Er zijn een paar voorwaarden verbonden aan het gebruik van de API:

* De API is bedoeld voor privegebruik, dus om jouw meterstanden naar je eigen account te uploaden.
* Commercieel gebruik is niet toegestaan tenzij anders overeengekomen met ons.
* Alleen normaal gebruik van de API: Het uploaden van een meterstand 1x per 1 of meer dagen.
* We behouden ons het recht voor om de toegang tot de API te kunnen blokkeren.

• Bitbucket: https://bitbucket.org/dennissiemensma/dsmr-reader/issue/10

Originally reported by: Dennis Siemensma (Bitbucket: dennissiemensma, GitHub: dennissiemensma)

Add:

• AUTHOR
• CREDITS
• LICENCE
• DEPENDENCIES

• Bitbucket: https://bitbucket.org/dennissiemensma/dsmr-reader/issue/8