emoncms / postprocess Goto Github PK

Post Processing module for emoncms

License: GNU Affero General Public License v3.0

PHP 86.39% JavaScript 13.56% Shell 0.06%

postprocess's Introduction

Emoncms

Emoncms is an open-source web application for processing, logging and visualising energy, temperature and other environmental data and is part of the OpenEnergyMonitor project.

Requirements

PHP (tested with 8.1.12)
MySQL or MariaDB (tested with 10.5.15)
Apache (tested with 2.4.54)
Redis* (tested with 6.0.16)

*Redis is recommended because it reduces the number of disk writes and therefore prolongs disk life (noticeably on SD cards e.g. Raspberry Pi). Some input-processors also require Redis and fail silently if Redis is not installed. Some environments such as shared hosting or as far as we have tried Windows servers don't support Redis hence why Emoncms has a fall back mode that allows core operation without Redis.

Documentation

View the Emoncms documentation at: https://docs.openenergymonitor.org/emoncms

Design

Emoncms timeseries database design (feed storage)

Other

Emoncms Terminology

Input: An incoming datasource. Each input has an associated "node" identifier and a "key" sub-identifier. Inputs are entry points, only the last value and time of the input is recorded. To record historic data a feed needs to be created from an input.
Input: Node: A grouping identifier for an input or feed.
Input: Key: A sub-identifier for items within each Node.
Input process list (or input processing): A list of processes* performed sequentially on each input value as it is received on that input.
Process: A function that can be attached to the process list of an input to change the value or to save the value to a feed*.
Feed: A place where data is recorded, a time-series of datapoints. The standard time-series databases used by Emoncms are PHPFina and PHPTimeSeries and were written as part of the Emoncms project.

For a description of what each input process does in Emoncms, see the helper note within the Emoncms input processing configuration interface.

Emoncms.org API Reference

Install

Emoncms is designed and tested to run on either Ubuntu Linux (Local, Dedicated machine or VPS) or RaspberryPi OS. It should work on other Debian Linux systems though we dont test or provide documentation for installation on these.

We do not recommend and are unable to support installation on shared hosting or XAMPP servers, shared hosting in particular has no or limited capabilities for running some of the scripts used by emoncms. There is now a large choice of low cost miniature Linux VPS hosting solutions that provide a much better installation environment at similar cost.

Recommended:

docker standalone container

An easy way to start with emoncms is to use the all-in-one docker container

A pipeline using github actions is producing builds with latest emoncms stable version for different architectures : amd64, arm64, arm/v7

These docker images, based on the alpine linux distribution, are designed for iot. Images are tagged using alpine and emoncms versions, for example alpine3.19_emoncms11.4.11.

The images have onboard :

the mariadb and redis databases,
the mosquitto mqtt broker,
the main modules : graph, sync, backup, dashboard and app,
the workers : emoncms_mqtt, service-runner and feedwriter.

You can easily :

deactivate the low-write
use an external broker.

To pull the latest image for testing :

sudo docker pull alexjunk/emoncms

More on https://emoncms-docker.github.io

Experimental

not currently up to date

Multi-platform using Docker Container

Modules

Modules can be installed by downloading or git cloning into the emoncms/Modules folder. Be sure to check for database updates in Administration menu after installing new modules. The following core modules are included on the emonSD image:

Graph module - Advanced graphing module that integrates with the emoncms feed list, highly recommended; examples of use can be found in emoncms guide [1][2][3][4].
Device module - Automatically configure inputs and feeds using device templates.
Dashboards module - Required for creating, viewing and publishing dashboards.
App module - Application specific dashboards e.g. MyElectric, MySolar.
Config - In-browser emonhub.conf editor and emonhub.log log viewer. Use git clone to install.
Wifi module - Wifi configuration interface designed for use on the emonPi
Raspberry Pi Backup / Restore module (emonPi / emonBase)
Sync module
Usefulscripts: Not strictly a module, more a collection of useful scripts for use with emoncms.
DemandShaper module - Schedule smartplugs, EmonEVSE smart EV chargers, heatpumps to run at best time in terms of: carbon, cost, grid strain. Based on day ahead forecasts.

There are many other available modules such as the event module and openbem (open source building energy modelling module): check out the Emoncms repo list.

Branches

master - The latest and greatest developments. Potential bugs, use at your own risk! All pull-requests should be made to the master branch.
stable - emonPi/emonBase release branch, regularly merged from master. Slightly more tried and tested. See release change log.

Tools

PHPFina data file viewer - Easily explore phpfina timeseries feed engine data files directly without a full Emoncms installation. Useful for checking backups and archived data.

Android App

Google Play

GitHub Repo

Development Forum

More information

Cloud hosted platform - http://emoncms.org
OpenEnergyMonitor Forums
OpenEnergyMonitor Homepage

postprocess's People

Contributors

Stargazers

Watchers

Forkers

wujason0416 pb66 neilw bluehawk simsasaile alexandrecuer emrysr reedy cassman42 ydraig cjthuys energylocal

postprocess's Issues

powertokwh: Missing data not handled

The postprocess powertokwh seems to have an error: The calculated energy continues to increase during a gap in the power data…

Already mentioned in the forum (https://community.openenergymonitor.org/t/bug-in-postprocess/7910), but this is obviously the better place to track bug reports.

Failed to sum two phptimeseries feeds

Trying to sum two phptimeseries feeds files without any user facing errors, console reports incorrect feed engine

Automate postprocess run

https://community.openenergymonitor.org/t/automate-postprocess/9732

trimfeedstart messed up feed, new data appears in future

I ran trimfeedstart on one of my feeds, and it worked well, it removed the old data I didn't want. Unfortunately, it seems like it messed up the feed, and now new data is being added in the future.

On Mar 7th, I trimmed the feed to start on Feb 17, removing about a month worth of messy data. That part worked, but now new data is showing up on Apr 8th, rather than Mar 8th. Note, it's not always one month of offset. I've tested other feeds, and the amount of "future offset" seems to be related to how much data was trimmed.

I think I've identified the problem (not updating npoints in the feed metadata after trimming the feed), and will be submitting a fix.

introducing python processes ?

@TrystanLea : would you be happy to also introduce python processes in postprocess ? right now, only php processes are supported, that's enough for many things, but with python processes, we could take advantages of the numpy library, which is very powerfull :-)

I think it could be quite simple to support python processes that but I wanted your opinion...

I recently investigate how to subclass the numpy ndarray object
cf https://numpy.org/devdocs/user/basics.subclassing.html

Here is a first proof of concept of a basic pyFina object deriving from the numpy ndarray object. No averaging in that class, but we could introduce some quite easily IMO

import numpy as np
import struct
import os
import math

class PyFina(np.ndarray):

    def __new__(cls, id, dir, start, step, npts):
        """
        decoding the .meta file

        id (4 bytes, Unsigned integer)
        npoints (4 bytes, Unsigned integer, Legacy : use instead filesize//4 )
        interval (4 bytes, Unsigned integer)
        start_time (4 bytes, Unsigned integer)

        """
        with open("{}/{}.meta".format(dir,id),"rb") as f:
            f.seek(8,0)
            hexa = f.read(8)
            aa= bytearray(hexa)
            if len(aa)==8:
                decoded=struct.unpack('<2I', aa)
            else:
                print("corrupted meta - aborting")
                return
        meta = {
                 "interval":decoded[0],
                 "start_time":decoded[1],
                 "npoints":os.path.getsize("{}/{}.dat".format(dir,id))//4
               }
        """
        decoding and sampling the .dat file
        values are 32 bit floats, stored on 4 bytes
        to estimate value(time), position in the dat file is calculated as follow :
        pos = (time - meta["start_time"]) // meta["interval"]
        Nota : no NAN value - if a NAN is detected, the algorithm will fetch the first non NAN value in the future
        """
        verbose = False
        obj = np.zeros(npts).view(cls)

        end = start + (npts-1) * step
        time = start
        i = 0
        with open("{}/{}.dat".format(dir,id), "rb") as ts:
            while time < end:
                time = start + step * i
                pos = (time - meta["start_time"]) // meta["interval"]
                if pos >=0 and pos < meta["npoints"]:
                    #print("trying to find point {} going to index {}".format(i,pos))
                    ts.seek(pos*4, 0)
                    hexa = ts.read(4)
                    aa= bytearray(hexa)
                    if len(aa)==4:
                      value=struct.unpack('<f', aa)[0]
                      if not math.isnan(value):
                          obj[i] = value
                      else:
                          if verbose:
                              print("NAN at pos {} uts {}".format(pos, meta["start_time"]+pos*meta["interval"]))
                          j=1
                          while True:
                              #print(j)
                              ramble=(pos+j)*4
                              ts.seek(ramble, 0)
                              hexa = ts.read(4)
                              aa= bytearray(hexa)
                              value=struct.unpack('<f', aa)[0]
                              if math.isnan(value):
                                  j+=1
                              else:
                                  break
                          obj[i] = value
                    else:
                      print("unpacking problem {} len is {} position is {}".format(i,len(aa),position))
                i += 1
        """
        storing the "signature" of the "sampled" feed
        """
        obj.start = start
        obj.step = step

        return obj

    def __array_finalize__(self, obj):
        if obj is None: return
        self.start = getattr(obj, 'start', None)
        self.step = getattr(obj, 'step', None)

    def timescale(self):
        """
        return the time scale of the feed as a numpy array
        """
        return np.arange(0,self.step*self.shape[0],self.step)

PyFina object creation in python is very easy. Here is an example with a feed number 66 starting on timestamp 1577404800, fetching the original recording and sampling a point each half an hour, for a duration of 200 hours :

start=1577404800
id = 66
dir = "/home/alexandrecuer/BIOS/labo/phpfina"
feed = PyFina(id,dir,start,10,72000)
feedS1 = PyFina(id,dir,start,1800,400)

import matplotlib.pylab as plt
plt.subplot(111)
plt.ylabel('instant power W')
plt.xlabel('time in seconds')
plt.plot(feed.timescale(),feed,label="original recording",color="orange")
plt.plot(feedS1.timescale(),feedS1,'x',color="blue",label="sampling with step equal to {} s".format(feedS1.step))
plt.plot(feedS1.timescale(),feedS1,color="blue")
plt.legend()
plt.show()

the below graph created with matplotlib is just for the example. On the Rpi of course, this would not be possible to do this

all calculations could be made using numpy,
The addition of two feeds feed1 and feed2, assuming sampling them with an equal step and from the same start, could be simply :

feed = feed1+feed2

Finally, the creation of a Fina feed could be possible with some simple methods like the following ones (plus the SQL command to inject the feed in the mariadb feed table)

def createMeta(nb,start,step,dir=dir):
    """
    create meta given :
    - a feed number
    - a unixtimestamp as start
    - a step
    """
    f=open("{}/{}.meta".format(dir,nb),"wb")
    data=np.array([0,0,step,start])
    format="<{}".format("I"*len(data))
    bin=struct.pack(format,*data)
    f.write(bin)
    f.close()

def createFeed(nb,data,dir=dir):
    """
    create a dat file given :
    - a feed number
    - a numpy vector of data
    """
    f=open("{}/{}.dat".format(dir,nb),"wb")
    format="<{}".format("f"*len(data))
    bin=struct.pack(format,*data)
    f.write(bin)
    f.close()

of course you need numpy to be installed

pip3 install numpy

Cannot run post process on VPS

Hi
I installed emonCMS on a VPS and it runs fine bar for the post process. I get this error

sudo: no tty present and no askpass program specified

How do I fix this please?

Invalid Node names

Emoncms V9.9.8 It appears that at least several actions in the Post Process module create invalid Feed names. Actually the Feed name itself is fine it is just they are placed in the “NoGroup”. This “NoGroup” appears to be invalid as any Feeds within this can’t be graphed.

I have tested with “powertokwh” and “mergefeeds”.

I haven’t found any way to rename Nodes or move feeds to an existing Node, so it would appear the best fix would be to allow the selection of an existing Node as part of the Post Process interface.

I tried editing/correcting the Feed name via the Feeds table in the MySQL data base. Even though I was able to change the name in the database it did not change in the local Emoncms.

PHP 7.3 issue with hyphen on regular expression

Hi,
there is an issue with PHP7.3 and hyphen in regex
check emoncms/emoncms#1180 for reference

postprocess.diff.txt

Documentation about post processes

Is there any documentation about what each processor does ?

Battery simulator process eats all the memory and causes the machine to crash.

When running the battery simulator it caused the Pi to crash. Having eaten all of the memory and caused various things to die!

My data goes from May 2016 - hence the process starts 31,449,986 data points behind!

I've looked at the code and added a new method in common.php, this will cause the existing buffered data to be written out
and the buffer cleared - so that it doesn't consume all the memory - it also makes the process significantly faster.

UI render failure

I have a weird issue, the UI does not render properly:

I've attached the HTML - it appears to contain unprocessed template data.
postprocess.html.zip

I'm not sure how to debug it.

Running process feedback required

Currently, when a process is running there is no user feedback

Parse error: syntax error, unexpected '=>' in latest version of postprocess

I've just updated the latest version of postprocess on an emonPi running emonSD-21Jul21 with PHP Version 7.3.31-1~deb10u1` and I get the following error

Parse error: syntax error, unexpected '=>' (T_DOUBLE_ARROW), expecting ')' in /opt/emoncms/modules/postprocess/postprocess-module/postprocess_model.php on line 222

This is the line in question which looks ok

usort($processes, fn($a, $b) => $a['order'] <=> $b['order']);

https://github.com/emoncms/postprocess/blob/master/postprocess-module/postprocess_model.php#L222C8-L222C70

However, on an emonPi running emonSD-10Nov22 with PHP Version 8.1.13 it works fine.