Issue with v1.10.2 about ha-ferroamp HOT 28 CLOSED

We ignored all lower values so I guess that's why you don't see any of this before updating to latest.

from ha-ferroamp.

@henricm haven't reported this to ferroamp, will do that now :)

from ha-ferroamp.

I also noticed this yesterday when i restarted by hub 😊

from ha-ferroamp.

Hmmm, maybe we should revert that change @henricm ? What versions do you guys have on your EnergyHubs? Just trying to find some kind of pattern here on why/when this is happening.

from ha-ferroamp.

I am running 5.0.3 if that makes a difference. Not sure where these erroneous values comes from as it is just one in the serie, then back to correct values again. There used to be similar issues before it was set to "total_Increasing", but that was fixed more than a year ago and it has worked like a charm ever since.

It would not be so bad if it did not mess up the daily, monthly, yearly aggregates.

from ha-ferroamp.

How does it continue? Will EH continue to report 22 MWh and increasing from there or does it go back to 27 MWh?

Could you post more data and possibly also how this looks in energy cloud?
When sensors are affected?

Home assistant uses 10% margin when trying to determine if there has been a reset or if there are error values for total-increasing sensors, so that's the reason for using the same.
Also, even if an actual reset happens it may take time before a system is back up again. And the counters may have moved during this time so I don't think we can use eg 90%.

If we should change this back it would be good to know more details when this happens and the nature of the incorrect values. Rather than a % maybe we may also need to use a time frame?

from ha-ferroamp.

Maybe we could ignore outliers somehow?

from ha-ferroamp.

As mentioned earlier, not sure why it occurred, never actually seen it before, although that does not mean much as you seldom look at the actual data -- row by row. But, at least in these two instances, it was just one value that stood out, before and after the series is correct.

from ha-ferroamp.

Thanks @u1malmberg!
Would be good to know how "wrong" the data could be. I guess ferroamp has no clue about this issue?

from ha-ferroamp.

Maybe we could ignore outliers somehow?

Is it doable to determine this scenario where a value is off a few seconds - and ignore it in that case? We do some average calculation already for values entering in between the configured fetch interval, right? Maybe it can bee hooked in there somehow?

from ha-ferroamp.

hmm, I cannot see any traces of this in the reports in FA portal -- although they are on a fairly high level.

Just looked at the dataset from 1st of jan this year up until today, exported it into csv and these two instances from today and yesterday are the only entries that are "off". well.. based on sorting all the rows and checking from lowest ... just these two.

Strange, I mean what are the odds!? only two errors this year are since after I updated the integration. I did update from 1.9-ish!? not sure exact "from" version (is it possible to check in HA somewhere!?), could it be one of the other patches that somehow could mess up the data?

Also, last two days have been the sunniest this year :) , so I guess we cannot rule out that the errors from FA could be more frequent under heavy load or something.

from ha-ferroamp.

A quick fix for this particular case would be to change the 10 % limit to 25 % instead and hope that's about how wrong the data can get.
I don't think I've seen this issue except in cases where there has been some issue with my EH. I recently got the SD card replaced for example, resetting some (not all) counters.

from ha-ferroamp.

I am running 5.3.2. I will paste a screenshot of my battery. It is the same with the values from the hub 😊

from ha-ferroamp.

from ha-ferroamp.

Maybe we could ignore outliers somehow?

Is it doable to determine this scenario where a value is off a few seconds - and ignore it in that case? We do some average calculation already for values entering in between the configured fetch interval, right? Maybe it can bee hooked in there somehow?

I'm thinking we could have a sliding average of the last minute or so and compare with that?

from ha-ferroamp.

The big question is if to leave it also for tomorrow to see if I get the same results around 1400 again! 😃

from ha-ferroamp.

Thanks! Have you reported this to ferroamp? Doesn't look healthy.

from ha-ferroamp.

Created a ticket to ferroamp now. Noticed that these issues only occur on data from SSO + ESO, not the hub itself.

from ha-ferroamp.

Maybe we could ignore outliers somehow?

Is it doable to determine this scenario where a value is off a few seconds - and ignore it in that case? We do some average calculation already for values entering in between the configured fetch interval, right? Maybe it can bee hooked in there somehow?

I'm thinking we could have a sliding average of the last minute or so and compare with that?

Maybe not work that well with the scenario @robinostlund is experiencing where values dropping to 0. I'm thinking that maybe this is happening to @u1malmberg too, but the average-calculation we do in between updates maybe hides this fact?

One idea could be to to max() of the x latest values. Then it will take a few updates before an actual reset is taken into account, but a few error values won't affect the sensor.

from ha-ferroamp.

or should we just add a if >= 0 skip or something? As it seems to go to zero when it is rebooted? 😄
As SSO and ESO is never 0 as they are tested in production so they have at least 3-5 kWh in their internal counters.

from ha-ferroamp.

Maybe we could ignore outliers somehow?

Is it doable to determine this scenario where a value is off a few seconds - and ignore it in that case? We do some average calculation already for values entering in between the configured fetch interval, right? Maybe it can bee hooked in there somehow?

I'm thinking we could have a sliding average of the last minute or so and compare with that?

Maybe not work that well with the scenario @robinostlund is experiencing where values dropping to 0. I'm thinking that maybe this is happening to @u1malmberg too, but the average-calculation we do in between updates maybe hides this fact?

One idea could be to to max() of the x latest values. Then it will take a few updates before an actual reset is taken into account, but a few error values won't affect the sensor.

Not sure how that would be handled with regards to average values etc. I think I'm more inclined to do the skip 0-values as @robinostlund suggested. That is if the actual low value seen by @u1malmberg was 0 (it was averaged out in the graphs of course and not sure how to know what the actual value was).

from ha-ferroamp.

Was going to tell you that I did not receive any wrong data today around 1400 (as usual), but then it happened around 16:30-ish instead. About the same "distance", ~20%, away from the correct values.

@argoyle, I am not seeing 0 values, just 20%-ish off values.

The 25% limit discussed above might work for me, but then again, since we do not know why the data is wrong, then we do not really know what the diff could be in the future...

from ha-ferroamp.

now after sending that, I can see that there are actually two bad datalines.... the one after is 26k, so it should have been filtered out if it was not for the 22k row just infront (I guess) -- cancelling the *1.1 fix!?

from ha-ferroamp.

Maybe we could ignore outliers somehow?

Is it doable to determine this scenario where a value is off a few seconds - and ignore it in that case? We do some average calculation already for values entering in between the configured fetch interval, right? Maybe it can bee hooked in there somehow?

I'm thinking we could have a sliding average of the last minute or so and compare with that?

Maybe not work that well with the scenario @robinostlund is experiencing where values dropping to 0. I'm thinking that maybe this is happening to @u1malmberg too, but the average-calculation we do in between updates maybe hides this fact?

One idea could be to to max() of the x latest values. Then it will take a few updates before an actual reset is taken into account, but a few error values won't affect the sensor.

Not sure how that would be handled with regards to average values etc. I think I'm more inclined to do the skip 0-values as @robinostlund suggested. That is if the actual low value seen by @u1malmberg was 0 (it was averaged out in the graphs of course and not sure how to know what the actual value was).

Maybe worth a try - if it's always 0-values that's causing this issue. Regardless this needs to be done in the average calc right? Since it's already too late if the incorrect values have been included in the average sum?

Otherwise I was thinking we could keep a list of the latest 10 or so values. Just as we keep values in between the configured update interval to do the average calc. And use Max instead of average for total-increasing energy counters which should make more sense than average calc. That could then also work if other types of junk values can appear.

But it will of course complicate things a bit 🙂.

from ha-ferroamp.

Was going to tell you that I did not receive any wrong data today around 1400 (as usual), but then it happened around 16:30-ish instead. About the same "distance", ~20%, away from the correct values.

@argoyle, I am not seeing 0 values, just 20%-ish off values.

The 25% limit discussed above might work for me, but then again, since we do not know why the data is wrong, then we do not really know what the diff could be in the future...

You do not see all values that EH sends since the sensor in HA is only updated every 30 second but values from EH arrives more often than that. The values that arrives in between are collected and used in average calculation every 30 second (or the interval you've configured). To see all values maybe you could connect an mqtt explorer or if it's possible to enable more logging of mqtt messages in HA?

from ha-ferroamp.

@henricm what version are you running? they did some changes regarding the measurement in latest version: https://support.ferroamp.com/en/support/solutions/articles/47001241579-software-release-5-3-2

from ha-ferroamp.

@robinostlund I'm on 5.0.3

from ha-ferroamp.

Closing. Please reopen or file new issue if the problem comes back.

from ha-ferroamp.