Include Smart Home / DECT devices? about fritzinfluxdb HOT 35 CLOSED

#52 (comment):

It's great that we get the data but I have no idea right now how to visualise for example the heating data for the thermostats. Any idea? should we move this to discussions?

I am going to start collecting data from the add-homautomation branch to another bucket this weekend. I am also interested in using this data instead of my own approach.
Anyway, I can't promise that I'll find the time to enhance the dashboard within this month, but I'll try.

from fritzinfluxdb.

Nice project! Thanks because that is exactly what I was looking for. I own some of these temperature sensors and tested your code. I would like to suggest some additions to services_lua_homeauto.py

• With the name field you can identify the single DECT301 devices with the room name you configured them with
• Additional to temperature the measured and offset values are interesting when you fine tune the settings: temperature.celsius is usually the value directly measured by the senor. When radiator is on the measurement will not be the same as room temperature and always be higher. Therefore you have an offset to make some corrections. Otherwise DECT301 device will turn heating down. Caluclated temperature field should be close to room temperature. It looks like Fritz calculates this value themselves and writes it into tist field

            "ha_name": {
                "data_path": "devicelist.device",
                "type": list,
                "next": {
                    # data struct type: dict
                    "type": str,
                    "tags_function": lambda data: {"name": data.get("name")},
                    "data_path": "name"
                },
                "exclude_filter_function": lambda data: "device" not in data.get("devicelist").keys()
            },
            "ha_id": {
                "data_path": "devicelist.device",
                "type": list,
                "next": {
                    # data struct type: dict
                    "type": str,
                    "tags_function": lambda data: {"name": data.get("name")},
                    "data_path": "@id"
                },
                "exclude_filter_function": lambda data: "device" not in data.get("devicelist").keys()
            },
            "ha_identifier": {
                "data_path": "devicelist.device",
                "type": list,
                "next": {
                    # data struct type: dict
                    "type": str,
                    "tags_function": lambda data: {"name": data.get("name")},
                    "data_path": "@identifier"
                },
                "exclude_filter_function": lambda data: "device" not in data.get("devicelist").keys()
            },
           "ha_temperature_sensor": {
                "data_path": "devicelist.device",
                "type": list,
                "next": {
                    # data struct type: dict
                    "type": float,
                    "tags_function": lambda data: {"name": data.get("name")},
                    "value_function": lambda data: (
                        float((int(grab(data, "temperature.celsius")))/10)
                    ),
                    "exclude_filter_function": lambda data: "temperature" not in data.keys()
                },
                "exclude_filter_function": lambda data: "device" not in data.get("devicelist").keys()
            },
           "ha_temperature_offset": {
                "data_path": "devicelist.device",
                "type": list,
                "next": {
                    # data struct type: dict
                    "type": float,
                    "tags_function": lambda data: {"name": data.get("name")},
                    "value_function": lambda data: (
                        float((int(grab(data, "temperature.offset")))/10)
                    ),
                    "exclude_filter_function": lambda data: "temperature" not in data.keys()
                },
                "exclude_filter_function": lambda data: "device" not in data.get("devicelist").keys()
            },

Here are some screenshots from my instance running the branch (room names are directly used from the name field content)

from fritzinfluxdb.

Thanks @bb-Ricardo, will check and give you feedback

from fritzinfluxdb.

Hey, just wanted to let you know that I have started working on a "Home Automation" dashboard. I haven't found any problems yet, just a typo: I think it should read ha_switch_state instead of ha_switch_sate ... and I wonder what unit ha_powermeter_energy has, but I haven't looked it up yet.

It is written in Flux as I am having problems with InfluxQL in my setup. I'm ignoring a migration to InfluxQL for now. It will work out somehow...

from fritzinfluxdb.

Hi,

Due to the lack of a Fritz!DECT Smart home device I can't get the readings and implement the measurements.

Can you post the Lua API output here?

You need to browser to the page of the FritzBox where the values are shown. Then open your browsers developer tools and get the full json output of the data.lua endpoint. This would help a lot.

Thank you.

from fritzinfluxdb.

Sorry, I tried to find that data.lua but didn't find it anywhere. Isn't there a .lua link to call dect devices?

from fritzinfluxdb.

Something like this:
http://fritz.box/webservices/homeautoswitch.lua?sid=[SesionID]&switchcmd=getdevicelistinfos

However, this wouldn't work for me and I don't know why.
Edit: Ok I've found out why, because my login sid is always 0 (0000000000000000) as per fritz.box/login_sid.lua which is according to AVM docs invalid. So I'm stuck here...

from fritzinfluxdb.

Ahh ok, for this data it seems to be homeautoswitch.lua. Did you use the Browsers developer tools to retrieve the data?

from fritzinfluxdb.

I actually can't get to the desired data, because I can't get a valid session id, like I mentioned in the above comment. There is no way to access any lua api data, w/o a valid sid. Mine is always reverted back to 000...., which is invalid.

from fritzinfluxdb.

I may have already accomplished what you are trying to do using the TR-069 protocol.

For each device I added the following to classes/fritzbox/services_tr069.py (NewIndex: 0-3, home prefix: 0-3):

 {
        "name": "X_AVM-DE_Homeauto",
        "interval": 120,
        "actions": [{
            "name": "GetGenericDeviceInfos",
            "params": {
                "NewIndex": 0
            }
        }],
        "value_instances": {
            "NewAIN": "home0_ain",
            "NewDeviceName": "home0_devicename",
            "NewPresent": "home0_present",
            "NewTemperatureIsEnabled": "home0_temp_enabled",
            "NewTemperatureIsValid": "home0_temp_valid",
            "NewTemperatureCelsius": "home0_temp_celsius",
            "NewTemperatureOffset": "home0_temp_offset",
            "NewMultimeterIsEnabled": "home0_multimeter_enabled",
            "NewMultimeterIsValid": "home0_multimeter_valid",
            "NewMultimeterPower": "home0_multimeter_power",
            "NewMultimeterEnergy": "home0_multimeter_energy",
            "NewHkrIsEnabled": "home0_heating_enabled",
            "NewHkrIsValid": "home0_heating_valid",
            "NewHkrIsTemperature": "home0_heating_istemperature",
            "NewHkrSetVentilStatus": "home0_heating_setventil",
            "NewHkrSetTemperature": "home0_heating_settemperature",
            "NewHkrSetTemperature": "home0_heating_settemperature",
            "NewHkrReduceVentilStatus": "home0_heating_reduceventil",
            "NewHkrReduceTemperature": "home0_heating_reducetemperature",
            "NewHkrComfortVentilStatus": "home0_heating_comfortventil",
            "NewHkrComfortTemperature": "home0_heating_comforttemperature"
        }
    }

My Flux query joins home{index}_devicename with home{index}_temp_celsius and creates this using pivot:

But if you just ignore the home{index}_devicename and add names statically, you can achieve this easier, I guess.

from fritzinfluxdb.

@matthiasstock Hi, this already looks promising. Are these values are also available via the web interface?

Are you able to check if this data can be extracted via homeautoswitch.lua in JSON?

Tank you.

from fritzinfluxdb.

I am able to query the temperatures using this endpoint.

curl -s -H 'Accept: application/json' "http://fritz.box/webservices/homeautoswitch.lua?sid=${mysid}&switchcmd=getdevicelistinfos"

However, I get XML instead of JSON. Does this already help or what else should I try?

<?xml version="1.0"?>
<devicelist version="1" fwversion="7.29">
  <device identifier="09995 0733992" id="16" functionbitmask="320" fwversion="05.02" manufacturer="AVM" productname="FRITZ!DECT 301">
    <present>1</present>
    <txbusy>0</txbusy>
    <name>Zimmer 1</name>
    <battery>70</battery>
    <batterylow>0</batterylow>
    <temperature>
      <celsius>260</celsius>
      <offset>0</offset>
    </temperature>
    <hkr>
      <tist>52</tist>
      <tsoll>253</tsoll>
      <absenk>38</absenk>
      <komfort>42</komfort>
      <lock>1</lock>
      <devicelock>0</devicelock>
      <errorcode>0</errorcode>
      <windowopenactiv>0</windowopenactiv>
      <windowopenactiveendtime>0</windowopenactiveendtime>
      <boostactive>0</boostactive>
      <boostactiveendtime>0</boostactiveendtime>
      <batterylow>0</batterylow>
      <battery>70</battery>
      <nextchange>
        <endperiod>1661979600</endperiod>
        <tchange>38</tchange>
      </nextchange>
      <summeractive>1</summeractive>
      <holidayactive>0</holidayactive>
    </hkr>
  </device>
  <device identifier="09995 0733974" id="17" functionbitmask="320" fwversion="05.02" manufacturer="AVM" productname="FRITZ!DECT 301">
    <present>1</present>
    <txbusy>0</txbusy>
    <name>Zimmer 2</name>
    <battery>70</battery>
    <batterylow>0</batterylow>
    <temperature>
      <celsius>270</celsius>
      <offset>0</offset>
    </temperature>
    <hkr>
      <tist>54</tist>
      <tsoll>253</tsoll>
      <absenk>38</absenk>
      <komfort>42</komfort>
      <lock>1</lock>
      <devicelock>0</devicelock>
      <errorcode>0</errorcode>
      <windowopenactiv>0</windowopenactiv>
      <windowopenactiveendtime>0</windowopenactiveendtime>
      <boostactive>0</boostactive>
      <boostactiveendtime>0</boostactiveendtime>
      <batterylow>0</batterylow>
      <battery>70</battery>
      <nextchange>
        <endperiod>1661979600</endperiod>
        <tchange>38</tchange>
      </nextchange>
      <summeractive>1</summeractive>
      <holidayactive>0</holidayactive>
    </hkr>
  </device>
  <device identifier="11630 0248598" id="18" functionbitmask="35712" fwversion="04.16" manufacturer="AVM" productname="FRITZ!DECT 200">
    <present>1</present>
    <txbusy>0</txbusy>
    <name>K8s</name>
    <switch>
      <state>0</state>
      <mode>manuell</mode>
      <lock>0</lock>
      <devicelock>0</devicelock>
    </switch>
    <simpleonoff>
      <state>0</state>
    </simpleonoff>
    <powermeter>
      <voltage>226873</voltage>
      <power>0</power>
      <energy>434</energy>
    </powermeter>
    <temperature>
      <celsius>290</celsius>
      <offset>0</offset>
    </temperature>
  </device>
  <device identifier="11657 0271380" id="19" functionbitmask="35712" fwversion="04.17" manufacturer="AVM" productname="FRITZ!DECT 210">
    <present>1</present>
    <txbusy>0</txbusy>
    <name>Schuppen</name>
    <switch>
      <state>0</state>
      <mode>auto</mode>
      <lock>0</lock>
      <devicelock>0</devicelock>
    </switch>
    <simpleonoff>
      <state>0</state>
    </simpleonoff>
    <powermeter>
      <voltage>223379</voltage>
      <power>0</power>
      <energy>107357</energy>
    </powermeter>
    <temperature>
      <celsius>245</celsius>
      <offset>0</offset>
    </temperature>
  </device>
</devicelist>

from fritzinfluxdb.

Hi,

well, this is not really what I expected. But a simple google search brought this: https://avm.de/fileadmin/user_upload/Global/Service/Schnittstellen/AHA-HTTP-Interface.pdf

Not sure If I'm willing to implement yet another Interface.

from fritzinfluxdb.

I guess this is in popular demand. Let's see if we can implement this in the next few weeks.

from fritzinfluxdb.

started implementing this.

from fritzinfluxdb.

Hey @SebastianChudoba, @matthiasstock, @4920441, @donnerhund,

I added the first version of support for homautomation values. Can you please checkout branch add-homautomation and let me know if if home automation values are read and written to InfluxDB? Just add -v and you will see what's written not InfluxDB.

Thank you

from fritzinfluxdb.

Hi Ricardo,
Thank you very much for the quick implementation of this function. I have just started the first attempt, but I get the following error:
Error: FritzBoxLuaService 'System Stats' instance has unsupported url_path: None

from fritzinfluxdb.

The following should fix the url_path: None problem, but the branch still doesn't work for me, yet. Maybe I'll have time to look at it this weekend.

diff --git a/fritzinfluxdb/classes/fritzbox/service_handler.py b/fritzinfluxdb/classes/fritzbox/service_handler.py
index a45d508..38644fd 100644
--- a/fritzinfluxdb/classes/fritzbox/service_handler.py
+++ b/fritzinfluxdb/classes/fritzbox/service_handler.py
@@ -154,7 +154,7 @@ class FritzBoxLuaService(FritzBoxService):

         super().__init__(service_data)

-        url_path = service_data.get("url_path")
+        url_path = service_data.get("url_path") or FritzBoxLuaURLPath.data
         if url_path == FritzBoxLuaURLPath.data:
             self.page = service_data.get("page")

from fritzinfluxdb.

Thank you for testing. Will have a look why it is not working.

from fritzinfluxdb.

My PR #50 fixes my requests, but leads to this error:

2022-09-16 17:17:25,566 - ERROR: FritzBox Lua returned: 403 : Forbidden
2022-09-16 17:17:25,566 - ERROR: FritzBox Lua returned body: None
2022-09-16 17:17:25,567 - ERROR: Unable to request FritzBox Lua service 'Home Automation', no data returned
2022-09-16 17:17:25,567 - INFO: FritzBox Lua service 'Home Automation' will be disabled.

from fritzinfluxdb.

I have changed @matthiasstock's change "by hand" in the corresponding file.
After starting ./fritzinfluxdb.py -v this time there are various DEBUG lines, in between the following error lines.
I use a Fritbox 7590 with the following firmware: 07.39-97058 BETA

2022-09-16 17:53:03,522 - ERROR: FritzBox Lua returned: 404 : Not Found
2022-09-16 17:53:03,522 - ERROR: FritzBox Lua returned body: None
2022-09-16 17:53:03,522 - ERROR: Unable to request FritzBox Lua service 'System Stats', no data returned
2022-09-16 17:53:03,522 - INFO: FritzBox Lua service 'System Stats' will be disabled.
2022-09-16 17:53:03,530 - ERROR: FritzBox Lua returned: 404 : Not Found
2022-09-16 17:53:03,530 - ERROR: FritzBox Lua returned body: None
2022-09-16 17:53:03,530 - ERROR: Unable to request FritzBox Lua service 'Energy Stats', no data returned
2022-09-16 17:53:03,530 - INFO: FritzBox Lua service 'Energy Stats' will be disabled.
2022-09-16 17:53:03,530 - DEBUG: FritzOS version 7.39 not in list of supported versions for System logs: ['7.29', '7.30', '7.31']
2022-09-16 17:53:03,538 - ERROR: FritzBox Lua returned: 404 : Not Found
2022-09-16 17:53:03,538 - ERROR: FritzBox Lua returned body: None
2022-09-16 17:53:03,538 - ERROR: Unable to request FritzBox Lua service 'System logs', no data returned
2022-09-16 17:53:03,538 - INFO: FritzBox Lua service 'System logs' will be disabled.
2022-09-16 17:53:03,538 - DEBUG: FritzOS version 7.39 not in list of supported versions for Internet connection logs: ['7.29', '7.30', '7.31']
2022-09-16 17:53:03,546 - ERROR: FritzBox Lua returned: 404 : Not Found
2022-09-16 17:53:03,546 - ERROR: FritzBox Lua returned body: None
2022-09-16 17:53:03,546 - ERROR: Unable to request FritzBox Lua service 'Internet connection logs', no data returned
2022-09-16 17:53:03,546 - INFO: FritzBox Lua service 'Internet connection logs' will be disabled.
2022-09-16 17:53:03,546 - DEBUG: FritzOS version 7.39 not in list of supported versions for Telephony logs: ['7.29', '7.30', '7.31']
2022-09-16 17:53:03,554 - ERROR: FritzBox Lua returned: 404 : Not Found
2022-09-16 17:53:03,554 - ERROR: FritzBox Lua returned body: None
2022-09-16 17:53:03,554 - ERROR: Unable to request FritzBox Lua service 'Telephony logs', no data returned
2022-09-16 17:53:03,554 - INFO: FritzBox Lua service 'Telephony logs' will be disabled.
2022-09-16 17:53:03,554 - DEBUG: FritzOS version 7.39 not in list of supported versions for WLAN logs: ['7.29', '7.30', '7.31']
2022-09-16 17:53:03,562 - ERROR: FritzBox Lua returned: 404 : Not Found
2022-09-16 17:53:03,562 - ERROR: FritzBox Lua returned body: None
2022-09-16 17:53:03,562 - ERROR: Unable to request FritzBox Lua service 'WLAN logs', no data returned
2022-09-16 17:53:03,562 - INFO: FritzBox Lua service 'WLAN logs' will be disabled.
2022-09-16 17:53:03,562 - DEBUG: FritzOS version 7.39 not in list of supported versions for USB Devices logs: ['7.29', '7.30', '7.31']
2022-09-16 17:53:03,606 - ERROR: FritzBox Lua returned: 404 : Not Found
2022-09-16 17:53:03,606 - ERROR: FritzBox Lua returned body: None
2022-09-16 17:53:03,606 - ERROR: Unable to request FritzBox Lua service 'USB Devices logs', no data returned
2022-09-16 17:53:03,606 - INFO: FritzBox Lua service 'USB Devices logs' will be disabled.
2022-09-16 17:53:03,614 - ERROR: FritzBox Lua returned: 404 : Not Found
2022-09-16 17:53:03,614 - ERROR: FritzBox Lua returned body: None
2022-09-16 17:53:03,614 - ERROR: Unable to request FritzBox Lua service 'Active network hosts', no data returned
2022-09-16 17:53:03,614 - INFO: FritzBox Lua service 'Active network hosts' will be disabled.
2022-09-16 17:53:03,622 - ERROR: FritzBox Lua returned: 404 : Not Found
2022-09-16 17:53:03,623 - ERROR: FritzBox Lua returned body: None
2022-09-16 17:53:03,623 - ERROR: Unable to request FritzBox Lua service 'Passive network hosts', no data returned
2022-09-16 17:53:03,623 - INFO: FritzBox Lua service 'Passive network hosts' will be disabled.
2022-09-16 17:53:03,623 - DEBUG: FritzOS version 7.39 not in list of supported versions for VPN Users: ['7.29', '7.30', '7.31']
2022-09-16 17:53:03,630 - ERROR: FritzBox Lua returned: 404 : Not Found
2022-09-16 17:53:03,630 - ERROR: FritzBox Lua returned body: None
2022-09-16 17:53:03,630 - ERROR: Unable to request FritzBox Lua service 'VPN Users', no data returned
2022-09-16 17:53:03,630 - INFO: FritzBox Lua service 'VPN Users' will be disabled.
2022-09-16 17:53:03,639 - ERROR: FritzBox Lua returned: 404 : Not Found
2022-09-16 17:53:03,639 - ERROR: FritzBox Lua returned body: None
2022-09-16 17:53:03,639 - ERROR: Unable to request FritzBox Lua service 'DSL Info', no data returned
2022-09-16 17:53:03,639 - INFO: FritzBox Lua service 'DSL Info' will be disabled.
2022-09-16 17:53:03,649 - ERROR: FritzBox Lua returned: 404 : Not Found
2022-09-16 17:53:03,649 - ERROR: FritzBox Lua returned body: None
2022-09-16 17:53:03,649 - ERROR: Unable to request FritzBox Lua service 'Cable Info', no data returned
2022-09-16 17:53:03,649 - INFO: FritzBox Lua service 'Cable Info' will be disabled.
2022-09-16 17:53:03,665 - ERROR: FritzBox Lua returned: 404 : Not Found
2022-09-16 17:53:03,665 - ERROR: FritzBox Lua returned body: None
2022-09-16 17:53:03,666 - ERROR: Unable to request FritzBox Lua service 'Home Automation', no data returned
2022-09-16 17:53:03,666 - INFO: FritzBox Lua service 'Home Automation' will be disabled.
2022-09-16 17:53:03,682 - DEBUG: Successfully wrote 75 measurements to InfluxDB
2022-09-16 17:53:03,682 - DEBUG: Current InfluxDB measurement buffer length: 0
2022-09-16 17:53:04,684 - DEBUG: InfluxDB data queue: No measurements found in queue
2022-09-16 17:53:04,684 - DEBUG: Current InfluxDB measurement buffer length: 0

from fritzinfluxdb.

damn, It's kinda difficult to write this stuff 'blind' without any actual real devices.

But I will have a look tonight. I have an idea. I must have implemented the url_path incorrectly.

if you run it with -vv you should also see the HTTP requests.

from fritzinfluxdb.

can you guys pull the latest add-homautomation branch and try again please?

thank you

from fritzinfluxdb.

With another fix in PR #52, this branch works for me and writes home automation data to InfluxDB.
Please note that I just took a random look at the data.

from fritzinfluxdb.

At first glance, the data from my two FRITZ!DECT devices arrive in InfluxDB :)
Now it's time to adapt the previously used Grafana Dashboard...

from fritzinfluxdb.

This is rendered from the fritzinfluxdata? Wow, nice 😬

from fritzinfluxdb.

During my test I got some error message when a DECT301 device temporary loses connection to FritzBox (could not find out why this sometimes happens so far). It seems that then some values are delivered back with "None" instead of 0:

2022-09-17 12:26:38,411 - ERROR: Unable to extract 'None' form '{ '@identifier': '09995 0906197', '@id': '16', '@functionbitmask': '320', '@fwversion': '05.02', '@manufacturer': 'AVM', '@productname': 'FRITZ!DECT 301', 'present': '0', 'txbusy': '0', 'name': 'Arbeitszimmer', 'battery': '60', 'batterylow': '0', 'temperature': {'celsius': None, 'offset': None}, 'hkr': {'tist': None, 'tsoll': None, 'absenk': None, 'komfort': None, 'lock': None, 'devicelock': None, 'errorcode': '0', 'windowopenactiv': '0', 'windowopenactiveendtime': '0', 'boostactive': '0', 'boostactiveendtime': '0', 'batterylow': '0', 'battery': '0', 'nextchange': {'endperiod': '0', 'tchange': '255'}, 'summeractive': None, 'holidayactive': None}}', got '<class 'NoneType'>'

from fritzinfluxdb.

Thank you for the testing. From I quick glace I didn't get the name and offset part. Will have a look later

from fritzinfluxdb.

The implemented temperature value is completely correct: (temperature.celsius + temperature.offset)/10. I just wanted to say that it will be useful also writing the separate values of temperature.celsius/10 and temperature.offset/10 to indfluxdb as well

from fritzinfluxdb.

Ahh, this is not a problem. Will add this as well.

from fritzinfluxdb.

Hi @schrdh,

I just pushed a new version to next-release including your suggestions regarding temperature and a fix for the ERROR messages.

Can you have a look if it behaves better now?

thank you

from fritzinfluxdb.

This seems to be mostly working with different versions of FritzOS. The "only" missing is the dashboard where I don't really know how to structure the data properly.

from fritzinfluxdb.

Thank you.

Then I might have to convert all dashboards to Flux and be done with it.

For ha_powermeter_energy the documentation states: <energy>Wert in 1.0 Wh (absoluter Verbrauch seit Inbetriebnahme)

Thank you so much.

from fritzinfluxdb.

Just fixed the typo for ha_switch_state in next-release

from fritzinfluxdb.

Hi @ll,

I consider this feature done. A dashboard (using Flux Query language) has been added: https://github.com/bb-Ricardo/fritzinfluxdb/blob/next-release/grafana/influx2_dashboards/fritzbox_home_automation_dashboard.json

Highly appreciating test and validation.

from fritzinfluxdb.