High CPU Utilization and Temperature about fritzinfluxdb HOT 41 CLOSED

Hi @bb-Ricardo

Thanks for your information. I didn't run Influx v2 at the moment. I use InfluxDB v1.8.10 because I thought it could be, because I setup a database inside the bucket. This means, that I have the utilization and temperature problem with v1.8.10 and v2.22.2.

I will now test with your former version of fritzinflux. It's now running and I will wait for some more data. Will come back then.

from fritzinfluxdb.

you can disable the collection of the Lua values here:
https://github.com/karrot-dev/fritzinfluxdb/blob/rewrite-adding-async/fritzinfluxdb.py#L96

Just comment it out and then the Lua data won't be requested.

Thank you for looking into this.

maybe cutting down the precession from "ms" to "s" helps.

from fritzinfluxdb.

you can disable the collection of the Lua values here: https://github.com/karrot-dev/fritzinfluxdb/blob/rewrite-adding-async/fritzinfluxdb.py#L96

Just comment it out and then the Lua data won't be requested.

I will give it a try...

from fritzinfluxdb.

I change the precision to "s" in the branch. Can you pull the the branch again and give it a try?

Thank you

already did... now running the container... will come back to you later...

from fritzinfluxdb.

now running with this lua services... maybe with this I will not run in high temp and utilization ;o)

fritzbox_services.append(
    {
        "name": "System Stats",
        "page": "ecoStat",
        "value_instances": {
            "cpu_temp": {
                "data_path": "data.cputemp.series.0.-1",
                "type": int
            },
            "cpu_utilization": {
                "data_path": "data.cpuutil.series.0.-1",
                "type": int
            },
            "ram_usage_fixed": {
                "data_path": "data.ramusage.series.0.-1",
                "type": int
            },
            "ram_usage_dynamic": {
                "data_path": "data.ramusage.series.1.-1",
                "type": int
            },
            "ram_usage_free": {
                "data_path": "data.ramusage.series.2.-1",
                "type": int
            }
        }
    })




# every 2 minutes
fritzbox_services.append(
    {
        "name": "Active network hosts",
        "page": "netDev",
        "params": {
            "useajax": 1,
            "xhrId": "all",
            "xhr": 1,
            "initial": True
        },
        "interval": 120,
        "value_instances": {
#            "active_host": {
#                "data_path": "data.active",
#                "type": list,
#                "next": extract_single_host
#            },
            "num_active_host": {
                "type": int,
                "value_function": lambda data: len(data.get("data", {}).get("active", [])),
            }
        }
    }
)

from fritzinfluxdb.

hi @bb-Ricardo looks good... with the reduced lua values I didn't face the temp and utilization problem. for me it's fine and I close this case...

from fritzinfluxdb.

Hey @bb-Ricardo

Nice to hear. I will definitely do it. But I need a bit time, I have limited access to my pi at the moment.

As soon as I set up the actual version I will give you a response.

BR
Ronald

from fritzinfluxdb.

Hello @ronaldn1969,

I noticed some similar issues that loading the dashboard got quite slow as soon as there was more data collected.

I would NOT think that this is an issue of fritzinfux and more a Influx/grafana issue.

First I blamed it on the shard group duration which was set to to something like 168 hours instead of 24. Now on creation of new buckets with the fritzinfluxdb they get set to 24H. -> d314333#diff-7e286a2207fa26a8ed2169a4f16250b27fb2564f10118236da39f3f7dc0ff006L210

Maybe you can check the shard group duration for your bucket in influxdb.

from fritzinfluxdb.

morning @bb-Ricardo

after collecting data over the weekend and check this morning, I didn't faced the issue on my Pi3 with the former version of fritzinflux...

the data appears directly and no impact to cpu utilization and temperature...

it's looks like, that this is maybe related to the "disordered" database structure or the lua values. Is it possible for me, to switch off the lua services? are the lua values responsible for the disordered database structure or is this because of influxdb v2?

from fritzinfluxdb.

@bb-Ricardo

Setup the latest version from Git and comment out L96.

I got following message...

 2022-05-02 08:06:23,349 - INFO: Starting fritzinfluxdb v1.0.0-alpha2 (2022-04-11)
 2022-05-02 08:06:23,351 - INFO: Done reading config files
 2022-05-02 08:06:23,416 - INFO: Successfully parsed config
 2022-05-02 08:06:23,436 - INFO: Database 'fritzbox' not found, trying to create it
 2022-05-02 08:06:23,491 - INFO: Connection to InfluxDB 1.8.10 established and database present
 2022-05-02 08:06:25,644 - INFO: Successfully established FritzBox TR-069 session
 2022-05-02 08:06:26,093 - INFO: Successfully established FritzBox Lua session
 2022-05-02 08:06:26,093 - INFO: Successfully connected to FritzBox '192.168.178.1' (fritz.box) Model: FRITZ!Box 7590 - FW: 7.29
 2022-05-02 08:06:26,096 - INFO: Starting main loop
!! 2022-05-02 08:06:27,010 - INFO: Requested invalid action 'GetInfo' for service: WANIPConnection:1
!! 2022-05-02 08:06:27,010 - INFO: Querying action 'GetInfo' will be disabled
!! 2022-05-02 08:06:27,010 - INFO: All actions for service 'WANIPConnection:1' are unavailable. Disabling service.

I took a look inside my fritzconnection output and find following:

Service:            WANIPConnection1
Action:             GetInfo
Parameters:

    Name                                  direction     data type

    NewEnable                                out ->     boolean
    NewConnectionStatus                      out ->     string
    NewPossibleConnectionTypes               out ->     string
    NewConnectionType                        out ->     string
    NewName                                  out ->     string
    NewUptime                                out ->     ui4
    NewLastConnectionError                   out ->     string
    NewRSIPAvailable                         out ->     boolean
    NewNATEnabled                            out ->     boolean
    NewExternalIPAddress                     out ->     string
    NewDNSServers                            out ->     string
    NewMACAddress                            out ->     string
    NewConnectionTrigger                     out ->     string
    NewRouteProtocolRx                       out ->     string
    NewDNSEnabled                            out ->     boolean
    NewDNSOverrideAllowed                    out ->     boolean

Is this is related to ":" in between?

from fritzinfluxdb.

Hi, this is a desired behaviour. WANIPConnection only works on FritzBoxes with a cable modem. Have to add a message that this is OK if the Model uses DSL and vice versa.

from fritzinfluxdb.

oh... yes... Monday morning... I see... WANPPPConnection:1 is for my FB...

Even without the Lua Service the database is not like in the old version... Is still fragmented... Maybe this is the reason for the issue. with the old unfragmented database structure I didn't face the issue. Could it be, that because of fragmentaion, Grafana/InfluxDB need more time and system ressources to display the data?

Setup: act. Fritzinfluxdb, InfluxDB v1.8.10, Grafana 8.5.0

from fritzinfluxdb.

I wouldn't call it 'fragmented'. It is more accurate compared to the old method.

maybe this page has some hints: https://docs.influxdata.com/influxdb/v2.2/write-data/best-practices/

I could try to reduce the precision to "seconds" instead of "milliseconds"

from fritzinfluxdb.

okay... understand... but I'm running v1.8.10 not v2.2

I could try to reduce the precision to "seconds" instead of "milliseconds"

if it's not to much for you... I'm here to test it...

from fritzinfluxdb.

It would reduce the points of data a bit. And it would be an easy change.

Ultimately I could reimplement the old behavior and could compare it as well.

from fritzinfluxdb.

Ultimately I could reimplement the old behavior and could compare it as well.

this means, that the database structure would be se same as in the former version? If you don't mind, let us start with the first of your suggestions ;o) maybe this solves the issue... Thanks a lot for your work, it's a great project...

from fritzinfluxdb.

I change the precision to "s" in the branch. Can you pull the the branch again and give it a try?

Thank you

from fritzinfluxdb.

Today the dashboard get a little bit slower and the CPU Utilization is around 35% and Temperature increase aboz 10°C as log as I display the dashboard.

from fritzinfluxdb.

Hi Ricardo

Unfortunately the issue is not solved by changing from ms to s....

from fritzinfluxdb.

Can you check how the data looks like in the database now?

from fritzinfluxdb.

of course... looks like before...

from fritzinfluxdb.

when you look at the time column you can see that the precision is now down to seconds.
Your view results probably from the different tags attached to each measurement.

And if you disable the lua_connection now and leave it running?

from fritzinfluxdb.

still some spaces

from fritzinfluxdb.

hi @bb-Ricardo

checked today again... without Lua I have no temp and utilization issue...

from fritzinfluxdb.

Hi,

Yes, that is correct. Just wanted to see if it improves the situation. Temp and cpu values are only available via LUa connection.

from fritzinfluxdb.

sorry... yes, what I want to say is, as log as I didn't collect Lua values, I didn't face the temperature and utilization issue, what I reported here. so, if I activate the Lua collection, I have the high temperature and utilization issue again.

from fritzinfluxdb.

Ahh, you're talking about your Raspi? I thought you ment your FritzBox.

Would be interesting to know which query causes the issue. Maybe I did something wrong creating the panels.

from fritzinfluxdb.

Yes... talking about Raspi... It happens already with this open row...

and there are only three lua values... cpu temp, utilization and log **... and I didn't belive, that there is something wrong in this panels... ;o)

But I can give it a tray... I switch on the lua collection and than run the dashboard without the lua panels. then we will see, what happens...

** and RAM and aktiv devices

Edit: Now running with lua collection and dashboard without lua panels. now have to wait... will come back to you...

from fritzinfluxdb.

Thank you for all the time and testing, this is highly valued.

from fritzinfluxdb.

morning Ricardo

It's in my own interest... And you did already a great job by publishing this project....

I took a look... The increase of temperature an utilization on the Raspi is not related to lua panels. Even without I have the issue...

from fritzinfluxdb.

It's looks like, that it is a Lua based issue. Maybe because the database have a lot more rows per timestamp instead of the old version, were it have only one row per timestamp.

from fritzinfluxdb.

Mmmmh, this is all not really good as it is a show stopper for the whole rewrite. I pretty much stopped now. Because if this continues then it is nice to have the values, you just can't look at them 🤷.

I'm out of ideas in how to enhance the data collection to not cause this issue.

from fritzinfluxdb.

I think, you can't do anything. Imho it's related to the fact, that it need one line for every data like energy consumption and active/passive hosts.

your project is absolutely great. so, my next step will be to comment out that lua data, which need more then one line and keep something like CPU temperature and utilization, number of active user and maybe the eventlog and then check again...

btw: today I took a look into my fritzinfluxdb log and found following:

 2022-05-25 11:19:37,350 - ERROR: Failed to write to InfluxDB '192.168.178.34': 400: {"error":"partial write: max-values-per-tag limit exceeded (100003/100000): measurement=\"fritzbox\" tag=\"ipv4_last_used\" value=\"1653475846\" dropped=15"}
 2022-05-25 11:21:37,738 - ERROR: Failed to write to InfluxDB '192.168.178.34': 400: {"error":"partial write: max-values-per-tag limit exceeded (100003/100000): measurement=\"fritzbox\" tag=\"ipv4_last_used\" value=\"1653475846\" dropped=15"}
 2022-05-25 11:23:37,842 - ERROR: Failed to write to InfluxDB '192.168.178.34': 400: {"error":"partial write: max-values-per-tag limit exceeded (100003/100000): measurement=\"fritzbox\" tag=\"ipv4_last_used\" value=\"1653475846\" dropped=15"}
 2022-05-25 11:25:39,628 - ERROR: Failed to write to InfluxDB '192.168.178.34': 400: {"error":"partial write: max-values-per-tag limit exceeded (100003/100000): measurement=\"fritzbox\" tag=\"ipv4_last_used\" value=\"1653475846\" dropped=15"}

do you know the meaning of it?

from fritzinfluxdb.

hey @ronaldn1969,

Thank you for testing and your encouraging words.

I might have found the issue with the high cpu utilisation. Can you pull the latest commits from the rewrite branch? I also added an updated dashboard with more information.

Thank you.

from fritzinfluxdb.

version 1.0.0 has been released

from fritzinfluxdb.

Hey @bb-Ricardo

Faster as thought. Build a docker container with version 1.0.0 from rewrite-adding-async and launched on my RasPi... Will now collected a few days and will come back and give you an update.

Edit: Checked the new Dashboard... Thanks for the rebuild an adding VPN/WLAN and modified Network Hosts.

from fritzinfluxdb.

In the reamde is a link to an prebuilt container on hub.docker.com.

Edit: but no Arm images, true. Let's see if I can build some as well.

from fritzinfluxdb.

Hi @bb-Ricardo

Thanks for the hint, I saw it later in the readme. But no problem to build it myself...

from fritzinfluxdb.

Now also multi arch linux

from fritzinfluxdb.

hi @bb-Ricardo, back from holiday and checking the cpu utilization and temperature from my pi after collecting 15 days data. both in normal condition and displaying of data is fast, when I switch between the sections or going back to dashboard.

looks like your adjustments were successful. thanks for your efforts and further development of the project.

from fritzinfluxdb.

These are great news. Thank you for all your help and testing efforts. This was very valuable.

from fritzinfluxdb.