PROXMOX: Memory shortage error -> Rendered image froze about valetudo_vacuum_camera HOT 33 CLOSED

@cdrfun and @N1c093 v1.5.8 is now available, hope you will enjoy it at your disposal for any future requirements or ideas.

from valetudo_vacuum_camera.

@cdrfun really good catch thanks a lot (grazie)!
I will follow you advice and anyhow confirm the size of the array (this patch's are coming out daily and as I work alone on this.. well..)
Thanks once again for pointing this.

from valetudo_vacuum_camera.

@sca075

When I noticed this error I just increased the ram from 3gb to 4gb of the virtual machine on Proxmox. I thought it was a miss configuration on my side.
Should I install the newest update and decrease the ram to 3gb again? If anything is working fine you don't need any log-files, right?

from valetudo_vacuum_camera.

Tested also the configuration that @cdrfun use since 5 years, with 2GB ram, of course proxmox (HA) will use use the swap area, and..

two vacuum at the same time running is still possible :) so V1.5.8 will go out as scheduled between today or tomorrow morning.

from valetudo_vacuum_camera.

@sca075 Thank you for your work.
I just installed v.1.5.8 and decreased my ram from 4GB back to 3 GB. Everything seems to work fine now 👍

from valetudo_vacuum_camera.

Just from looking at it seems like this calculation is wrong - I'd guess margin + (...) would be correct.:

total_memory_mb = round((margin * (total_memory_bytes / (1024 * 1024))), 1)

https://github.com/sca075/valetudo_vacuum_camera/blob/31e9dfbd22efb4799d599534a3b7a8059958d867/custom_components/valetudo_vacuum_camera/valetudo/hypfer/image_handler.py#L636C1-L636C13

from valetudo_vacuum_camera.

@cdrfun I review the code anyway let me explain it in the details:
the aim is calculating the size on one element (in the case of the image is (y, x, 4) each element is an int32 (4 bytes) so this line return the element_size_bytes.

element_size_bytes = array.itemsize * 4  # int32 is 4 bytes

then we need the total size of the Numpy array per the element size as above:

total_memory_bytes = array.size * element_size_bytes

Total required memory for the next step, it's consider and defined at the begin of the class in the constant MEMORY_WARN_LIMIT this is what will be pass as "margin" (and it's the number of arrays that I reduced now from 12 to 2, it is possible to set 0 there to clear the warning) so we calculate number of arrays * array sizes in bytes than scaled in MiB.

total_memory_mb = round(((margin * total_memory_bytes) / (1024 * 1024)), 1)

Instead of an error will raise a warning, with that the camera should not stop completely.
v1.5.7.5 is going to be release with those changes.. on the S5 this would mean more than 800MiB should be free for the Camera to do not raise the warning..
Hope this is going to be okay also for @N1c093
If you want to disable this check for good just set the MEMORY_WARN_LIMIT that right now is hard coded but I count to enable / disable it from the options.. to 0.

from valetudo_vacuum_camera.

V1.5.7.5 released, please confirm if this will work as above described and let me know if we close this.

from valetudo_vacuum_camera.

Thanks for elaborating. Using a warning instead of an error is a good measure to actually test that number.

I'll not try to argue with you, but the number seems to be quite high. This would calculate to 17 Million elements in the array - is this correct?

from valetudo_vacuum_camera.

Please wait I made a mistake on the commits give me a few minute prior to download... the number returned is / not * 10241204 the calculation returns values in bytes and then to covert them in Mib we need to dived 1024**2 = 10241024. do you agree? This is why it looks as it is.. anyhow margin = 2 ( one margin = one array).. I will change the name for clarity to number_of_arrays.

from valetudo_vacuum_camera.

Okay now finally it is corrected..
models5: Estimated Margin of Memory usage: 800.0 MiB
models5: Available memory: 6275.3 MiB

1 * array would result in 400.0MiB
V1.5.7.5 finally out for your verification.

Edit:
models5 Camera Memory usage in GB: 0.6
this is at the end of the process when the array is resized.. hope this make now sense to you.

from valetudo_vacuum_camera.

Ok, I just wanted to double check. After installing my camera feed is gray:

Log:

2024-02-09 13:21:36.527 INFO (MainThread) [custom_components.valetudo_vacuum_camera.valetudo.MQTT.connector] Received valetudo/rocky image data from MQTT
2024-02-09 13:21:39.317 INFO (MainThread) [custom_components.valetudo_vacuum_camera.valetudo.MQTT.connector] valetudo/rocky: Extracting JSON Complete
2024-02-09 13:21:39.322 INFO (rocky_camera_0) [custom_components.valetudo_vacuum_camera.valetudo.hypfer.image_handler] rocky: Composing the image for the camera.
2024-02-09 13:21:39.322 INFO (rocky_camera_0) [custom_components.valetudo_vacuum_camera.valetudo.hypfer.image_handler] Vacuum JSon ID: 65ca0255-c792-46fd-b5b5-ace04f7ea379 at Frame 3.
2024-02-09 13:21:39.332 INFO (rocky_camera_0) [custom_components.valetudo_vacuum_camera.valetudo.hypfer.image_handler] rocky: Calculating the hash of the image.
2024-02-09 13:21:39.337 WARNING (rocky_camera_0) [custom_components.valetudo_vacuum_camera.valetudo.hypfer.image_handler] Memory shortage error: Not enough memory available (Margin: 800.0 MiB)
2024-02-09 13:21:39.338 INFO (MainThread) [custom_components.valetudo_vacuum_camera.camera] /config/www/snapshot_rocky.png: searching Snapshot image
2024-02-09 13:21:39.340 INFO (MainThread) [custom_components.valetudo_vacuum_camera.camera] rocky: Starting up ...

The system has about 700 MB of unused memory, but I'm out of time for today, so I'll revert to 1.5.7 for now. Many thanks nonetheless.

from valetudo_vacuum_camera.

okay I think I will let you guys decide when to activate or deactivate this function ;) meanwhile thanks a lot for the feedback.

from valetudo_vacuum_camera.

@cdrfun I will release v1.5.8 without CPU and memory checks I will use them only for my tests or troubleshooting.

from valetudo_vacuum_camera.

@cdrfun if you have time, could you please confirm the "main" 6e98dba
works for you?
Thanks in advance for you time and cooperation here.
I'm not done yet for the planned V1.5.8 that will hopefully thanks also to you confirmation close this issue, max next Friday we should be okay with the new release.

from valetudo_vacuum_camera.

Hey, just checked. I don't get they gray map now - but the map doesn't update, too. I'm not seeing any errors or earnings in the log:

2024-02-11 18:38:02.732 INFO (MainThread) [custom_components.valetudo_vacuum_camera.valetudo.MQTT.connector] valetudo/rocky: Received vacuum moving status.

2024-02-11 18:38:22.125 INFO (MainThread) [custom_components.valetudo_vacuum_camera.valetudo.MQTT.connector] valetudo/rocky: Received vacuum idle status.

I'll hopefully have a bit more time to dig into it tomorrow.

from valetudo_vacuum_camera.

If you can provide a log from the beginning to the end and explain what you mean by the map do not update I would be checking and trying to help you out 😉 it's a little strange.

The camera logs now provide the architecture of the system where it runs (you just post the received vacuum states change)

I suppose it can be faster if I can have the full log. Thanks in advance.

from valetudo_vacuum_camera.

@cdrfun Buongiorno,
I could do some test and the only possible reason when the image isn't updated is when the camera auto suspend because the CPU go over 80% (camera service turn ON should restore the camera) note that the calculation of the used CPU was review on the new code you downloaded, so if CPU went upper than 80% (it is unluckily) the question is on what do you run Home Assistant?

from valetudo_vacuum_camera.

@N1c093 good morning,
Would be really nice of you if you could try it too and give feedback on this topic.
I understand you faced a similar issue of what reported here.
If the problem is still there, and this modification isn't working also for you please be so kind to provide the logs of the camera. Note that the camera save the logs in a filtered way.. and the export of the those logs is possible by checking the log export function in the options menu. There will be then a zip file in www with the data (json and HA filtered logs).
I did recently removed the rooms data from the logs as well.
Please let me know also on what kind of machine you run Home Assistant.
Thanks in advance for your (both of you) cooperation.

from valetudo_vacuum_camera.

What I meant with not updating the map: the rendered Map was not updated when I sent the robot out. This morning, however, the map was updated with the last path as expected. I'll do some test later today.

I've created a log zip file and this is what I got:
rocky.zip

I'm using a VM to host Home Assistant within a proxmox host. I've allocated 2 GB of RAM and 2 CPU cores of an i5-3470 (3,2 GhHz). Looking at my CPU utilisation I hardly go above 15% - sitting at arround 10% most of the time.

from valetudo_vacuum_camera.

@cdrfun
It looks that the camera run twice according to this:

2024-02-11 20:48:20.480 INFO (MainThread) [custom_components.valetudo_vacuum_camera.camera] System Release: homeassistant, 6.1.74-haos
2024-02-11 20:48:20.480 INFO (MainThread) [custom_components.valetudo_vacuum_camera.camera] System Version: #1 SMP PREEMPT_DYNAMIC Mon Feb  5 11:44:53 UTC 2024
2024-02-11 20:48:20.480 INFO (MainThread) [custom_components.valetudo_vacuum_camera.camera] System Machine: x86_64
2024-02-11 20:48:20.480 INFO (MainThread) [custom_components.valetudo_vacuum_camera.camera] Python Version: 3.12.1
2024-02-11 20:48:20.485 INFO (MainThread) [custom_components.valetudo_vacuum_camera.camera] Memory Available: 850.8 and In Use: 1089.2
2024-02-11 20:48:20.486 INFO (MainThread) [custom_components.valetudo_vacuum_camera.camera] System Release: homeassistant, 6.1.74-haos
2024-02-11 20:48:20.486 INFO (MainThread) [custom_components.valetudo_vacuum_camera.camera] System Version: #1 SMP PREEMPT_DYNAMIC Mon Feb  5 11:44:53 UTC 2024
2024-02-11 20:48:20.486 INFO (MainThread) [custom_components.valetudo_vacuum_camera.camera] System Machine: x86_64
2024-02-11 20:48:20.486 INFO (MainThread) [custom_components.valetudo_vacuum_camera.camera] Python Version: 3.12.1
2024-02-11 20:48:20.489 INFO (MainThread) [custom_components.valetudo_vacuum_camera.camera] Memory Available: 850.9 and In Use: 1089.2
2024-02-11 20:48:23.040 INFO (MainThread) [custom_components.valetudo_vacuum_camera.valetudo.MQTT.connector] Received valetudo/rocky image data from MQTT
2024-02-11 20:48:23.041 INFO (MainThread) [custom_components.valetudo_vacuum_camera.valetudo.MQTT.connector] valetudo/rocky: Received vacuum docked status.
2024-02-11 20:48:23.041 INFO (MainThread) [custom_components.valetudo_vacuum_camera.valetudo.MQTT.connector] valetudo/rocky: Received vacuum Error: No error
2024-02-11 20:48:23.082 INFO (MainThread) [custom_components.valetudo_vacuum_camera.valetudo.MQTT.connector] Received valetudo/robo image data from MQTT
2024-02-11 20:48:23.082 INFO (MainThread) [custom_components.valetudo_vacuum_camera.valetudo.MQTT.connector] valetudo/robo: Received vacuum paused status.
2024-02-11 20:48:23.082 INFO (MainThread) [custom_components.valetudo_vacuum_camera.valetudo.MQTT.connector] valetudo/robo: Received vacuum Error: No error
2024-02-11 20:48:23.093 INFO (MainThread) [custom_components.valetudo_vacuum_camera.valetudo.MQTT.connector] valetudo/robo: Received vacuum idle status.
2024-02-11 20:48:24.127 INFO (MainThread) [custom_components.valetudo_vacuum_camera.valetudo.MQTT.connector] valetudo/rocky: Extracting JSON Complete
2024-02-11 20:48:24.146 INFO (MainThread) [custom_components.valetudo_vacuum_camera.valetudo.MQTT.connector] valetudo/robo: Extracting JSON Complete
2024-02-11 20:48:24.179 INFO (robo_camera_0) [custom_components.valetudo_vacuum_camera.valetudo.hypfer.image_handler] robo: Composing the image for the camera.
2024-02-11 20:48:24.179 INFO (robo_camera_0) [custom_components.valetudo_vacuum_camera.valetudo.hypfer.image_handler] Vacuum JSon ID: 177b7aa5-ca05-4551-96d1-435c6005de5f at Frame 0.
2024-02-11 20:48:24.189 INFO (robo_camera_0) [custom_components.valetudo_vacuum_camera.valetudo.hypfer.image_handler] robo: Calculating the hash of the image.
2024-02-11 20:48:24.189 INFO (robo_camera_0) [custom_components.valetudo_vacuum_camera.valetudo.hypfer.image_handler] robo: Empty image with background color
2024-02-11 20:48:24.210 INFO (rocky_camera_0) [custom_components.valetudo_vacuum_camera.valetudo.hypfer.image_handler] rocky: Composing the image for the camera.
2024-02-11 20:48:24.210 INFO (rocky_camera_0) [custom_components.valetudo_vacuum_camera.valetudo.hypfer.image_handler] Vacuum JSon ID: 6e38e372-7e0d-4eeb-981e-7083d66ca17f at Frame 0.
2024-02-11 20:48:24.261 INFO (rocky_camera_0) [custom_components.valetudo_vacuum_camera.valetudo.hypfer.image_handler] rocky: Calculating the hash of the image.
2024-02-11 20:48:24.262 INFO (rocky_camera_0) [custom_components.valetudo_vacuum_camera.valetudo.hypfer.image_handler] rocky: Empty image with background color
2024-02-11 20:48:24.996 INFO (robo_camera_0) [custom_components.valetudo_vacuum_camera.valetudo.hypfer.image_handler] robo: Overlapping Layers
2024-02-11 20:48:25.260 INFO (rocky_camera_0) [custom_components.valetudo_vacuum_camera.valetudo.hypfer.image_handler] rocky: Overlapping Layers
2024-02-11 20:48:25.557 INFO (robo_camera_0) [custom_components.valetudo_vacuum_camera.valetudo.hypfer.image_handler] robo: Drawing No Go area.
2024-02-11 20:48:25.564 INFO (robo_camera_0) [custom_components.valetudo_vacuum_camera.valetudo.hypfer.image_handler] robo: Completed base Layers
2024-02-11 20:48:25.564 INFO (robo_camera_0) [custom_components.valetudo_vacuum_camera.valetudo.hypfer.image_handler] robo: Copying the array.
2024-02-11 20:48:25.758 INFO (rocky_camera_0) [custom_components.valetudo_vacuum_camera.valetudo.hypfer.image_handler] rocky: Completed base Layers
2024-02-11 20:48:25.758 INFO (rocky_camera_0) [custom_components.valetudo_vacuum_camera.valetudo.hypfer.image_handler] rocky: Copying the array.
2024-02-11 20:48:25.820 INFO (robo_camera_0) [custom_components.valetudo_vacuum_camera.valetudo.hypfer.image_handler] robo: Copying the array.
2024-02-11 20:48:26.101 INFO (robo_camera_0) [custom_components.valetudo_vacuum_camera.utils.draweble] Drawing robo Robot With Angle: 82.0
2024-02-11 20:48:26.138 INFO (rocky_camera_0) [custom_components.valetudo_vacuum_camera.valetudo.hypfer.image_handler] rocky: Copying the array.
2024-02-11 20:48:26.683 INFO (rocky_camera_0) [custom_components.valetudo_vacuum_camera.utils.draweble] Drawing rocky Robot With Angle: 1.0
2024-02-11 20:48:31.599 INFO (robo_camera_0) [custom_components.valetudo_vacuum_camera.valetudo.hypfer.image_handler] Getting robo Calibrations points.
2024-02-11 20:48:31.799 INFO (rocky_camera_0) [custom_components.valetudo_vacuum_camera.valetudo.hypfer.image_handler] Getting rocky Calibrations points.
2024-02-11 20:48:31.799 INFO (rocky_camera_0) [custom_components.valetudo_vacuum_camera.camera_processing] Suspended the camera data processing for: rocky.

I'm waiting for the hardware that should arrive tomorrow to test PROXMOX. Please give me some time to determinate:

1. if there is something wrong on the logging I do.. I didn't change anything there but I will confirm.
2. Why the entry in the logs are doubled. How many camera entities are configured? I guess only one.. and it is a kind of strange that the data are duplicated. There is also no entry of any overload and this was expected, but some entry in the logs is missing and this is strange.

EDIT: Two different robots rocky and robo?? in 2Gb Ram?? with two robots around 0.57GB are needed for both cameras. I would advise to give 4Gb to HA also because this is officially the limit for it to run stable.

from valetudo_vacuum_camera.

@N1c093

Okay so here we go there is an issue raised also on this.. PROXMOX is actually to be checked but if you try the main would be appreciated. Just one robot right?

from valetudo_vacuum_camera.

By the way I got to admit that one correction that might work better from the test I did.. in the case of the camera detect that there's a possibility that Home Assistant slow down or is going hanging the camera will self turn off and then back on after one second (this means wear CPU values got) I repeat as per I can't right now confirm it on PROXMOX I can't promise this will help or solve the issue but we could try.

I need to go at customer now but later today I will post a new update of the camera.py with the mentioned measures for PROXMOX

from valetudo_vacuum_camera.

Guys I was able to reproduce the issue :) so it could be we can get it fix soon..

from valetudo_vacuum_camera.

@cdrfun I removed the CPU check and therefore it will work as v1.7.0 for you... the one thing I still do not get is, you are using PROXMOX is my understanding correct? And in your setup there are 2GB memory allocated for HA to run.. what instruction you followed for setting up the VE?

from valetudo_vacuum_camera.

Just to update I just started HA on the PROXMOX VM following this guide: https://www.derekseaman.com/2023/10/home-assistant-proxmox-ve-8-0-quick-start-guide-2.html
the current setup is this (and on this I will validate the camera).

Using Advanced Settings
Using HAOS Version: 11.5
Virtual Machine ID: 100
Using Machine Type: i440fx
Using Disk Cache: None
Using Hostname: haos11.5
Using CPU Model: KVM64
Allocated Cores: 2
Allocated RAM: 4096
Using Bridge: vmbr0
Using MAC Address: 02:[REDACTED]91
Using Vlan: Default
Using Interface MTU Size: Default
Start VM when completed: yes
Creating a HAOS VM using the above advanced settings
 ✓ Using local-lvm for Storage Location.
 ✓ Virtual Machine ID is 100.
 ✓ https://github.com/home-assistant/operating-system/releases/download/11.5/haos_ova-11.5.qcow2.xz
 ✓ Downloaded haos_ova-11.5.qcow2.xz
 ✓ Extracted KVM Disk Image
 ✓ Created HAOS VM (haos11.5)
 ✓ Started Home Assistant OS VM
 ✓ Completed Successfully!

from valetudo_vacuum_camera.

@cdrfun, @N1c093 and And right now.. I can confirm it works also on PROXMOX.

I will complete V1.5.8 within Friday, I still need to check the RAND firmware.. but actually I can upload the changes already done later today on the repository as soon I'm done will let you know.

Note VSCODE isn't running at this time.. and I will do some additional test of course.. interesting that the memory usage repot in proxmox looks mmmm quite busy.. and in HA is exactly at the same level it is on the PI4 (I did restore the backup of the PI4 test environment on this VE).

from valetudo_vacuum_camera.

@sca075 First: Thanks for your dedication and going through replicating our environment. I really appreciate your hard work.

I've set up HA around 5 years ago, I'm not sure if I can figure out why I choose 2 GB of RAM. The current X86-46 installation guide doesn't give a recommended RAM size. But looking here creating a Virtual Machine with at least 2 GB of RAM is recommended. But I'm quite sure the current documentation is different now, compared to 5 years ago.

Yeah, I forgot to mention I have two robos setup (rocky and robo) - I will remove one for testing.

I've never had issues with RAM, but I certainly can increase what's available to the VM. Right now, Proxmox reports around 45% used. Which seems to match the "active" RAM displayed in glances (HA Add-on).

I'll wait for 1.5.8 until further testing.

from valetudo_vacuum_camera.

@sca075 First: Thanks for your dedication and going through replicating our environment. I really appreciate your hard work.

I've set up HA around 5 years ago, I'm not sure if I can figure out why I choose 2 GB of RAM. The current X86-46 installation guide doesn't give a recommended RAM size. But looking here creating a Virtual Machine with at least 2 GB of RAM is recommended. But I'm quite sure the current documentation is different now, compared to 5 years ago.

Yeah, I forgot to mention I have two robos setup (rocky and robo) - I will remove one for testing.

I've never had issues with RAM, but I certainly can increase what's available to the VM. Right now, Proxmox reports around 45% used. Which seems to match the "active" RAM displayed in glances (HA Add-on).

I'll wait for 1.5.8 until further testing.

@cdrfun I really appreciate this comment, and sure "never touch a running system" is the rule number one. I don't know how Proxmox calculate the ram and there was already a discussion on discord for it as well on the forum, the code I'm currently testing is on the main repository, and please do not remove the second vacuum, we need to got it run with both vacuums ;) and also with the two GB configured as max ram.

from valetudo_vacuum_camera.

@cdrfun and @N1c093,
After the reboot of proxmox VE (the first screenshot was taken just after the first startup of the VE) the memory is correctly display. Although there is to note the following.. this is the graph with both vacuums I own in standby overnight, there is one peak (as per I tested shortly to move the S5 this morning). The memory with the HAOS11.5 is stable at around 2GB (on the VE there is just some ESP device, Visual Studio, HACS and the Camera. So probably experience done and safety wise for your implementations you may consider to use 4GB at least.

Within Friday will go out v1.5.8 and will close this issue, as per now, it is possible to confirm the camera works also on Proxmox (I will also update the docs for future references).

from valetudo_vacuum_camera.

Two vacuums running at he same time..

from valetudo_vacuum_camera.

This might be semi-related: I run home assistant on a raspberry pi 3 with 1GB. So far everything worked fine, and usually not more than around 600 MB memory is used. When trying to run this custom component, I get OOM kills of python, resulting in a crash/restart loop of home assistant. But I guess my hardware is too limited. I just wanted to leave a note here for this information.

from valetudo_vacuum_camera.

This might be semi-related: I run home assistant on a raspberry pi 3 with 1GB. So far everything worked fine, and usually not more than around 600 MB memory is used. When trying to run this custom component, I get OOM kills of python, resulting in a crash/restart loop of home assistant. But I guess my hardware is too limited. I just wanted to leave a note here for this information.

@bluesceada
Well I'm aware, I don't know if you can adjust the swap area to give to the camera a little more room to run. I think it is about 1Gb more should be fine.
Anyway in the read me I pointed out that there are this limitations (min hardware required).
And thanks really much for your report.

from valetudo_vacuum_camera.