motioneye running out of memory (referring to bitbucket issue tracker) about motioneye HOT 26 CLOSED

Yes, bisecting is the way to go, but not at a source code level as the issue dates way back (heck, it could be there from the start for all we know). Bisecting the settings could lead us to the source of the issue but requires a lot of patience. That's why I proposed those changes to the settings, to rule out a few internal motionEye components that can leak memory.

from motioneye.

Could you please run the following command (as root) on your machine when the memory usage is high (due to motionEye): netstat -antp | grep 808? I would like to to see the output.

Also, would you be able to set up SSH remote access for me on this device of yours that leaks memory? I would like to run some tests and see if I can identify the source of the memory leak, as it's a problem that appears to affect more users.

from motioneye.

I had to restart everything in order to setup it at home. Now i can control it better.

[root@mp-01c27e37 data]# cat mem.sh
!#/usr/bin/bash
while true
do
    top -b -n 1 | fgrep python
    sleep 600
done

I'm logging it now with this snippet, to see, how memory grows. I'll post the log the next days ...

After one hour, netstat -antp | grep 808 shows the following:

[root@mp-01c27e37 ~]# netstat -antp | grep 808
tcp        0      0 127.0.0.1:8081          0.0.0.0:*               LISTEN      4870/motion-mrdave
tcp        0      0 127.0.0.1:8082          0.0.0.0:*               LISTEN      4870/motion-mrdave
tcp        0      0 127.0.0.1:46999         127.0.0.1:8082          ESTABLISHED 599/python2.7
tcp       18      0 127.0.0.1:8081          127.0.0.1:51456         ESTABLISHED 4870/motion-mrdave
tcp       18      0 127.0.0.1:8082          127.0.0.1:46999         ESTABLISHED 4870/motion-mrdave
tcp        0      0 127.0.0.1:51456         127.0.0.1:8081          ESTABLISHED 599/python2.7

And, i didn't apply your changes to settings.py yet. I'll do that in the next run, becuase first i have to remount it.
You can let me do the tests, lets talk about ssh when we end in a dead end ;)
rellla

from motioneye.

For info, this is the result after 30min:

  599     1 root     S    38400   5%   0% /usr/bin/python2.7 /programs/motioneye/motioneye.py --port 80 --motion-binary /usr/bin/motion --log-level info --mjpg-client-idle-timeout 10
  599     1 root     S    39572   5%   0% /usr/bin/python2.7 /programs/motioneye/motioneye.py --port 80 --motion-binary /usr/bin/motion --log-level info --mjpg-client-idle-timeout 10
  599     1 root     S    40876   5%   0% /usr/bin/python2.7 /programs/motioneye/motioneye.py --port 80 --motion-binary /usr/bin/motion --log-level info --mjpg-client-idle-timeout 10
  599     1 root     S    42192   5%   0% /usr/bin/python2.7 /programs/motioneye/motioneye.py --port 80 --motion-binary /usr/bin/motion --log-level info --mjpg-client-idle-timeout 10
  599     1 root     S    43492   6%   0% /usr/bin/python2.7 /programs/motioneye/motioneye.py --port 80 --motion-binary /usr/bin/motion --log-level info --mjpg-client-idle-timeout 10

Following top, i come to the conclusion, that memory usage sometimes goes one step up, after the second python process appeared.

rellla

from motioneye.

There definitly seems to exist a leak ... https://gist.github.com/rellla/ddd531503e59979822ba
I'll adapt the settings.py and try again.
rellla

from motioneye.

1. Why did you close the issue?
2. What was the command that yielded all those lines that you captured in the gist file?

from motioneye.

Sry, was a "mis"click ;)
It's the output oft mem.sh &> mem.log posted above.

from motioneye.

Same issue with the settings, you recommended: https://gist.github.com/rellla/99eaf13f7eefcb85983c
netstat shows the same as above...

from motioneye.

I'll try to include some memory info dumping functionality in a test branch or something, just to find out what the heck is leaking that much ram.

from motioneye.

In order to help finding the memory leak I have added a small /memusage page to a separate branch of motionEye. The branch is part of 0.27 release series so it goes with the new install instructions available here: https://github.com/ccrisan/motioneye/wiki/Manual-Download-And-Installation.

Please download the branch from here: https://github.com/ccrisan/motioneye/archive/aafd26348b3f0bdd8d6c2d1365dbb4c09178ac56.zip.

Then check for the memory usage every now and then. Whenever you see that memory has crossed a certain common sense boundary, simply access http://your_ip:8765/memusage?by=class and http://your_ip:8765/memusage?by=via. I need to see these two pages then so please save them and send them back to me via email or pastebin.com.

from motioneye.

I'll try that, thanks.

from motioneye.

I forgot to mention that you need to pip install guppy for this to work.

from motioneye.

Hi, maybe i forgot to say, that i'm using motionpie. So buildroot does not provide pip imho. I edited server.py and handlers.py manually but now i need some hint, how to get guppy in my system, without creating a new image!?
Btw., motioneye stopped working at 712MB...
Thanks.

from motioneye.

Since you posted the issue on motionEye's issue tracker, I just assumed that you were running motionEye on raspbian or on some common distro (although you were right to associate the issue with motionEye rather than motionPie).

Nevertheless, there's no (easy) way to upgrade motionEye on an existing motionPie installation. It would be very helpful if you took 15 minutes and temporarily install Rasbian with the memusage branch of motionEye and configure it the same way you had your motionPie configured. Then see if the problem persists and report back the /memysage... output as indicated.

Thanks.

from motioneye.

Yes, i thought it's a motioneye related issue, so i opened the issue here. I'll do a manual installation on raspberry or cubieboard as soon i have found another sd card :p

from motioneye.

Ok. Set up finally. I'm waiting for some logs ...

from motioneye.

Damn... I've set up manually on a debian jessie and a BananaPi and could not reproduce this memory consuming. Memory usage is staying constant. I have to check, if my motioneye config is exactly the same as before, and if so, it's maybe a motionpie related issue after all. Maybe it has something to do with the python version or something else ...
If you can give me a hint, how to get guppy and dependencies onto the running motionpie system, this will probably be the fastest way, to find out whats wrong.
Regards rellla

from motioneye.

Installing anything on motionPie is practically impossible. Especially when it comes to stuff written for CPython. I have however some new leads regarding this memory leak. One is related to server-side stream resizing and the other one to the working schedule setting.

In the meantime you could try the new version of the OS which is now called motionEyeOS. Many things have improved since and this memory leak could have silently disappeared.

from motioneye.

-> working schedule: iirc i haven't activated this on the manually installed system, but on motionpie...
In the meantime i can check if the issue disappears when disabling the schedule table.
I will try motionEyeOS then.

from motioneye.

Alright. Keep me posted.

from motioneye.

Enabling working schedule on my manually installed machine (which i had no problems as told) gives me the following output of

top -b -n 1 | grep meyectl

each 600 seconds:

 3198 root      20   0   42292  29636   2864 S   0,0  3,0  12:40.85 meyectl
 3198 root      20   0   42292  29636   2864 S   0,0  3,0  12:42.09 meyectl
 3198 root      20   0   42292  29636   2864 S   0,0  3,0  12:43.50 meyectl
 3198 root      20   0   42292  29640   2868 S   5,6  3,0  12:52.13 meyectl
 3198 root      20   0   55756  43100   2868 S   0,0  4,4  13:47.01 meyectl
 3198 root      20   0   57316  44588   2872 S   0,0  4,6  14:05.87 meyectl

The last two lines are caught after activating the working schedule....

Here you can find the corresponding /memusage outputs.
1st by=via https://gist.github.com/rellla/4ecd6003866d53f6a08d
2nd by=via https://gist.github.com/rellla/97502bba3e9be90748a9
1st by=class https://gist.github.com/rellla/2ed83135fc2a84f021ff
2nd by=class https://gist.github.com/rellla/0ea1da25dc7a0ea99f56

Hope this info helps...
Regards rellla

from motioneye.

While time of writing, you can add another line ;)

 3198 root      20   0   58040  45240   2872 S   5,1  4,6  15:12.29 meyectl

from motioneye.

Nice. Thanks for putting this all together. I'll take a look over it during the weekend.

from motioneye.

Thanks, too. I can confirm, that this issue is caused by the working schedule. Memory usage constantly increases. Disabling it again let the usage jump back to a smaller value and there it stays constantly.
I think, we finally found, what is repsonsible for the leak...
Good luck on your search ;)

from motioneye.

Good news everyone! The memory leak was caused by a misuse of the synchronous tornado HTTPClient, which happened only when having working schedule enabled. The problem has been fixed now and I'll release 0.28.2 later today. Please test it and let me know if the problem still occurs.

from motioneye.

Thanks for fixing this! Memory usage is constant during the last hour.
Regards rellla

from motioneye.