Run all broker modules in external about alignak HOT 9 CLOSED

You will have issue with pipe and very large env

from alignak.

I have begin to rewrite part of broker to use only external modules.
The problem is when add broks to the queue of the external modules and it's not very fast and consume lots of memory.

As you can see here: http://taotetek.net/2011/02/03/python-multiprocessing-zeromq-vs-queue/ there is a big difference between queue (used actually) and zeromq. zeromq use a little more memory (15%) but is execution time is 12 times faster than queue

So use zeromq seems a good idea, but mean each external module use own TCP port.

I think the memory leaks and performances problem come from that

An opinion on this?

from alignak.

Another solution is to use pipe: http://stackoverflow.com/questions/8463008/python-multiprocessing-pipe-vs-queue and seems more simple than zeromq but have less performances than zeromq

from alignak.

Yep, for the moment, I have only remove the internal module code and clean do_loop_turn of broker and the leak not propagated to modules (little leak again in broker itself) and win 30% of memory on broker and each module.

I need add tests before PR

from alignak.

Some modules, as they are implemented, HAVE to be threads...

from alignak.

You can have thread but in another process than the principal broker process

from alignak.

yeah "effectively" (different threads in different processes != different threads in the same process..), but I meant : some (broker we talk about here) modules are (others are not) actually "threads" based (in the same process than the broker (main) process so).. Ok.

Now, given (EDIT: python->CPython) CPython's GIL (I guess everyone knows all the ins and outs about it ;)), if a module is more IO bound than CPU then moving it (its "working" thread(s)) to another process won't make it that faster (and there will be the added cost of serializing&deserialazing of data transfered from the broker process to the module one). Though if serializing cost is (enoughly) lower than the "working" cost then it could effectively be valuable..

Also, regarding the memory issue(+s) we are facing here and there : this solution is more a workaround against it than the real solution on the root problem (which actually we can only guess that there are some objects which are never "unreferenced" up to a zero count (so that the python garbage collector can take them away (if their actual occupied memory can be freed/taken away..!).

Now you have a point: zmq (and others solutions like that) could be our friends (to lower cost of serialiazing data between different processes)..

from alignak.

To be re-discussed after version 1.0 and some more testing on huge configuration

from alignak.

Will be re-opened if necessary

from alignak.