Gridmap with large output objects about gridmap HOT 1 OPEN

I think the only reason we're using bz2 compression is that that was what the original pythongrid (which GridMap is forked from) did it that way.

I hadn't thought about the problem of needing to send back large objects before. Unfortunately, one of the reasons PyZMQ is so easy to use is that it is inherently single-threaded, so there's no way to parallelize JobMonitor by allowing for more threads/processes. What we could do, would be to break things up into multiple messages somehow, so that the the JobMonitor reconstructs the data piecemeal. That way it's never spending all of its time receiving one particular message.

As for adding some functionality so that numpy arrays are treated specially, that's something I'd be open to, but keep in mind that the non-copying stuff talked about in the PyZMQ documentation doesn't apply if you're sending things from one machine to another.

from gridmap.