Comments (1)
I think the only reason we're using bz2 compression is that that was what the original pythongrid (which GridMap is forked from) did it that way.
I hadn't thought about the problem of needing to send back large objects before. Unfortunately, one of the reasons PyZMQ is so easy to use is that it is inherently single-threaded, so there's no way to parallelize JobMonitor by allowing for more threads/processes. What we could do, would be to break things up into multiple messages somehow, so that the the JobMonitor reconstructs the data piecemeal. That way it's never spending all of its time receiving one particular message.
As for adding some functionality so that numpy arrays are treated specially, that's something I'd be open to, but keep in mind that the non-copying stuff talked about in the PyZMQ documentation doesn't apply if you're sending things from one machine to another.
from gridmap.
Related Issues (20)
- grid_map submits jobs twice HOT 2
- Support arbitrary DRMAA native specification
- Python cluster manager
- Request: Update PyPi package HOT 3
- Travis build not working HOT 2
- Handle case where temp_dir exists but is not writable
- runpy.py:125: RuntimeWarning: 'gridmap.runner' found in sys.modules HOT 2
- JobMonitor Port and Job Native Specifications. HOT 1
- Update anaconda python support HOT 2
- Start making conda packages
- Error: too many values to unpack (expected 2) HOT 1
- conda env vars seemly not copied causing "Could not import drmaa" error HOT 15
- specifying resources? HOT 3
- Open up customization of resource requests
- Encountered ConnectionRefusedError, so killing all jobs HOT 4
- Encountered ConnectionRefusedError instead of handle_resubmit
- gridmap_web requires cherrypy
- Documentation needs updating: missing `project` option
- Option of passing shared value for the executed jobs
- Submit a job and return without waiting for completion
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from gridmap.