Comments (5)
I thought a bit about it. I guess deleting a constant on all workers might be difficult, because you need to ensure that the deletion is propagated to all workers in a reasonable amount of time. But (for me) it would already be sufficient to be able to delete a constant on the current worker.
In order to do that would it suffice to add the following function to the shared
module?
def deleteConst(name):
""" Deletes a constant with a particular `name` on the **current** worker. """
for key in elements:
elements[key].pop(name, None)
If so, I can open a pull request with these changes, including a unit test, of course.
from scoop.
Well, an alternative proposal is the following function:
def deleteConst(name):
"""Deletes a constant with a particular `name` on the current worker.
Deletion does **not** propagate to other workers.
Deleted variables can also not be reset by `setConst`.
"""
for key in elements:
var_dict = elements[key]
if name in var_dict:
var_dict[name] = None
To avoid potential side effects by the user trying to reset a deleted constant via shared.setConst
.
from scoop.
Hello,
The issue with constant deletion is not exactly the propagation assertion, but the synchronization between the tasks. If an API offered constant deletion, a user could want to perform variable sharing (or "updating") through constant setting and deletion. In this case, there is no way to determine if the worker will use version 1 or version 2 of the variable when it executes. I guess this will wreck havoc in the code's logic and/or bewilder most of the non-acquainted with parallel programming.
I had planned integrating shared variables using vector clocks or playing around with the operational transformation, but lacked the resources to do so. The API that should be proposed is also not completely trivial, at least it is not evident to me.
I'm not quite sure of the use case for your second proposal? If I understand correctly, a worker can set a constant, and it is the job of every other to delete their constant when they don't need it anymore? This kind of asymmetry is something to think about in the API coherence... Aside from very large variables that could be dropped to free some memory, is there any other use cases?
from scoop.
Well, the use case I had in mind is where you have repeated but independent batches of tasks. So the first batch with multiple tasks is distributed among SCOOP workers with shared variables. These variables could be big. So after a full batch ended, all variables can be deleted to free memory and make space for the next batch.
from scoop.
I understand your use case, and I like your last example of deleteConst. It is indeed important not to allow the reset of a variable, otherwise non-aware users may simply fall into problems easily...
The only strange behaviour I see is that setConst applies to every worker, while deleteConst only applies to a the current worker.
Well, I think it could be interesting. Let me sleep over this decision and I'll keep you posted on any API change.
from scoop.
Related Issues (20)
- If there any possibility to support IBM LSF system? HOT 3
- In a PBS job context, pbs_tmrsh should be used in favour of ssh
- source prolog file not working HOT 1
- [**Time Sensitive**] pyscoop.org domain expired HOT 3
- Is this project abandoned? HOT 2
- Failure trying to distribute work in HPC environment HOT 4
- Cannot use on a single node in a SLURM cluster HOT 2
- Scoop not working on Windows
- TypeError: 'NoneType' object is not callable in Python 3.5 HOT 2
- How to wait for the finish of subprocess.run (["./executer"]) when lunched by SCOOP? HOT 1
- Fix simple typo: tupel -> tuple
- Why it is much slower when using python -m scoop deap_ga_onemax.py
- Another question, help me please. HOT 1
- cpu_usage 100% In centos HOT 1
- Could not successfully launch the remote worker HOT 1
- requested remote group process id received:b''
- problems using SCOOP shared object API
- Python 3.10 Cannot import name 'Iterable' from 'collections'
- Warnings: lost track of future
- Backport collections patch to 0.7.1.1 HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from scoop.