Comments (3)
After some try-and-trial, finally, I find out the solution. The key point is creating hostfile
dynamically.
Here is the step-by-step and hope it helpful for others.
- Create a job script file
deap_scoop_lsf_demo.lsf
.
#!/bin/sh
#BSUB -J TEST_SCOOP
#BSUB -q rhel6normal
source ~/.bash_profile
# Path to executable
cd /wps/home/zhulj/demo/scoop_test
# Add local python installation paht
export PATH=/home/zhulj/soft/python-2.7.13/bin:$PATH
# start a new host file from scratch
SCOOPHOST_FILE=hosts_$LSB_JOBID
rm -f $SCOOPHOST_FILE
touch $SCOOPHOST_FILE
echo "# SCOOP hostfile created by LSF on `date`"
# check if we were able to start writing the conf file
if [ -f $SCOOPHOST_FILE ]; then
:
else
echo "$0: can't create $SCOOPHOST_FILE"
exit 1
fi
HOST=""
NUM_PROC=""
FLAG=""
TOTAL_CPUS=0
for TOKEN in $LSB_MCPU_HOSTS
do
if [ -z "$FLAG" ]; then # -z means string is empty
HOST="$TOKEN"
FLAG="0"
else
NUM_PROC="$TOKEN"
FLAG="1"
fi
if [ "$FLAG" = "1" ]; then
_x=0
if [ $_x -lt $NUM_PROC ]; then
TOTAL_CPUS=`expr "$TOTAL_CPUS" + "$NUM_PROC"`
echo "$HOST $NUM_PROC" >> $SCOOPHOST_FILE
fi
# get ready for the next host
FLAG=""
HOST=""
NUM_PROC=""
fi
done
echo "Your SCOOP boot hostfile looks like:"
echo "TOTAL_CPUS: ${TOTAL_CPUS}"
# Python script
script=onemax_island_scoop.py
# SCOOP command
python -m scoop --debug --hostfile $SCOOPHOST_FILE $script > testscoop_$LSB_JOBID.stdout.log
- cd to the directory of
deap_scoop_lsf_demo.lsf
. - Submit job by
bsub
with the specification of processor number.$ bsub -n 48 ./deap_scoop_lsf_demo.lsf
The hostfile
created is something like this. Be caution, no other lines should be existed in this file.
b10n07.cluster.com 12
b06n03.cluster.com 12
b07n12.cluster.com 8
b07n06.cluster.com 12
b07n03.cluster.com 4
from scoop.
It would still be great to support LSF! Can this issue be reopened?
from scoop.
It would still be great to support LSF! Can this issue be reopened?
Currently, the above solution I posted can work. It would be great if SCOOP can support LSF natively.
from scoop.
Related Issues (20)
- In a PBS job context, pbs_tmrsh should be used in favour of ssh
- source prolog file not working HOT 1
- [**Time Sensitive**] pyscoop.org domain expired HOT 3
- Is this project abandoned? HOT 2
- Failure trying to distribute work in HPC environment HOT 4
- Cannot use on a single node in a SLURM cluster HOT 2
- Scoop not working on Windows
- TypeError: 'NoneType' object is not callable in Python 3.5 HOT 2
- How to wait for the finish of subprocess.run (["./executer"]) when lunched by SCOOP? HOT 1
- Fix simple typo: tupel -> tuple
- Why it is much slower when using python -m scoop deap_ga_onemax.py
- Another question, help me please. HOT 1
- cpu_usage 100% In centos HOT 1
- Could not successfully launch the remote worker HOT 1
- requested remote group process id received:b''
- problems using SCOOP shared object API
- Python 3.10 Cannot import name 'Iterable' from 'collections'
- Warnings: lost track of future
- Backport collections patch to 0.7.1.1 HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from scoop.