Git Product home page Git Product logo

Comments (3)

crazyzlj avatar crazyzlj commented on July 21, 2024

After some try-and-trial, finally, I find out the solution. The key point is creating hostfile dynamically.
Here is the step-by-step and hope it helpful for others.

  • Create a job script file deap_scoop_lsf_demo.lsf.
#!/bin/sh
#BSUB -J TEST_SCOOP
#BSUB -q rhel6normal

source ~/.bash_profile
# Path to executable
cd /wps/home/zhulj/demo/scoop_test
# Add local python installation paht
export PATH=/home/zhulj/soft/python-2.7.13/bin:$PATH
# start a new host file from scratch
SCOOPHOST_FILE=hosts_$LSB_JOBID
rm -f $SCOOPHOST_FILE
touch $SCOOPHOST_FILE
echo "# SCOOP hostfile created by LSF on `date`"
# check if we were able to start writing the conf file
if [ -f $SCOOPHOST_FILE ]; then
    :
else
    echo "$0: can't create $SCOOPHOST_FILE"
    exit 1
fi
HOST=""
NUM_PROC=""
FLAG=""
TOTAL_CPUS=0
for TOKEN in $LSB_MCPU_HOSTS
    do
        if [ -z "$FLAG" ]; then # -z means string is empty
            HOST="$TOKEN"
            FLAG="0"
        else
            NUM_PROC="$TOKEN"
            FLAG="1"
        fi
        if [ "$FLAG" = "1" ]; then
            _x=0
            if [ $_x -lt $NUM_PROC ]; then
                TOTAL_CPUS=`expr "$TOTAL_CPUS" + "$NUM_PROC"`
                echo "$HOST $NUM_PROC" >> $SCOOPHOST_FILE
            fi
            # get ready for the next host
            FLAG=""
            HOST=""
            NUM_PROC=""
        fi
    done
echo "Your SCOOP boot hostfile looks like:"
echo "TOTAL_CPUS: ${TOTAL_CPUS}"
# Python script
script=onemax_island_scoop.py
# SCOOP command
python -m scoop --debug --hostfile $SCOOPHOST_FILE $script > testscoop_$LSB_JOBID.stdout.log
  • cd to the directory of deap_scoop_lsf_demo.lsf.
  • Submit job by bsub with the specification of processor number.
    $ bsub -n 48 ./deap_scoop_lsf_demo.lsf

The hostfile created is something like this. Be caution, no other lines should be existed in this file.

b10n07.cluster.com 12
b06n03.cluster.com 12
b07n12.cluster.com 8
b07n06.cluster.com 12
b07n03.cluster.com 4

from scoop.

louisabraham avatar louisabraham commented on July 21, 2024

It would still be great to support LSF! Can this issue be reopened?

from scoop.

crazyzlj avatar crazyzlj commented on July 21, 2024

It would still be great to support LSF! Can this issue be reopened?

Currently, the above solution I posted can work. It would be great if SCOOP can support LSF natively.

from scoop.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.