The pbstools from tabaer

pbs-spark-submit should respect PBS_NUM_PPN

when pbs-spark-submit is spawning the workers it should respect the env PBS_NUM_PPN and set that to SPARK_WORKER_CORES accordingly.

currently this is not the case, in current implementation SPARK_WORKER_CORES always gets set to the maxim cores on the available machine

the current code will simply count all the cpus

        cpus = 0
        if ( os.path.exists("/proc/self/cpuset") ):
            cpusetfile = open("/proc/self/cpuset")
            cpuset = cpusetfile.read().rstrip("\n")
            cpusetfile.close()
        if ( os.path.exists("/dev/cpuset") ):
            cpusetroot = "/dev/cpuset"
        elif ( os.path.exists("/sys/fs/cgroup/cpuset") ):
            cpusetroot = "/sys/fs/cgroup/cpuset"
        if ( cpusetroot is not None and cpuset is not None ):
            cpusfile = None
            if ( os.path.exists(cpusetroot+cpuset+"/cpus") ):
                cpusfile = open(cpusetroot+cpuset+"/cpus")
            elif ( os.path.exists(cpusetroot+cpuset+"/cpuset.cpus") ):
                cpusfile = open(cpusetroot+cpuset+"/cpuset.cpus")
            if ( cpusfile is not None ):
                allcpus = cpusfile.read()
                cpusfile.close()
                for cgroup in allcpus.split(","):
                    cpurange = cgroup.split("-")
                    if ( len(cpurange)==1 ):
                        cpus += 1
                    elif ( len(cpurange)==2 ):
                        cpus += int(cpurange[1])-int(cpurange[0])+1

Assuming above code calculated cpu = MAX_CPU_AVAILABLE this is != 0
Then even if PBS_NP is set in the environ nothing will happen

        if ( cpus==0 and "PBS_NP" in os.environ.keys() ):
            try:
                cpus = int(os.environ["PBS_NP"])
            except e,Exception:
                pass
        if ( cpus>0 ):
            os.environ["SPARK_WORKER_CORES"] = str(cpus)
            env["SPARK_WORKER_CORES"] = str(cpus)

I believe that the environment variable name is actually PBS_NUM_PPN (number of processors per node) and SPARK_WORKER_CORES should be set to the min(cpus, os.environ["PBS_NUM_PPN"])

Usage of PBS_NP can lead to different results as PBS_NP = PBS_NUM_NODES * PBS_NUM_PPN
and we should not spawn workers with total of PBS_NP on each node

Expected behaviour:
For example:

#PBS -l nodes=2:ppn=2

In this case pbs-spark-submit should behave like this:

spawn 2 workers one per machine with 2 cores each

tabaer / pbstools Goto Github PK

pbstools's People

Contributors

Stargazers

Watchers

Forkers

pbstools's Issues

pbs-spark-submit should respect PBS_NUM_PPN

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent