SLURM_ARRAY_TASK_ID = NoneType? about camel HOT 3 CLOSED

Are you running your code on a Slurm based cluster? If so are you running an array of jobs? The way we currently run this code is by bashing an array of Slurm jobs and use the array_idx to chunk the jobs (so we can benefit from running on multiple CPUs). I could share bash script for that if you want. Let me know

from camel.

There are probably plenty of us that gets excited about this, and despite us having no right to expect such a contribution, it would be fantastic to get the bash script and maybe even a small instruction set regarding setting up/using a Slurm cluster as well.
Are there any simpler, standalone, versions of the script for the code example that you just haven't released yet?

from camel.

Sorry for the late reply @phober ! This is a sample slurm script to run the generation of AI Society. In this example we chunk the 50 roles into 5 jobs each containing 10 assistant roles. Even though this used to be very helpful before, it seems that with the rate limit we are not really benefiting from this much anymore as we exceed rate limit when we do that. Setting up a slurm cluster is a way bigger thing (we ourselves never set one but rather use the one we have at the university).

#!/bin/bash --login
#SBATCH --job-name=ChatGPT
#SBATCH --output=Output/%A_%a.out
#SBATCH --error=Error/%A_%a.err
#SBATCH --array=0-4
#SBATCH --time=1:59:00
#SBATCH --mem=50G
#SBATCH --cpus-per-task=20


export OPENAI_API_KEY="..."

conda activate camel
python examples/ai_society/role_playing_multiprocess.py

from camel.