xenonnt / env_starter Goto Github PK

Now we have three shell scripts to start the notebook on midway2, midway3 and dali, which is a dirty solution. We should use the same script but just modify the settings according to the partition.

I think the function make_executable is declared at 2 places?

I see the function declared here

Nicely pointed out by @DonNabla. The issue is that we are not binding X509_USER_PROXY when we are on dali compute node, which is supposed to locate at /project2/lgrandi/grid_proxy/xenon_service_proxy. A quick solution would be just to bind the X509_USER_PROXY copy on dali manually.

Not sure this is a problem within env_starter

-- | -- | -- | -- | --
python | 3.9.16 | /opt/XENONnT/anaconda/envs/XENONnT_development... | None
strax | 1.5.1 | /opt/XENONnT/anaconda/envs/XENONnT_development... | None
straxen | 2.1.0 | /opt/XENONnT/anaconda/envs/XENONnT_development... | None
cutax | 1.15.0 | /dali/lgrandi/xenonnt/software/cutax/v1.15.0/c... | None```

• In README.md, symbolic link generator is:

  [ershockley@dali-login1 ~]$ ln -s start_jupyter.py /home/ershockley/nt/computing/env_starter/start_jupyter.py
  

  But I think it shall be:

  [ershockley@dali-login1 ~]$ ln -s /home/ershockley/nt/computing/env_starter/start_jupyter.py start_jupyter.py
  

• If I set the symbolic link correctly, and run

  ssh [email protected] /dali/lgrandi/strax/miniconda3/bin/python start_jupyter.py
  

  on my laptop, an error shows:

  Starting jupyter job
      /tmp/slurmd/job12221635/slurm_script: line 18: /start_notebook.sh: No such file or directory
  Traceback (most recent call last):
    File "start_jupyter.py", line 310, in <module>
      main()
    File "start_jupyter.py", line 275, in main
      raise RuntimeError("Jupyter did not start inside your job!")
  RuntimeError: Jupyter did not start inside your job!
  

  I think this is because the directory of the symbolic link does not include start_notebook.sh.

If one want to start more than one job for testing different container if would be handy to have various log files.

env_starter is doing a similar thing as batchq, and the only difference is that the job string here is always jupyter .......

So here is no reason to make env_starter an independent package, which increases the complexity of maintenance.

Currently, starting a new start_jupyter job when an old one is already running results in the extension of that old job and does not start a new one.

The idea suggested in this issue is to give the user a way to launch parallel jobs, which can be useful to run code on, e.g., the main and backup environments at the same time, thus allowing one to test new features in slightly different scopes (for example, with different strax/straxen versions which may not be deployed simultaneously on both environments).

As suggested by @JelleAalbers on the XENON1T/analysis Gitter Room, this new feature could be implemented as a flag set by the user when running start_jupyter.

Deleting a file from within the Jupyter environment can give a permission error, under certain conditions. The error is as follows:

This happens both for deleting newly created files as well as existing files. I have checked the permissions of these files from the command line when I am logged into dali. I can delete files directly from the terminal.

I start the Jupyter environment by calling the "start_jupyter.py" script on the dali cluster. I am using the latest version from the master branch. I am calling the script from within my own folder on dali (/dali/lgrandi/bderonde) with the following command:

python3 /dali/lgrandi/bderonde/env_starter/start_jupyter.py --tag 2023.11.1 --partition dali --cpu 6 --ram 45000 --exclude_node dali001

Using the stable env_starter branch located in "/project2/lgrandi/xenonnt/development/env_starter/start_jupyter.py" gives the same problem.

Supposedly, the issue arises from the fact that the Jupyter environment tries to move a file to the dali trash folder, which I don't have permission to, rather than actually deleting the file. A similar issue is discussed here: (jupyter-server/jupyter_server#1338).

A solution to the problem was suggested on the linked github page as well: adding the "--FileContentsManager.delete_to_trash=False" option to the line of code that calls Jupyter changes the behaviour of Jupyter, where files are now directly deleted instead of moved to a trash folder. This line of code is in the "start_notebook_dali.sh" shell script; the suggested change is highlighted in the picture below:

I tested this solution myself on a local copy of the scripts and this resolved the issue.

It's already implemented for utilix, but not for env_starter yet