warm-up JASMIN tutorial for Zhenkun about wrf-suews HOT 39 CLOSED

@zhenkunl @sunt05

Here is a brief instruction on Preprocessing scripts:

• After generating the original wrfinput files, you should follow the following to modify them and use them for the runs:

Under WRF-SUEWS/wrfinput-processor/: there are 4 main folders with different functionalities:

---> /input-checker This folder contains a script that check if the SUEWS parameters are being inputted to WRF are in current range and logic. It is still going on, and not completed but you do not need this step to modify the inputs.

---> /param_extractor_SuPy: this folder contains scripts that (first) runs SUEWS offline using 2012 London or Swindon parameters to spin up the model and (second) extracts all the parameters needed for SUEWS to be inputted in the WRF. Finally it puts them in two files SUEWS_param_new.json and namelist.suews.new. The first one (SUEWS_param_new.json) contains parameters that are in grid-level and needed to be put directly inside wrfinputs (change_to_SUEWS folder that I will explain next). You need to copy this file under WRF-SUEWS/wrfinput-processor/ and make sure the script in change_to_SUEWS has the right name for it. The other file (namelist.suews.new) contains the run-level parameters of SUEWS, and you need to put this file in the WRF-SUEWS run folder (change its name to namelist.suews). Note that you also need a namelist.suews under WRF-SUEWS/wrfinput-processor/ to run the script of this folder because it uses its structure to generate the new namelist file.

---> /change_to_SUEWS: the script in this folder modifies original wrfinputs and adds SUEWS related parameters to them. As I mentioned, it uses SUEWS_param.json under WRF-SUEWS/wrfinput-processor/. After running the script, you should copy new wrfinputs in the WRF-SUEWS run folder.

---> /London-Land-Cover-Modify: the script in this folder is just for the London run, and uses a high resolution land use fraction data to modify the third domain (London focused domain). If you are using the original land use data generated by WPS for Shanghai, you can ignore this folder; otherwise you can use it to modify your inputs for Shanghai.

Please let me know if you get to any difficulties running any of the scripts.

from wrf-suews.

@zhenkunl once you get some results for Shanghai, we can close this issue.

from wrf-suews.

Great!
@zhenkunl
I assumed you have used WPS. So the first step is to get standard wrfinputs for Shanghai using WPS. After you do that, then you should modify the inputs for the coupled version. I will write you a complete tutorial on this later one. But Let's start with having standard wrfinputs first.
In addition, here some tips for running WRF-WPS in Jasmin:

• When configuring WPS, and WRF in Jasmin, we need to use Intel compilers. For this purpose, before starting to configure or compile WPS or WRF, put it in .bashrc file and source it.

module load intel/15.1
module load intel/mpi/5.1.2.150
export NETCDF=/apps/libs/netCDF/intel15/fortran/4.4.1
export WRFIO_NCD_NO_LARGE_FILE_SUPPORT=1
export J='-j 6'
export NETCDF_classic=1
export WRF_EM_CORE=1

• For Runs, use jasmin-sci3.ceda.ac.uk otherwise you would get to memory problems.

• WRF4 has a new method for the number of processors. The total number grids assign in to each processor in x or y direction should not be less than 10. You might get to this problem, but it is easy to fix.

• Here a simple bash script for running jobs in Jasmin:

#!/bin/bash 
#BSUB -q par-multi 
#BSUB -n 49
#BSUB -o %J.out 
#BSUB -e %J.err 
#BSUB -W 24:00

echo "Running WRF"
# (./real.exe for generating wrfinputs )
mpirun ./wrf.exe 

from wrf-suews.

Thanks for your detailed explanation @hamidrezaomidvar. I will try to get with Jasmin first. I will ask for your help when I experience difficulties.

from wrf-suews.

Hi @hamidrezaomidvar. What is the difference of wrf.exe under hamid/xx-test-xx-2 or xx-test-xx-3 or xx-test-xx-4? Which one is the newest?

from wrf-suews.

These are some of the local test I am doing right now. Try xx-test-xx-2 if like to run a case. Others are the test that I have not merge to the master! also please clone master of WRF-SUEWS since the test-dev is still have some problems that I am fixing now.

from wrf-suews.

BTW, I'd like to comment on the "best practise" for organising our WRF runs as I can see more regions will be tested and applied with our coupled system.

1. split wrf.exe and other related static data files (e.g., those profile-like data generated by WRF itself for a specific version) from your cases with input and output files and ; so all binaries stay in one place;
2. furthermore, split input files from cases; for e.g., a specific case usually only have one set of forcing conditions, which should come from a single place.

Then, ideally, we would have a structure like this:

├── WRF-exe
│   ├── wrf.exe.orig-4.0
│   ├── wrf.exe.orig-4.1
│   ├── wrf.exe.suews-4.0
│   └── wrf.exe.suews-4.1
├── cases
│   ├── London-GMD-paper
│   └── London-test-201504
├── wrf-data
│   ├── CAM_ABS_DATA
│   ├── CAM_AEROPT_DATA
│   ├── CAMtr_volume_mixing_ratio.A1B
│   ├── CAMtr_volume_mixing_ratio.A2
│   ├── CAMtr_volume_mixing_ratio.RCP4.5
│   ├── ...many other files...
│   ├── tr49t85
│   ├── tr67t85
│   └── wind-turbine-1.tbl
├── wrfbdy
│   ├── London
│   │   ├── 201504
│   │   └── 201507
│   └── Shanghai
│       └── 201509
└── wrfinput
    ├── London
    │   ├── MODIS
    │   ├── MODIS-SUEWS
    │   └── MODIS-updated
    └── Shanghai
        ├── MODIS
        └── MODIS-updated

By adopting such a structure, we can set up different runs under the cases folder and link configurations and binaries from other places; also, as we are linking files, we know what original information is and how we can proceed from there.

In the above structure, the wrfinput part might need to be changed according to different initial conditions for specific cases, but I put it separately for the geographic data, which usually needs quite amount of work to set up but won't change across runs of a specific region. So instead of link, under certain scenarios, we'd better copy them to the cases folder.

from wrf-suews.

I felt puzzled at the relationship among them this afternoon. It's very thoughtful for you to inform me of these promptly.

from wrf-suews.

Some errors occurred when I submitted real.exe to Jasmin using bsub < bsub_run_real. The error log showed as the following:

Fatal error in MPI_Init: Other MPI error, error stack:
MPIR_Init_thread(784).................:
MPID_Init(1326).......................: channel initialization failed
MPIDI_CH3_Init(141)...................:
dapl_rc_setup_all_connections_20(1396): generic failure with errno = 671107855
MPID_nem_dapl_get_from_bc(1309).......: Missing port or invalid host/port description in business card
Fatal error in MPI_Init: Other MPI error, error stack:
MPIR_Init_thread(784).................:

Any hints?

from wrf-suews.

from wrf-suews.

I used less cores last night, but still failed. Maybe I should keep trying as you said.

from wrf-suews.

Another problem when run wrf.exe:

INITIALIZE SUEWS NAMELIST
-------------- FATAL CALLED ---------------
FATAL CALLED FROM FILE: LINE: 1270
ERROR reading sector coeff of namelist.suews

I used the WRF version under xx-test-xx-2, and namelist.suews was generated for Shanghai. I am wondering if the WRF version is too old to read in the namelist.suews correctly or namelist.suews has changed since WRF was compiled.

from wrf-suews.

from wrf-suews.

I tried many times. Sometimes jobs can be submitted successfully, but exit soon. The error files always say "ERROR reading sector coeff of namelist.suews". I suppose there might be something wrong with the code itself or wrf.exe(in xx-test-xx-2) is not consistent with the one in the repo.

from wrf-suews.

from wrf-suews.

I did attempt to change the number of processors, unfortunately the jobs cannot be submitted no matter what the number is. Even no *.err or *.out files are outputted. Can you have a try to see if it is a problem of Jasmin now?

from wrf-suews.

from wrf-suews.

I copied the Apr-London-Swindon folder to my own directory and changed the forcing data and namelist files(include namelist.input and namelist.suews). All the others remain the same. Then I submitted the job, however, it looked like I didn't do anything. No jobs can be found when execute jobs command, no logs are generated. It's really tricky!

from wrf-suews.

We need to find a solution for this instability. Let's work on it together on Monday and try to solve it.

from wrf-suews.

My wrf.exe run for a while, wrote the wrfout_d01* file for the outmost domain and then exited. I find one of the rsl.error.* file ends with

-------------- FATAL CALLED ---------------
FATAL CALLED FROM FILE: LINE: 29365
fatal error in SUEWS:Problem with (z-zd) and/or z0.

application called MPI_Abort(MPI_COMM_WORLD, 1) - process 32

and one ends with

-------------- FATAL CALLED ---------------
FATAL CALLED FROM FILE: LINE: 29365
fatal error in SUEWS:Inappropriate value calculated.

application called MPI_Abort(MPI_COMM_WORLD, 1) - process 33

Is there anything I might have done wrong?

from wrf-suews.

from wrf-suews.

from wrf-suews.

A: 1)Only one time step was outputted
2)bldgH_SUEWS = 35.9 in SUEWS_param_new.json
3)Yes, it is.
So I need to change the bldgH_SUEWS to a lower value and modify wrfinput_d0* again, right?

from wrf-suews.

The bldgH_SUEWS was reset to 25 or 22 successively and still the same errors. Maybe try a little lower value?

from wrf-suews.

heights of trees also matter.
check these variables: EveTreeH_SUEWS and DecTreeH_SUEWS.

from wrf-suews.

Both of these two variables in London run are 13.1, and they are 9.1 and 10.9 respectively in my case. What's the direction?

from wrf-suews.

then try to set a higher debug value to see what height the first/lowest atmospheric level is.

from wrf-suews.

Can you see something from the log?

d01 2012-12-01_00:00:00 after SuMin, qn_SUEWS= 76.0966107299998
d01 2012-12-01_00:00:00 after SuMin, qf_SUEWS= 0.000000000000000E+000
d01 2012-12-01_00:00:00 after SuMin, qs_SUEWS= 6.56633480928586
d01 2012-12-01_00:00:00 after SuMin, qh_SUEWS= -129.063718135778
d01 2012-12-01_00:00:00 after SuMin, qe_SUEWS= 198.593994056492
d01 2012-12-01_00:00:00 qn_out = 76.0966107299998
d01 2012-12-01_00:00:00 qf_out = 0.000000000000000E+000
d01 2012-12-01_00:00:00 qs_out = 6.56633480928586
d01 2012-12-01_00:00:00 qh_out = -129.063718135778
d01 2012-12-01_00:00:00 qe_out = 198.593994056492
d01 2012-12-01_00:00:00 First vertical level is 25.4766330718994
d01 2012-12-01_00:00:00 in SuMin, before calculation, OHM_coef: 0.718999981880188 0.718999981880188 0.718999981880188 0.718999981880188 0.194000005722046 0.194000005722046 0.194000005722046 0.194000005722046 -36.5999984741211 -36.5999984741211 -36.5999984741211 -36.5999984741211
d01 2012-12-01_00:00:00 Problem: In stability subroutine, (z-zd) < z0.
d01 2012-12-01_00:00:00 ERROR! Program stopped: Problem with (z-zd) and/or z0.
d01 2012-12-01_00:00:00 Values: 0.4766 3.6000
d01 2012-12-01_00:00:00 17
d01 2012-12-01_00:00:00 ERROR! SUEWS run stopped.
-------------- FATAL CALLED ---------------
FATAL CALLED FROM FILE: LINE: 29365
fatal error in SUEWS:Problem with (z-zd) and/or z0.

application called MPI_Abort(MPI_COMM_WORLD, 1) - process 32

from wrf-suews.

from wrf-suews.

from wrf-suews.

Eta levels have been decreased from 33 to 28. The error now becomes:

d01 2012-12-01_00:00:00 call cumulus_driver
d01 2012-12-01_00:00:00 in cu_tiedtke
d01 2012-12-01_00:00:00 returning from cumulus_driver
d01 2012-12-01_00:00:00 call shallow_cumulus_driver
d01 2012-12-01_00:00:00 calling inc/HALO_EM_FDDA_SFC_inline.inc
d01 2012-12-01_00:00:00 call fddagd_driver
d01 2012-12-01_00:00:00 call calculate_phy_tend
d01 2012-12-01_00:00:00 call compute_diff_metrics
d01 2012-12-01_00:00:00 calling inc/HALO_EM_TKE_C_inline.inc
Fatal error in PMPI_Wait: A process has failed, error stack:
PMPI_Wait(198)............: MPI_Wait(request=0x53a8d5c, status=0x7ffcbca84170) failed
MPIR_Wait_impl(79)........:
dequeue_and_set_error(933): Communication error with rank 29

Is this a MPI problem or not?

from wrf-suews.

I have run the model twice and errors are the same

from wrf-suews.

Looks like so. I think nothing we help with this.

from wrf-suews.

But still I am seeing the z-zd<0 in rsl.out.024. Check this: grep "First vertical level" rsl.out.00* and look what is the lowest value. Maybe try to decrease the eta level of the second grid in your namelist.input lower than 0.90, and see what happens.

from wrf-suews.

It has been running for two or more time steps for all three domains and still continues. I can have a good sleep! Thank you all.

from wrf-suews.

Good Job! what was the final problem? is it still running?

from wrf-suews.

I modified the eta levels as you said then it succeeded. It is still running now.

from wrf-suews.

@zhenkunl can we close this?

from wrf-suews.

Sure.

from wrf-suews.