error while run split about eqp-cluster HOT 12 CLOSED

Hi Chai,

It seems that there are some problems with the shell scripts which run fine in my environment (for instance, the “[[ -z ]]” syntax is supposed to be standard for ksh but does not seem to work for you). I have now made a couple of changes to ameliorate potential problems:

1.   Replaced the ‘[[ -z <variable> ]]’ syntax by  ‘[ “<variable>” = “” ]’  in process-samples.sh
   

2.   Replaced “source” by “.”
   

3.   Changed “#!/bin/sh” to “#!/bin/bash” to avoid problems with “set –o pipefail”
   

The affected/changed scripts are:

bin/process-samples.sh
bin/create-bowtie2-indices.sh
exon-pipeline-scripts/bin/align-scripts/bowtie2-align.sh
exon-pipeline-scripts/bin/process-scripts/compute-counts.sh
exon-pipeline-scripts/bin/process-scripts/compute-genomic-alignments.sh
exon-pipeline-scripts/bin/process-scripts/compute-genomic-counts.sh
exon-pipeline-scripts/bin/process-scripts/compute-intron-counts.sh
exon-pipeline-scripts/bin/process-scripts/util-scripts/compute-alignment-sam-stats.sh
exon-pipeline-scripts/bin/process-scripts/util-scripts/compute-combined-weight-file.sh
exon-pipeline-scripts/bin/process-scripts/util-scripts/compute-intersection-bed-file.sh
exon-pipeline-scripts/bin/process-scripts/util-scripts/filter-junction-map-files.sh

I hope that this enables you to run process-samples.sh. Please note, however, the configuration of your UGE/SGE instance may not be compatible with the parameters passed to qsub in process-samples.sh. In this case, you will have to edit process-samples.sh and adapt the function submitJob to fit your UGE/SGE instance.

Best regards,

Sven

From: Arkarachai Fungtammasan [mailto:[email protected]]
Sent: Tuesday, November 17, 2015 10:54 PM
To: Novartis/EQP-cluster
Subject: [EQP-cluster] error while run split (#2)

Hi,

I successfully build the bowtie2 index and I try to run analysis using your sample in samples.tgz. However, I encounter this error.

sh bin/process-samples.sh split

bin/process-samples.sh: 546: bin/process-samples.sh: source: not found
bin/process-samples.sh: 640: bin/process-samples.sh: [[: not found
bin/process-samples.sh: 647: bin/process-samples.sh: [[: not found
bin/process-samples.sh: 653: bin/process-samples.sh: [[: not found
bin/process-samples.sh: 780: bin/process-samples.sh: [[: not found
bin/process-samples.sh: 1107: bin/process-samples.sh: [[: not found
Samples:

I tried to set up $PROG_DIR to the location where setup.sh is located. However, I still get the same error message.
I try both just unpacked samples.tgz and samples that contain only two fastq.gz files.

Is there any recommendation that I should try?

Thank you,
Chai

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/2.

from eqp-cluster.

Hi Sven,

Thank you so much for your help on this. However, I cannot see the attach file. It may possible that it doesn't go through the github. Would you mind forward your modify scripts to my E-mail?
[email protected]

Thank you,
Chai

from eqp-cluster.

Hi Sven,

I also tried ksh this morning, but I got the same error message.
May I ask what type/version of shell script that these scripts were written? Also, what is the type and version of computer that you was able to run?
Also, in general how do you run your script? I tried bin/process-samples.sh split and I got "No files or directory". When I try sh bin/process-samples.sh split, I got the error message that I post earlier in the thread.

Thank you,
Chai

from eqp-cluster.

Hi Chai,

Sorry, about all your trouble. I updated all the files on GitHub. So if you just download the GitHub files that I mentioned in my email, it is worth another try.

Best regards,

Sven

From: Arkarachai Fungtammasan [mailto:[email protected]]
Sent: Wednesday, November 18, 2015 6:54 PM
To: Novartis/EQP-cluster
Cc: Schuierer, Sven
Subject: Re: [EQP-cluster] error while run split (#2)

Hi Sven,

Thank you so much for your help on this. However, I cannot see the attach file. It may possible that it doesn't go through the github. Would you mind forward your modify scripts to my E-mail?
[email protected]:[email protected]

Thank you,
Chai

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/2#issuecomment-157800676.

from eqp-cluster.

Hi Chai,

Sorry, I should have made this more clear. It does not depend on which shell you use as a login shell. The script process-samples.sh runs under /bin/ksh (since the first line is the pseudo comment “#!/bin/ksh”). Our version of ksh is:

ksh --version
version sh (AT&T Research) 93t+ 2010-06-21

I run my script just as you by calling

bin/process-samples.sh split

and, of course, since you get an error message, nothing happens.

Best regards,

Sven

From: Arkarachai Fungtammasan [mailto:[email protected]]
Sent: Wednesday, November 18, 2015 8:17 PM
To: Novartis/EQP-cluster
Cc: Schuierer, Sven
Subject: Re: [EQP-cluster] error while run split (#2)

Hi Sven,

I also tried ksh this morning, but I got the same error message.
May I ask what type/version of shell script that these script were written? Also, what is the type and version of computer that you were able to run?
Also, in general how do you run your script? I tried bin/process-samples.sh split and I got No files or directory. When I try sh bin/process-samples.sh split, I got the error message that I post earlier in the thread.

Thank you,
Chai

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/2#issuecomment-157827158.

from eqp-cluster.

Thanks Sven. I will give a try.

Best,
Chai

from eqp-cluster.

Dear Sven,

I think you are right. I got the new error message about the qsub as below.

sh bin/process-samples.sh split
Samples: sample1 sample2
Sample sample1
Making directory EQP-cluster/samples/sample1/fastq-files
Creating directory EQP-cluster/samples/sample1/log-files
bin/process-samples.sh: 77: bin/process-samples.sh: qsub: not found
Sample sample2
Making directory EQP-cluster/samples/sample2/fastq-files
Creating directory EQP-cluster/samples/sample2/log-files
bin/process-samples.sh: 77: bin/process-samples.sh: qsub: not found

I guess, I need to modify the script to fit the UGE/SGE, right.

Just a quick comment. My run-test.sh does't store in "bin" but in "samples". From your Readme, I understood that it should store in bin. I tried to move it around or run it on where it was, but they all didn't work.

Best,
Chai

from eqp-cluster.

Hi Sven,

It seems that I can make it work now (normal run, not the run-test.sh). I used the "sh process-samples.sh -noqsub 20 split " Thank you for your help.
Also, I have to modify this script "exon-pipeline-scripts/bin/process-scripts/split-fastq-file.sh" from ksh to bash.

Just another quick question here. When I run split function, I got fq.gz files which look fine. However, it also create directory C001 which is empty. Is that normal?

Here is my log file

'''
Creating directory /home/dnanexus/EQP-cluster/samples/sample1/fastq-files
FASTQ_FILE_1=sample1_1.fastq.gz, GZIP_SUFFIX=sample1_1.fastq.gz, CAT=zcat, GZIP=gzip
Fri Nov 20 01:18:13 UTC 2015
Creating new fastq file(s) for sample1_1.fastq.gz and sample1_2.fastq.gz
Writing to file /home/dnanexus/EQP-cluster/samples/sample1/fastq-files/sample1-C001_1.fq.gz (line: 0)
Writing to file /home/dnanexus/EQP-cluster/samples/sample1/fastq-files/sample1-C001_2.fq.gz (line: 0)

Fastq input complete.
Number of entries filtered since at least one read of a pair has less than 5 non-As or non-Ts: 1
Fri Nov 20 01:18:21 UTC 2015
Making directory /home/dnanexus/EQP-cluster/samples/sample1/C001
Done
Fri Nov 20 01:18:21 UTC 2015
'''

Best,
Chai

from eqp-cluster.

Hi Svan,

Just a quick question, should I change all shebang ksh to bash? I notice that process-samples.sh also has it.

Thanks,
Chai

from eqp-cluster.

Hi Chai,

The empty C001 directory is normal. All subsequent alignment and quantification files will be written into (subdirectories of) this directory.

Best regards,

Sven

From: Arkarachai Fungtammasan [mailto:[email protected]]
Sent: Friday, November 20, 2015 2:29 AM
To: Novartis/EQP-cluster
Cc: Schuierer, Sven
Subject: Re: [EQP-cluster] error while run split (#2)

Hi Sven,

It seems that I can make it work now (normal run, not the run-test.sh). I used the "sh process-samples.sh -noqsub 20 split " Thank you for your help.
Also, I have to modify this script "exon-pipeline-scripts/bin/process-scripts/split-fastq-file.sh" from ksh to bash.

Just another quick question here. When I run split function, I got fq.gz files which look fine. However, it also create directory C001 which is empty. Is that normal?

Here is my log file

'''
Creating directory /home/dnanexus/EQP-cluster/samples/sample1/fastq-files
FASTQ_FILE_1=sample1_1.fastq.gz, GZIP_SUFFIX=sample1_1.fastq.gz, CAT=zcat, GZIP=gzip
Fri Nov 20 01:18:13 UTC 2015
Creating new fastq file(s) for sample1_1.fastq.gz and sample1_2.fastq.gz
Writing to file /home/dnanexus/EQP-cluster/samples/sample1/fastq-files/sample1-C001_1.fq.gz (line: 0)
Writing to file /home/dnanexus/EQP-cluster/samples/sample1/fastq-files/sample1-C001_2.fq.gz (line: 0)

Fastq input complete.
Number of entries filtered since at least one read of a pair has less than 5 non-As or non-Ts: 1
Fri Nov 20 01:18:21 UTC 2015
Making directory /home/dnanexus/EQP-cluster/samples/sample1/C001
Done
Fri Nov 20 01:18:21 UTC 2015
'''

Best,
Chai

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/2#issuecomment-158252936.

from eqp-cluster.

Hi Chai,

There should be only one file with first line “#!/bin/ksh” and this is process-samples.sh. After the changes I made it should run normally.

Best regards,

Sven

From: Arkarachai Fungtammasan [mailto:[email protected]]
Sent: Friday, November 20, 2015 3:37 AM
To: Novartis/EQP-cluster
Cc: Schuierer, Sven
Subject: Re: [EQP-cluster] error while run split (#2)

Hi Svan,

Just a quick question, should I change all shebang ksh to bash? I notice that process-samples.sh also has it.

Thanks,
Chai

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/2#issuecomment-158262063.

from eqp-cluster.

Thanks Svan. I got everything works nicely now.

Best,
Chai

from eqp-cluster.