comprna / cheui Goto Github PK

Hi,
When I use C++ version run the training, it gave me the error as this:
error while loading shared libraries: PicklingTools/libptools.so: cannot open shared object file: No such file or directory
How should I solve that?

Thanks,
Xiangru

Hi,

I obtained around 22,000 m5C midification sites using CHEUI-solo model, and I overlapped it with m5C-atlas database, only 1786 overlapped site obtained.
I wonder how can I adjust parmeters to get high overlapped site? or any tips should I attention ?

Thanks a lot,
Bai

Hi,

Nice tool!

Will Cheui work with the new RNA004 chemistry or is there a timeline for adapting/training Cheui for the new SQK-RNA004? I assume that this also depends on the adaptation of nanopolish, correct?

Thanks!

Hello,

Thank you for developping CHEUI.

I was wondering if the reads must be basecalled with Guppy v4.0.14?
Can I use a later version of Guppy (in my case v6.0.1)? Will it impact the performance of CHEUI?

Best,
Lolita

**Hi, i used the follow command:
python3 ../scripts/CHEUI_predict_model2.py -i /media/logen/sd16b/Hp/cheui/nanopolish_out_26695.txt -m ../kmer_models/model_kmer.csv -o out_A_signals+IDs.p -n 15

it reports wrong:**

2022-12-05 18:18:02.188272: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.1
2022-12-05 18:18:03.597052: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2022-12-05 18:18:03.598138: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2022-12-05 18:18:03.639249: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:04:00.0 name: NVIDIA GeForce GTX 1080 computeCapability: 6.1
coreClock: 1.7845GHz coreCount: 20 deviceMemorySize: 7.92GiB deviceMemoryBandwidth: 298.32GiB/s
2022-12-05 18:18:03.639297: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.1
2022-12-05 18:18:03.641257: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.10
2022-12-05 18:18:03.641297: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.10
2022-12-05 18:18:03.642909: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2022-12-05 18:18:03.643223: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2022-12-05 18:18:03.645643: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2022-12-05 18:18:03.646885: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.10
2022-12-05 18:18:03.651311: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.7
2022-12-05 18:18:03.651775: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2022-12-05 18:18:03.652146: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: SSE4.1 SSE4.2 AVX
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-12-05 18:18:03.653928: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2022-12-05 18:18:03.654179: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:04:00.0 name: NVIDIA GeForce GTX 1080 computeCapability: 6.1
coreClock: 1.7845GHz coreCount: 20 deviceMemorySize: 7.92GiB deviceMemoryBandwidth: 298.32GiB/s
2022-12-05 18:18:03.654208: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.1
2022-12-05 18:18:03.654237: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.10
2022-12-05 18:18:03.654262: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.10
2022-12-05 18:18:03.654287: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2022-12-05 18:18:03.654311: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2022-12-05 18:18:03.654362: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2022-12-05 18:18:03.654402: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.10
2022-12-05 18:18:03.654428: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.7
2022-12-05 18:18:03.654756: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2022-12-05 18:18:03.654798: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.1
2022-12-05 18:18:04.339973: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2022-12-05 18:18:04.340033: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267] 0
2022-12-05 18:18:04.340045: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0: N
2022-12-05 18:18:04.340764: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7130 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce GTX 1080, pci bus id: 0000:04:00.0, compute capability: 6.1)
2022-12-05 18:18:04.993160: W tensorflow/core/util/tensor_slice_reader.cc:95] Could not open ../kmer_models/model_kmer.csv: Data loss: not an sstable (bad magic number): perhaps your file is in a different file format and you need to use a different restore operator?
Traceback (most recent call last):
File "../scripts/CHEUI_predict_model2.py", line 114, in
model.load_weights(secondML_path)
File "/home/logen/anaconda3/envs/cheui/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py", line 2227, in load_weights
with h5py.File(filepath, 'r') as f:
File "/home/logen/anaconda3/envs/cheui/lib/python3.7/site-packages/h5py/_hl/files.py", line 408, in init
swmr=swmr)
File "/home/logen/anaconda3/envs/cheui/lib/python3.7/site-packages/h5py/_hl/files.py", line 173, in make_fid
fid = h5f.open(name, flags, fapl=fapl)
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py/h5f.pyx", line 88, in h5py.h5f.open
OSError: Unable to open file (file signature not found)

It shouldn't be the problem of model_kmer.csv. since I could open it.
And the test sentence works:
python3 ../scripts/CHEUI_preprocess_m6A.py -i nanopolish_output_test.txt -m ../kmer_models/model_kmer.csv -o out_A_signals+IDs.p -n 15

the first ten row of nanopolish_out_26695.txt is:

contig position reference_kmer read_name strand event_index event_level_mean event_stdv event_length model_kmer model_mean model_stdv standardized_level start_idx end_idx samples
lcl|NC_018939.1_cds_WP_001165637.1_2 299 TATTG 166588ca-0972-47be-b76d-9a9124773666 t 1 92.61 2.517 0.00830 TATTG 94.79 5.53 -0.34 23504 23529 91.0363,91.3671,88.7207,93.0212,96.4946,90.0439,92.525,90.2093,94.0136,92.6904,97.3216,92.8558,93.0212,94.5098,99.4719,87.5629,95.1714,88.0591,93.8482,91.3671,89.5477,95.006,94.3444,92.1942,90.8709
lcl|NC_018939.1_cds_WP_001165637.1_2 300 ATTGC 166588ca-0972-47be-b76d-9a9124773666 t 2 85.50 1.788 0.00199 ATTGC 83.88 2.65 0.53 23498 23504 85.9088,86.9013,85.0818,86.7358,87.0667,81.2775
lcl|NC_018939.1_cds_WP_001165637.1_2 301 TTGCC 166588ca-0972-47be-b76d-9a9124773666 t 3 90.04 2.549 0.00332 TTGCC 89.55 3.13 0.14 23488 23498 89.5477,92.525,86.405,84.5856,92.1942,89.7131,95.006,91.5325,89.3823,89.5477
lcl|NC_018939.1_cds_WP_001165637.1_2 301 TTGCC 166588ca-0972-47be-b76d-9a9124773666 t 4 87.49 2.808 0.00564 TTGCC 89.55 3.13 -0.58 23471 23488 92.0287,90.5401,86.9013,84.5856,86.0742,86.9013,88.3899,94.0136,89.2169,88.5553,83.924,89.8785,80.9467,85.578,89.3823,84.5856,85.7434
lcl|NC_018939.1_cds_WP_001165637.1_2 301 TTGCC 166588ca-0972-47be-b76d-9a9124773666 t 5 93.08 2.956 0.00498 TTGCC 89.55 3.13 0.99 23456 23471 91.6979,94.3444,93.0212,93.6828,99.3065,90.7055,97.3216,87.5629,93.8482,92.1942,88.7207,92.3596,90.8709,99.3065,91.2017
lcl|NC_018939.1_cds_WP_001165637.1_2 301 TTGCC 166588ca-0972-47be-b76d-9a9124773666 t 6 87.81 1.698 0.00332 TTGCC 89.55 3.13 -0.49 23446 23456 87.7283,90.3747,86.0742,89.8785,87.0667,87.3975,83.7586,88.2245,90.0439,87.5629
lcl|NC_018939.1_cds_WP_001165637.1_2 301 TTGCC 166588ca-0972-47be-b76d-9a9124773666 t 7 90.20 2.765 0.00896 TTGCC 89.55 3.13 0.18 23419 23446 87.0667,87.8937,91.5325,88.7207,85.0818,90.2093,83.924,88.7207,86.7358,91.6979,88.3899,90.3747,93.8482,90.0439,95.5022,87.0667,88.2245,95.5022,92.0287,87.8937,92.8558,95.006,90.2093,95.1714,89.2169,89.7131,92.8558
lcl|NC_018939.1_cds_WP_001165637.1_2 301 TTGCC 166588ca-0972-47be-b76d-9a9124773666 t 8 84.27 1.651 0.00764 TTGCC 89.55 3.13 -1.48 23396 23419 84.0894,83.7586,90.0439,84.5856,85.2472,84.751,84.751,84.4202,80.2851,81.2775,82.27,85.7434,83.4278,85.7434,85.578,84.4202,84.0894,82.9316,84.0894,86.0742,84.0894,83.924,82.6008
lcl|NC_018939.1_cds_WP_001165637.1_2 301 TTGCC 166588ca-0972-47be-b76d-9a9124773666 t 9 81.79 1.298 0.00896 TTGCC 89.55 3.13 -2.17 23369 23396 82.7662,81.7738,80.2851,81.4429,83.924,83.924,79.9543,80.9467,83.5932,79.4581,84.2548,82.4354,79.9543,82.27,81.4429,80.6159,82.4354,78.7965,81.9392,81.9392,82.9316,81.2775,80.4505,84.0894,80.7813,81.7738,82.9316

Hello Team,

I have basic question regarding the stoichiometry, if i just consider the probability threshold above 0.9 in the probability column of 'site-level prediction file ' without considering any threshold for stoichiometry, is that accurate way to analyze the data or if i consider the stoichiometry threshold above 0.7 and probability threshold above 0.9, which is more accurate way to analyze the data. It would be nice to have your inputs on the same.
Looking forward to your reply.

Thanks & Regards
Priyanka Roy

Hi developer,

I tried to install CHEUI and its dependencies, but when I installed tensorflow-gpu==2.4.1, numpy is downgraded to version1.19.5.
Is it normal that the versions of tensorflow-gpu and numpy that you're recommending are not compatible ?
I haven't tested CHEUI yet, I hope it won't cause any errors when running the different scripts.

Thank you,
Rania

Hello, i was wondering if the --samples name option in nanopolish eventalign is really required? and if so why?
Other m6A callers don't seem to require this input and in my case, it is greatly increasing the output file size.

Thanks

Hi developer,
Sorry to bother you again as I'm new to RNA modification and have one more question. If I were to look for one site which is an m6A/m5C modification site that differs between two groups, then should this site not only appears in the output file of CHEUI-diff, but also should has a probability greater than the 0.99 which was mentioned in your article in the result of CHEUI-solo model2. Is that the right understanding?
Thanks again!
Zaifeng

Hi,
I have a small query regarding the tool, can I use the genomic fasta file as a reference instead of transcriptomic fasta file for detection of methylation events.
Looking forward to hearing from you.

Thanks

Hi team,

I was trying to run the command:
python3 ../scripts/CHEUI_predict_model2.py -i read_level_m6A_predictions_sorted.txt -m ../CHEUI_trained_models/CHEUI_m6A_model2.h5 -o site_level_m6A_predictions.txt

The command got executed, and the site-level file was made, but it also gave an error. I've attached a screenshot for the same.

Any assistance with this will be highly appreciated.

Thanks,

Dear CHEUI,

Any chance you have created a docker, or could create one?

Thanks

Hi developer,
Have you added multithreading function to your preprocessing program？
I give 40 cores to it, but it only uses one.

If we wish to get the read level information of the methylated transcript, can we use the second column i.e. read levele and k-mer probability of the middle A of the file read_level_prediction_m6A_sorted to filter the methylation probability of >=0.9 ? As in, is the methylation probability of site level in the site-level prediction file same as that of probability (second column) in the read level prediction file ?

Hello,
I'm trying to compile preprocessing C++ file, but I got an expected identifier error. This occurs in ocavlhash.h and ocordavlhash.h file used in PicklingTools (line 924 in the code below). Are there any way to successfully compile the code?
To note, I compiled the C++ code in Ubuntu 22.04 with g++ 11.4.0.

Hi team,

I would like to have clarity on the column header position in the output file of site_level_prediction (CHEUI solo outputs). Does it indicate the position of modified base from the start of that particular transcript ID or the coordinate on the genome where that modified base is predicted. If it is the position of the base from the start of that particular transcript, does it start from the mRNA coordinates (starting from the 5' UTR region) on the genome ?

Any idea on this concern further will be highly appreciated.

Thanks,

Sukriti

I was trying to execute the following command for the prediction of m6A probability and stoichiometry at every A nucleotide site in the reference transcriptome y using following command, but unfortunately the command executed with error(attached herewith). I need help regarding the same.

python3 ../scripts/CHEUI_predict_model2.py -i read_level_m6A_predictions_sorted.txt -m ../CHEUI_trained_models/CHEUI_m6A_model2.h5 -o site_level_m6A_predictions.txt

Hi,

I predicted around 150000 m6a modification site using CHEUI, and using the probability cut-off of 0.8, but it should be range from 20000 to 50000 modification site. It seems too much, is there any some parameters to adjust when I predicted?

Thanks,
Bai

Hi I have run CHEUI pipeline on our own dataset, overall it's very smooth.
However, I found issue regarding the sort command used to sort the read level prediction in the README.md:
sort -k1 --parallel=15 ./read_level_m6A_predictions_combined.txt > ./read_level_m6A_predictions_sorted.txt

When the contig name is start with character smaller than K, the head line "KMER Prediction label" will not keep in head, and lead afterward analysis crashed, so I would suggest using this sorting command instead:

(head -1 ./read_level_m5C_predictions.txt && tail -n +2 ./read_level_m5C_predictions.txt| sort -k1 --parallel=15) > ./read_level_m5C_predictions_sorted.txt

Hello CHEUI team,
Thank you so much for developing this wonderful software for the concurrent detection of m6A and m5C modification from nanopore dRNA-Seq. I'm running CHEUI on our in-house dRNA-seq data. I've got a few questions regarding the choice of reference transcriptome file and about the output file of CHEUI for site-level predictions.

1. I've used the UCSC mouse transcriptome reference file - "mRNA.fa" (https://hgdownload.soe.ucsc.edu/goldenPath/mm10/bigZips/) and then aligned using minimap2. I'm just wondering about the Ensemble cDNA FASTA file "Mus_musculus.GRCm39.cdna.all.fa.gz"(https://ftp.ensembl.org/pub/release-109/fasta/mus_musculus/cdna/Mus_musculus.GRCm39.cdna.all.fa.gz) are also transcriptome reference file can be used for minimap2 transcriptome mapping? Are there any relative benefits of using Ensemble over the UCSC reference file?
2. I've attached below the site-level m6A predictions by CHEUI. The contig is UCSC transcript ID, how can I change this to gene symbol or ensemble gene ID? In the output file, I see only contig and position. What does this mean? Is there any way of getting the chrosome as well in the output file?
3. What is the well-acceptable threshold to identify a significant m6A-modified site based on probability and stoichiometry?
   I beg pardon as some questions are not relevant to CHEUI.

I would appreciate your time and help.

Kind regards
Reza
PhD Candidate at Queensland Brain Institute, The University of Queensland.

Hi developer,

Thank you for developing such an excellent tool! I have obtained the different sites from CHEUI-diff, but could you tell me how to differentiate which group the each different site belongs to in output files of CHEUI-diff?

Thank you in advance,
Zaifeng

Hi,
thanks for developing CHEUI. I run CHEUI on my own data and the nanopolish result file is about 700G. I have been running the preprocess_m6A step using 20 cpus for 10 days and it hasn't finished yet. How long will this step take normally, and is there any way to accelerate this step? thank you.

Chujie

Hi developer,

Thank you for developing such an excellent tool. I am new to m6A RNA modification. Could you tell me what's the difference between stoichiometry and modification probability for m6A (or m5C) in output files for CHEUI models?

Thank you in advance,
Weipeng

I successfully ran model 1 to get the site-level predictions, but model 2's run was cut short due to this error. Here's the full traceback:
Traceback (most recent call last):
File "/Users/noahlittman/CHEUI/scripts/CHEUI_predict_model2.py", line 197, in
lr_probs = model.predict(prob_vectors)
File "/Users/noahlittman/tensorflow-metal/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 67, in error_handler
raise e.with_traceback(filtered_tb) from None
File "/Users/noahlittman/tensorflow-metal/lib/python3.9/site-packages/tensorflow/python/eager/execute.py", line 54, in quick_execute
tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.InvalidArgumentError: Graph execution error:

Detected at node 'model/conv1d/Conv1D' defined at (most recent call last):
File "/Users/noahlittman/CHEUI/scripts/CHEUI_predict_model2.py", line 197, in
lr_probs = model.predict(prob_vectors)
File "/Users/noahlittman/tensorflow-metal/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
return fn(*args, **kwargs)
File "/Users/noahlittman/tensorflow-metal/lib/python3.9/site-packages/keras/engine/training.py", line 2033, in predict
tmp_batch_outputs = self.predict_function(iterator)
File "/Users/noahlittman/tensorflow-metal/lib/python3.9/site-packages/keras/engine/training.py", line 1845, in predict_function
return step_function(self, iterator)
File "/Users/noahlittman/tensorflow-metal/lib/python3.9/site-packages/keras/engine/training.py", line 1834, in step_function
outputs = model.distribute_strategy.run(run_step, args=(data,))
File "/Users/noahlittman/tensorflow-metal/lib/python3.9/site-packages/keras/engine/training.py", line 1823, in run_step
outputs = model.predict_step(data)
File "/Users/noahlittman/tensorflow-metal/lib/python3.9/site-packages/keras/engine/training.py", line 1791, in predict_step
return self(x, training=False)
File "/Users/noahlittman/tensorflow-metal/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
return fn(*args, **kwargs)
File "/Users/noahlittman/tensorflow-metal/lib/python3.9/site-packages/keras/engine/training.py", line 490, in call
return super().call(*args, **kwargs)
File "/Users/noahlittman/tensorflow-metal/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
return fn(*args, **kwargs)
File "/Users/noahlittman/tensorflow-metal/lib/python3.9/site-packages/keras/engine/base_layer.py", line 1014, in call
outputs = call_fn(inputs, *args, **kwargs)
File "/Users/noahlittman/tensorflow-metal/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
return fn(*args, **kwargs)
File "/Users/noahlittman/tensorflow-metal/lib/python3.9/site-packages/keras/engine/functional.py", line 458, in call
return self._run_internal_graph(
File "/Users/noahlittman/tensorflow-metal/lib/python3.9/site-packages/keras/engine/functional.py", line 596, in _run_internal_graph
outputs = node.layer(*args, **kwargs)
File "/Users/noahlittman/tensorflow-metal/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
return fn(*args, **kwargs)
File "/Users/noahlittman/tensorflow-metal/lib/python3.9/site-packages/keras/engine/base_layer.py", line 1014, in call
outputs = call_fn(inputs, *args, **kwargs)
File "/Users/noahlittman/tensorflow-metal/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
return fn(*args, **kwargs)
File "/Users/noahlittman/tensorflow-metal/lib/python3.9/site-packages/keras/layers/convolutional/base_conv.py", line 250, in call
outputs = self.convolution_op(inputs, self.kernel)
File "/Users/noahlittman/tensorflow-metal/lib/python3.9/site-packages/keras/layers/convolutional/base_conv.py", line 225, in convolution_op
return tf.nn.convolution(
Node: 'model/conv1d/Conv1D'
Detected at node 'model/conv1d/Conv1D' defined at (most recent call last):
File "/Users/noahlittman/CHEUI/scripts/CHEUI_predict_model2.py", line 197, in
lr_probs = model.predict(prob_vectors)
File "/Users/noahlittman/tensorflow-metal/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
return fn(*args, **kwargs)
File "/Users/noahlittman/tensorflow-metal/lib/python3.9/site-packages/keras/engine/training.py", line 2033, in predict
tmp_batch_outputs = self.predict_function(iterator)
File "/Users/noahlittman/tensorflow-metal/lib/python3.9/site-packages/keras/engine/training.py", line 1845, in predict_function
return step_function(self, iterator)
File "/Users/noahlittman/tensorflow-metal/lib/python3.9/site-packages/keras/engine/training.py", line 1834, in step_function
outputs = model.distribute_strategy.run(run_step, args=(data,))
File "/Users/noahlittman/tensorflow-metal/lib/python3.9/site-packages/keras/engine/training.py", line 1823, in run_step
outputs = model.predict_step(data)
File "/Users/noahlittman/tensorflow-metal/lib/python3.9/site-packages/keras/engine/training.py", line 1791, in predict_step
return self(x, training=False)
File "/Users/noahlittman/tensorflow-metal/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
return fn(*args, **kwargs)
File "/Users/noahlittman/tensorflow-metal/lib/python3.9/site-packages/keras/engine/training.py", line 490, in call
return super().call(*args, **kwargs)
File "/Users/noahlittman/tensorflow-metal/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
return fn(*args, **kwargs)
File "/Users/noahlittman/tensorflow-metal/lib/python3.9/site-packages/keras/engine/base_layer.py", line 1014, in call
outputs = call_fn(inputs, *args, **kwargs)
File "/Users/noahlittman/tensorflow-metal/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
return fn(*args, **kwargs)
File "/Users/noahlittman/tensorflow-metal/lib/python3.9/site-packages/keras/engine/functional.py", line 458, in call
return self._run_internal_graph(
File "/Users/noahlittman/tensorflow-metal/lib/python3.9/site-packages/keras/engine/functional.py", line 596, in _run_internal_graph
outputs = node.layer(*args, **kwargs)
File "/Users/noahlittman/tensorflow-metal/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
return fn(*args, **kwargs)
File "/Users/noahlittman/tensorflow-metal/lib/python3.9/site-packages/keras/engine/base_layer.py", line 1014, in call
outputs = call_fn(inputs, *args, **kwargs)
File "/Users/noahlittman/tensorflow-metal/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
return fn(*args, **kwargs)
File "/Users/noahlittman/tensorflow-metal/lib/python3.9/site-packages/keras/layers/convolutional/base_conv.py", line 250, in call
outputs = self.convolution_op(inputs, self.kernel)
File "/Users/noahlittman/tensorflow-metal/lib/python3.9/site-packages/keras/layers/convolutional/base_conv.py", line 225, in convolution_op
return tf.nn.convolution(
Node: 'model/conv1d/Conv1D'
2 root error(s) found.
(0) INVALID_ARGUMENT: input must be 4-dimensional[1,32,99]
[[{{node model/conv1d/Conv1D}}]]
[[model/activation_13/Relu/_20]]
(1) INVALID_ARGUMENT: input must be 4-dimensional[1,32,99]
[[{{node model/conv1d/Conv1D}}]]
0 successful operations.
0 derived errors ignored. [Op:__inference_predict_function_34672]

Let me know if you have any advice!

Thanks,
Noah

Hi CHEUI team,

I have a few questions about the preprocessing step.

I have finished the preprocessing for the nanopolish files with the C++ script and there are 26 outputs for that. I tried to rejoin the signal files by using the combine_split_files.py but there is an error message:

"combine_split_files: error: unrecognised arguments:"

I tried to input the directory and single file of those files and they turned out with the same message. Only the first split file seems to have no error. I can confirm that I have added headers into the split file during the preprocessing while I put the individual signal files into model 1 and they are working.

Bw,
Dominic

Hi CHEUI team,

I'm running the below cheui script for almost 24 hours with 28 CPU in HPC, however, it seems slow. Is there any way to specify CPU core as we can do in CHEUI preprocessing steps?

python /scratch/qbi/uqmrah19/CHEUI/scripts/CHEUI_predict_model1.py
-i /scratch/qbi/uqmrah19/CHEUI/cheui/ONT_Reza/cheui_analysis/ctrl1/out_A_signals+IDs.p/eventalign_signals+IDS.p
-m /scratch/qbi/uqmrah19/CHEUI/CHEUI_trained_models/CHEUI_m6A_model1.h5
-o ./read_level_m6A_predictions_WT_rep1.txt
-l WT_rep1

I appreciate your time.

Best regards
Reza

Hello there,

I came across an error message while running the final site-based prediction: '../scripts/CHEUI_predict_model2.py'
"transpose expects a vector of size 3. But input(1) is a vector of size 4"

I found the fix. In '../scripts/CHEUI_predict_model2.py', change the following
lr_probs = model.predict(prob_vectors)
to:
lr_probs = model.predict(prob_vectors.reshape(prob_vectors.shape[0], 99, 1))

I am not sure if others have also experienced the same, but just wanted to share this anyway.
Hope this helps.

Hi Dear CHEUI developers,

Thank you so much for creating this magical tool to detect RNA modifications.

I would like to ask whether CHEUI models work on Human only or it might work on Bacteria too?

Thanks,

Josh

Hi developer,

Thanks for developing CHEUI. I ran the CHEUI_predict_model1.py step with the command:

python /work/bio-mowp/software/CHEUI/scripts/CHEUI_predict_model1.py -i out_A_signals+IDs.p/nanopolish_output_signals+IDS.p -m /work/bio-mowp/software/CHEUI/CHEUI_trained_models/CHEUI_m6A_model1.h5 -o ./read_level_m6A_predictions_gpu.txt -l WT

The stderr log output is as follows:

2022-12-02 09:44:41.223660: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-12-02 09:45:16.648827: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2022-12-02 09:45:16.650310: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2400000000 Hz

It has not used any GPU yet:

Fri Dec  2 09:55:16 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.73.01    Driver Version: 460.73.01    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla V100-PCIE...  Off  | 00000000:2F:00.0 Off |                    0 |
| N/A   28C    P0    23W / 250W |      0MiB / 16160MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  Tesla V100-PCIE...  Off  | 00000000:86:00.0 Off |                    0 |
| N/A   30C    P0    23W / 250W |      0MiB / 16160MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

How do I use TensorFlow GPU? I have installed tensorflow and tensorflow-gpu.

# packages in environment at /work/bio-mowp/anaconda3/envs/cheui:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
_tflow_select             2.1.0                       gpu    defaults
abseil-cpp                20210324.2           h9c3ff4c_0    conda-forge
absl-py                   1.3.0              pyhd8ed1ab_0    conda-forge
aiohttp                   3.8.3            py37h540881e_0    conda-forge
aiosignal                 1.3.1              pyhd8ed1ab_0    conda-forge
astunparse                1.6.3              pyhd8ed1ab_0    conda-forge
async-timeout             4.0.2              pyhd8ed1ab_0    conda-forge
asynctest                 0.13.0                     py_0    conda-forge
attrs                     22.1.0             pyh71513ae_1    conda-forge
blinker                   1.5                pyhd8ed1ab_0    conda-forge
brotlipy                  0.7.0           py37h540881e_1004    conda-forge
c-ares                    1.18.1               h7f98852_0    conda-forge
ca-certificates           2022.10.11           h06a4308_0    defaults
cachetools                4.2.4              pyhd8ed1ab_0    conda-forge
certifi                   2022.9.24          pyhd8ed1ab_0    conda-forge
cffi                      1.15.1           py37h43b0acd_1    conda-forge
charset-normalizer        2.1.1              pyhd8ed1ab_0    conda-forge
click                     8.1.3            py37h89c1867_0    conda-forge
cryptography              38.0.2           py37h38fbfac_1    conda-forge
exceptiongroup            1.0.4              pyhd8ed1ab_0    conda-forge
frozenlist                1.3.1            py37h540881e_0    conda-forge
gast                      0.3.3                      py_0    conda-forge
giflib                    5.2.1                h36c2ea0_2    conda-forge
google-auth               1.35.0             pyh6c4a22f_0    conda-forge
google-auth-oauthlib      0.4.6              pyhd8ed1ab_0    conda-forge
google-pasta              0.2.0              pyh8c360ce_0    conda-forge
grpc-cpp                  1.37.1               hfc4d69e_3    conda-forge
grpcio                    1.37.1           py37hb27c1af_0    conda-forge
h5py                      2.10.0          nompi_py37ha3df211_106    conda-forge
hdf5                      1.10.6          nompi_h6a2412b_1114    conda-forge
icu                       68.2                 h9c3ff4c_0    conda-forge
idna                      3.4                pyhd8ed1ab_0    conda-forge
importlib-metadata        4.11.4           py37h89c1867_0    conda-forge
importlib_metadata        4.11.4               hd8ed1ab_0    conda-forge
iniconfig                 1.1.1              pyh9f0ad1d_0    conda-forge
jpeg                      9e                   h166bdaf_2    conda-forge
keras-preprocessing       1.1.2              pyhd8ed1ab_0    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
krb5                      1.19.3               h3790be6_0    conda-forge
ld_impl_linux-64          2.39                 hcc3a1bd_1    conda-forge
libblas                   3.9.0           16_linux64_openblas    conda-forge
libcblas                  3.9.0           16_linux64_openblas    conda-forge
libcurl                   7.86.0               h7bff187_1    conda-forge
libedit                   3.1.20210910         h7f8727e_0    defaults
libev                     4.33                 h516909a_1    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 12.2.0              h65d4601_19    conda-forge
libgfortran-ng            12.2.0              h69a702a_19    conda-forge
libgfortran5              12.2.0              h337968e_19    conda-forge
libgomp                   12.2.0              h65d4601_19    conda-forge
liblapack                 3.9.0           16_linux64_openblas    conda-forge
libnghttp2                1.47.0               hdcd2b5c_1    conda-forge
libopenblas               0.3.21          pthreads_h78a6416_3    conda-forge
libpng                    1.6.39               h753d276_0    conda-forge
libprotobuf               3.15.8               h780b84a_1    conda-forge
libsqlite                 3.40.0               h753d276_0    conda-forge
libssh2                   1.10.0               haa6b8db_3    conda-forge
libstdcxx-ng              12.2.0              h46fd767_19    conda-forge
libzlib                   1.2.13               h166bdaf_4    conda-forge
markdown                  3.4.1              pyhd8ed1ab_0    conda-forge
markupsafe                2.1.1            py37h540881e_1    conda-forge
multidict                 6.0.2            py37h540881e_1    conda-forge
ncurses                   6.3                  h27087fc_1    conda-forge
numpy                     1.21.6           py37h976b520_0    conda-forge
oauthlib                  3.2.2              pyhd8ed1ab_0    conda-forge
openssl                   1.1.1s               h166bdaf_0    conda-forge
opt_einsum                3.3.0              pyhd8ed1ab_1    conda-forge
packaging                 21.3               pyhd8ed1ab_0    conda-forge
pandas                    1.3.4            py37he8f5f7f_1    conda-forge
pip                       22.3.1             pyhd8ed1ab_0    conda-forge
pluggy                    1.0.0            py37h89c1867_3    conda-forge
protobuf                  3.15.8           py37hcd2ae1e_0    conda-forge
pyasn1                    0.4.8                      py_0    conda-forge
pyasn1-modules            0.2.8                      py_0    defaults
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pyjwt                     2.6.0              pyhd8ed1ab_0    conda-forge
pyopenssl                 22.1.0             pyhd8ed1ab_0    conda-forge
pyparsing                 3.0.9              pyhd8ed1ab_0    conda-forge
pysocks                   1.7.1            py37h89c1867_5    conda-forge
pytest                    7.2.0            py37h89c1867_0    conda-forge
python                    3.7.15               h7a1cb2a_1    defaults
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-flatbuffers        1.12               pyhd8ed1ab_1    conda-forge
python_abi                3.7                     2_cp37m    conda-forge
pytz                      2022.6             pyhd8ed1ab_0    conda-forge
pyu2f                     0.1.5              pyhd8ed1ab_0    conda-forge
re2                       2021.08.01           h9c3ff4c_0    conda-forge
readline                  8.2                  h5eee18b_0    defaults
requests                  2.28.1             pyhd8ed1ab_1    conda-forge
requests-oauthlib         1.3.1              pyhd8ed1ab_0    conda-forge
rsa                       4.9                pyhd8ed1ab_0    conda-forge
scipy                     1.7.3            py37hf2a6cf1_0    conda-forge
setuptools                59.8.0           py37h89c1867_1    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
snappy                    1.1.9                hbd366e4_2    conda-forge
sqlite                    3.40.0               h4ff8645_0    conda-forge
tensorboard               2.4.1              pyhd8ed1ab_1    conda-forge
tensorboard-plugin-wit    1.8.1              pyhd8ed1ab_0    conda-forge
tensorflow                2.4.1            py37h89c1867_0    conda-forge
tensorflow-base           2.4.1            py37he2fe834_0    conda-forge
tensorflow-estimator      2.4.1              pyheb71bc4_0    defaults
tensorflow-gpu            2.4.1                h30adc30_0    defaults
termcolor                 2.1.1              pyhd8ed1ab_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
typing-extensions         4.4.0                hd8ed1ab_0    conda-forge
typing_extensions         4.4.0              pyha770c72_0    conda-forge
urllib3                   1.26.13            pyhd8ed1ab_0    conda-forge
werkzeug                  2.2.2              pyhd8ed1ab_0    conda-forge
wheel                     0.38.4             pyhd8ed1ab_0    conda-forge
wrapt                     1.14.1           py37h540881e_0    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
yarl                      1.8.1            py37h5eee18b_0    defaults
zipp                      3.11.0             pyhd8ed1ab_0    conda-forge
zlib                      1.2.13               h166bdaf_4    conda-forge

I appreciate your time and help.

Thanks a lot.
Best regards
Weipeng

Dear CHEUI,

Can you add an optional flag for your preprocessing script that allows designation of a temporary directory for writing temporary files that is separate from the final output directory? This is with clusters and/or cloud computing in mind, and/or users that have multiple discs at their disposal.

Thanks

Hi,

When I try to run CHEUI_diffenrentialRNAMod.py I get this error without any output:

error at reading yml file
mapping values are not allowed here
  in "config.yml", line 7, column 13
Traceback (most recent call last):
  File "/home/rania/softs/CHEUI/scripts/CHEUI_diffenrentialRNAMod.py", line 52, in <module>
    input_file = config_dic['input']
NameError: name 'config_dic' is not defined

Could you help please ?
Rania