fhalab / mlde Goto Github PK

Hi @brucejwittmann ,

I have installed the MLDE as per your instructions. However, when I get to the tests It fails

This is what I get for for:

>mlde
Traceback (most recent call last):
File "generate_encoding.py", line 65, in
main()
File "generate_encoding.py", line 16, in main
from code.encode.encoding_generator import EncodingGenerator
File "/home/ubuntu/MLDE/code/encode/encoding_generator.py", line 13, in
from Bio import SeqIO
File "/home/ubuntu/anaconda3/envs/mlde/lib/python3.7/site-packages/Bio/SeqIO/init.py", line 387, in
from Bio.Align import MultipleSeqAlignment
File "/home/ubuntu/anaconda3/envs/mlde/lib/python3.7/site-packages/Bio/Align/init.py", line 22, in
from Bio.Align import _aligners
ImportError: /home/ubuntu/anaconda3/envs/mlde/lib/python3.7/site-packages/Bio/Align/_aligners.cpython-37m-x86_64-linux-gnu.so: invalid ELF header

>mlde2
/home/ubuntu/anaconda3/envs/mlde2/lib/python3.7/site-packages/sklearn/linear_mod el/least_angle.py:30: DeprecationWarning: np.float is a deprecated alias for t he builtin float. To silence this warning, use float by itself. Doing this w ill not modify any behavior and is safe. If you specifically wanted the numpy sc alar type, use np.float64 here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdo cs/release/1.20.0-notes.html#deprecations
method='lar', copy_X=True, eps=np.finfo(np.float).eps,
/home/ubuntu/anaconda3/envs/mlde2/lib/python3.7/site-packages/sklearn/linear_mod el/least_angle.py:167: DeprecationWarning: np.float is a deprecated alias for the builtin float. To silence this warning, use float by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy s calar type, use np.float64 here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdo cs/release/1.20.0-notes.html#deprecations
method='lar', copy_X=True, eps=np.finfo(np.float).eps,
/home/ubuntu/anaconda3/envs/mlde2/lib/python3.7/site-packages/sklearn/linear_mod el/least_angle.py:284: DeprecationWarning: np.float is a deprecated alias for the builtin float. To silence this warning, use float by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy s calar type, use np.float64 here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdo cs/release/1.20.0-notes.html#deprecations
eps=np.finfo(np.float).eps, copy_Gram=True, verbose=0,
/home/ubuntu/anaconda3/envs/mlde2/lib/python3.7/site-packages/sklearn/linear_mod el/least_angle.py:862: DeprecationWarning: np.float is a deprecated alias for the builtin float. To silence this warning, use float by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy s calar type, use np.float64 here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdo cs/release/1.20.0-notes.html#deprecations
eps=np.finfo(np.float).eps, copy_X=True, fit_path=True,
/home/ubuntu/anaconda3/envs/mlde2/lib/python3.7/site-packages/sklearn/linear_mod el/least_angle.py:1101: DeprecationWarning: np.float is a deprecated alias for the builtin float. To silence this warning, use float by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use np.float64 here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdo cs/release/1.20.0-notes.html#deprecations
eps=np.finfo(np.float).eps, copy_X=True, fit_path=True,
/home/ubuntu/anaconda3/envs/mlde2/lib/python3.7/site-packages/sklearn/linear_mod el/least_angle.py:1127: DeprecationWarning: np.float is a deprecated alias for the builtin float. To silence this warning, use float by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use np.float64 here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdo cs/release/1.20.0-notes.html#deprecations
eps=np.finfo(np.float).eps, positive=False):
/home/ubuntu/anaconda3/envs/mlde2/lib/python3.7/site-packages/sklearn/linear_mod el/least_angle.py:1362: DeprecationWarning: np.float is a deprecated alias for the builtin float. To silence this warning, use float by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use np.float64 here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdo cs/release/1.20.0-notes.html#deprecations
max_n_alphas=1000, n_jobs=None, eps=np.finfo(np.float).eps,
/home/ubuntu/anaconda3/envs/mlde2/lib/python3.7/site-packages/sklearn/linear_mod el/least_angle.py:1602: DeprecationWarning: np.float is a deprecated alias for the builtin float. To silence this warning, use float by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use np.float64 here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdo cs/release/1.20.0-notes.html#deprecations
max_n_alphas=1000, n_jobs=None, eps=np.finfo(np.float).eps,
/home/ubuntu/anaconda3/envs/mlde2/lib/python3.7/site-packages/sklearn/linear_mod el/least_angle.py:1738: DeprecationWarning: np.float is a deprecated alias for the builtin float. To silence this warning, use float by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use np.float64 here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdo cs/release/1.20.0-notes.html#deprecations
eps=np.finfo(np.float).eps, copy_X=True, positive=False):
Traceback (most recent call last):
File "predict_zero_shot.py", line 49, in
main()
File "predict_zero_shot.py", line 12, in main
from code.zero_shot.support_funcs import check_args, run_zero_shot
File "/home/ubuntu/MLDE/code/zero_shot/support_funcs.py", line 8, in
from .zero_shot_predictor import ZeroShotPredictor
File "/home/ubuntu/MLDE/code/zero_shot/zero_shot_predictor.py", line 12, in
from ..encode.sequence_loader import SequenceLoader
File "/home/ubuntu/MLDE/code/encode/sequence_loader.py", line 9, in
from Bio import SeqIO
File "/home/ubuntu/anaconda3/envs/mlde2/lib/python3.7/site-packages/Bio/SeqIO/ init.py", line 387, in
from Bio.Align import MultipleSeqAlignment
File "/home/ubuntu/anaconda3/envs/mlde2/lib/python3.7/site-packages/Bio/Align/ init.py", line 22, in
from Bio.Align import _aligners
ImportError: /home/ubuntu/anaconda3/envs/mlde2/lib/python3.7/site-packages/Bio/A lign/_aligners.cpython-37m-x86_64-linux-gnu.so: invalid ELF header

from both tests, it looks that I have issues with the import of "_aligners.cpython-37m-x86_64-linux-gnu.so" file that is empty.

I have to mention that I am running the mlde within a AWS EC2 (I use a VM because our workstation has an AMD RX570 which makes it incompatible with the cudatoolkit and TensorFlow).
I would highly appreciate your help.

Cordially

When test the scirtp `generate_encoding.py', it always interrupt with the error: Blas GEMM launch failed: a.shape=(56, 512), b.shape](512,512), m=56, k=512. We test the code with GPU 3090 that has 24G free memory, so I think it is not due to the out of memory.

Hi, I followed your guide to install MLDE. The installation was successful and I run the first 2 data test. As far as I could tell everything was fine. When I started the pytest scripts it run into problems. It seems it can't locate the Support module. Inside the MLDE directory is a directory called Support though.

Do you have any idea what could have gone wrong?

Regards Christoph

(mlde) christoph@computer:~/bioinformatik/MLDE/MLDE$ PYTHONPATH=$home/christoph/bioinformatik/MLDE/MLDE/ pytest
============================================================================================= test session starts =============================================================================================
platform linux -- Python 3.7.3, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /home/christoph/bioinformatik/MLDE/MLDE, configfile: pytest.ini
plugins: cov-2.12.1
collected 1 item / 7 errors

=================================================================================================== ERRORS ====================================================================================================
_____________________________________________________________________ ERROR collecting Validation/pytest/Encode/test_EncodingGenerator.py _____________________________________________________________________
ImportError while importing test module '/home/christoph/bioinformatik/MLDE/MLDE/Validation/pytest/Encode/test_EncodingGenerator.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../../anaconda3/envs/mlde/lib/python3.7/importlib/init.py:127: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
Validation/pytest/Encode/test_EncodingGenerator.py:16: in
from Support.Encode.EncodingGenerator import EncodingGenerator
E ModuleNotFoundError: No module named 'Support'
__________________________________________________________________________ ERROR collecting Validation/pytest/Run/test_FinalizeX.py ___________________________________________________________________________
ImportError while importing test module '/home/christoph/bioinformatik/MLDE/MLDE/Validation/pytest/Run/test_FinalizeX.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../../anaconda3/envs/mlde/lib/python3.7/importlib/init.py:127: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
Validation/pytest/Run/test_FinalizeX.py:5: in
from Support.Params.Defaults import cpu_models, gpu_models
E ModuleNotFoundError: No module named 'Support'
__________________________________________________________________________ ERROR collecting Validation/pytest/Run/test_LossFuncs.py ___________________________________________________________________________
ImportError while importing test module '/home/christoph/bioinformatik/MLDE/MLDE/Validation/pytest/Run/test_LossFuncs.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../../anaconda3/envs/mlde/lib/python3.7/importlib/init.py:127: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
Validation/pytest/Run/test_LossFuncs.py:5: in
from Support.RunMlde.LossFuncs import mse
E ModuleNotFoundError: No module named 'Support'
_________________________________________________________________________ ERROR collecting Validation/pytest/Run/test_MldeClasses.py __________________________________________________________________________
ImportError while importing test module '/home/christoph/bioinformatik/MLDE/MLDE/Validation/pytest/Run/test_MldeClasses.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../../anaconda3/envs/mlde/lib/python3.7/importlib/init.py:127: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
Validation/pytest/Run/test_MldeClasses.py:5: in
from Support.RunMlde.MldeClasses import (MldeModel, KerasModel,
E ModuleNotFoundError: No module named 'Support'
_________________________________________________________________________ ERROR collecting Validation/pytest/Run/test_MldeHyperopt.py _________________________________________________________________________
ImportError while importing test module '/home/christoph/bioinformatik/MLDE/MLDE/Validation/pytest/Run/test_MldeHyperopt.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../../anaconda3/envs/mlde/lib/python3.7/importlib/init.py:127: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
Validation/pytest/Run/test_MldeHyperopt.py:12: in
from Support.Params.Defaults import (cpu_models, gpu_models, default_model_params,
E ModuleNotFoundError: No module named 'Support'
___________________________________________________________________________ ERROR collecting Validation/pytest/Run/test_RunFuncs.py ___________________________________________________________________________
ImportError while importing test module '/home/christoph/bioinformatik/MLDE/MLDE/Validation/pytest/Run/test_RunFuncs.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../../anaconda3/envs/mlde/lib/python3.7/importlib/init.py:127: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
Validation/pytest/Run/test_RunFuncs.py:14: in
from Support.RunMlde.RunFuncs import (process_results, get_training_from_design,
E ModuleNotFoundError: No module named 'Support'
_______________________________________________________________________ ERROR collecting Validation/pytest/Run/test_TrainAndPredict.py ________________________________________________________________________
ImportError while importing test module '/home/christoph/bioinformatik/MLDE/MLDE/Validation/pytest/Run/test_TrainAndPredict.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../../anaconda3/envs/mlde/lib/python3.7/importlib/init.py:127: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
Validation/pytest/Run/test_TrainAndPredict.py:6: in
from Support.Params.Defaults import (gpu_models, cpu_models,
E ModuleNotFoundError: No module named 'Support'
============================================================================================== warnings summary ===============================================================================================
../../../anaconda3/envs/mlde/lib/python3.7/site-packages/Bio/Alphabet/init.py:26
/home/christoph/anaconda3/envs/mlde/lib/python3.7/site-packages/Bio/Alphabet/init.py:26: PendingDeprecationWarning: We intend to remove or replace Bio.Alphabet in 2020, ideally avoid using it explicitly in your code. Please get in touch if you will be adversely affected by this. biopython/biopython#2046
PendingDeprecationWarning)

-- Docs: https://docs.pytest.org/en/stable/warnings.html
=========================================================================================== short test summary info ===========================================================================================
ERROR Validation/pytest/Encode/test_EncodingGenerator.py
ERROR Validation/pytest/Run/test_FinalizeX.py
ERROR Validation/pytest/Run/test_LossFuncs.py
ERROR Validation/pytest/Run/test_MldeClasses.py
ERROR Validation/pytest/Run/test_MldeHyperopt.py
ERROR Validation/pytest/Run/test_RunFuncs.py
ERROR Validation/pytest/Run/test_TrainAndPredict.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 7 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
======================================================================================== 1 warning, 7 errors in 0.75s =========================================================================================

If a large portion of my experimental data is single site mutation data, should I use these single site mutation data when running execute_mlde.py. In addition to the single-site mutation data, I also have some data for five-site mutations.

The specified versions of packages in mlde.yml are quite outdated, making them unavailable. Additionally, newer versions of these packages may offer improved performance. It might be advisable to update the versions of these packages.

when I run "conda activate mlde
python generate_encoding.py transformer GB1_T2Q --fasta ./code/validation/basic_test_data/2GI9.fasta --positions V39 D40 --batches 1
"
I got an error:

FileNotFoundError: [Errno 2] No such file or directory: '/data/qm/MLDE/20211129-131456/Encodings/GB1_T2Q_transformer_TempOutputs.pkl'

log :
"
/data/qm/MLDE/code/encode/encoding_generator.py:32: UserWarning: Could not load TRANSFORMER_TO_CLASS. This is expected if you are running in the mlde environment or a custom environment without PyTorch installed. Otherwise, you might have a problem. It will not be possible to build encodings from ESM and ProtTrans with this import not working.
warnings.warn("Could not load TRANSFORMER_TO_CLASS. This is expected if "
Batch#: 0%| | 0/1 [00:00<?, ?it/s]Transformer with Parameters:
n_layers: 12
n_heads: 8
d_model: 512
d_filter: 2048
dropout: 0.1
WARNING:tensorflow:From /data/qm/anaconda3/envs/mlde/lib/python3.7/site-packages/tensorflow/python/ops/resource_variable_ops.py:435: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
2021-11-29 13:17:26.387501: E tensorflow/stream_executor/cuda/cuda_blas.cc:698] failed to run cuBLAS routine cublasSgemm_v2: CUBLAS_STATUS_EXECUTION_FAILED
2021-11-29 13:17:26.387706: E tensorflow/stream_executor/cuda/cuda_blas.cc:698] failed to run cuBLAS routine cublasSgemm_v2: CUBLAS_STATUS_EXECUTION_FAILED
2021-11-29 13:17:26.387830: E tensorflow/stream_executor/cuda/cuda_blas.cc:698] failed to run cuBLAS routine cublasSgemm_v2: CUBLAS_STATUS_EXECUTION_FAILED
Traceback (most recent call last):
File "/data/qm/anaconda3/envs/mlde/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call
return fn(*args)
File "/data/qm/anaconda3/envs/mlde/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/data/qm/anaconda3/envs/mlde/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InternalError: Blas GEMM launch failed : a.shape=(56, 512), b.shape=(512, 512), m=56, n=512, k=512
[[{{node transformer/transformer_encoder/encoder_stack/transformer_encoder_block/transformer_self_attention/self_attention/multi_head_attention/attention_qkv_projection/weight_norm_dense_26/Tensordot/MatMul}}]]
[[{{node transformer/transformer_encoder/encoder_stack/transformer_encoder_block_11/transformer_feed_forward_11/add}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/data/qm/anaconda3/envs/mlde/bin/tape-embed", line 33, in
sys.exit(load_entry_point('tape', 'console_scripts', 'tape-embed')())
File "/data/qm/MLDE/code/tape-neurips2019/tape/run_embed.py", line 91, in main
embeddings = run_embed(args.datafile, args.model, args.load_from, args.task)
File "/data/qm/MLDE/code/tape-neurips2019/tape/run_embed.py", line 47, in run_embed
protein_length: [int_sequence.shape[1]]})
File "/data/qm/anaconda3/envs/mlde/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 929, in run
run_metadata_ptr)
File "/data/qm/anaconda3/envs/mlde/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1152, in _run
feed_dict_tensor, options, run_metadata)
File "/data/qm/anaconda3/envs/mlde/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
run_metadata)
File "/data/qm/anaconda3/envs/mlde/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InternalError: Blas GEMM launch failed : a.shape=(56, 512), b.shape=(512, 512), m=56, n=512, k=512
[[node transformer/transformer_encoder/encoder_stack/transformer_encoder_block/transformer_self_attention/self_attention/multi_head_attention/attention_qkv_projection/weight_norm_dense_26/Tensordot/MatMul (defined at /data/qm/anaconda3/envs/mlde/lib/python3.7/site-packages/rinokeras/core/v1x/common/layers/normalization.py:76) ]]
[[node transformer/transformer_encoder/encoder_stack/transformer_encoder_block_11/transformer_feed_forward_11/add (defined at /data/qm/anaconda3/envs/mlde/lib/python3.7/site-packages/rinokeras/core/v1x/models/transformer/transformer_ff.py:67) ]]

Caused by op 'transformer/transformer_encoder/encoder_stack/transformer_encoder_block/transformer_self_attention/self_attention/multi_head_attention/attention_qkv_projection/weight_norm_dense_26/Tensordot/MatMul', defined at:
File "/data/qm/anaconda3/envs/mlde/bin/tape-embed", line 33, in
sys.exit(load_entry_point('tape', 'console_scripts', 'tape-embed')())
File "/data/qm/MLDE/code/tape-neurips2019/tape/run_embed.py", line 91, in main
embeddings = run_embed(args.datafile, args.model, args.load_from, args.task)
File "/data/qm/MLDE/code/tape-neurips2019/tape/run_embed.py", line 37, in run_embed
output = embedding_model({'primary': primary, 'protein_length': protein_length})
File "/data/qm/anaconda3/envs/mlde/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 554, in call
outputs = self.call(inputs, *args, **kwargs)
File "/data/qm/MLDE/code/tape-neurips2019/tape/models/Transformer.py", line 83, in call
encoder_output = self.encoder(sequence, mask=attention_mask)
File "/data/qm/anaconda3/envs/mlde/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 554, in call
outputs = self.call(inputs, *args, **kwargs)
File "/data/qm/anaconda3/envs/mlde/lib/python3.7/site-packages/rinokeras/core/v1x/models/transformer/transformer_encoder.py", line 243, in call
inputs, mask=(encoder_mask, conv_mask))
File "/data/qm/anaconda3/envs/mlde/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 554, in call
outputs = self.call(inputs, *args, **kwargs)
File "/data/qm/anaconda3/envs/mlde/lib/python3.7/site-packages/rinokeras/core/v1x/common/layers/stack.py", line 42, in call
output = layer(output, **layer_args)
File "/data/qm/anaconda3/envs/mlde/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 554, in call
outputs = self.call(inputs, *args, **kwargs)
File "/data/qm/anaconda3/envs/mlde/lib/python3.7/site-packages/rinokeras/core/v1x/models/transformer/transformer_encoder.py", line 87, in call
inputs, mask=self_attention_mask, return_attention_weights=True)
File "/data/qm/anaconda3/envs/mlde/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 554, in call
outputs = self.call(inputs, *args, **kwargs)
File "/data/qm/anaconda3/envs/mlde/lib/python3.7/site-packages/rinokeras/core/v1x/models/transformer/transformer_attention.py", line 47, in call
attn_inputs, mask=mask, return_attention_weights=True)
File "/data/qm/anaconda3/envs/mlde/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 554, in call
outputs = self.call(inputs, *args, **kwargs)
File "/data/qm/anaconda3/envs/mlde/lib/python3.7/site-packages/rinokeras/core/v1x/common/attention.py", line 510, in call
return self.multi_attention((inputs, inputs, inputs), mask=mask, return_attention_weights=return_attention_weights)
File "/data/qm/anaconda3/envs/mlde/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 554, in call
outputs = self.call(inputs, *args, **kwargs)
File "/data/qm/anaconda3/envs/mlde/lib/python3.7/site-packages/rinokeras/core/v1x/common/attention.py", line 454, in call
qkv_projection = self.compute_qkv((qa, ma, va))
File "/data/qm/anaconda3/envs/mlde/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 554, in call
outputs = self.call(inputs, *args, **kwargs)
File "/data/qm/anaconda3/envs/mlde/lib/python3.7/site-packages/rinokeras/core/v1x/common/attention.py", line 116, in call
values = self.value_norm(self.value_layer(value_antecedent))
File "/data/qm/anaconda3/envs/mlde/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 554, in call
outputs = self.call(inputs, *args, **kwargs)
File "/data/qm/anaconda3/envs/mlde/lib/python3.7/site-packages/rinokeras/core/v1x/common/layers/normalization.py", line 76, in call
outputs = standard_ops.tensordot(inputs, self.kernel, [[rank - 1], [0]])
File "/data/qm/anaconda3/envs/mlde/lib/python3.7/site-packages/tensorflow/python/ops/math_ops.py", line 3583, in tensordot
ab_matmul = matmul(a_reshape, b_reshape)
File "/data/qm/anaconda3/envs/mlde/lib/python3.7/site-packages/tensorflow/python/ops/math_ops.py", line 2455, in matmul
a, b, transpose_a=transpose_a, transpose_b=transpose_b, name=name)
File "/data/qm/anaconda3/envs/mlde/lib/python3.7/site-packages/tensorflow/python/ops/gen_math_ops.py", line 5333, in mat_mul
name=name)
File "/data/qm/anaconda3/envs/mlde/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "/data/qm/anaconda3/envs/mlde/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/data/qm/anaconda3/envs/mlde/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 3300, in create_op
op_def=op_def)
File "/data/qm/anaconda3/envs/mlde/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 1801, in init
self._traceback = tf_stack.extract_stack()

InternalError (see above for traceback): Blas GEMM launch failed : a.shape=(56, 512), b.shape=(512, 512), m=56, n=512, k=512
[[node transformer/transformer_encoder/encoder_stack/transformer_encoder_block/transformer_self_attention/self_attention/multi_head_attention/attention_qkv_projection/weight_norm_dense_26/Tensordot/MatMul (defined at /data/qm/anaconda3/envs/mlde/lib/python3.7/site-packages/rinokeras/core/v1x/common/layers/normalization.py:76) ]]
[[node transformer/transformer_encoder/encoder_stack/transformer_encoder_block_11/transformer_feed_forward_11/add (defined at /data/qm/anaconda3/envs/mlde/lib/python3.7/site-packages/rinokeras/core/v1x/models/transformer/transformer_ff.py:67) ]]

Traceback (most recent call last):
File "generate_encoding.py", line 65, in
main()
File "generate_encoding.py", line 61, in main
batch_size = args.batch_size)
File "/data/qm/MLDE/code/encode/encoding_generator.py", line 332, in generate_encodings
unnormalized_embeddings = self._generate_tape(n_batches)
File "/data/qm/MLDE/code/encode/encoding_generator.py", line 284, in _generate_tape
with open(temp_filename, "rb") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/data/qm/MLDE/20211129-131456/Encodings/GB1_T2Q_transformer_TempOutputs.pkl'
"

When I test the pytest, it always reported an error.

1. The error is about > test_encoding_generator.py

subprocess.CalledProcessError: Command '['python', './code/esm/extract.py', 'esm1b_t33_650M_UR50S', './code/validation/pytest/encode/test_data/DummyFastaEmbeddingTest.fasta', './code/validation/pytest/encode/test_data/EncodingOutput/', '--include', 'per_tok', '--toks_per_batch', '29']' returned non-zero exit status 1.

../../software/anaconda/miniconda/envs/mlde/lib/python3.7/subprocess.py:512: CalledProcessError

FAILED code/validation/pytest/encode/test_encoding_generator.py::test_generate_encodings - subprocess.CalledProcessError: Command '['python', './code/esm/extract.py', 'esm1b_t33_650M_UR50S', './code/vali...
================================================================================== 1 failed, 6 passed in 758.24s (0:12:38) ===

Look forward to your reply.
Thanks!