A bioinformatics API to interface with public multi-omics bio databases for wicked fast data integration.
Home Page: https://openomics.readthedocs.io/en/latest/
License: MIT License
![deepsource-autofix[bot] avatar](https://avatars.githubusercontent.com/in/57168?v=4 "deepsource-autofix[bot]")
The docs have https://openomics.readthedocs.io/en/latest/api/openomics.database.disease.OMIM.html
Regardless, the class is empty, so we should fill out OMIM. Thank you!
Installation with pip install openomics worked fine on the Linux system I am using. However, using my windows computer, I got the following error: error: Microsoft Visual C++ 14.0 is required, but I cannot find this in the requirements
from openomics import MultiOmics results in the "UserWarning: Tensorflow not installed; ParametricUMAP will be unavailable"
Hey, all! I'm a fan of this package and hope to use it within my work. I saw the tests were failing so I took a stab at trying to get that working. I was able to fix some flake8 stuff in #166 which at least got to the testing. From there, I found that the imports of the modules was causing testing to not even run. I've submitted a separate PR (#167) for that.
Below are just some of the errors from the tests. I'm guessing some of that is just due to some changes in the packages API, but since I'm not extremely familiar with everything, I figured I'd just share it here it to help.
If it's preferable to leave that PR open to debug, please let me know - or I can close it.
============================================================================= short test summary info =============================================================================
FAILED tests/test_annotations.py::test_annotate_GeneOntology - KeyError: 'gene_name'
ERROR tests/test_annotations.py::test_import_rnacentral_db - Exception: Unable to download file at ftp://ftp.ebi.ac.uk/pub/databases/RNAcentral/current_release/go_annotations/r...
ERROR tests/test_annotations.py::test_annotate_rnacentral - Exception: Unable to download file at ftp://ftp.ebi.ac.uk/pub/databases/RNAcentral/current_release/go_annotations/rn...
ERROR tests/test_annotations.py::test_import_MessengerRNA_Dask - TypeError: initialize_annotations() got an unexpected keyword argument 'gene_list'
ERROR tests/test_disease.py::test_import_HMDD - Exception: Unable to download file at http://www.cuilab.cn/static/hmdd3/data/alldata.txt. Please try manually downloading the fi...
ERROR tests/test_disease.py::test_annotate_HMDD - Exception: Unable to download file at http://www.cuilab.cn/static/hmdd3/data/alldata.txt. Please try manually downloading the ...
ERROR tests/test_disease.py::test_LncRNADisease - Exception: Unable to download file at http://www.cuilab.cn/files/images/ldd/data_v2017.txt. Please try manually downloading th...
ERROR tests/test_disease.py::test_annotate_LncRNADisease - Exception: Unable to download file at http://www.cuilab.cn/files/images/ldd/data_v2017.txt. Please try manually downl...
ERROR tests/test_disease.py::test_import_MalaCards - Exception: Unable to download file at http://zdzlab.einstein.yu.edu/1/hedd/download.action.php?filename=DataDownload/MalaCa...
ERROR tests/test_disease.py::test_import_MessengerRNA_Dask - TypeError: initialize_annotations() got an unexpected keyword argument 'gene_list'
ERROR tests/test_interaction.py::test_import_LncRNA2Target - Exception: Unable to download file at http://123.59.132.21/lncrna2target/data/lncrna_target.rar. Please try manuall...
ERROR tests/test_interaction.py::test_import_STRING - networkx.exception.NetworkXError: Invalid edge_attr argument: No columns found with name: []
ERROR tests/test_interaction.py::test_annotate_STRING - networkx.exception.NetworkXError: Invalid edge_attr argument: No columns found with name: []
ERROR tests/test_interaction.py::test_get_interactions_lnc2target - Exception: Unable to download file at http://123.59.132.21/lncrna2target/data/lncrna_target.rar. Please try ...
ERROR tests/test_interaction.py::test_import_MessengerRNA_Dask - TypeError: initialize_annotations() got an unexpected keyword argument 'gene_list'
ERROR tests/test_multiomics.py::test_import_MessengerRNA_Dask - TypeError: initialize_annotations() got an unexpected keyword argument 'gene_list'
ERROR tests/test_sequences.py::test_import_MessengerRNA_Dask - TypeError: initialize_annotations() got an unexpected keyword argument 'gene_list'
ERROR tests/test_sequences.py::test_import_mirbase_db - Exception: Unable to download file at ftp://mirbase.org/pub/mirbase/CURRENT/aliases.txt.gz. Please try manually download...
ERROR tests/test_sequences.py::test_import_GENCODE - Exception: Unable to download file at ftp://ftp.ebi.ac.uk/pub/databases/gencode/Gencode_human/release_32/gencode.v32.long_n...
ERROR tests/test_sequences.py::test_import_dask_GENCODE - TypeError: __init__() got an unexpected keyword argument 'npartitions'
ERROR tests/test_sequences.py::test_annotate_GENCODE - Exception: Unable to download file at ftp://ftp.ebi.ac.uk/pub/databases/gencode/Gencode_human/release_32/gencode.v32.long...
ERROR tests/test_sequences.py::test_annotate_dask_GENCODE - TypeError: __init__() got an unexpected keyword argument 'npartitions'
ERROR tests/test_sequences.py::test_annotate_sequence_GENCODE - Exception: Unable to download file at ftp://ftp.ebi.ac.uk/pub/databases/gencode/Gencode_human/release_32/gencode...
========================================================= 1 failed, 15 passed, 5 warnings, 22 errors in 757.03s (0:12:37) =========================================================
It is not completely clear what OpenOmics can be specifically used for. An overview of the available functions and methods would be great: What exactly does OpenOmics do? What are specific usage examples (also e.g. after using OpenOmics: What's next?)?:
'OpenOmics facilitates the common coding tasks when preparing data for bioinformatics analysis.': For which bioinformatic analyses exactly?
This should be included in the README file.
Running MiRTarBase(path="/data/datasets/Bioinformatics_ExternalData/miRTarBase/") in the generate_MiRTarBase test results in an error.
Some tests are available and run as part of the Travis CI pipeline, though coverage isn't amazing and would benefit from additional work. Focusing on test driven development is a good way to ensure greater coverage. Running the tests locally took a long time, and returned various warnings and an error.
☁ OpenOmics [master] python -m pytest --cov=./
============================================================================================================ test session starts =============================================================================================================
platform darwin -- Python 3.8.7, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
rootdir: /Users/stephenmoss/Dropbox/Code/OpenOmics, configfile: setup.cfg
plugins: cov-2.11.1, dash-1.19.0
collected 35 items
tests/test_annotations.py ......... [ 25%]
tests/test_disease.py .......... [ 54%]
tests/test_interaction.py .E..... [ 74%]
tests/test_multiomics.py ... [ 82%]
tests/test_sequences.py ...... [100%]
======================================================= ERRORS =======================================================
______________________________________ ERROR at setup of test_import_MiRTarBase ______________________________________
@pytest.fixture
def generate_MiRTarBase():
> return MiRTarBase(path="/data/datasets/Bioinformatics_ExternalData/miRTarBase/", strip_mirna_name=True,
filters={"Species (Target Gene)": "Homo sapiens"})
tests/test_interaction.py:19:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
openomics/database/interaction.py:611: in __init__
super(MiRTarBase, self).__init__(path=path, file_resources=file_resources,
openomics/database/interaction.py:40: in __init__
self.validate_file_resources(path, file_resources, verbose=verbose)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <openomics.database.interaction.MiRTarBase object at 0x1aa5036d0>
path = '/data/datasets/Bioinformatics_ExternalData/miRTarBase/'
file_resources = {'miRTarBase_MTI.xlsx': '/data/datasets/Bioinformatics_ExternalData/miRTarBase/miRTarBase_MTI.xlsx'}
npartitions = None, verbose = False
def validate_file_resources(self, path, file_resources, npartitions=None, verbose=False) -> None:
"""For each file in file_resources, fetch the file if path+file is a URL
or load from disk if a local path. Additionally unzip or unrar if the
file is compressed.
Args:
path (str): The folder or url path containing the data file
resources. If url path, the files will be downloaded and cached
to the user's home folder (at ~/.astropy/).
file_resources (dict): default None, Used to list required files for
preprocessing of the database. A dictionary where keys are
required filenames and value are file paths. If None, then the
class constructor should automatically build the required file
resources dict.
npartitions:
verbose:
"""
if validators.url(path):
for filename, filepath in copy.copy(file_resources).items():
data_file = get_pkg_data_filename(path, filepath,
verbose=verbose) # Download file and replace the file_resource path
filetype_ext = filetype.guess(data_file)
# This null if-clause is needed incase when filetype_ext is None, causing the next clause to fail
if filetype_ext is None:
file_resources[filename] = data_file
elif filetype_ext.extension == 'gz':
file_resources[filename] = gzip.open(data_file, 'rt')
elif filetype_ext.extension == 'zip':
zf = zipfile.ZipFile(data_file, 'r')
for subfile in zf.infolist():
if os.path.splitext(subfile.filename)[-1] == os.path.splitext(filename)[-1]: # If the file extension matches
file_resources[filename] = zf.open(subfile.filename, mode='r')
elif filetype_ext.extension == 'rar':
rf = rarfile.RarFile(data_file, 'r')
for subfile in rf.infolist():
if os.path.splitext(subfile.filename)[-1] == os.path.splitext(filename)[-1]: # If the file extension matches
file_resources[filename] = rf.open(subfile.filename, mode='r')
else:
file_resources[filename] = data_file
elif os.path.isdir(path) and os.path.exists(path):
for _, filepath in file_resources.items():
if not os.path.exists(filepath):
raise IOError(filepath)
else:
> raise IOError(path)
E OSError: /data/datasets/Bioinformatics_ExternalData/miRTarBase/
openomics/database/base.py:113: OSError
================================================== warnings summary ==================================================
../../../.pyenv/versions/3.8.7/lib/python3.8/site-packages/_pytest/config/__init__.py:1233
/Users/stephenmoss/.pyenv/versions/3.8.7/lib/python3.8/site-packages/_pytest/config/__init__.py:1233: PytestConfigWarning: Unknown config option: collect_ignore
self._warn_or_fail_if_strict(f"Unknown config option: {key}\n")
tests/test_annotations.py: 6 warnings
tests/test_disease.py: 6 warnings
tests/test_interaction.py: 4 warnings
tests/test_multiomics.py: 3 warnings
tests/test_sequences.py: 4 warnings
/Users/stephenmoss/Dropbox/Code/OpenOmics/openomics/transcriptomics.py:108: ParserWarning: Falling back to the 'python' engine because the 'c' engine does not support sep=None with delim_whitespace=False; you can avoid this warning by specifying engine='python'.
df = pd.read_table(data, sep=None)
tests/test_annotations.py::test_import_GTEx
tests/test_annotations.py::test_GTEx_annotate
/Users/stephenmoss/Dropbox/Code/OpenOmics/openomics/database/annotation.py:239: FutureWarning: The default value of regex will change from True to False in a future version.
gene_exp_medians["Name"] = gene_exp_medians["Name"].str.replace("[.].*", "")
tests/test_disease.py::test_import_HMDD
tests/test_disease.py::test_annotate_HMDD
/Users/stephenmoss/.pyenv/versions/3.8.7/lib/python3.8/encodings/unicode_escape.py:26: DeprecationWarning: invalid escape sequence '\ '
return codecs.unicode_escape_decode(input, self.errors)[0]
tests/test_disease.py::test_import_HMDD
tests/test_disease.py::test_annotate_HMDD
/Users/stephenmoss/.pyenv/versions/3.8.7/lib/python3.8/encodings/unicode_escape.py:26: DeprecationWarning: invalid escape sequence '\s'
return codecs.unicode_escape_decode(input, self.errors)[0]
tests/test_interaction.py::test_import_LncRNA2Target
/Users/stephenmoss/Dropbox/Code/OpenOmics/openomics/database/interaction.py:476: FutureWarning: Your version of xlrd is 1.2.0. In xlrd >= 2.0, only the xls format is supported. As a result, the openpyxl engine will be used if it is installed and the engine argument is not specified. Install openpyxl instead.
table = pd.read_excel(file_resources["lncRNA_target_from_low_throughput_experiments.xlsx"])
tests/test_interaction.py::test_import_LncRNA2Target
/Users/stephenmoss/Dropbox/Code/OpenOmics/openomics/database/interaction.py:480: FutureWarning: The default value of regex will change from True to False in a future version.
table["Target_official_symbol"] = table["Target_official_symbol"].str.replace("(?i)(mir)", "hsa-mir-")
-- Docs: https://docs.pytest.org/en/stable/warnings.html
---------- coverage: platform darwin, python 3.8.7-final-0 -----------
Name Stmts Miss Cover
--------------------------------------------------------------
openomics/__init__.py 25 7 72%
openomics/clinical.py 57 19 67%
openomics/database/__init__.py 7 0 100%
openomics/database/annotation.py 197 110 44%
openomics/database/base.py 140 37 74%
openomics/database/disease.py 61 1 98%
openomics/database/interaction.py 330 196 41%
openomics/database/ontology.py 152 93 39%
openomics/database/sequence.py 128 66 48%
openomics/genomics.py 26 8 69%
openomics/imageomics.py 63 47 25%
openomics/multicohorts.py 0 0 100%
openomics/multiomics.py 111 66 41%
openomics/proteomics.py 13 2 85%
openomics/transcriptomics.py 111 32 71%
openomics/utils/GTF.py 53 53 0%
openomics/utils/__init__.py 0 0 100%
openomics/utils/df.py 23 11 52%
openomics/utils/io.py 40 19 52%
openomics/utils/read_gtf.py 107 24 78%
openomics/visualization/__init__.py 1 0 100%
openomics/visualization/heatmat.py 11 8 27%
openomics/visualization/umap.py 29 24 17%
openomics_web/__init__.py 0 0 100%
openomics_web/app.py 69 69 0%
openomics_web/callbacks.py 0 0 100%
openomics_web/layouts/__init__.py 0 0 100%
openomics_web/layouts/annotation_view.py 0 0 100%
openomics_web/layouts/app_layout.py 7 7 0%
openomics_web/layouts/clinical_view.py 10 10 0%
openomics_web/layouts/control_tabs.py 5 5 0%
openomics_web/layouts/datatable_view.py 28 28 0%
openomics_web/server.py 2 2 0%
openomics_web/utils/__init__.py 0 0 100%
openomics_web/utils/io.py 62 62 0%
openomics_web/utils/str_utils.py 25 25 0%
setup.py 44 44 0%
tests/__init__.py 0 0 100%
tests/data/__init__.py 0 0 100%
tests/data/test_dask_dataframes.py 0 0 100%
tests/test_annotations.py 39 0 100%
tests/test_disease.py 34 0 100%
tests/test_interaction.py 20 1 95%
tests/test_multiomics.py 46 1 98%
tests/test_sequences.py 18 1 94%
--------------------------------------------------------------
TOTAL 2094 1078 49%
============================================== short test summary info ===============================================
ERROR tests/test_interaction.py::test_import_MiRTarBase - OSError: /data/datasets/Bioinformatics_ExternalData/miRTa...
================================ 34 passed, 32 warnings, 1 error in 662.47s (0:11:02) ================================
The main error seemed to be a missing dataset. On further inspection of the codebase it seems that the package is supposed to download the miRTarBase data (although it appears to have the version 7 release URL hardcoded when version 8 is now available). I wondered whether this was a permissions issue with not being able to create the /data/datasets/Bioinformatics_ExternalData/miRTarBase/ path on my system. I tried with sudo python -m pytest --cov=./ tests/test_interaction.py and got the same? I tried sudo mkdir -p /data/datasets/Bioinformatics_ExternalData/miRTarBase beforehand, which returned:
mkdir: /data/datasets/Bioinformatics_ExternalData/miRTarBase: Read-only file system
This is likely due to macOS system integrity protection.
However, it seems I am also unable to resolve http://mirtarbase.mbc.nctu.edu.tw/cache/download/7.0/. I believe the URL should actually be http://mirtarbase.cuhk.edu.cn/cache/download/7.0/ (or even http://mirtarbase.cuhk.edu.cn/cache/download/8.0/)? I manually updated to the working version 7.0 release and updated the path in test_interaction.py before running the following:
mkdir -p tests/data/datasets/Bioinformatics_ExternalData/miRTarBase
sudo python -m pytest --cov=./ tests/test_interaction.py
I still received the error, so something needs looking at in more detail here.
The following code was available in an earlier version of the README (now https://openomics.readthedocs.io/en/latest/) and contained a few incorrect points:
'# Load each expression dataframe': additional ')' in lines should be removed! as current form results in an error
mRNA = MessengerRNA(data=folder_path+"LUAD__geneExp.txt", transpose=True, usecols="GeneSymbol|TCGA", gene_index="GeneSymbol", gene_level="gene_name")
Results in warnings (only in first time use):
'/homes/.local/lib/python3.7/site-packages/openomics/transcriptomics.py:95: FutureWarning:
read_table is deprecated, use read_csv instead.
/homes/.local/lib/python3.7/site-packages/openomics/transcriptomics.py:95: ParserWarning:
Falling back to the 'python' engine because the 'c' engine does not support sep=None with delim_whitespace=False; you can avoid this warning by specifying engine='python'.'
som = SomaticMutation(data=folder_path+"LUAD__somaticMutation_geneLevel.txt", transpose=True, usecols="GeneSymbol|TCGA", gene_index="gene_name")
Results in: 'KeyError: 'gene_name''. This should probably be 'gene_index="GeneSymbol"'.
luad_data.add_clinical_data(clinical_data=folder_path+"nationwidechildrens.org_clinical_patient_luad.txt")
Results in warning: '/homes/.local/lib/python3.7/site-packages/openomics/clinical.py:51: FutureWarning:
read_table is deprecated, use read_csv instead, passing sep='\t'.'
gencode = GENCODE(path="ftp://ftp.ebi.ac.uk/pub/databases/gencode/Gencode_human/release_32/",file_resources={"long_noncoding_RNAs.gtf": "gencode.v32.long_noncoding_RNAs.gtf.gz","basic.annotation.gtf": "gencode.v32.basic.annotation.gtf.gz","lncRNA_transcripts.fa": "gencode.v32.lncRNA_transcripts.fa.gz","transcripts.fa": "gencode.v32.transcripts.fa.gz"},remove_version_num=True,npartitions=5)
Results in: _'AttributeError: 'io.TextIOWrapper' object has no attribute 'startswith''.
The MalaCards object throws an Exception when instantiating it:
Exception: Unable to download file at http://zdzlab.einstein.yu.edu/1/hedd/download.action.php?filename=DataDownload/MalaCards.csv. Please try manually downloading the files.
<urlopen error Unable to open any source! Exceptions were {'http://zdzlab.einstein.yu.edu/1/hedd/download.action.php?filename=DataDownload/MalaCards.csv': <HTTPError 404: 'Not Found'>, 'http://www.astropy.org/astropy-data/download.action.php?filename=DataDownload/MalaCards.csv': <HTTPError 404: 'Not Found'>}>
from openomics.database.disease import MalaCards
MalaCards()This doesn't work because http://zdzlab.einstein.yu.edu/1/hedd/download.action.php?filename=DataDownload/MalaCards.csv is throwing a 404 error.
The bot created this issue to inform you that pyup.io has been set up on this repo.
Once you have closed it, the bot will open pull requests for updates as soon as they are available.
When raising a PR the build in Travis CI failed for all python versions. There is some work needed to get these issues resolved though I didn't inspect the output in detail. See https://travis-ci.com/github/gawbul/OpenOmics/builds/216404830
Running import umap results in a chain of errors as pip install on Mac OS X doesn't correctly sets up BLAS and LAPACK libraries.
When trying to run an openomics_test.py file with the from openomics import MultiOmics statement I received the following:
Creating directory /Users/stephenmoss/Library/Application Support/bioservices
Matplotlib is building the font cache; this may take a moment.
Traceback (most recent call last):
File "/Users/stephenmoss/Dropbox/Code/openomics_test.py", line 1, in <module>
from openomics import MultiOmics
File "/Users/stephenmoss/Dropbox/Code/OpenOmics/openomics/__init__.py", line 40, in <module>
from .visualization import (
File "/Users/stephenmoss/Dropbox/Code/OpenOmics/openomics/visualization/umap.py", line 3, in <module>
import umap
File "/Users/stephenmoss/.pyenv/versions/3.9.0/lib/python3.9/site-packages/umap/__init__.py", line 2, in <module>
from .umap_ import UMAP
File "/Users/stephenmoss/.pyenv/versions/3.9.0/lib/python3.9/site-packages/umap/umap_.py", line 47, in <module>
from pynndescent import NNDescent
File "/Users/stephenmoss/.pyenv/versions/3.9.0/lib/python3.9/site-packages/pynndescent/__init__.py", line 3, in <module>
from .pynndescent_ import NNDescent, PyNNDescentTransformer
File "/Users/stephenmoss/.pyenv/versions/3.9.0/lib/python3.9/site-packages/pynndescent/pynndescent_.py", line 21, in <module>
import pynndescent.sparse as sparse
File "/Users/stephenmoss/.pyenv/versions/3.9.0/lib/python3.9/site-packages/pynndescent/sparse.py", line 330, in <module>
def sparse_alternative_jaccard(ind1, data1, ind2, data2):
File "/Users/stephenmoss/.pyenv/versions/3.9.0/lib/python3.9/site-packages/numba/core/decorators.py", line 218, in wrapper
disp.compile(sig)
File "/Users/stephenmoss/.pyenv/versions/3.9.0/lib/python3.9/site-packages/numba/core/compiler_lock.py", line 32, in _acquire_compile_lock
return func(*args, **kwargs)
File "/Users/stephenmoss/.pyenv/versions/3.9.0/lib/python3.9/site-packages/numba/core/dispatcher.py", line 819, in compile
cres = self._compiler.compile(args, return_type)
File "/Users/stephenmoss/.pyenv/versions/3.9.0/lib/python3.9/site-packages/numba/core/dispatcher.py", line 82, in compile
raise retval
File "/Users/stephenmoss/.pyenv/versions/3.9.0/lib/python3.9/site-packages/numba/core/dispatcher.py", line 92, in _compile_cached
retval = self._compile_core(args, return_type)
File "/Users/stephenmoss/.pyenv/versions/3.9.0/lib/python3.9/site-packages/numba/core/dispatcher.py", line 105, in _compile_core
cres = compiler.compile_extra(self.targetdescr.typing_context,
File "/Users/stephenmoss/.pyenv/versions/3.9.0/lib/python3.9/site-packages/numba/core/compiler.py", line 627, in compile_extra
return pipeline.compile_extra(func)
File "/Users/stephenmoss/.pyenv/versions/3.9.0/lib/python3.9/site-packages/numba/core/compiler.py", line 363, in compile_extra
return self._compile_bytecode()
File "/Users/stephenmoss/.pyenv/versions/3.9.0/lib/python3.9/site-packages/numba/core/compiler.py", line 425, in _compile_bytecode
return self._compile_core()
File "/Users/stephenmoss/.pyenv/versions/3.9.0/lib/python3.9/site-packages/numba/core/compiler.py", line 405, in _compile_core
raise e
File "/Users/stephenmoss/.pyenv/versions/3.9.0/lib/python3.9/site-packages/numba/core/compiler.py", line 396, in _compile_core
pm.run(self.state)
File "/Users/stephenmoss/.pyenv/versions/3.9.0/lib/python3.9/site-packages/numba/core/compiler_machinery.py", line 341, in run
raise patched_exception
File "/Users/stephenmoss/.pyenv/versions/3.9.0/lib/python3.9/site-packages/numba/core/compiler_machinery.py", line 332, in run
self._runPass(idx, pass_inst, state)
File "/Users/stephenmoss/.pyenv/versions/3.9.0/lib/python3.9/site-packages/numba/core/compiler_lock.py", line 32, in _acquire_compile_lock
return func(*args, **kwargs)
File "/Users/stephenmoss/.pyenv/versions/3.9.0/lib/python3.9/site-packages/numba/core/compiler_machinery.py", line 291, in _runPass
mutated |= check(pss.run_pass, internal_state)
File "/Users/stephenmoss/.pyenv/versions/3.9.0/lib/python3.9/site-packages/numba/core/compiler_machinery.py", line 264, in check
mangled = func(compiler_state)
File "/Users/stephenmoss/.pyenv/versions/3.9.0/lib/python3.9/site-packages/numba/core/typed_passes.py", line 92, in run_pass
typemap, return_type, calltypes = type_inference_stage(
File "/Users/stephenmoss/.pyenv/versions/3.9.0/lib/python3.9/site-packages/numba/core/typed_passes.py", line 70, in type_inference_stage
infer.propagate(raise_errors=raise_errors)
File "/Users/stephenmoss/.pyenv/versions/3.9.0/lib/python3.9/site-packages/numba/core/typeinfer.py", line 1071, in propagate
raise errors[0]
numba.core.errors.TypingError: Failed in nopython mode pipeline (step: nopython frontend)
Failed in nopython mode pipeline (step: nopython frontend)
Failed in nopython mode pipeline (step: nopython mode backend)
Failed in nopython mode pipeline (step: nopython mode backend)
Failed in nopython mode pipeline (step: nopython frontend)
No implementation of function Function(<function make_quicksort_impl.<locals>.run_quicksort at 0x13e3a6940>) found for signature:
>>> run_quicksort(array(int32, 1d, C))
There are 2 candidate implementations:
- Of which 2 did not match due to:
Overload in function 'register_jitable.<locals>.wrap.<locals>.ov_wrap': File: numba/core/extending.py: Line 150.
With argument(s): '(array(int32, 1d, C))':
Rejected as the implementation raised a specific error:
UnsupportedError: Failed in nopython mode pipeline (step: analyzing bytecode)
Use of unsupported opcode (LOAD_ASSERTION_ERROR) found
File "../../../.pyenv/versions/3.9.0/lib/python3.9/site-packages/numba/misc/quicksort.py", line 180:
def run_quicksort(A):
<source elided>
while high - low >= SMALL_QUICKSORT:
assert n < MAX_STACK
^
raised from /Users/stephenmoss/.pyenv/versions/3.9.0/lib/python3.9/site-packages/numba/core/byteflow.py:269
During: resolving callee type: Function(<function make_quicksort_impl.<locals>.run_quicksort at 0x13e3a6940>)
During: typing of call at /Users/stephenmoss/.pyenv/versions/3.9.0/lib/python3.9/site-packages/numba/np/arrayobj.py (5007)
File "../../../.pyenv/versions/3.9.0/lib/python3.9/site-packages/numba/np/arrayobj.py", line 5007:
def array_sort_impl(arr):
<source elided>
# Note we clobber the return value
sort_func(arr)
^
During: lowering "$14call_method.5 = call $12load_method.4(func=$12load_method.4, args=[], kws=(), vararg=None)" at /Users/stephenmoss/.pyenv/versions/3.9.0/lib/python3.9/site-packages/numba/np/arrayobj.py (5017)
During: lowering "$8call_method.3 = call $4load_method.1(arr, func=$4load_method.1, args=[Var(arr, sparse.py:28)], kws=(), vararg=None)" at /Users/stephenmoss/.pyenv/versions/3.9.0/lib/python3.9/site-packages/pynndescent/sparse.py (28)
During: resolving callee type: type(CPUDispatcher(<function arr_unique at 0x13e1d4280>))
During: typing of call at /Users/stephenmoss/.pyenv/versions/3.9.0/lib/python3.9/site-packages/pynndescent/sparse.py (41)
During: resolving callee type: type(CPUDispatcher(<function arr_unique at 0x13e1d4280>))
During: typing of call at /Users/stephenmoss/.pyenv/versions/3.9.0/lib/python3.9/site-packages/pynndescent/sparse.py (41)
File "../../../.pyenv/versions/3.9.0/lib/python3.9/site-packages/pynndescent/sparse.py", line 41:
def arr_union(ar1, ar2):
<source elided>
else:
return arr_unique(np.concatenate((ar1, ar2)))
^
During: resolving callee type: type(CPUDispatcher(<function arr_union at 0x13e1d4820>))
During: typing of call at /Users/stephenmoss/.pyenv/versions/3.9.0/lib/python3.9/site-packages/pynndescent/sparse.py (331)
During: resolving callee type: type(CPUDispatcher(<function arr_union at 0x13e1d4820>))
During: typing of call at /Users/stephenmoss/.pyenv/versions/3.9.0/lib/python3.9/site-packages/pynndescent/sparse.py (331)
File "../../../.pyenv/versions/3.9.0/lib/python3.9/site-packages/pynndescent/sparse.py", line 331:
def sparse_alternative_jaccard(ind1, data1, ind2, data2):
num_non_zero = arr_union(ind1, ind2).shape[0]
^
This turned out to be an issue with python 3.9, which the package is supposed to support. I tried with python 3.8 instead, but it failed initially with the scipy install as it needed the BLAS and LAPACK libraries. I needed to install using:
brew install openblas lapack
LDFLAGS="-L/usr/local/opt/openblas/lib -L/usr/local/opt/lapack/lib" CPPFLAGS="-I/usr/local/opt/openblas/include -I/usr/local/opt/lapack/include" LLVM_CONFIG=/usr/local/opt/llvm@9/bin/llvm-config pip install openomics
Now running python openomics_test.py gives me only the following warning:
/Users/stephenmoss/.pyenv/versions/3.8.7/lib/python3.8/site-packages/umap/__init__.py:9: UserWarning: Tensorflow not installed; ParametricUMAP will be unavailable
warn("Tensorflow not installed; ParametricUMAP will be unavailable")
Running the following rectifies this:
brew install libtensorflow
pip install tensorflow
I feel, therefore, a dependency on python 3.8 should be specified in the documentation and setup.py, as it looks like there are issues with python 3.9 at present. It would also be useful to include tensorflow in the list of package dependencies (i.e. requirements.txt) to avoid this warning. Using something like pipenv might be an ideal solution here? Though explicitly stating the external library dependencies for scipy would still be necessary.
Running pip install requirements.txt initially received error.
Perhaps an external dependency on this can be specified (required by llvmlite)? The assumption is that the end-user has a working python installation and relevant compilers etc. installed, though this doesn't seem to be specified anywhere?
Installing collected packages: MarkupSafe, Werkzeug, numpy, Jinja2, itsdangerous, zope.interface, zope.event, urllib3, threadpoolctl, scipy, retrying, PyYAML, pytz, python-dateutil, pyparsing, ptyprocess, llvmlite, joblib, idna, heapdict, greenlet, Flask, chardet, certifi, brotli, zict, typing-extensions, tornado, toolz, tblib, soupsieve, sortedcontainers, scikit-learn, requests, psutil, plotly, pillow, pexpect, patsy, pandas, packaging, numba, msgpack, locket, gevent, future, flask-compress, decorator, dask, dash-table, dash-renderer, dash-html-components, dash-core-components, colorlog, cloudpickle, xmltodict, xlsxwriter, xlrd, wrapt, wget, suds-jurko, statsmodels, requests-cache, pynndescent, pyerfa, pydot, partd, networkx, lxml, kiwisolver, grequests, fsspec, easydev, docopt, distributed, dash, cython, cycler, cachetools, bokeh, beautifulsoup4, appdirs, validators, umap-learn, typing, sqlalchemy, scikit-allel, rarfile, obonet, matplotlib, large-image, h5py, gunicorn, gtfparse, goatools, filetype, dash-daq, dash-bootstrap-components, bioservices, biopython, astropy, openomics
Running setup.py install for retrying ... done
Running setup.py install for llvmlite ... error
ERROR: Command errored out with exit status 1:
command: /Users/stephenmoss/.pyenv/versions/3.9.0/bin/python3.9 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/bf/cl_g6mhx7zd9_1mhhvppbzdh0000gn/T/pip-install-sm6pl876/llvmlite_0bff606e31a6496399a22ccfcec04d59/setup.py'"'"'; __file__='"'"'/private/var/folders/bf/cl_g6mhx7zd9_1mhhvppbzdh0000gn/T/pip-install-sm6pl876/llvmlite_0bff606e31a6496399a22ccfcec04d59/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/bf/cl_g6mhx7zd9_1mhhvppbzdh0000gn/T/pip-record-rgduko9u/install-record.txt --single-version-externally-managed --compile --install-headers /Users/stephenmoss/.pyenv/versions/3.9.0/include/python3.9/llvmlite
cwd: /private/var/folders/bf/cl_g6mhx7zd9_1mhhvppbzdh0000gn/T/pip-install-sm6pl876/llvmlite_0bff606e31a6496399a22ccfcec04d59/
Complete output (29 lines):
running install
running build
got version from file /private/var/folders/bf/cl_g6mhx7zd9_1mhhvppbzdh0000gn/T/pip-install-sm6pl876/llvmlite_0bff606e31a6496399a22ccfcec04d59/llvmlite/_version.py {'version': '0.34.0', 'full': 'c5889c9e98c6b19d5d85ebdd982d64a03931f8e2'}
running build_ext
/Users/stephenmoss/.pyenv/versions/3.9.0/bin/python3.9 /private/var/folders/bf/cl_g6mhx7zd9_1mhhvppbzdh0000gn/T/pip-install-sm6pl876/llvmlite_0bff606e31a6496399a22ccfcec04d59/ffi/build.py
LLVM version... Traceback (most recent call last):
File "/private/var/folders/bf/cl_g6mhx7zd9_1mhhvppbzdh0000gn/T/pip-install-sm6pl876/llvmlite_0bff606e31a6496399a22ccfcec04d59/ffi/build.py", line 105, in main_posix
out = subprocess.check_output([llvm_config, '--version'])
File "/Users/stephenmoss/.pyenv/versions/3.9.0/lib/python3.9/subprocess.py", line 420, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
File "/Users/stephenmoss/.pyenv/versions/3.9.0/lib/python3.9/subprocess.py", line 501, in run
with Popen(*popenargs, **kwargs) as process:
File "/Users/stephenmoss/.pyenv/versions/3.9.0/lib/python3.9/subprocess.py", line 947, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "/Users/stephenmoss/.pyenv/versions/3.9.0/lib/python3.9/subprocess.py", line 1819, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'llvm-config'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/private/var/folders/bf/cl_g6mhx7zd9_1mhhvppbzdh0000gn/T/pip-install-sm6pl876/llvmlite_0bff606e31a6496399a22ccfcec04d59/ffi/build.py", line 191, in <module>
main()
File "/private/var/folders/bf/cl_g6mhx7zd9_1mhhvppbzdh0000gn/T/pip-install-sm6pl876/llvmlite_0bff606e31a6496399a22ccfcec04d59/ffi/build.py", line 185, in main
main_posix('osx', '.dylib')
File "/private/var/folders/bf/cl_g6mhx7zd9_1mhhvppbzdh0000gn/T/pip-install-sm6pl876/llvmlite_0bff606e31a6496399a22ccfcec04d59/ffi/build.py", line 107, in main_posix
raise RuntimeError("%s failed executing, please point LLVM_CONFIG "
RuntimeError: llvm-config failed executing, please point LLVM_CONFIG to the path for llvm-config
error: command '/Users/stephenmoss/.pyenv/versions/3.9.0/bin/python3.9' failed with exit code 1
----------------------------------------
ERROR: Command errored out with exit status 1: /Users/stephenmoss/.pyenv/versions/3.9.0/bin/python3.9 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/bf/cl_g6mhx7zd9_1mhhvppbzdh0000gn/T/pip-install-sm6pl876/llvmlite_0bff606e31a6496399a22ccfcec04d59/setup.py'"'"'; __file__='"'"'/private/var/folders/bf/cl_g6mhx7zd9_1mhhvppbzdh0000gn/T/pip-install-sm6pl876/llvmlite_0bff606e31a6496399a22ccfcec04d59/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/bf/cl_g6mhx7zd9_1mhhvppbzdh0000gn/T/pip-record-rgduko9u/install-record.txt --single-version-externally-managed --compile --install-headers /Users/stephenmoss/.pyenv/versions/3.9.0/include/python3.9/llvmlite Check the logs for full command output.
I needed to run the following to fix the issue:
brew install llvm@9
LLVM_CONFIG=/usr/local/opt/llvm@9/bin/llvm-config pip install openomics
I tried with brew install llvm (version 11.0.1 and it failed with RuntimeError: Building llvmlite requires LLVM 10.0.x or 9.0.x, got '11.0.1'. Be sure to set LLVM_CONFIG to the right executable path.
Docstrings seem to be available throughout the codebase for all relevant functions, however, I found the documentation on Read the Docs to be lacking in comparison, particularly in providing detail of function parameters that are otherwise available in the docstrings.
There is no link to the contribution guidelines from the main README, which I feel would be beneficial, however contributing guidelines are available in https://github.com/BioMeCIS-Lab/OpenOmics/blob/master/CONTRIBUTING.rst and on Read the Docs.
Just being finicky, I personally would prefer a common format for the README and CONTRIBUTING documents etc. Either both in reStructuredText or both in Markdown. There seems to be an outdated README.rst that could probably be replaced with the current README.md?
Trying to run the vignettes from the README:
# Import GENCODE database (from URL)
from openomics.database import GENCODE
gencode = GENCODE(path="ftp://ftp.ebi.ac.uk/pub/databases/gencode/Gencode_human/release_32/",
file_resources={"long_noncoding_RNAs.gtf": "gencode.v32.long_noncoding_RNAs.gtf.gz",
"basic.annotation.gtf": "gencode.v32.basic.annotation.gtf.gz",
"lncRNA_transcripts.fa": "gencode.v32.lncRNA_transcripts.fa.gz",
"transcripts.fa": "gencode.v32.transcripts.fa.gz"},
remove_version_num=True,
npartitions=5)
# Annotate LncRNAs with GENCODE by gene_id
luad_data.LncRNA.annotate_genomics(gencode, index="gene_id",
columns=['feature', 'start', 'end', 'strand', 'tag', 'havana_gene'])
luad_data.LncRNA.annotations.info()
I added the text from the vignette above to a file called openomics_test.py and ran the following command:
python openomics_test.py
I received the following output:
Downloading ftp://ftp.ebi.ac.uk/pub/databases/gencode/Gencode_human/release_32/gencode.v32.long_noncoding_RNAs.gtf.gz
|========================================================================================================================================================================================================| 4.4M/4.4M (100.00%) 0s
Downloading ftp://ftp.ebi.ac.uk/pub/databases/gencode/Gencode_human/release_32/gencode.v32.basic.annotation.gtf.gz
|========================================================================================================================================================================================================| 26M/ 26M (100.00%) 7s
Downloading ftp://ftp.ebi.ac.uk/pub/databases/gencode/Gencode_human/release_32/gencode.v32.lncRNA_transcripts.fa.gz
|========================================================================================================================================================================================================| 14M/ 14M (100.00%) 3s
Downloading ftp://ftp.ebi.ac.uk/pub/databases/gencode/Gencode_human/release_32/gencode.v32.transcripts.fa.gz
|========================================================================================================================================================================================================| 72M/ 72M (100.00%) 15s
INFO:root:<_io.TextIOWrapper name='/Users/stephenmoss/.astropy/cache/download/url/141581d04d4001254d07601dfa7d983b/contents' encoding='UTF-8'>
Traceback (most recent call last):
File "openomics_test.py", line 34, in <module>
gencode = GENCODE(path="ftp://ftp.ebi.ac.uk/pub/databases/gencode/Gencode_human/release_32/",
File "/Users/stephenmoss/.pyenv/versions/3.8.7/lib/python3.8/site-packages/openomics/database/sequence.py", line 67, in __init__
super(GENCODE, self).__init__(path=path, file_resources=file_resources, col_rename=col_rename,
File "/Users/stephenmoss/.pyenv/versions/3.8.7/lib/python3.8/site-packages/openomics/database/sequence.py", line 17, in __init__
super(SequenceDataset, self).__init__(**kwargs)
File "/Users/stephenmoss/.pyenv/versions/3.8.7/lib/python3.8/site-packages/openomics/database/base.py", line 39, in __init__
self.data = self.load_dataframe(file_resources, npartitions=npartitions)
File "/Users/stephenmoss/.pyenv/versions/3.8.7/lib/python3.8/site-packages/openomics/database/sequence.py", line 74, in load_dataframe
df = read_gtf(file_resources[gtf_file], npartitions=npartitions)
File "/Users/stephenmoss/.pyenv/versions/3.8.7/lib/python3.8/site-packages/openomics/utils/read_gtf.py", line 349, in read_gtf
result_df = parse_gtf_and_expand_attributes(
File "/Users/stephenmoss/.pyenv/versions/3.8.7/lib/python3.8/site-packages/openomics/utils/read_gtf.py", line 290, in parse_gtf_and_expand_attributes
result = parse_gtf(
File "/Users/stephenmoss/.pyenv/versions/3.8.7/lib/python3.8/site-packages/openomics/utils/read_gtf.py", line 195, in parse_gtf
chunk_iterator = dd.read_table(
File "/Users/stephenmoss/.pyenv/versions/3.8.7/lib/python3.8/site-packages/dask/dataframe/io/csv.py", line 659, in read
return read_pandas(
File "/Users/stephenmoss/.pyenv/versions/3.8.7/lib/python3.8/site-packages/dask/dataframe/io/csv.py", line 464, in read_pandas
paths = get_fs_token_paths(urlpath, mode="rb", storage_options=storage_options)[
File "/Users/stephenmoss/.pyenv/versions/3.8.7/lib/python3.8/site-packages/fsspec/core.py", line 619, in get_fs_token_paths
path = cls._strip_protocol(urlpath)
File "/Users/stephenmoss/.pyenv/versions/3.8.7/lib/python3.8/site-packages/fsspec/implementations/local.py", line 147, in _strip_protocol
if path.startswith("file://"):
AttributeError: '_io.TextIOWrapper' object has no attribute 'startswith'
Running gtex = GTEx(path="https://storage.googleapis.com/gtex_analysis_v8/rna_seq_data/") Results in: OSError: Not enough free space in /homes/.astropy/cache/download/url to download a 3.6G file, only 2.7G left.
Is there a possibility to choose the directory in which the files should be downloaded? This would be great.
gtex = GTEx(path="https://storage.googleapis.com/gtex_analysis_v8/rna_seq_data/")
OSError: Not enough free space in /homes/.astropy/cache/download/url to download a 3.6G file, only 2.7G left.
What I was trying to get done?
I was trying to use the function load_data() of class Multiomics. Instead it gave the error elaborated below
//The command that raised the error [Here, luad_data is an object of class Multiomics]:
X_multiomics, y = luad_data.load_data(omics="all", target=["pathologic_stage"])
//And it gave the below error:
~/*****************/site-packages/openomics/multiomics.py in load_data(self, omics, target, pathologic_stages, histological_subtypes, predicted_subtypes, tumor_normal, samples_barcode)
251
252 # Filter y target column labels
--> 253 y = y.filter(target)
254 y.dropna(axis=0, inplace=True)
255 matched_samples = y.index
AttributeError: 'tuple' object has no attribute 'filter'
I tried goiong through the function description for the arguments of the function given, and it seems I'm not passing absurd arguments.
I'm guessing that this error is raised becuase the y used in line 253 of multiomics.py is an tuple of an Index object (pandas.core.indexes.base.Index) and numpy.ndarray object instead of DataFrame object (pandas.core.frame.DataFrame).
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.