The issue seems to be this:

If I don't run

pip install --force-reinstall --upgrade protobuf

after tensorflow install. I am met with the following error from protobuf

Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/IPython/core/ultratb.py", line 1101, in get_records
    return _fixed_getinnerframes(etb, number_of_lines_of_context, tb_offset)
  File "/opt/conda/lib/python3.7/site-packages/IPython/core/ultratb.py", line 319, in wrapped
    return f(*args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/IPython/core/ultratb.py", line 353, in _fixed_getinnerframes
    records = fix_frame_records_filenames(inspect.getinnerframes(etb, context))
  File "/opt/conda/lib/python3.7/inspect.py", line 1502, in getinnerframes
    frameinfo = (tb.tb_frame,) + getframeinfo(tb, context)
  File "/opt/conda/lib/python3.7/inspect.py", line 1460, in getframeinfo
    filename = getsourcefile(frame) or getfile(frame)
  File "/opt/conda/lib/python3.7/inspect.py", line 696, in getsourcefile
    if getattr(getmodule(object, filename), '__loader__', None) is not None:
  File "/opt/conda/lib/python3.7/inspect.py", line 733, in getmodule
    if ismodule(module) and hasattr(module, '__file__'):
  File "/opt/conda/lib/python3.7/site-packages/tensorflow/__init__.py", line 50, in __getattr__
    module = self._load()
  File "/opt/conda/lib/python3.7/site-packages/tensorflow/__init__.py", line 44, in _load
    module = _importlib.import_module(self.__name__)
  File "/opt/conda/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/opt/conda/lib/python3.7/site-packages/tensorflow_core/__init__.py", line 42, in <module>
    from . _api.v2 import audio
  File "/opt/conda/lib/python3.7/site-packages/tensorflow_core/_api/v2/audio/__init__.py", line 10, in <module>
    from tensorflow.python.ops.gen_audio_ops import decode_wav
  File "/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/ops/gen_audio_ops.py", line 11, in <module>
    from tensorflow.python.eager import context as _context
  File "/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/eager/context.py", line 29, in <module>
    from tensorflow.core.protobuf import config_pb2
  File "/opt/conda/lib/python3.7/site-packages/tensorflow_core/core/protobuf/config_pb2.py", line 17, in <module>
    from tensorflow.core.framework import graph_pb2 as tensorflow_dot_core_dot_framework_dot_graph__pb2
  File "/opt/conda/lib/python3.7/site-packages/tensorflow_core/core/framework/graph_pb2.py", line 16, in <module>
    from tensorflow.core.framework import node_def_pb2 as tensorflow_dot_core_dot_framework_dot_node__def__pb2
  File "/opt/conda/lib/python3.7/site-packages/tensorflow_core/core/framework/node_def_pb2.py", line 16, in <module>
    from tensorflow.core.framework import attr_value_pb2 as tensorflow_dot_core_dot_framework_dot_attr__value__pb2
  File "/opt/conda/lib/python3.7/site-packages/tensorflow_core/core/framework/attr_value_pb2.py", line 16, in <module>
    from tensorflow.core.framework import tensor_pb2 as tensorflow_dot_core_dot_framework_dot_tensor__pb2
  File "/opt/conda/lib/python3.7/site-packages/tensorflow_core/core/framework/tensor_pb2.py", line 16, in <module>
    from tensorflow.core.framework import resource_handle_pb2 as tensorflow_dot_core_dot_framework_dot_resource__handle__pb2
  File "/opt/conda/lib/python3.7/site-packages/tensorflow_core/core/framework/resource_handle_pb2.py", line 45, in <module>
    serialized_options=None, file=DESCRIPTOR),
  File "/opt/conda/lib/python3.7/site-packages/google/protobuf/descriptor.py", line 534, in __new__
    return _message.default_pool.FindFieldByName(full_name)
KeyError: "Couldn't find field tensorflow.ResourceHandleProto.DtypeAndShape.dtype"

Now, for some reason, the previous error can be fixed by forcing a reinstall of protobuf with pip. However, forcing the reinstall causes nucleus to throw this error anytime I try to use a function (import is fine).

from nucleus.

CONTEXT: After converting jupyter notebook to python source code, I notice it recommends to install rather old version of google-nucleus 0.2.2 and tensorflow 2.0.0-alpha0. However, I have upgraded the nucleus to depend on pip3, which does not allow me to downgrade to install a version of google-nucleus below 0.3.0. The latest version of google-nucleus is 0.5.1, protobuf is 3.12.2 and the tensorflow is 2.2.0.

PROBLEM: Although the ticket is closed, I still experience the same problem reported by @TimSmole. When running the test program posted by @ThomasColthurst, it reports the following errors:

python3 dna_sequencing_error_correction.txt

Generating data...
Traceback (most recent call last):
File "dna_sequencing_error_correction.txt", line 381, in
run(hparams)
File "dna_sequencing_error_correction.txt", line 312, in run
generate_tfrecord_datasets(hparams)
File "dna_sequencing_error_correction.txt", line 91, in generate_tfrecord_datasets
for example in all_examples:
File "dna_sequencing_error_correction.txt", line 120, in make_ngs_examples
for read in sam_reader:
File "/usr/local/lib/python3.5/dist-packages/nucleus/io/clif_postproc.py", line 67, in next
record, not_done = self._raw_next()
File "/usr/local/lib/python3.5/dist-packages/nucleus/io/clif_postproc.py", line 126, in _raw_next
not_done = self._cc_iterable.PythonNext(record)
RuntimeError: PythonNext() argument read is not valid: Could not load PyProto API

python3 t.py

2020-05-30 21:24:02.271801: W ./nucleus/util/proto_clif_converter.h:60] Failed to cast type N6google8protobuf14DynamicMessageE
Traceback (most recent call last):
File "t.py", line 3, in
for v in r:
File "/usr/local/lib/python3.5/dist-packages/nucleus/io/clif_postproc.py", line 67, in next
record, not_done = self._raw_next()
File "/usr/local/lib/python3.5/dist-packages/nucleus/io/clif_postproc.py", line 134, in _raw_next
not_done = self._cc_iterable.PythonNext(record)
RuntimeError: PythonNext() argument variant is not valid: Dynamic cast failed

from nucleus.

Thank you for this error report. I have reproduced the "Could not load PyProto API" error, and we hope to have a fix out for it soon.

Do you encounter this problem outside of a colab? (So far that's the only place I've been able to reproduce it.)

from nucleus.

Yes, I experience same error when running on local machine with Ubuntu 18.04 inside fresh vritualenv and also when running inside docker image based on tensorflow/tensorflow:1.13.1-gpu-py3-jupyter.

from nucleus.

Interesting! I wasn't able to reproduce the problem on an Ubuntu 18.04 machine, either inside or outside of a virtualenv.

For the virtualenv version, I ran:

$ gcloud compute instances create thomaswc-1804 --scopes "compute-rw,storage-full,cloud-platform" --image-family ubuntu-1804-lts --image-project ubuntu-os-cloud --machine-type n1-standard-32 --boot-disk-size=200GB
$ gcloud compute ssh thomaswc-1804
(thomaswc-1804) $ sudo apt-get update
(thomaswc-1804) $ sudo apt-get install python3-venv
(thomaswc-1804) $ python3 -m venv ven
(thomaswc-1804) $ source ven/bin/active
(ven) $ pip3 install google-nucleus
(ven) $ wget https://raw.githubusercontent.com/google/nucleus/master/nucleus/testdata/test_samples.vcf
(ven) $ python3

from nucleus.io import vcf
r = vcf.VcfReader('test_samples.vcf')
for v in r:
... print(v.start)

and it worked fine.

from nucleus.

bfa5687 resolves the problem!

from nucleus.

Cool! I've released a new version and pip package (0.4.1) to go along with that commit, in case that is helpful to you.

from nucleus.

@ThomasColthurst Thank you very much for your help!

It seems to be working now, however I found out that it works only if nucleus is imported before tensorflow, otherwise the same error is thrown. I am using tensorflow==1.13.1 and nucleus==0.4.1.

from nucleus.

Having this same issue with whatever the most recent version of nucleus is in the pypi

Ubuntu 18.04 and also with a docker based notebook env

from nucleus.

I have got the same issue as described by @jamesthegiantpeach .

from nucleus.

@JianpingCAI

This is caused by bfa5687

The nucleus install strips the protobuf installed by tensorflow and installs an older version. At least that’s what I thought the issue was last time I looked at it. Reverting to an older version of TF will “fix” the issue.

There’s already a ticket in on it.

from nucleus.

@JianpingCAI @jamesthegiantpeach Has the problem been resolved for you?

from nucleus.

Hi @yijunyu I'm trying to reproduce your issue but I'm not completely sure what your current setup is. Could you provide simple steps to reproduce your issue starting from a clean Python3 virtual environment if it's possible (names and versions of the packages to install, the order of the installation, the script and data to use)?

from nucleus.

Thanks, @tedyun.

What I am trying to do is to create a Docker image at "yijun/nucleus" for my bioinformatics colleagues at Open University. It was built using the following Dockerfile: https://github.com/yijunyu/fastseq/blob/master/docker/Dockerfile,

This spec was adapted from the historical Dockerfile.build_dev spec in nucleus repository https://github.com/google/nucleus/blob/master/Dockerfile.build_env.
I have pushed the built image into the dockerhub, see below.

To reproduce the error as I see it you can do the following steps 1-3:

1. Get the python script from the Jupyter notebook in the tutorial:
   docker run -v $(PWD):/host -it yijun/nucleus bash -c "jupyter nbconvert --to script nucleus/examples/dna_sequencing_error_correction.ipynb && mv nucleus/examples/dna_sequencing_error_correction.txt /host"

In the converted python script, some shell commands are prefixed with "!", which need to be commented out. So I have moved these shell commands into the following script to:

2. obtain the test data files:
   docker run -v $(PWD):/host -it yijun/nucleus bash -c "bash /host/t.sh"
   , where "t.sh" simply downloads the data files: https://github.com/yijunyu/fastseq/blob/master/t.sh

3. Finally, I could run nucleus using the following command in the docker image:
   docker run -v $(PWD):/host -w /host -it yijun/nucleus bash -c "python3 dna_sequencing_error_correction.txt"
   ==
   % docker run -v $(PWD):/host -w /host -it yijun/nucleus bash -c "python3 /host/dna_sequencing_error_correction.txt"
   Generating data...
   Traceback (most recent call last):
   File "/host/dna_sequencing_error_correction.txt", line 381, in
   run(hparams)
   File "/host/dna_sequencing_error_correction.txt", line 312, in run
   generate_tfrecord_datasets(hparams)
   File "/host/dna_sequencing_error_correction.txt", line 91, in generate_tfrecord_datasets
   for example in all_examples:
   File "/host/dna_sequencing_error_correction.txt", line 120, in make_ngs_examples
   for read in sam_reader:
   File "/usr/local/lib/python3.5/dist-packages/nucleus/io/clif_postproc.py", line 67, in next
   record, not_done = self._raw_next()
   File "/usr/local/lib/python3.5/dist-packages/nucleus/io/clif_postproc.py", line 126, in _raw_next
   not_done = self._cc_iterable.PythonNext(record)
   RuntimeError: PythonNext() argument read is not valid: Could not load PyProto API
   ==
   Sorry for the lengthy explanation, hope so you can reproduce the error.
   Thank you.

Best regards,
Yijun
[email protected]

from nucleus.

@yijunyu I can see you're using jupyter nbconvert --to script nucleus/examples/dna_sequencing_error_correction.ipynb as an example. Can you please try a simpler script like this one in your environment and see if you can reproduce the issue:

test.py:

from nucleus.io import vcf
r = vcf.VcfReader('test_samples.vcf')
for v in r:
  print(v.start)

$ wget https://raw.githubusercontent.com/google/nucleus/master/nucleus/testdata/test_samples.vcf
$ python3 test.py

from nucleus.

@tedyun This one works, thanks!

docker run -v $(PWD):/host -w /host -it yijun/nucleus bash -c "python3 test.py"

It prints a number of numbers to the screen.
I wonder why the other one didn't work.

from nucleus.

@yijunyu That's great! I'll close this as it could be related to the particular code you were running (I'd recommend manually looking at the converted python code to see if there are any obvious errors).

If you find an issue in a normal Python workflow using Nucleus, please feel free to open a new issue here. Thank you!

from nucleus.

Sure @tedyun, I tried the other readers for FASTQ and SAM as well, they all seem to be working now. Thank you for pointing me to the right direction !

from nucleus.

@yijunyu I'm glad to hear that :) No problem!

from nucleus.