Comments (5)
Related is also #1513.
Can you post the full output?
from returnn.
Can you try to run on CPU only (export DISABLE_CUDA=1
)?
Can you try to run test_TFNativeOp.py?
from returnn.
Please find here the full output of the compilation on CUDA.
Answering your questions:
- A non-CUDA environment, CPU only works!
python3 -m pytest test_TFNativeOp.py
also works, but I'm not sure if I'm running the test with CUDA enabled (if that makes any difference for the test). Besides, there are some skipped tests as well as some warnings. I'm running these in a machine that has GPUs available. Please see the results below.
test_TFNativeOp.py ......................................................sssssss [100%]
============================================================================ warnings summary =============================================================================
../../../../../../../../../../../.venvs/singularity/returnn_test_native_op/lib/python3.10/site-packages/nose/plugins/manager.py:418
/home/nbeneitez/.venvs/singularity/returnn_test_native_op/lib/python3.10/site-packages/nose/plugins/manager.py:418: DeprecationWarning: pkg_resources is deprecated as an
API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
import pkg_resources
../../../../../../../../../../../.venvs/singularity/returnn_test_native_op/lib/python3.10/site-packages/nose/importer.py:12
/home/nbeneitez/.venvs/singularity/returnn_test_native_op/lib/python3.10/site-packages/nose/importer.py:12: DeprecationWarning: the imp module is deprecated in favour of
importlib and slated for removal in Python 3.12; see the module's documentation for alternative uses
from imp import find_module, load_module, acquire_lock, release_lock
../../../../../../../../../../../.venvs/singularity/returnn_test_native_op/lib/python3.10/site-packages/numpy/__config__.py:155
/home/nbeneitez/.venvs/singularity/returnn_test_native_op/lib/python3.10/site-packages/numpy/__config__.py:155: UserWarning: Install `pyyaml` for better output
warnings.warn("Install `pyyaml` for better output", stacklevel=1)
tests/test_TFNativeOp.py::test_py_viterbi
/home/nbeneitez/work/returnn/native_op_issue/work/i6_core/tools/git/CloneGitRepositoryJob.nH5B7CKRCU89/output/repository/tests/test_TFNativeOp.py:2224: RuntimeWarning: d
ivide by zero encountered in log
am_scores = numpy.log(am_scores) # in +log space
tests/test_TFNativeOp.py::test_fast_viterbi
/home/nbeneitez/work/returnn/native_op_issue/work/i6_core/tools/git/CloneGitRepositoryJob.nH5B7CKRCU89/output/repository/tests/test_TFNativeOp.py:2277: RuntimeWarning: d
ivide by zero encountered in log
am_scores = numpy.log(am_scores) # in +log space
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
========================================================== 54 passed, 7 skipped, 5 warnings in 193.13s (0:03:13) ==========================================================
Should the test_TFNativeOp.py
fail for me? As I said, I might be doing something wrong.
from returnn.
python3 -m pytest test_TFNativeOp.py
also works
I assume you tested that with export DISABLE_CUDA=1
, i.e. only for CPU? Can you also try with CUDA?
from returnn.
Note, the main error is error: name followed by "::" must be a class or namespace name
on perftools
:
/home/nbeneitez/work/returnn/native_op_issue/work/i6_core/tools/git/CloneGitRepositoryJob.nH5B7CKRCU89/output/repository/returnn/native_op.cpp(240): error: name followed by "::" must be a class or namespace name
perftools::gputools::DeviceMemory<T> AsDeviceMemory(const T* cuda_memory) {
^
I guess they moved/renamed that. I see in other TF code that it is se::DeviceMemory<T>
(or maybe tensorflow::se::DeviceMemory<T>
or stream_executor::DeviceMemory<T>
or so) now.
Similarly, in our static perftools::gputools::blas::Transpose get_transpose
, I think it is stream_executor::blas::Transpose
or so now.
from returnn.
Related Issues (20)
- PyTorch/RF (?): choosing on which epochs to save optimizer state
- Datasets: blocklist in addition to allowlist for segment list file
- Make batch_size configurable for cross validation HOT 1
- Ignore a single broken gradient HOT 2
- DistributeFilesDataset: _distribute_evenly_by_size suboptimal for multi-gpu sharding HOT 8
- multiprocessing: OSError: AF_UNIX path too long HOT 11
- ConcatSeqsDataset with extended functionality HOT 3
- Torch: print model at log verbosity 3 HOT 1
- RuntimeError: CUDA error: an illegal memory access was encountered HOT 1
- Torch gradient_checkpoint_scope _unregister_custom_saved_tensors_hooks error HOT 4
- RF parametrization breaks Conv
- Torch gradient_checkpoint_scope could trigger segmentation fault? HOT 16
- Torch gradient_checkpoint_scope potential memory leak
- Torch multiple simultaneous gradient_checkpoint_scope
- `rf.pack_padded` with PyTorch takes a lot of memory HOT 1
- `rf.RelPosCausalSelfAttention` fails with `single_step_dim` HOT 9
- Torch `report_profile` `check_events` based tests maybe unstable HOT 1
- Torch: gradient_clip wrong when grad_scaler is used
- Torch print step info on crash
- Make `FileCache` able to detect updated remote files HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from returnn.