Comments (4)
albertz
commented on August 11, 2024
We could move this logic to an own subprocess, if it has an effect. But maybe it does not matter? Then this here could also be closed.
from returnn.
albertz
commented on August 11, 2024
We currently have time.sleep(5) in there. Maybe increase that? But should already be high enough?
from returnn.
albertz
commented on August 11, 2024
The watch_memory runs already in an own thread, and the _read_smaps_file should do most work with the GIL released, so this should really not have much impact.
from returnn.
albertz
commented on August 11, 2024
We could move this logic to an own subprocess
I did that now. So this issue here can be closed.
from returnn.
Related Issues (20)
- RuntimeError: CUDA error: unknown error
- PyTorch debug_add_check_numerics_ops
- Compilation of custom operations failing on TF 2.15/CUDA 12 HOT 5
- `ConcatFilesDataset` combines poorly with `MetaDataset` HOT 6
- RF torch `lstm` fails with torch amp option. HOT 6
- `DistributeFilesDataset` with sharding on file level HOT 6
- ConcatFilesDataset: Reshuffle files per subepoch after every full epoch HOT 2
- `ConcatFilesDataset` needs a better name HOT 10
- RF BatchNorm running var small diff between TF-layers, pure RF and direct PyTorch, biased vs unbiased
- `DistributeFilesDataset`, allow kwargs in `get_sub_epoch_dataset` HOT 10
- Tensor deepcopy does not copy raw_tensor
- Possible race condition in `FileCache`? HOT 5
- Ideas for generic `CachedFile` support across all datasets HOT 18
- `FileCache`: better cleaning, free more than just the minimum
- `FileCache`: avoid cache-wide dir lock
- `DistributeFilesDataset`: copying files blocks `init_seq_order` HOT 2
- `FileCache`: Race condition when removing empty directories HOT 5
- Gradient checkpointing for weight noise etc in PyTorch HOT 7
- SlowMo (BMUF) support for PyTorch distributed training
- DistributeFilesDataset has issues with DataLoader and `num_workers > 0` HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from returnn.