Comments (3)
@MKDJr could please try to interrupt it and get the stack trace?
also, could you run it with export DVCLIVE_LOGLEVEL=TRACE
please, see if we can get more information, thanks!
if you can try also on some super small script that would be also helpful (just to see if this is about DVCLive or there is something specific yo your environment).
from dvclive.
Thanks for the response! I set the loglevel to trace and when I interrupted the command after about 60 seconds this is the error I got:
100% Adding...|██████████████████████████████████████████████████████████████████████████████████████████████████|1/1 [00:00, 21.31file/s]
^CTraceback (most recent call last):
File "/home/mkdjr/technical-language-vs-sensed-values-alignment/run_experiment.py", line 225, in <module>
live.end()
File "/home/mkdjr/.pyenv/versions/pytorch-env/lib/python3.11/site-packages/dvclive/live.py", line 949, in end
self.save_dvc_exp()
File "/home/mkdjr/.pyenv/versions/pytorch-env/lib/python3.11/site-packages/dvclive/utils.py", line 182, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/mkdjr/.pyenv/versions/pytorch-env/lib/python3.11/site-packages/dvclive/live.py", line 979, in save_dvc_exp
self._experiment_rev = self._dvc_repo.experiments.save(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mkdjr/.pyenv/versions/pytorch-env/lib/python3.11/site-packages/dvc/repo/experiments/__init__.py", line 359, in save
return save(self.repo, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mkdjr/.pyenv/versions/pytorch-env/lib/python3.11/site-packages/dvc/repo/experiments/save.py", line 36, in save
save_result = executor.save(
^^^^^^^^^^^^^^
File "/home/mkdjr/.pyenv/versions/pytorch-env/lib/python3.11/site-packages/dvc/repo/experiments/executor/base.py", line 302, in save
with cls.auto_push(dvc):
File "/home/mkdjr/.pyenv/versions/3.11.9/lib/python3.11/contextlib.py", line 144, in __exit__
next(self.gen)
File "/home/mkdjr/.pyenv/versions/pytorch-env/lib/python3.11/site-packages/dvc/repo/experiments/executor/base.py", line 692, in auto_push
cls._auto_push(dvc, git_remote)
File "/home/mkdjr/.pyenv/versions/pytorch-env/lib/python3.11/site-packages/dvc/repo/experiments/executor/base.py", line 713, in _auto_push
dvc.experiments.push(
File "/home/mkdjr/.pyenv/versions/pytorch-env/lib/python3.11/site-packages/dvc/repo/experiments/__init__.py", line 364, in push
return push(self.repo, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mkdjr/.pyenv/versions/pytorch-env/lib/python3.11/site-packages/dvc/repo/__init__.py", line 58, in wrapper
return f(repo, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mkdjr/.pyenv/versions/pytorch-env/lib/python3.11/site-packages/dvc/repo/scm_context.py", line 143, in run
return method(repo, *args, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mkdjr/.pyenv/versions/pytorch-env/lib/python3.11/site-packages/dvc/repo/experiments/push.py", line 126, in push
result["uploaded"] = _push_cache(repo, pushed_refs_info, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mkdjr/.pyenv/versions/pytorch-env/lib/python3.11/site-packages/dvc/repo/experiments/push.py", line 182, in _push_cache
return repo.push(
^^^^^^^^^^
File "/home/mkdjr/.pyenv/versions/pytorch-env/lib/python3.11/site-packages/dvc/repo/__init__.py", line 58, in wrapper
return f(repo, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mkdjr/.pyenv/versions/pytorch-env/lib/python3.11/site-packages/dvc/repo/push.py", line 98, in push
used_run_cache = self.stage_cache.push(remote) if run_cache else []
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mkdjr/.pyenv/versions/pytorch-env/lib/python3.11/site-packages/dvc/stage/cache.py", line 281, in push
return self.transfer(self.repo.cache.legacy, dest_odb)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mkdjr/.pyenv/versions/pytorch-env/lib/python3.11/site-packages/dvc/stage/cache.py", line 264, in transfer
if to_fs.exists(key) and first(to_fs.find(key)):
^^^^^^^^^^^^^^^^^^^^^^
File "/home/mkdjr/.pyenv/versions/pytorch-env/lib/python3.11/site-packages/funcy/seqs.py", line 63, in first
return next(iter(seq), None)
^^^^^^^^^^^^^^^^^^^^^
File "/home/mkdjr/.pyenv/versions/pytorch-env/lib/python3.11/site-packages/dvc_objects/fs/base.py", line 529, in find
yield from self.fs.find(path)
File "/home/mkdjr/.pyenv/versions/pytorch-env/lib/python3.11/site-packages/dvc_objects/fs/local.py", line 78, in find
for root, _, files in self.walk(path, **kwargs):
File "/home/mkdjr/.pyenv/versions/pytorch-env/lib/python3.11/site-packages/dvc_objects/fs/local.py", line 56, in walk
for root, dirs, files in os.walk(
File "<frozen os>", line 358, in _walk
KeyboardInterrupt
After waiting for no more than an hour and a half, pushing did finally finish with the following message:
100% Adding...|██████████████████████████████████████████████████████████████████████████████████████████████████|1/1 [00:00, 26.70file/s]
Collecting |2.60M [02:58, 14.6kentry/s]
WARNING: Some of the cache files do not exist neither locally nor on remote. Missing cache files:
md5: a47bf585eda8e10aaa30cdc7e8d469e5.dir
md5: 942738b9064eacca5fbc81116da2bcc7.dir
md5: 6d2f4af81551a0db2fb4fe619b170a44.dir
md5: 573a5adbfdc20e7cb4f6bd00c3ff4242.dir
md5: e614a27f24b38624e91efc347d234afe.dir
md5: a6c7ce3d01e43ee3b11bc9d150be82e8.dir
md5: 830f328697740e3e8784ae050019b2fd.dir
md5: 2370c7201fb662516b962d576580eb7a.dir
md5: cb6185032f0071cbf16c65b8253c266c.dir
md5: a8765898706eda9536b4663db181e291.dir
md5: b842c74ada14da3e2fbee2dba22dca37.dir
md5: 4d0e83696cf45200524646a18c44cdca.dir
md5: 2f24f25b84879aaecd0db0a429ac41fd
md5: ce570dc2720216b7027eb87e5d66adcf.dir
md5: 5cdc0918a13fea894176803ed7677af3.dir
md5: 8a93dbebc046b5ffa0e9134fd9044ca8.dir
md5: 3672004bcc42945481f7e94a50a1b228.dir
md5: a9a3ad024dc69e2cb116eb710b20ea1c.dir
md5: fd0545d65e6bdb26b6552d5c4b76a8dc.dir
md5: e268e8cf24c4ad4a113200eb1604ef58.dir
md5: 548121f283fa53bfead3145e1e385408.dir
md5: 9aba1c782098bdb0ba6a13acbbc262dc.dir
md5: 1f7e8cbefb9eabb688fbaf642bd43876.dir
md5: af596aaa1e251b0799d74a53078edb33.dir
md5: 138cb203e2e6744e0d3d479611d3a5a5.dir
md5: a4e0c75b40834ef21392a1b7c53bb809.dir
md5: f8f6d94af6759deb65b241f9139c9902.dir
md5: 430a804f11d99c0ad85bab105fce7436.dir
md5: bb340a69a3c21502a5a6f049ff0803bc.dir
md5: 06906c90cba632f31ebd9fa3e21940b7.dir
md5: c20cad6350046bfc1df63af6c0dee3d5.dir
md5: cd6b61422f387a9ac11988c71bcd4fcf.dir
md5: d7d62aa0cb2003264562971a896cbad1.dir
md5: a2fe28d609382459eca10fd6d9627208.dir
md5: 3cdd4dd76627cf74d0f09a5012cdedc4.dir
md5: b1d1bc97d0de33a8b9ae04e23f954ab0.dir
md5: d54642c401dd41f53f4a33fc6ca891d8.dir
md5: a6d0fcb7ffeae9ac1fabd48156a1a081.dir
md5: d751713988987e9331980363e24189ce.dir
md5: 99e0012da3764659f8d3d2442fdfdf62.dir
md5: 76fcf0f391c620a3801545728e68c7b7.dir
md5: f038560ccfe371248021ea9820ceb1ff.dir
md5: 72bcfe0becd167bb7b1274f9b6ec0399.dir
md5: c8f09e42ded208823838f96c39e507d8.dir
md5: 6e8a71e9b81ecb94c56a8e637599004e.dir
md5: dd7d5a0e384dde906a03bd26611e2ad2.dir
md5: c26a713ecc0b5fb8ce88ddfecae216c2.dir
md5: 570a2a9e56e88d423c5d167bd5f49a8e
md5: af72bdd60ffe4e9286c985401a57661d.dir
md5: 430c93f785a4db98ae62ba632c898d1c.dir
md5: 01bad613e0b60fe0f4111371dc7dd5d4.dir
md5: 132c4a69581b62798c9820ffab679a97.dir
md5: c9da98a0276c74ebd31c4d4785e65166.dir
md5: 67d238bc3501f8f3a1575fd8abe4c87a.dir
md5: d6a13ece470ac76b91b8e4e9b79b774d.dir
md5: 28c5dc2686f41b0dbed51360e9d61f7c.dir
md5: 709725ba19bb5f97aedbed7d2edd8f54.dir
md5: fb13ba103951ef9603e8b71be75adffa.dir
md5: 0e90846e5c2944dd7f9f68f7fc30634a.dir
md5: 384c040285e6f2eb9f13a51d219100c2.dir
PushingWARNING:dvc_data.index.fetch:Some of the cache files do not exist neither locally nor on remote. Missing cache files:
md5: a47bf585eda8e10aaa30cdc7e8d469e5.dir
md5: 942738b9064eacca5fbc81116da2bcc7.dir
md5: 6d2f4af81551a0db2fb4fe619b170a44.dir
md5: 573a5adbfdc20e7cb4f6bd00c3ff4242.dir
md5: e614a27f24b38624e91efc347d234afe.dir
md5: a6c7ce3d01e43ee3b11bc9d150be82e8.dir
md5: 830f328697740e3e8784ae050019b2fd.dir
md5: 2370c7201fb662516b962d576580eb7a.dir
md5: cb6185032f0071cbf16c65b8253c266c.dir
md5: a8765898706eda9536b4663db181e291.dir
md5: b842c74ada14da3e2fbee2dba22dca37.dir
md5: 4d0e83696cf45200524646a18c44cdca.dir
md5: 2f24f25b84879aaecd0db0a429ac41fd
md5: ce570dc2720216b7027eb87e5d66adcf.dir
md5: 5cdc0918a13fea894176803ed7677af3.dir
md5: 8a93dbebc046b5ffa0e9134fd9044ca8.dir
md5: 3672004bcc42945481f7e94a50a1b228.dir
md5: a9a3ad024dc69e2cb116eb710b20ea1c.dir
md5: fd0545d65e6bdb26b6552d5c4b76a8dc.dir
md5: e268e8cf24c4ad4a113200eb1604ef58.dir
md5: 548121f283fa53bfead3145e1e385408.dir
md5: 9aba1c782098bdb0ba6a13acbbc262dc.dir
md5: 1f7e8cbefb9eabb688fbaf642bd43876.dir
md5: af596aaa1e251b0799d74a53078edb33.dir
md5: 138cb203e2e6744e0d3d479611d3a5a5.dir
md5: a4e0c75b40834ef21392a1b7c53bb809.dir
md5: f8f6d94af6759deb65b241f9139c9902.dir
md5: 430a804f11d99c0ad85bab105fce7436.dir
md5: bb340a69a3c21502a5a6f049ff0803bc.dir
md5: 06906c90cba632f31ebd9fa3e21940b7.dir
md5: c20cad6350046bfc1df63af6c0dee3d5.dir
md5: cd6b61422f387a9ac11988c71bcd4fcf.dir
md5: d7d62aa0cb2003264562971a896cbad1.dir
md5: a2fe28d609382459eca10fd6d9627208.dir
md5: 3cdd4dd76627cf74d0f09a5012cdedc4.dir
md5: b1d1bc97d0de33a8b9ae04e23f954ab0.dir
md5: d54642c401dd41f53f4a33fc6ca891d8.dir
md5: a6d0fcb7ffeae9ac1fabd48156a1a081.dir
md5: d751713988987e9331980363e24189ce.dir
md5: 99e0012da3764659f8d3d2442fdfdf62.dir
md5: 76fcf0f391c620a3801545728e68c7b7.dir
md5: f038560ccfe371248021ea9820ceb1ff.dir
md5: 72bcfe0becd167bb7b1274f9b6ec0399.dir
md5: c8f09e42ded208823838f96c39e507d8.dir
md5: 6e8a71e9b81ecb94c56a8e637599004e.dir
md5: dd7d5a0e384dde906a03bd26611e2ad2.dir
md5: c26a713ecc0b5fb8ce88ddfecae216c2.dir
md5: 570a2a9e56e88d423c5d167bd5f49a8e
md5: af72bdd60ffe4e9286c985401a57661d.dir
md5: 430c93f785a4db98ae62ba632c898d1c.dir
md5: 01bad613e0b60fe0f4111371dc7dd5d4.dir
md5: 132c4a69581b62798c9820ffab679a97.dir
md5: c9da98a0276c74ebd31c4d4785e65166.dir
md5: 67d238bc3501f8f3a1575fd8abe4c87a.dir
md5: d6a13ece470ac76b91b8e4e9b79b774d.dir
md5: 28c5dc2686f41b0dbed51360e9d61f7c.dir
md5: 709725ba19bb5f97aedbed7d2edd8f54.dir
md5: fb13ba103951ef9603e8b71be75adffa.dir
md5: 0e90846e5c2944dd7f9f68f7fc30634a.dir
md5: 384c040285e6f2eb9f13a51d219100c2.dir
Pushing
WARNING: The following untracked files were present in the workspace before saving but will not be included in the experiment commit:
run_experiment_cmd.py, cache/embeddings/Lowercase-NFD-StripAccents-Whitespace-all-MiniLM-L6-v2-100/embedding.json, cache/embeddings/Lowercase-NFD-StripAccents-Whitespace-all-MiniLM-L6-v2-100/embedding.csv, cache/embeddings/Lowercase-NFD-StripAccents-Whitespace-all-MiniLM-L6-v2-100/embedding.parquet, cache/corpora/Lowercase-NFD-StripAccents-Whitespace/corpus.json, cache/corpora/Lowercase-NFD-StripAccents-Whitespace/corpus.parquet, cache/corpora/Lowercase-NFD-StripAccents-Whitespace/corpus.csv, models/Lowercase-NFD-StripAccents-Whitespace-all-MiniLM-L6-v2-100-Linear/study.db
WARNING:dvc.repo.experiments.executor.base:The following untracked files were present in the workspace before saving but will not be included in the experiment commit:
run_experiment_cmd.py, cache/embeddings/Lowercase-NFD-StripAccents-Whitespace-all-MiniLM-L6-v2-100/embedding.json, cache/embeddings/Lowercase-NFD-StripAccents-Whitespace-all-MiniLM-L6-v2-100/embedding.csv, cache/embeddings/Lowercase-NFD-StripAccents-Whitespace-all-MiniLM-L6-v2-100/embedding.parquet, cache/corpora/Lowercase-NFD-StripAccents-Whitespace/corpus.json, cache/corpora/Lowercase-NFD-StripAccents-Whitespace/corpus.parquet, cache/corpora/Lowercase-NFD-StripAccents-Whitespace/corpus.csv, models/Lowercase-NFD-StripAccents-Whitespace-all-MiniLM-L6-v2-100-Linear/study.db
from dvclive.
So, it looks like it's just pushing a lot of data to the remote storage.
Do you have DVC_EXP_AUTO_PUSH
or config.auto_push
enabled by chance?
Does your pipeline produce a lot of data (every time new?)
from dvclive.
Related Issues (20)
- integrations: hf accelerate HOT 4
- Log metrics/models etc... to studio directly with Live or DVCLiveCallbacks in training script HOT 9
- integrations: lightning fabric
- `log_image` fails to save an image if the path was provided, but the `name` does not have an extension HOT 1
- 4.0 Checklist HOT 1
- Add user friendly warning/error messages and helpers for log_plot() HOT 3
- Foreach to run multiple experiments HOT 3
- DVCLive throws a weird warning when origin is not defined HOT 6
- Add docstrings HOT 2
- Invalid path causes `TypeError` due to shadowed builtin `type` HOT 2
- `Live.log_artifact` raises `ValueError` by logging to closed `stderr` when Tensorflow is imported first HOT 6
- `log_image`: log bounding boxes HOT 5
- Test falling for Huggingface
- Path not supported for some path arguments HOT 2
- Error RuntimeError: SQLite version is too old, please upgrade to >= (3, 8, 3) HOT 7
- Provide examples or quick code snippets in the public methods
- System metrics should be plotted with `timestamp` rather than `step`
- log_sklearn_plot: exception when plot params are passed. HOT 3
- What is the proper way of enabling intermediate checkpointing when using pytorch lightning+dvc+dvclive? HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dvclive.