Git Product home page Git Product logo

Comments (3)

shcheklein avatar shcheklein commented on July 17, 2024

@MKDJr could please try to interrupt it and get the stack trace?

also, could you run it with export DVCLIVE_LOGLEVEL=TRACE please, see if we can get more information, thanks!

if you can try also on some super small script that would be also helpful (just to see if this is about DVCLive or there is something specific yo your environment).

from dvclive.

MKDJr avatar MKDJr commented on July 17, 2024

Thanks for the response! I set the loglevel to trace and when I interrupted the command after about 60 seconds this is the error I got:

100% Adding...|██████████████████████████████████████████████████████████████████████████████████████████████████|1/1 [00:00, 21.31file/s]
^CTraceback (most recent call last):
  File "/home/mkdjr/technical-language-vs-sensed-values-alignment/run_experiment.py", line 225, in <module>
    live.end()
  File "/home/mkdjr/.pyenv/versions/pytorch-env/lib/python3.11/site-packages/dvclive/live.py", line 949, in end
    self.save_dvc_exp()
  File "/home/mkdjr/.pyenv/versions/pytorch-env/lib/python3.11/site-packages/dvclive/utils.py", line 182, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/mkdjr/.pyenv/versions/pytorch-env/lib/python3.11/site-packages/dvclive/live.py", line 979, in save_dvc_exp
    self._experiment_rev = self._dvc_repo.experiments.save(
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mkdjr/.pyenv/versions/pytorch-env/lib/python3.11/site-packages/dvc/repo/experiments/__init__.py", line 359, in save
    return save(self.repo, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mkdjr/.pyenv/versions/pytorch-env/lib/python3.11/site-packages/dvc/repo/experiments/save.py", line 36, in save
    save_result = executor.save(
                  ^^^^^^^^^^^^^^
  File "/home/mkdjr/.pyenv/versions/pytorch-env/lib/python3.11/site-packages/dvc/repo/experiments/executor/base.py", line 302, in save
    with cls.auto_push(dvc):
  File "/home/mkdjr/.pyenv/versions/3.11.9/lib/python3.11/contextlib.py", line 144, in __exit__
    next(self.gen)
  File "/home/mkdjr/.pyenv/versions/pytorch-env/lib/python3.11/site-packages/dvc/repo/experiments/executor/base.py", line 692, in auto_push
    cls._auto_push(dvc, git_remote)
  File "/home/mkdjr/.pyenv/versions/pytorch-env/lib/python3.11/site-packages/dvc/repo/experiments/executor/base.py", line 713, in _auto_push
    dvc.experiments.push(
  File "/home/mkdjr/.pyenv/versions/pytorch-env/lib/python3.11/site-packages/dvc/repo/experiments/__init__.py", line 364, in push
    return push(self.repo, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mkdjr/.pyenv/versions/pytorch-env/lib/python3.11/site-packages/dvc/repo/__init__.py", line 58, in wrapper
    return f(repo, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mkdjr/.pyenv/versions/pytorch-env/lib/python3.11/site-packages/dvc/repo/scm_context.py", line 143, in run
    return method(repo, *args, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mkdjr/.pyenv/versions/pytorch-env/lib/python3.11/site-packages/dvc/repo/experiments/push.py", line 126, in push
    result["uploaded"] = _push_cache(repo, pushed_refs_info, **kwargs)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mkdjr/.pyenv/versions/pytorch-env/lib/python3.11/site-packages/dvc/repo/experiments/push.py", line 182, in _push_cache
    return repo.push(
           ^^^^^^^^^^
  File "/home/mkdjr/.pyenv/versions/pytorch-env/lib/python3.11/site-packages/dvc/repo/__init__.py", line 58, in wrapper
    return f(repo, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mkdjr/.pyenv/versions/pytorch-env/lib/python3.11/site-packages/dvc/repo/push.py", line 98, in push
    used_run_cache = self.stage_cache.push(remote) if run_cache else []
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mkdjr/.pyenv/versions/pytorch-env/lib/python3.11/site-packages/dvc/stage/cache.py", line 281, in push
    return self.transfer(self.repo.cache.legacy, dest_odb)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mkdjr/.pyenv/versions/pytorch-env/lib/python3.11/site-packages/dvc/stage/cache.py", line 264, in transfer
    if to_fs.exists(key) and first(to_fs.find(key)):
                             ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mkdjr/.pyenv/versions/pytorch-env/lib/python3.11/site-packages/funcy/seqs.py", line 63, in first
    return next(iter(seq), None)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/mkdjr/.pyenv/versions/pytorch-env/lib/python3.11/site-packages/dvc_objects/fs/base.py", line 529, in find
    yield from self.fs.find(path)
  File "/home/mkdjr/.pyenv/versions/pytorch-env/lib/python3.11/site-packages/dvc_objects/fs/local.py", line 78, in find
    for root, _, files in self.walk(path, **kwargs):
  File "/home/mkdjr/.pyenv/versions/pytorch-env/lib/python3.11/site-packages/dvc_objects/fs/local.py", line 56, in walk
    for root, dirs, files in os.walk(
  File "<frozen os>", line 358, in _walk
KeyboardInterrupt

After waiting for no more than an hour and a half, pushing did finally finish with the following message:

100% Adding...|██████████████████████████████████████████████████████████████████████████████████████████████████|1/1 [00:00, 26.70file/s]
Collecting                                                                                                    |2.60M [02:58, 14.6kentry/s]
WARNING: Some of the cache files do not exist neither locally nor on remote. Missing cache files:
md5: a47bf585eda8e10aaa30cdc7e8d469e5.dir
md5: 942738b9064eacca5fbc81116da2bcc7.dir
md5: 6d2f4af81551a0db2fb4fe619b170a44.dir
md5: 573a5adbfdc20e7cb4f6bd00c3ff4242.dir
md5: e614a27f24b38624e91efc347d234afe.dir
md5: a6c7ce3d01e43ee3b11bc9d150be82e8.dir
md5: 830f328697740e3e8784ae050019b2fd.dir
md5: 2370c7201fb662516b962d576580eb7a.dir
md5: cb6185032f0071cbf16c65b8253c266c.dir
md5: a8765898706eda9536b4663db181e291.dir
md5: b842c74ada14da3e2fbee2dba22dca37.dir
md5: 4d0e83696cf45200524646a18c44cdca.dir
md5: 2f24f25b84879aaecd0db0a429ac41fd
md5: ce570dc2720216b7027eb87e5d66adcf.dir
md5: 5cdc0918a13fea894176803ed7677af3.dir
md5: 8a93dbebc046b5ffa0e9134fd9044ca8.dir
md5: 3672004bcc42945481f7e94a50a1b228.dir
md5: a9a3ad024dc69e2cb116eb710b20ea1c.dir
md5: fd0545d65e6bdb26b6552d5c4b76a8dc.dir
md5: e268e8cf24c4ad4a113200eb1604ef58.dir
md5: 548121f283fa53bfead3145e1e385408.dir
md5: 9aba1c782098bdb0ba6a13acbbc262dc.dir
md5: 1f7e8cbefb9eabb688fbaf642bd43876.dir
md5: af596aaa1e251b0799d74a53078edb33.dir
md5: 138cb203e2e6744e0d3d479611d3a5a5.dir
md5: a4e0c75b40834ef21392a1b7c53bb809.dir
md5: f8f6d94af6759deb65b241f9139c9902.dir
md5: 430a804f11d99c0ad85bab105fce7436.dir
md5: bb340a69a3c21502a5a6f049ff0803bc.dir
md5: 06906c90cba632f31ebd9fa3e21940b7.dir
md5: c20cad6350046bfc1df63af6c0dee3d5.dir
md5: cd6b61422f387a9ac11988c71bcd4fcf.dir
md5: d7d62aa0cb2003264562971a896cbad1.dir
md5: a2fe28d609382459eca10fd6d9627208.dir
md5: 3cdd4dd76627cf74d0f09a5012cdedc4.dir
md5: b1d1bc97d0de33a8b9ae04e23f954ab0.dir
md5: d54642c401dd41f53f4a33fc6ca891d8.dir
md5: a6d0fcb7ffeae9ac1fabd48156a1a081.dir
md5: d751713988987e9331980363e24189ce.dir
md5: 99e0012da3764659f8d3d2442fdfdf62.dir
md5: 76fcf0f391c620a3801545728e68c7b7.dir
md5: f038560ccfe371248021ea9820ceb1ff.dir
md5: 72bcfe0becd167bb7b1274f9b6ec0399.dir
md5: c8f09e42ded208823838f96c39e507d8.dir
md5: 6e8a71e9b81ecb94c56a8e637599004e.dir
md5: dd7d5a0e384dde906a03bd26611e2ad2.dir
md5: c26a713ecc0b5fb8ce88ddfecae216c2.dir
md5: 570a2a9e56e88d423c5d167bd5f49a8e
md5: af72bdd60ffe4e9286c985401a57661d.dir
md5: 430c93f785a4db98ae62ba632c898d1c.dir
md5: 01bad613e0b60fe0f4111371dc7dd5d4.dir
md5: 132c4a69581b62798c9820ffab679a97.dir
md5: c9da98a0276c74ebd31c4d4785e65166.dir
md5: 67d238bc3501f8f3a1575fd8abe4c87a.dir
md5: d6a13ece470ac76b91b8e4e9b79b774d.dir
md5: 28c5dc2686f41b0dbed51360e9d61f7c.dir
md5: 709725ba19bb5f97aedbed7d2edd8f54.dir
md5: fb13ba103951ef9603e8b71be75adffa.dir
md5: 0e90846e5c2944dd7f9f68f7fc30634a.dir
md5: 384c040285e6f2eb9f13a51d219100c2.dir
PushingWARNING:dvc_data.index.fetch:Some of the cache files do not exist neither locally nor on remote. Missing cache files:
md5: a47bf585eda8e10aaa30cdc7e8d469e5.dir
md5: 942738b9064eacca5fbc81116da2bcc7.dir
md5: 6d2f4af81551a0db2fb4fe619b170a44.dir
md5: 573a5adbfdc20e7cb4f6bd00c3ff4242.dir
md5: e614a27f24b38624e91efc347d234afe.dir
md5: a6c7ce3d01e43ee3b11bc9d150be82e8.dir
md5: 830f328697740e3e8784ae050019b2fd.dir
md5: 2370c7201fb662516b962d576580eb7a.dir
md5: cb6185032f0071cbf16c65b8253c266c.dir
md5: a8765898706eda9536b4663db181e291.dir
md5: b842c74ada14da3e2fbee2dba22dca37.dir
md5: 4d0e83696cf45200524646a18c44cdca.dir
md5: 2f24f25b84879aaecd0db0a429ac41fd
md5: ce570dc2720216b7027eb87e5d66adcf.dir
md5: 5cdc0918a13fea894176803ed7677af3.dir
md5: 8a93dbebc046b5ffa0e9134fd9044ca8.dir
md5: 3672004bcc42945481f7e94a50a1b228.dir
md5: a9a3ad024dc69e2cb116eb710b20ea1c.dir
md5: fd0545d65e6bdb26b6552d5c4b76a8dc.dir
md5: e268e8cf24c4ad4a113200eb1604ef58.dir
md5: 548121f283fa53bfead3145e1e385408.dir
md5: 9aba1c782098bdb0ba6a13acbbc262dc.dir
md5: 1f7e8cbefb9eabb688fbaf642bd43876.dir
md5: af596aaa1e251b0799d74a53078edb33.dir
md5: 138cb203e2e6744e0d3d479611d3a5a5.dir
md5: a4e0c75b40834ef21392a1b7c53bb809.dir
md5: f8f6d94af6759deb65b241f9139c9902.dir
md5: 430a804f11d99c0ad85bab105fce7436.dir
md5: bb340a69a3c21502a5a6f049ff0803bc.dir
md5: 06906c90cba632f31ebd9fa3e21940b7.dir
md5: c20cad6350046bfc1df63af6c0dee3d5.dir
md5: cd6b61422f387a9ac11988c71bcd4fcf.dir
md5: d7d62aa0cb2003264562971a896cbad1.dir
md5: a2fe28d609382459eca10fd6d9627208.dir
md5: 3cdd4dd76627cf74d0f09a5012cdedc4.dir
md5: b1d1bc97d0de33a8b9ae04e23f954ab0.dir
md5: d54642c401dd41f53f4a33fc6ca891d8.dir
md5: a6d0fcb7ffeae9ac1fabd48156a1a081.dir
md5: d751713988987e9331980363e24189ce.dir
md5: 99e0012da3764659f8d3d2442fdfdf62.dir
md5: 76fcf0f391c620a3801545728e68c7b7.dir
md5: f038560ccfe371248021ea9820ceb1ff.dir
md5: 72bcfe0becd167bb7b1274f9b6ec0399.dir
md5: c8f09e42ded208823838f96c39e507d8.dir
md5: 6e8a71e9b81ecb94c56a8e637599004e.dir
md5: dd7d5a0e384dde906a03bd26611e2ad2.dir
md5: c26a713ecc0b5fb8ce88ddfecae216c2.dir
md5: 570a2a9e56e88d423c5d167bd5f49a8e
md5: af72bdd60ffe4e9286c985401a57661d.dir
md5: 430c93f785a4db98ae62ba632c898d1c.dir
md5: 01bad613e0b60fe0f4111371dc7dd5d4.dir
md5: 132c4a69581b62798c9820ffab679a97.dir
md5: c9da98a0276c74ebd31c4d4785e65166.dir
md5: 67d238bc3501f8f3a1575fd8abe4c87a.dir
md5: d6a13ece470ac76b91b8e4e9b79b774d.dir
md5: 28c5dc2686f41b0dbed51360e9d61f7c.dir
md5: 709725ba19bb5f97aedbed7d2edd8f54.dir
md5: fb13ba103951ef9603e8b71be75adffa.dir
md5: 0e90846e5c2944dd7f9f68f7fc30634a.dir
md5: 384c040285e6f2eb9f13a51d219100c2.dir
Pushing
WARNING: The following untracked files were present in the workspace before saving but will not be included in the experiment commit:
        run_experiment_cmd.py, cache/embeddings/Lowercase-NFD-StripAccents-Whitespace-all-MiniLM-L6-v2-100/embedding.json, cache/embeddings/Lowercase-NFD-StripAccents-Whitespace-all-MiniLM-L6-v2-100/embedding.csv, cache/embeddings/Lowercase-NFD-StripAccents-Whitespace-all-MiniLM-L6-v2-100/embedding.parquet, cache/corpora/Lowercase-NFD-StripAccents-Whitespace/corpus.json, cache/corpora/Lowercase-NFD-StripAccents-Whitespace/corpus.parquet, cache/corpora/Lowercase-NFD-StripAccents-Whitespace/corpus.csv, models/Lowercase-NFD-StripAccents-Whitespace-all-MiniLM-L6-v2-100-Linear/study.db
WARNING:dvc.repo.experiments.executor.base:The following untracked files were present in the workspace before saving but will not be included in the experiment commit:
        run_experiment_cmd.py, cache/embeddings/Lowercase-NFD-StripAccents-Whitespace-all-MiniLM-L6-v2-100/embedding.json, cache/embeddings/Lowercase-NFD-StripAccents-Whitespace-all-MiniLM-L6-v2-100/embedding.csv, cache/embeddings/Lowercase-NFD-StripAccents-Whitespace-all-MiniLM-L6-v2-100/embedding.parquet, cache/corpora/Lowercase-NFD-StripAccents-Whitespace/corpus.json, cache/corpora/Lowercase-NFD-StripAccents-Whitespace/corpus.parquet, cache/corpora/Lowercase-NFD-StripAccents-Whitespace/corpus.csv, models/Lowercase-NFD-StripAccents-Whitespace-all-MiniLM-L6-v2-100-Linear/study.db

from dvclive.

shcheklein avatar shcheklein commented on July 17, 2024

So, it looks like it's just pushing a lot of data to the remote storage.

Do you have DVC_EXP_AUTO_PUSH or config.auto_push enabled by chance?
Does your pipeline produce a lot of data (every time new?)

from dvclive.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.