john-hewitt / backpacks-flash-attn Goto Github PK
View Code? Open in Web Editor NEWThe original Backpack Language Model implementation, a fork of FlashAttention
License: BSD 3-Clause "New" or "Revised" License
backpacks-flash-attn's People
Contributors
Stargazers
Watchers
backpacks-flash-attn's Issues
Fix some figures in the paper from arXiv
Hi, thanks for your excellent work. When I read the paper from arXiv version (https://arxiv.org/pdf/2305.16765.pdf), the Figure 1 and Figure 3 are not visible. Could you fix the figures in arxiv?
CUDA version in nvcc -V and nvidia-smi different
I'm using the lab's server from my school. The server contains different gpus with different cuda version. RTX A6000 has cuda version of 11.7. But the nvcc is still version of 10.1. I checked online. It is because that nvcc's version needs to be compatible with all GPUs (since some GPUs in the server has verson of 10.1). Thus, nvcc is the lowest version. When running the setup.py file, the cuda version is detecting nvcc‘s version. Then, it always pop up errors. I do not have authority to modify the server's cuda version. How to make the setup.py file to detect and use the environment from nvidia-smi?
Hydra's cascading AttributeErrors
Hello Authors,
We are trying to reproduce the micro model on 2 GPUs and run into a cascade of AttributeErrors as pasted below. We wonder if you could provide us some pointers on how to fix this issue. Many thanks!
(flenv) [nuanwen@d14-03 training]$ torchrun --nproc_per_node=2 run.py experiment=owt/backpack-micro-flash trainer.devices=2 name=backpack-micro-flash-fp16 datamodule.batch_size=128
...
[2023-11-21 19:44:28,586][lightning_lite.utilities.seed][INFO] - [rank: 0] Global seed set to 1111
[2023-11-21 19:44:28,617][src.tasks.seq][INFO] - Instantiating model <src.models.backpack.BackpackLMHeadModel>
Error executing job with overrides: ['experiment=owt/backpack-micro-flash', 'trainer.devices=2', 'name=backpack-micro-flash-fp16', 'datamodule.batch_size=128']
Error executing job with overrides: ['experiment=owt/backpack-micro-flash', 'trainer.devices=2', 'name=backpack-micro-flash-fp16', 'datamodule.batch_size=128']
Traceback (most recent call last):
File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/utils.py", line 644, in _locate
Traceback (most recent call last):
File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/utils.py", line 644, in _locate
obj = getattr(obj, part)
AttributeError: module 'src.models' has no attribute 'backpack'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/utils.py", line 650, in _locate
obj = getattr(obj, part)
AttributeError: module 'src.models' has no attribute 'backpack'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/utils.py", line 650, in _locate
obj = import_module(mod)
File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/importlib/__init__.py", line 127, in import_module
obj = import_module(mod)
File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 790, in exec_module
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/training/src/models/backpack.py", line 19, in <module>
File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 790, in exec_module
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/training/src/models/backpack.py", line 19, in <module>
from flash_attn.modules.mlp import Mlp, FusedDenseGeluDense, ParallelFusedDenseGeluDense
ImportError: cannot import name 'FusedDenseGeluDense' from 'flash_attn.modules.mlp' (/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/flash_attn/modules/mlp.py)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 134, in _resolve_target
from flash_attn.modules.mlp import Mlp, FusedDenseGeluDense, ParallelFusedDenseGeluDense
ImportError: cannot import name 'FusedDenseGeluDense' from 'flash_attn.modules.mlp' (/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/flash_attn/modules/mlp.py)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 134, in _resolve_target
target = _locate(target)
File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/utils.py", line 658, in _locate
target = _locate(target)
File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/utils.py", line 658, in _locate
raise ImportError(
ImportError: Error loading 'src.models.backpack.BackpackLMHeadModel':
ImportError("cannot import name 'FusedDenseGeluDense' from 'flash_attn.modules.mlp' (/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/flash_attn/modules/mlp.py)")
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 92, in _call_target
raise ImportError(
ImportError: Error loading 'src.models.backpack.BackpackLMHeadModel':
ImportError("cannot import name 'FusedDenseGeluDense' from 'flash_attn.modules.mlp' (/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/flash_attn/modules/mlp.py)")
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 92, in _call_target
return _target_(*args, **kwargs)
File "/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/training/src/tasks/seq.py", line 33, in __init__
return _target_(*args, **kwargs)
File "/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/training/src/tasks/seq.py", line 33, in __init__
self.instantiate_model()
File "/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/training/src/tasks/seq.py", line 56, in instantiate_model
self.instantiate_model()
File "/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/training/src/tasks/seq.py", line 56, in instantiate_model
self.model = hydra.utils.instantiate(self.model_cfg, _recursive_=recursive)
File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 226, in instantiate
self.model = hydra.utils.instantiate(self.model_cfg, _recursive_=recursive)
File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 226, in instantiate
return instantiate_node(
File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 333, in instantiate_node
return instantiate_node(
File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 333, in instantiate_node
_target_ = _resolve_target(node.get(_Keys.TARGET), full_key)
File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 139, in _resolve_target
_target_ = _resolve_target(node.get(_Keys.TARGET), full_key)
File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 139, in _resolve_target
raise InstantiationException(msg) from eraise InstantiationException(msg) from e
hydra.errors
.InstantiationExceptionhydra.errors.: Error locating target 'src.models.backpack.BackpackLMHeadModel', set env var HYDRA_FULL_ERROR=1 to see chained exception.
full_key: cfg.modelInstantiationException
The above exception was the direct cause of the following exception:
: Traceback (most recent call last):
Error locating target 'src.models.backpack.BackpackLMHeadModel', set env var HYDRA_FULL_ERROR=1 to see chained exception.
full_key: cfg.model
File "/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/training/run.py", line 68, in <module>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/training/run.py", line 68, in <module>
main()
File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/main.py", line 94, in decorated_main
main()
File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/main.py", line 94, in decorated_main
_run_hydra(
File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
_run_hydra(
File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
_run_app(
File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/utils.py", line 457, in _run_app
_run_app(
File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/utils.py", line 457, in _run_app
run_and_report(
File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
run_and_report(
File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
raise ex
File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
raise ex
File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
return func()
File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
return func()
File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
lambda: hydra.run(
File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/hydra.py", line 132, in run
lambda: hydra.run(
File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/hydra.py", line 132, in run
_ = ret.return_value
File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/core/utils.py", line 260, in return_value
_ = ret.return_value
File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/core/utils.py", line 260, in return_value
raise self._return_value
File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/core/utils.py", line 186, in run_job
raise self._return_value
File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/core/utils.py", line 186, in run_job
ret.return_value = task_function(task_cfg)ret.return_value = task_function(task_cfg)
File "/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/training/run.py", line 62, in main
File "/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/training/run.py", line 62, in main
return train(config)
File "/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/training/src/train.py", line 50, in train
return train(config)
File "/project/jonmay_231/nuanwen/current/classes/csci662/project/backpacks-flash-attn/training/src/train.py", line 50, in train
model: LightningModule = hydra.utils.instantiate(config.task, cfg=config, _recursive_=False)
File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 226, in instantiate
model: LightningModule = hydra.utils.instantiate(config.task, cfg=config, _recursive_=False)
File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 226, in instantiate
return instantiate_node(
File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 347, in instantiate_node
return instantiate_node(
File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 347, in instantiate_node
return _call_target(_target_, partial, args, kwargs, full_key)
File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 97, in _call_target
raise InstantiationException(msg) from e
hydra.errors.InstantiationException: Error in call to target 'src.tasks.seq.SequenceLMModel':
InstantiationException("Error locating target 'src.models.backpack.BackpackLMHeadModel', set env var HYDRA_FULL_ERROR=1 to see chained exception.\nfull_key: cfg.model")
full_key: task
return _call_target(_target_, partial, args, kwargs, full_key)
File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 97, in _call_target
raise InstantiationException(msg) from e
hydra.errors.InstantiationException: Error in call to target 'src.tasks.seq.SequenceLMModel':
InstantiationException("Error locating target 'src.models.backpack.BackpackLMHeadModel', set env var HYDRA_FULL_ERROR=1 to see chained exception.\nfull_key: cfg.model")
full_key: task
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 46950) of binary: /home1/nuanwen/.conda/envs/flenv/bin/python
Traceback (most recent call last):
File "/home1/nuanwen/.conda/envs/flenv/bin/torchrun", line 8, in <module>
sys.exit(main())
File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
return f(*args, **kwargs)
File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/distributed/run.py", line 762, in main
run(args)
File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/distributed/run.py", line 753, in run
elastic_launch(
File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
File "/home1/nuanwen/.conda/envs/flenv/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 246, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
run.py FAILED
------------------------------------------------------------
Failures:
[1]:
time : 2023-11-21_19:44:32
host : d14-03.hpc.usc.edu
rank : 1 (local_rank: 1)
exitcode : 1 (pid: 46951)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
time : 2023-11-21_19:44:32
host : d14-03.hpc.usc.edu
rank : 0 (local_rank: 0)
exitcode : 1 (pid: 46950)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
EOFError when np.load
Did you train on a single CPU or multiple CPUs? On a single CPU, it takes too long to tokenize. But when on multiple CPUs, data load from npy file pops up an error. (EOF error) It seems like multi-process access the file at the same time. How to resolve the problem?
Running tokenizer on dataset (num_proc=32): 100%|██████████| 8009762/8009762 [2:34:45<00:00, 862.59 examples/s]
Running tokenizer on dataset (num_proc=32): 100%|██████████| 8009762/8009762 [2:34:45<00:00, 862.59 examples/s]
Running tokenizer on dataset (num_proc=32): 100%|██████████| 8009762/8009762 [2:34:45<00:00, 862.59 examples/s]
Running tokenizer on dataset (num_proc=32): 100%|██████████| 4007/4007 [01:51<00:00, 36.07 examples/s] ]ples/s]
Running tokenizer on dataset (num_proc=32): 100%|██████████| 4007/4007 [00:11<00:00, 335.22 examples/s]
Running tokenizer on dataset (num_proc=32): 100%|██████████| 4007/4007 [00:10<00:00, 388.29 examples/s]
Running tokenizer on dataset (num_proc=32): 97%|█████████▋| 3882/4007 [00:03<00:00, 1052.70 examples/s]
Running tokenizer on dataset (num_proc=32): 100%|██████████| 8009762/8009762 [2:35:31<00:00, 858.32 examples/s]
Concatenating examples (num_proc=32): 100%|██████████| 8032/8032 [27:28<00:00, 4.87 examples/s]
Concatenating examples (num_proc=32): 100%|██████████| 32/32 [00:03<00:00, 10.07 examples/s]les]
Concatenating examples (num_proc=32): 100%|██████████| 32/32 [00:03<00:00, 10.03 examples/s]s/s]
Concatenating examples (num_proc=32): 100%|██████████| 8032/8032 [27:35<00:00, 4.85 examples/s]
Concatenating examples (num_proc=32): 100%|██████████| 32/32 [00:00<00:00, 35.47 examples/s]s/s]
Concatenating examples (num_proc=32): 100%|██████████| 32/32 [00:00<00:00, 49.27 examples/s]s/s]
Concatenating examples (num_proc=32): 100%|██████████| 8032/8032 [26:52<00:00, 4.98 examples/s]
Concatenating examples (num_proc=32): 100%|██████████| 8032/8032 [26:32<00:00, 5.04 examples/s]
Concatenating examples (num_proc=32): 100%|██████████| 8032/8032 [26:41<00:00, 5.01 examples/s]
Concatenating examples (num_proc=32): 100%|██████████| 8032/8032 [26:50<00:00, 4.99 examples/s]
Concatenating examples (num_proc=32): 100%|██████████| 8032/8032 [27:51<00:00, 4.80 examples/s]
Concatenating examples (num_proc=32): 100%|██████████| 32/32 [00:00<00:00, 108.61 examples/s]
Concatenating examples (num_proc=32): 100%|██████████| 32/32 [00:00<00:00, 77.63 examples/s] /s]
Concatenating examples (num_proc=32): 100%|██████████| 32/32 [00:00<00:00, 73.01 examples/s]]/s]
Concatenating examples (num_proc=32): 100%|██████████| 32/32 [00:00<00:00, 87.52 examples/s]
Concatenating examples (num_proc=32): 100%|██████████| 32/32 [00:00<00:00, 118.30 examples/s]/s]
Concatenating examples (num_proc=32): 100%|██████████| 32/32 [00:00<00:00, 110.60 examples/s]
Concatenating examples (num_proc=32): 100%|██████████| 32/32 [00:00<00:00, 99.78 examples/s] /s]
[2023-11-24 02:07:49,615][src.utils.utils][INFO] - Saving to cache at /home/tangyimi/backpacks-flash-attn/training/data/openwebtext/cache/tokenizer_name-gpt2-val_ratio-0.0005-val_split_seed-2357-add_eos-True-detokenize-False
Concatenating examples (num_proc=32): 100%|██████████| 32/32 [00:00<00:00, 73.51 examples/s]s/s]
Concatenating examples (num_proc=32): 100%|██████████| 32/32 [00:00<00:00, 65.07 examples/s]
Concatenating examples (num_proc=32): 100%|██████████| 32/32 [00:00<00:00, 120.52 examples/s]/s]
Concatenating examples (num_proc=32): 100%|██████████| 8032/8032 [27:59<00:00, 4.78 examples/s]
Concatenating examples (num_proc=32): 100%|██████████| 32/32 [00:00<00:00, 123.27 examples/s]
Concatenating examples (num_proc=32): 100%|██████████| 32/32 [00:00<00:00, 122.61 examples/s]
Error executing job with overrides: ['experiment=owt/backpack-mini-flash', 'trainer.devices=8', 'name=backpack-mini-flash-fp16', 'datamodule.batch_size=128', 'datamodule.num_workers=32']
Traceback (most recent call last):
File "/home/tangyimi/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 92, in _call_target
return _target_(*args, **kwargs)
File "/home/tangyimi/backpacks-flash-attn/training/src/tasks/seq.py", line 32, in __init__
self.instantiate_datamodule() # Uncomment this for reloading data when pretraining
File "/home/tangyimi/backpacks-flash-attn/training/src/tasks/seq.py", line 43, in instantiate_datamodule
self._datamodule.setup()
File "/home/tangyimi/backpacks-flash-attn/training/src/datamodules/language_modeling_hf.py", line 89, in setup
concat_ids, self.tokenizer = self.process_dataset()
File "/home/tangyimi/backpacks-flash-attn/training/src/datamodules/language_modeling_hf.py", line 101, in process_dataset
return self._load_from_cache(cache_dir)
File "/home/tangyimi/backpacks-flash-attn/training/src/datamodules/language_modeling_hf.py", line 243, in _load_from_cache
concat_ids = {split: np.load(cache_dir / f'{split}.npy', mmap_mode='r')
File "/home/tangyimi/backpacks-flash-attn/training/src/datamodules/language_modeling_hf.py", line 243, in <dictcomp>
concat_ids = {split: np.load(cache_dir / f'{split}.npy', mmap_mode='r')
File "/home/tangyimi/.conda/envs/flenv/lib/python3.9/site-packages/numpy/lib/npyio.py", line 436, in load
raise EOFError("No data left in file")
EOFError: No data left in file
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/tangyimi/backpacks-flash-attn/training/run.py", line 68, in <module>
main()
File "/home/tangyimi/.conda/envs/flenv/lib/python3.9/site-packages/hydra/main.py", line 94, in decorated_main
_run_hydra(
File "/home/tangyimi/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
_run_app(
File "/home/tangyimi/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/utils.py", line 457, in _run_app
run_and_report(
File "/home/tangyimi/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
raise ex
File "/home/tangyimi/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
return func()
File "/home/tangyimi/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
lambda: hydra.run(
File "/home/tangyimi/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/hydra.py", line 132, in run
_ = ret.return_value
File "/home/tangyimi/.conda/envs/flenv/lib/python3.9/site-packages/hydra/core/utils.py", line 260, in return_value
raise self._return_value
File "/home/tangyimi/.conda/envs/flenv/lib/python3.9/site-packages/hydra/core/utils.py", line 186, in run_job
ret.return_value = task_function(task_cfg)
File "/home/tangyimi/backpacks-flash-attn/training/run.py", line 62, in main
return train(config)
File "/home/tangyimi/backpacks-flash-attn/training/src/train.py", line 50, in train
model: LightningModule = hydra.utils.instantiate(config.task, cfg=config, _recursive_=False)
File "/home/tangyimi/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 226, in instantiate
return instantiate_node(
File "/home/tangyimi/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 347, in instantiate_node
return _call_target(_target_, partial, args, kwargs, full_key)
File "/home/tangyimi/.conda/envs/flenv/lib/python3.9/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 97, in _call_target
raise InstantiationException(msg) from e
hydra.errors.InstantiationException: Error in call to target 'src.tasks.seq.SequenceLMModel':
EOFError('No data left in file')
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.