Comments (6)
nshazeer
commented on September 26, 2024
1
(None, 0, None) indicates the sharding of the tensor-dimensions. The first
and third tensor-dimensions are not split. The second tensor-dimension
(batch=4) is split across the 0-th mesh-dimension.
Say that this is using model_parallelism=1 (i.e. pure data-parallelism) on
an 8-core TPU. This would mean that the batch needs to be split 8 ways.
But the batch size is 4, so this is impossible.
Solutions would be to either double the batch size, or to increase
model-parallelism to 2 or 4.
…On Wed, Apr 1, 2020 at 11:11 AM Colin Raffel ***@***.***> wrote:
@orsk-moscow <https://github.com/orsk-moscow> Are you running on a TPU or
CPU? What model parallelism are you using?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#67 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AHDH33WDYZVDXNFIFA3VXKLRKN7UXANCNFSM4KP37GSQ>
.
from text-to-text-transfer-transformer.
adarob
commented on September 26, 2024
@nshazeer do you have any ideas?
from text-to-text-transfer-transformer.
nshazeer
commented on September 26, 2024
I think that the mesh_shape needs to be empty "" for single-device CPU execution. It looks like the MtfModel constructor is hard-coded to always call utils.tpu_mesh_shape(). That should probably be changed.
from text-to-text-transfer-transformer.
orsk-moscow
commented on September 26, 2024
I've got exactly same error:
Tensor dimension size not divisible by mesh dimension size: tensor_shape=Shape[outer_batch=1, batch=4, length=128] tensor_layout=TensorLayout(None, 0, None)
I saw your patch 15c1d96, it refer to TPU topology in case of using a TPU, and i use standart Google Colab TPU, so i do not have any ideas why I 've got (None,0,None) as a mesh_dim.size
from text-to-text-transfer-transformer.
craffel
commented on September 26, 2024
@orsk-moscow Are you running on a TPU or CPU? What model parallelism are you using?
from text-to-text-transfer-transformer.
orsk-moscow
commented on September 26, 2024
Hi @nshazeer , thanks for clarification
from text-to-text-transfer-transformer.
Related Issues (20)
- ValueError when evaluating tuning model using Mtf library
- using A100（40G）*8 gpus server to train T5-3b，it reports OOM resource is exhausted problem HOT 2
- How should I speed up T5 exported saved_model by using TF-TRT ?
- model.finetune(...) does not show the loss of the model HOT 6
- CUDA OOM with HF Model
- Predictions are inconsistent unless model is reloaded for each prediction HOT 1
- how to change teacher forcing fashion to autogressive fashion in training stage?
- ERROR:root:Path not found: gs://t5-data/pretrained_models/large/operative_config.gin HOT 6
- Fine tuning t5 without TPU
- About "seqio" in "hf_model.py"
- Question about the metric reported in the paper?
- All attempts to get a Google authentication bearer token failed, returning an empty token. HOT 2
- How to fine-tune T5 with a Casual Language Modeling object?
- cmd vs entrypoint youtube video suggestion HOT 1
- Question about cross-node(multi-node) data parallelism on GPU HOT 1
- Dependencies in `setup.py` have module conflicts.
- How can I get the best checkpoint in Squad?
- Custom Model
- Columns and DataType Not Explicitly Set on line 163 of eval_utils_test.py
- Clarification on T5 Model Pre-training Objective and Denoising Process
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from text-to-text-transfer-transformer.