Comments (2)
I've been dealing with this error for like three days, and now I cannot reproduce it. I suspect it may have been due to an older version of tensorflow datasets. Thanks anyway.
from seqio.
Ok, I have done some more digging and it looks like the error only occurs for some datasets: they MUST be on GCS in the bucket mentioned above for the error to occur. In this case, the user should NOT be providing a data_dir anyway, since there is no reason to download.
Here is an example of something that will trigger an error:
import os
import seqio
import functools
task_name = "wiki40b_en"
data_dir = os.path.abspath(os.path.expanduser("~/my_tfds_test_dir"))
tfds_id = "wiki40b/en:1.3.0"
tfds_col = "text"
vocab = seqio.ByteVocabulary() # whatever
seqio.TaskRegistry.add(
task_name,
source=seqio.TfdsDataSource(tfds_id, tfds_data_dir=data_dir),
preprocessors=[
functools.partial(seqio.preprocessors.rekey, key_map={"targets": tfds_col}),
seqio.preprocessors.tokenize,
],
output_features={"targets": seqio.Feature(vocabulary=vocab)},
)
ds = seqio.TaskRegistry.get(task_name).get_dataset(sequence_length=None, split="train", shuffle=False)
So it seems we can keep this issue closed! Thanks again.
from seqio.
Related Issues (20)
- Please include installation instructions HOT 2
- import seqio
- how to decide ideal mixture rates ? HOT 1
- FunctionDataSource does not allow function with 3 positional arguments thus shuffling does not work HOT 2
- unable to train mt5 from t5x using mixtures ValueError: Dataset is missing an expected feature during input_validation validation: 'inputs' HOT 3
- Tokenizer is not behaving as expected on special tokens (doesn't recognize `pad` and `eos` tokens) HOT 1
- Using a registered task to add another HOT 1
- seqio 0.0.13 cannot be installed on Apple Silicon due to transitive tensorflow dependency of clu HOT 2
- How to apply the huggingface tokenizer in seqio.vocabulary
- Different preprocessors for each dataset split HOT 2
- import seqio: AttributeError: module 'typing' has no attribute 'get_origin' HOT 1
- Concatenating Tasks? HOT 2
- caching tasks goes out of memory due to apache beam HOT 2
- How to choose minimum sequence length while avoiding truncation
- Dataset performance
- seqio.get_mixture_or_task('bool_q_template_0_no_opt_five_shot') failed
- unimax sampling ?
- ValueError: mutable default <class 'seqio.vocabularies.PassThroughVocabulary'> for field vocabulary is not allowed: use default_factory HOT 3
- Unimax sampler implementation?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from seqio.