Load data from a list of lists and build a vocabulary about text HOT 6 CLOSED

I think this can be closed. As of 0.9.0 there's a build_vocab_from_iterator which accepts a List[List[str]].

from text.

Is the number of sentences in each sample fixed? Because if not, torchtext doesn't have a good way to do this right now (if yes, use Example.from_list in a custom Dataset). There might be a way to hack up something that would work for the variable-number-of-sentences case, but it would involve subclassing Field, Example, and Dataset and that's probably not worth it.

from text.

Yes, the number of sentences in each sequence is fixed but the number of words in each sentence is variable. So once I convert the list into Examples, is there any streamlined way to convert the Examples into, say, a TabularDataset that I can iterate over?

from text.

OK, if the number of sentences per example is fixed, then use a single Example object per sequence with one column for each sentence. You can create Examples like this:

ex = SequentialExample.fromlist(raw_list_of_sentences, [('sentence' + str(i), text_field) for i in range(len(raw_list_of_sentences))])

then build an iterator as normal and use the batches like this

for sentence_index in range(num_sentences):
    do_something(getattr(batch, 'sentence' + str(sentence_index)))

from text.

Hi, SequentialExample is an object in TorchText? Would you please

OK, if the number of sentences per example is fixed, then use a single Example object per sequence with one column for each sentence. You can create Examples like this:

ex = SequentialExample.fromlist(raw_list_of_sentences, [('sentence' + str(i), text_field) for i in range(len(raw_list_of_sentences))])

then build an iterator as normal and use the batches like this

for sentence_index in range(num_sentences):
    do_something(getattr(batch, 'sentence' + str(sentence_index)))

Hi, SequentialExample is an object in TorchText? Would you please say more clearly, thank you for advance.

from text.

I think this can be closed. As of 0.9.0 there's a build_vocab_from_iterator which accepts a List[List[str]].

Thanks for identifying stale issues @erip. Moving forward, we should make an effort to clean up non-relevant issues and PRs. I will create an issue for this soon 😃

from text.