Hi, I appreciate your sharing this project! It is a very thoughtful work and friendly to newers!
I have some questions when reading the code.
In dataset.py
Instead of writing:
new_instances = [i for i in self.instances]
return self.__class__(new_instances[:max_instances])
Why don't you write:
return self.__class__(self.instances[:max_instances])
At dataset.py line 180:
label_counts = [(label, len([x for x in group]))
for label, group
in itertools.groupby(labels, lambda x: x[0])]
label_count_str = str(label_counts)
if len(label_count_str) > 100:
label_count_str = label_count_str[:100] + '...'
logger.info("Finished reading dataset; label counts: %s",
label_count_str)
Why do you do above process?
Correct me if i am wrong: (FAKE EXAMPLE)
>>label_counts
[(0,300),(1,300),(2,300)]
>>label_count_str
'[(0,300),(1,300),(2,300)]'
300 is the number of times a specific type of label appears
I don't understand why all these steps mean for? How can this calculate labels counts?
In sts_instance.py line 108:
Instead of writing:
fields = list(csv.reader([line]))[0]
Why not writing:
fields = [x for x in csv.reader([line])]
I run a trial:
>>> field = list(csv.reader(['asdf','fddf','ddd']))
>>> field
[['asdf'], ['fddf'], ['ddd']]
>>> field[0]
['asdf']
It seems list(csv.reader(['asdf','fddf','ddd']))[0]
can't be used to count the number of fields? Correct me if i am wrong.
4.Why create character-level dictionary/instance?
When you tokenize the string 'Today has a good weather' to character level, the result would be ['T','o','d','a','y','h','a'............]
The character-level seems can't be used to encode contextual information?
Could you tell me some cases where character-level tokenization plays a difference?
Thank you!