How is `doc_length` different from other length measures such as `n_characters`? about textdescriptives HOT 3 CLOSED

doc_length is len(doc), I.e. the number of tokens. However, it is not the same as n_tokens (from descriptive stats) as
it filters punctuations and similar non-word tokens. Hopefully this answer your question.

@HLasse we should probably make n_tokens --> n_words and make a n_tokens in descriptive stats as well which is equal to len(doc). In the quality component, we should rename the doc_length to n_tokens. Do you agree with this?

from textdescriptives.

Thank you for your reply.

doc_length is len(doc), I.e. the number of tokens.

Is this the lengh of SpaCy tokens?

However, it is not the same as n_tokens (from descriptive stats) as it filters punctuations and similar non-word tokens.

Could you please tell which one does what? Does n_tokens filer puncts?

from textdescriptives.

Yes n_tokens filters puncts, while doc_length doesn't filter anything.

And yes len(doc)==doc_length is the number of spacy tokens. That means that it is dependent on the tokenizer that you use.

from textdescriptives.