Add features from macroetym. Check stanza version for implementation last time (but could probably use a rewrite)

• Auto bump version ID, added in #110
• Auto accept dependabot, added in #110 (or was added beforehand, but at least here)
• Add concurrency to pytests, added in #110
• CI for tutorials
• #98
• #99
  • #108

Use spacy's word embeddings to calculate the similarity of the previous n words following

Alberto, P., Mary, L. J., Arndis, S., Vibeke, B., Yuan, Z., Huiling, W., Lana, I., Katja, K., & Riccardo, F. (2022). Speech disturbances in schizophrenia: Assessing cross-linguistic generalizability of NLP automated measures of coherence. MedRxiv.

We should consider separating some of the components from each other, so the user doesn't get overloaded. (also, speed)

I propose:

1. textdescriptives: includes everything but `quality``
2. quality: only quality

Thoughts, @KennethEnevoldsen ?

Currently waiting on pradyunsg/furo#500 to get merged

How to use TD for quality filtering -- other cool use cases of quality filtering?

Usage for feature extraction for model training?

TextDescriptives/docs/_build/html/usingthepackage.html#available-attributes

To ensure that everything works across languages.

(following PR #19 )

For instance instead of returning:

{'n_stop_words': 435,
 'alpha_ratio': 0.79,
 'mean_word_length': 1.3,
 'doc_length': 894,
...
}

We could return the values as a tuple with the second value indicating whether it passed (given the current threshold)

{'n_stop_words': (435, True),
 'alpha_ratio': (0.79, True),
 'mean_word_length': (1.3, False),
 'doc_length': (894, True),

Alternatively, return a dataclass with a .value and a .passed attribute?

Create a wrapper function to remove spacy boilerplate. Suggested API

td.extract(texts, lang="en", metrics=["dependency_distance", "readability"])

Outputs a dataframe.

Can use the language tag to infer which spacy model to download (always lang_core_news/web_lg). Alternatively, supply a spacy pipeline

td.extract(texts, spacy_model="da_core_news_sm", metrics=["descriptive_stats"]

Make clever checks to only add spacy models if metrics actually need it (e.g. no model for descriptive stats and readability).

Could be easily extended to a CLI

textdescriptives extract -i file1.txt file2.txt -m readability quality -l en

Hello,
I am tying textdescriptives on a Python 3.6 + spaCy3.1 on a Linux system through JupyterLab.

I found this error. Not quite sure how to deal with spaCy decorator. Could you please help? Thanks!

nlp.add_pipe("textdescriptives")

ValueError: [E002] Can't find factory for 'textdescriptives' for language English (en). This usually happens when spaCy calls nlp.create_pipe with a custom component name that's not registered on the current language class. If you're using a Transformer, make sure to install 'spacy-transformers'. If you're using a custom component, make sure you've added the decorator @Language.component (for function components) or @Language.factory (for class components).

Make the syllables attributes available for Span as well

In descriptive_statistics the issue was saving spans/tokens to an attribute. As they are not serializable, we get errors.

Workarounds:

1. Don't set attributes with tokens/spans..
2. Set the attribute as (token, start_idx, end_idx). This is serializable and can be reconstructed to be the original span

Use a generative model to calculate the average perplexity of documents (or pseudo perplexity for masked language models).

• Load a model from huggingface
• Calculate token level perplexity

@KennethEnevoldsen Any other ideas for calculating surprise?

>>> nlp = spacy.blank("en")
>>> nlp("hello my name is kenneth. How are you?")._.coherence
>>> 

This should probably raise an error.

Hi,

first of all thanks for this very helpful library. I have a question regarding the way dependency distance (DD) for Doc objects is calculated.

Your function on calculating DD for a Doc, returns the DD value here: "dependency_distance_mean": np.mean(dep_dists). The mean returned is as far as I can see, the mean over mean DD of every sentence (contained in dep_dist) constituting the Doc object.

the two sources you cite on dependency distance in your documentation (Liu 2008 and Oya, 2008), however, have a different approach.

For calculating DD of a text, Liu seems to take the sum of the absolute DD found in the whole text and multiplies by (1 / (number of words - the number of sentences). Oya seems to takes a mean of means like you do, but for a sentence averages sum of absolute DD over the number of dependency links in an utterance. In your documentation nor in your code I can retrieve how you exactly calculate DD for a text.

Would you please be so kind as to explain with what approach you calculate DD for Doc objects, and provide pointers on how we may adapt the code to e.g. implement approaches by Liu and Oya? Thanks!

Which page or section is this issue related to?

https://hlasse.github.io/TextDescriptives/dependencydistance.html

All the other pages have an example

extract_df does not extract POS metrics, due to mismatch between metric name pos_stats and Extractor.unpack_extension logic, which checks for the original component name pos_proportions in spaCy.

Make the components check whether e.g. has_annotation is set instead of checking tagger component is there to allow for custom POS pipelines/support for languages without one

(following PR #19 )

Perhaps the following link to the corresponding Arxiv article can be included in the readme:

https://arxiv.org/abs/2301.02057

Remove the index (look more like dacy)

• Check if the right keys are there.
• Check if correct number of rows are there (if not already done)

Some of the metrics are bit hard to understand - mainly the repetitious text metrics:

Duplicate lines character fraction (duplicate_lines_chr_fraction): Fraction of characters in a document which are contained within duplicate lines.

Duplicate paragraphs character fraction (duplicate_paragraphs_chr_fraction): Fraction of characters in a document which are contained within duplicate paragraphs.

Duplicate n-gram character fraction (duplicate_{n}_gram_chr_fraction): Fraction of characters in a document which are contained within duplicate n-grams. For a speciifed n-gram range.

Top n-gram character fraction (top_{n}_gram_chr_fraction): Fraction of characters in a document which are contained within the top n-grams. For a speciifed n-gram range.

Calculating POS stats seems to slow things down significantly. TODO:

• Profile the package, what causes the slowdown?
• Calculate the sum of values once and then call in the dict comprehension in PosStatistics

Ideas for speedup:

• Identify which pos_tags the model can make and predefine the counter/dictionary with those keys (would also solve the issue of different numbers of keys across docs/sentences)
• Alternatives to Counter?

Other options:

• Remove posstats from default TextDescriptives and make it an optional component that takes in which specific POS tags the user is interested in and extracts those (+ 'others')

• Summary describing the purpose of the software
• Statement of need
• Package features & functionality
• Target audience
• References to other software addressing related needs
• Past or ongoing research projects using the software

Tutorial describing basic functionality -- any good examples?

E.g. development of sentence complexity over time in historical texts? (ADL data)

Currently, a series of metrics use the same underlying metrics or variables, e.g. document length. There is no reason that these should be calculated repeatedly, if we change the underlying getter function to a setter we can avoid this.

Update docs to correctly add and format docstrings, and document all functionality that has been added gradually.

Very minimal example in README -- move focus to documentation

ssort auto commit fucked me over big time..

Which page or section is this issue related to?

Allow threshold for the quality pipelien to be changed after data threshold have been set.
E.g. by adding a set_quality_thresholds method

How to reproduce the behaviour

import spacy
import textdescriptives as td
nlp = spacy.load("en_core_web_sm")
nlp.add_pipe("textdescriptives")
doc = nlp("The world is changed. I feel it in the water. I feel it in the earth. I smell it in the air. Much that once was is lost, for none now live who remember it.")
doc..readability
doc..token_length

Environment

Name: textdescriptives Version: 0.1.1
Windows 10
Python 3.6

Error message

ValueError: [E002] Can't find factory for 'textdescriptives' for language English (en). This usually happens when spaCy calls nlp.create_pipe with a custom component name that's not registered on the current language class. If you're using a Transformer, make sure to install 'spacy-transformers'. If you're using a custom component, make sure you've added the decorator @Language.component (for function components) or @Language.factory (for class components).

Available factories: attribute_ruler, tok2vec, merge_noun_chunks, merge_entities, merge_subtokens, token_splitter, parser, beam_parser, entity_linker, ner, beam_ner, entity_ruler, lemmatizer, tagger, morphologizer, senter, sentencizer, textcat, spancat, textcat_multilabel, en.lemmatizer

ValueError Traceback (most recent call last)
in
6 import textdescriptives as td
7 nlp = spacy.load('en_core_web_sm')
----> 8 nlp.add_pipe('textdescriptives')
9 doc = nlp('This is a short test text')
10 doc._.readability # access some of the values

~.conda\envs\tdstanza\lib\site-packages\spacy\language.py in add_pipe(self, factory_name, name, before, after, first, last, source, config, raw_config, validate)
780 config=config,
781 raw_config=raw_config,
--> 782 validate=validate,
783 )
784 pipe_index = self._get_pipe_index(before, after, first, last)

~.conda\envs\tdstanza\lib\site-packages\spacy\language.py in create_pipe(self, factory_name, name, config, raw_config, validate)
639 lang_code=self.lang,
640 )
--> 641 raise ValueError(err)
642 pipe_meta = self.get_factory_meta(factory_name)
643 config = config or {}

Add the functionality from pos_statistics to Spans as well

Can't spot the difference with timeseriesflattener.
Perhaps @MartinBernstorff can help when he gets back?

Loosen the tolerance for pos-proportions in the tests or find a way to always download the same spacy model for tests.
Will be fixed in next release together with #24

it actually caught a bug when I did it with dacy with files not being properly added to the manifest.in file.

I don't want to do this if you don't agree @HLasse ?