Fails for empty strings about textdescriptives HOT 4 CLOSED

We should handle empty strings as it seems like something that could easily happen in an automated system. Either by a) removing them, or b) first excluding them and then adding them back to the result as np.nan for all metrics, or c) raising an error.

a) is the easiest but if the end user expect the same number of rows as they inputted, it's problematic.

b) is probably the best as you would maintain the number of rows in the output with the order matching the input. But this increases complexity of the code as you first have to exclude the empty strings, remember their index and then insert them in the output. Technically you just have to store the indices I guess, although insertion of multiple empty rows might be tricky?

c) would require the user to decide between a) and b) (and then implement it). Seems annoying.

I would probably go with b

Further, perhaps we should have a look at whether to add a check for the length of span in the calculation or just allow it to fail with zero-division?

from textdescriptives.

Completely agree that b is the way to go. We actually already have handling for this in all components except quality and information_theory. If a metric can't be calculated (e.g. because of empty string) we return np.nan.

@KennethEnevoldsen you know the code for those components the best - is this something you'd be able to take a look at?

from textdescriptives.

And add a unit test when you're at it @KennethEnevoldsen ;)

from textdescriptives.

🫡

from textdescriptives.