Can't plot a comparison word cloud on grouped DFM with TF-IDF weighting about quanteda HOT 4 CLOSED

Here's a workaround, which removes the weighted flag. Note that you have many zeroed features if you are applying this to the grouped dfm, since the inverse document frequency weight will zero the features that occur in both groups. (Did you want to weight before applying dfm_group()?)

library("quanteda")
#> Package version: 3.3.1
#> Unicode version: 14.0
#> ICU version: 71.1
#> Parallel computing: 12 of 12 threads used.
#> See https://quanteda.io for tutorials and examples.
library("quanteda.textplots")

my_dfm <- data_corpus_inaugural[54:59] %>%
    tokens() %>%
    dfm() %>%
    dfm_group(groups = Party) %>%
    dfm_tfidf()

# drops the "weighted" flag
my_dfm <- as.dfm(as.matrix(my_dfm))
my_dfm
#> Document-feature matrix of: 2 documents, 2,320 features (65.56% sparse) and 0 docvars.
#>             features
#> docs         president clinton , distinguished guests and my fellow citizens
#>   Democratic         0 0       0             0      0   0  0      0        0
#>   Republican         0 1.20412 0             0      0   0  0      0        0
#>             features
#> docs         the
#>   Democratic   0
#>   Republican   0
#> [ reached max_nfeat ... 2,310 more features ]

# necessary to remove zero-weighted featurs
my_dfm <- dfm_trim(my_dfm)

textplot_wordcloud(my_dfm, comparison = TRUE, min_count = 0)

^{Created on 2023-08-23 with reprex v2.0.2}

from quanteda.

Wow, thank you so much for the lightning fast response @kbenoit! I'm very happy with this workaround, and the results look sensible to me.

On your question about the order of operations - I think this way round makes sense to me - I'm basically treating each group as one big document and then running TF-IDF weighting on that (I agreed with the comments you were making in the other issue about doing the operations the other way around not making a whole lot of sense, or at least not having a clear interpretation).

I did notice a lot of zero weighted terms coming through - I don't fully understand why the IDF should wipe them out completely, I would have thought it would have just down-weighted features that occur in multiple groups. If you're able to explain why that happens that would be great, but you've already been a huge help - thanks again!

from quanteda.

It's because you are multiplying by the log of inverse document frequency, which for words that occur in both groups, would be log(2/2) = 0. tf-idf will always zero out features that occur across all documents, regardless of how imbalanced are those features.

See p485 of https://kenbenoit.net/pdfs/CURINI_FRANZESE_Ch26.pdf where I wrote more about this "feature" of tf-idf.

from quanteda.

Gotcha! One of those things that's obvious once it's explained...

Since, all of this is by way of exploratory analysis, and the final product (a topic model) will proceed from the individual documents anyway, I think I'll just bowl on and treat this as a feature rather than a bug of TF-IDF, since I'm going to be keeping an eye on the unadjusted counts as well.

Thanks again for all of your help, I would have been a bit lost without it!

from quanteda.