LDA and tf-idf document term matrix about text_mining HOT 3 CLOSED

Try this stackoverflow explanation with a workaround. I have never done it myself.
Apparently, LDA requires TF not TfIdf because its measuring distributions.
I wouldn't recommend using LDA this way. I suppose you could do some data wrangling to get it into a useable format for LDA but the authors of LDA clearly wants Tf.
What exactly are you trying to accomplish?

from text_mining.

from text_mining.

Was giving this some thought and I think you could perform some sort of tf-idf TDM, then apply a heuristic to identify the low quality terms.

library(tm)
data("crude")
dtm <- DocumentTermMatrix(crude,
                          control = list(weighting =
                                           function(x)
                                             weightTfIdf(x, normalize =
                                                           FALSE),
                                         stopwords = TRUE))

dtmM<-as.matrix(dtm)
tfScores<-colSums(dtmM)
tfScores<-data.frame(term=names(tfScores),tfScoring=tfScores)
tfScores<-tfScores[order(tfScores$tfScoring),]

# Then perform a subset based on deciling, or other heuristic for example
drops<- subset(tfScores$term,tfScores$tfScoring<=5) #or change to 0 etc.
drops<-as.character(drops)

drops is a vector of terms that can be concatenated to the stop words list. The example above has no Corpus Cleaning functions applied so you would have to do that before. Then you would have a tfIDF version for LDA.

from text_mining.