arnim / hdp Goto Github PK

In mallet branch, HDPGibbsSampler line 71-73
simply adding the length of data to totalNumberOfWords should be faster than traversing an index and adding one each time, especially for very large documents
might look like this:

totalNumberOfWords += ((FeatureSequence) corpus.get(d).getData()).getLength();

java.lang.ArrayIndexOutOfBoundsException: XXX
at de.uni_leipzig.informatik.asv.hdp.HDPGibbsSampler.addInstances(HDPGibbsSampler.java:85)
at de.uni_leipzig.informatik.asv.hdp.HDPGibbsSampler.main(HDPGibbsSampler.java:455)

TODO update the addInstances

may I ask what is the format of input file?
I only know one line is a document

thanks for your patient.

I have an input text file where to load it....

Hi,
I have read the code. The original paper build a gibbs chain for two level HDP with two sampling stage, sampling k and sampling t. whereas in this implementation only t variables are sampled and sampling table-topic assignment subroutine is called only when new tables are created. The original paper sampled all table-topic assignment during every gibbs swap.
I'm confused. Is this an implementation bug or it is just another algorithm (maybe simplified) ?