blei-lab / online-hdp Goto Github PK

When I run pubmed dataset, it raises memory error.
It shows:
creating directory pubmed/corpus-pubmed-kappa-0.5-tau-1-batchsize-100
creating online hdp instance.
T = 300, K = 20, D = 8200000, W = 737869083
Traceback (most recent call last):
File "../run_online_hdp.py", line 262, in <module> run_online_hdp()
File "../run_online_hdp.py", line 138, in run_online_hdp options.adding_noise)
File "/home/nlp/topic/online-hdp-master/onlinehdp.py", line 184, in __init__ self.m_lambda = np.random.gamma(1.0, 1.0, (T, W)) * D*100/(T*W)-eta
File "mtrand.pyx", line 2233, in mtrand.RandomState.gamma (numpy/random/mtrand/mtrand.c:20292)
File "mtrand.pyx", line 234, in mtrand.cont2_array_sc (numpy/random/mtrand/mtrand.c:3092)
MemoryError
But my memory is 250G.
Anyone knows why?

For example, in the doc-count-1000.topic(the topic model file after processing 1000 documents), it shows all the probability of all the words.
I don't know why.
Anyone could help?

I searched on the Internet, the time.clock() returns time by seconds. But when I run online-hdp, the time generated by time.clock() and showed in log file can't be expressed by seconds obviously. I think it is about tenfold of second.
Anyone knows why?

In the code, when calculating the doc-level parameters, the solution is
if iter < 3:
var_phi = np.dot(phi.T, (Elogbeta_doc * doc.counts).T)
(log_var_phi, log_norm) = utils.log_normalize(var_phi)
var_phi = np.exp(log_var_phi)
else:
var_phi = np.dot(phi.T, (Elogbeta_doc * doc.counts).T) + Elogsticks_1st
(log_var_phi, log_norm) = utils.log_normalize(var_phi)
var_phi = np.exp(log_var_phi)
Why it calculate parameters in this way?(different solutions between iter<3 and iter>=3)

when I run python run_online_hdp.py ,I got such error:
File "run_online_hdp.py", line 261, in
run_online_hdp()
File "run_online_hdp.py", line 88, in run_online_hdp
train_filenames = glob(options.data_path)
File "/usr/lib/python2.7/glob.py", line 27, in glob
return list(iglob(pathname))
File "/usr/lib/python2.7/glob.py", line 38, in iglob
dirname, basename = os.path.split(pathname)
File "/usr/lib/python2.7/posixpath.py", line 85, in split
i = p.rfind('/') + 1
AttributeError: 'NoneType' object has no attribute 'rfind'

I find that it is because I didn't have the correct 'corpus_name','data_path' and ‘test_data_path’ in the following function,
parser.set_defaults(T=300, K=20, D=-1, W=-1, eta=0.01, alpha=1.0, gamma=1.0,
kappa=0.5, tau=1.0, batchsize=100, max_time=-1,
max_iter=-1, var_converge=0.0001, random_seed=999931111,
corpus_name=None, data_path=None, test_data_path=None,
test_data_path_in_folds=None, directory=None, save_lag=500, pass_ratio=0.5,
new_init=False, scale=1.0, adding_noise=False,
seq_mode=False, fixed_lag=False)
so what inputs are needed to make it run?
thanks a lot.

I run the code to train nytimes dataset(lda-c format). But it shows:
starting online variational inference.
iteration: 000000001
warning, likelihood is decreasing!
warning, likelihood is decreasing!
warning, likelihood is decreasing!
warning, likelihood is decreasing!
working on predictions.
working on fixed test data.
working on predictions.
working on fixed test data.
working on predictions.
working on fixed test data.
working on predictions.
working on fixed test data.
warning, likelihood is decreasing!
warning, likelihood is decreasing!
warning, likelihood is decreasing!
working on predictions.
working on fixed test data.
working on predictions.
working on fixed test data.
working on predictions.
working on fixed test data.
working on predictions.
working on fixed test data.
Anyone could help?

In the paper, the author said, the number of topics can be determined with cross validation or held-out likelihood.
But I run the code, and just set default T and K. The number of topics is always equal to T.
Anyone knows why?