percyliang / brown-cluster Goto Github PK

Hello, I would like to use your algorithm to categorize job titles. Do you still make updates and maintain the library ?

Bets Regards,
Evangelia

Hi,

Can you please guide me how can I pass multiple text files to generate output files on them?

The code fails (with core dump: segmentation fault message) when I run it on a huge txt file (about 20M types and 14GB file size). I already used wcluster for different files with much less types and it worked pretty well.

Is there any limit for the vocabulary size (#types)?

Hello! I was browsing the code and I saw the
opt_define_bool(paths2map, "paths2map", false, "Take the paths file and generate a map file.");
Is it possible to be used? What is the output? Something like the tree presented in brown algorithm paper?
Thank you very much

In case anyone is clustering large datasets:

in my experiments (40M corpus and NofClusters=1000), turning on compiler optimization with "-O3" yields speed-ups of around 3.

I changed the following lines in my Makefile:

wcluster: $(files)
    g++ -Wall -g -O3 -o wcluster $(files)

%.o: %.cc
    g++ -Wall -g -O3 -o $@ -c $<

I am using Cygwin on windows and trying to run this code. On the first step when running "make" command, getting following error.

basic/prob-utils.cc: In function ‘double rand_gaussian(double, double)’:
basic/prob-utils.cc:8:37: error: ‘M_PI’ was not declared in this scope
double z = sqrt(-2log(x1))cos(2M_PIx2);
^~~~
make: *** [Makefile:13: basic/prob-utils.o] Error 1

Can you guide in this regard?

http://www.cs.berkeley.edu/~pliang/papers/meng-thesis.pdf is borken.

Does the package return (or write in the log file) the perplexity or any other goodness of fit measure? If yes, would it be a good idea to run a BayesOpt optimizer to find the best clustering this way? Or is it ill-posed?

Thanks

I was wondering if it's possible to make a library out of this code in order to be able to include it into other projects?

I'm looking for some way to run the clustering algorithm while using previously learned collocs, map, and paths. I tried pointing to the paths file with the --paths flag, but this just overwrote it with a newly learned one.

I don't have time to relearn everything from scratch: it takes days!

I'm trying to compile on Windows 7 using g++ 4.7.2 and GNU Make 3.8.1. When I do I get the following errors:

C:\Users\ameasure\brown-cluster-master>make
g++ -Wall -g -o wcluster.o -c wcluster.cc
wcluster.cc: In function 'void repcheck()':
wcluster.cc:431:3: error: '__STRING' was not declared in this scope
wcluster.cc:432:3: error: '__STRING' was not declared in this scope
wcluster.cc: In function 'int main(int, char*)':
wcluster.cc:1072:3: error: '__STRING' was not declared in this scope
make: ** [wcluster.o] Error 1

Any idea what's going on?

the length of text is defined int in src, so what happened if length of text is bigger than INT_MAX ?

I'm not sure whether this can be called an issue or the matter of understanding, I ran the clustering on Persian text and after couple of hours I got these results in map output:
بینبریج 00111111-L 5.54361 00111111-R 2.82232 00111111-freq 1
گروهان 00111111-L 5.20714 00111111-R 2.7586 00111111-freq 1
می‌دهده 00111111-L 4.15732 00111111-R 6.05444 00111111-freq 1
...
and I'm not sure what each column means and which one exactly is the cluster group?!