adolenc / topotext Goto Github PK

Use bottleneck distance to compute distance between two diagrams (in all dimensions).

Also apply clustering to check if persistence diagrams make sense?

Put the data in folder ./data/<folder_name>

Names specified below:

• Sports articles - "sport"
• Movie reviews - "movie"
• Science articles abstracts - "abstracts"

Perhaps by using http://www.netlib.org/voronoi/hull.html

Meaning it takes texts, converts to feature matrix, splits matrices and generates diagrams appropriately.

I am having serious trouble figuring out how .data[0] component for 3d alpha_shapes is computed and what its maximum value could be. This then messes up our division of the interval on 10 equal parts and produces a handful of erroneously large birth/death pairs per each domain.

You can play around with this by changing line 14 in main.py to cx_method = alpha_shapes, which will then print out matrix of points (PCA'd X) on which it is building alpha_shapes, and all the ''bad'' simplices along with the square root of their data[0] and the estimated maximum R for this domain. Note that the number of such simplices is very small (so we could in the worst case just ignore them). Note also, that changing line 15 to dims = 2 now works correctly for alpha_shapes.

Any help is appreciated.

Reference: http://www.mrzv.org/software/dionysus/python/alphashapes.html#alphashapes

Our method doesn't work even on linearly separable iris classes. But that doesnt necessary mean anything is wrong, as we are comparing the structures of examples of both classes individually. We should create a custom set of points, which would have a clear difference in its structure within classes (e.g. samples from one class would lie in a circle, and samples from the other in a straight line)

Write code to draw plots of both persistence diagrams and bar diagrams.

Since implementations of alpha shapes in higher than 3 dimensions are rare/non-existent implement PCA technique to project features of texts into 2(3) dimensions and then build aplha-shapes on top of that.

Use sklearn, dyonisis...

Instead of defining R as max distance between two points in domain, define it as max distance in filtration.

Preprocessor should accept:

• a path to directory of files (which it loads and sends to pre-preprocessor);
• a pre-preprocessor, which takes as input a text and produces a list of words;
• a list of functions which generate features;
• optional parameter which makes it also include tf/idf features;

and combines all of the features into matrices X and y.

Change and upload the code for calculating the tf-idf matrix.

It should accept an array of string arrays, where one row represents one text. Obviously rows will be of different lenghts.

It will return an tf-idf matrix (mxn) where m is the number of samples and n is the number of words selected as features.