The ekphrasis from cbaziotis

Ekphrasis downloads statistics in /usr/local

If the user does not install ekphrasis using a virtualenv it downloads statistics in /usr/local and requires sudo permissions to run.

Statistics should be downloaded in /home//.ekphrasis to resolve this.

Please add a LICENSE to this repo

I really like this tool. Can you add a license to this repo so that it is easy to use it in other projects as a dependency?

This is a good resource around this: https://help.github.com/en/github/creating-cloning-and-archiving-repositories/licensing-a-repository

Failed during generate_stats.py

computing statistics for file: text8.txt
100%|███████████████████████████████████████████████████████████████████| 1/1 [12:41<00:00, 761.71s/it]

Writing 1-grams...
entries:250,982 - tokens:16,996,178
writing stats to file /home/jj301440/ekphrasis-master/ekphrasis/tools/../stats/text8/counts_1grams.txt
Writing 2-grams...
entries:4,136,483 - tokens:16,996,178
writing stats to file /home/jj301440/ekphrasis-master/ekphrasis/tools/../stats/text8/counts_2grams.txt
Writing 3-grams...
entries:10,327,876 - tokens:16,996,178
writing stats to file /home/jj301440/ekphrasis-master/ekphrasis/tools/../stats/text8/counts_3grams.txt
Traceback (most recent call last):
File "generate_stats.py", line 191, in
write_stats(stats)
File "generate_stats.py", line 147, in write_stats
write_stats_to_file(filename, counter, args.mincount[int(k) - 1])
IndexError: list index out of range

Can you please tell me where the problem is?

what's wrong with "Word statistics files not found!"

teach me， thx

how to get the word statistics?

Excuse me, where can I get the word statistics?😢
In other Issue, they said

but I can't find the word statistics in such url...
Thanks in advance.!

Segmentation: Preserve case?

The Segmentation tool you provide is excellent. One feature request:

Unless I am mistaken, the tool always provided the split words in 1. lower case, and 2. does not provide information for where spaces were inserted. Instead, a preserve_case or capitalize parameter would be helpful (for 1). The following code capitalizes the split string according to the capitalization used in the hashtag.


from ekphrasis.classes.segmenter import Segmenter
segmenter = Segmenter(corpus="twitter")


def word_segmentation(text, fix_case=True):
    words_string = segmenter.segment(text)
    if not fix_case:
        return words_string

    fixed = ""

    n_add = 0
    for i in range(len(words_string)):
        if words_string[i] == " " and text[i+n_add] != " ":
            n_add += 1
            fixed += " "
            continue

        is_capital = text[i-n_add].isupper()
        if is_capital:
            fixed += words_string[i].upper()
        else:
            fixed += words_string[i]
    return fixed

Of course, if the user is using camelCase or PascalCase, the capitalization may not be meaningful, but in other cases, this can be. For instance:

I #eatsomuch food --> I eat so much food.
I care so much. #IranProtests --> I care so much. Iran Protests

Arguably, the use of a stand-alone hashtag approximately refers to a proper noun, in which case the adopted capitalization is meaningful.

Word Statistics File not Found. | Receiving 404 error while dowloading the file.

While using the library, the word statistics file is again missing from its original source:

Please fix this as soon as possible and please make some more robust solutions for hosting this file. It looks like people faced this problem in the past as well.

Due to this issue, the production service is stopped. Please fix this asap!

Thanks in advance.!

"maximum recursion depth exceeded" Error

Hi,
I am running a segmenter on a big file and I am facing "maximum recursion depth exceeded" error. I am running on a big file. Is there any way to resolve this?

RecursionError Traceback (most recent call last)
in ()
1 message_p = []
2 for i in range(0,len(message), 1000):
----> 3 message_p.append(tweet2lang(message[i:i+1000]))

482 frames
in tweet2lang(message)
18 for w in msg_tokens:
19 if len(w) >3:
---> 20 temp_seg = seg_tw.segment(w)
21 # print(w,temp_seg)
22 message_seg.append(temp_seg)

/usr/local/lib/python3.6/dist-packages/ekphrasis/classes/segmenter.py in segment(self, word)
133 def segment(self, word):
134 if word.islower():
--> 135 return " ".join(self.find_segment(word)[1])
136 else:
137 return self.case_split.sub(r' \1', word).lower()

/usr/local/lib/python3.6/dist-packages/ekphrasis/classes/segmenter.py in find_segment(self, text, prev)
126 return 0.0, []
127 candidates = [self.combine((log10(self.condProbWord(first, prev)), first), self.find_segment(rem, first))
--> 128 for first, rem in self.splits(text)]
129 return max(candidates)
130