derwenai / pytextrank Goto Github PK
View Code? Open in Web Editor NEWPython implementation of TextRank algorithms ("textgraphs") for phrase extraction
Home Page: https://derwen.ai/docs/ptr/
License: MIT License
Python implementation of TextRank algorithms ("textgraphs") for phrase extraction
Home Page: https://derwen.ai/docs/ptr/
License: MIT License
I believe there is a serious bug in the line 284 in pytextrank.py.
Indeed once the unneeded words are filtered out (keeps = list(filter(lambda w: w.word_id > 0, graf))
) the word indices have holes.
The corrected line should be
if (j - i) < size:
yield (w0.root, w1.root)
Question: I am new at this and working from the example script to extract key phrases from about 800 separate documents. The tool is great! I notice in the results that some documents will produce a singular word in a key phrase (like "donor") while other documents in the set produce the plural of the same word ("donors"). In the ideal case I would like to collapse those into just one version, either singular or plural. Can you suggest an approach to do that? Is there any feature that would convert the found keywords to their singular, for example?
hi,
It seems the function load_stopwords
in pytextrank.py
is never called.
Would you consider passing an argument when initializing to enable and disable removing the stopwords? And another argument to choose among pos and tag?
Besides, is it possible to make the pharse
similar to a span
, to allow access token.pos_
for each token in a pharse
?
When trying out the examples, the line graph, ranks = pytextrank.text_rank(path_stage1)
throws the following error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/user/.local/lib/python3.6/site-packages/pytextrank/pytextrank.py", line 349, in text_rank
graph = build_graph(json_iter(path))
File "/home/user/.local/lib/python3.6/site-packages/pytextrank/pytextrank.py", line 308, in build_graph graph.edge[pair[0]][pair[1]]["weight"] += 1.0
AttributeError: 'DiGraph' object has no attribute 'edge'
Red Hat Enterprise Linux Server release 7.3 (Maipo)
core-4.1-amd64:core-4.1-noarch
python 3.6.4
pytextrank==1.1.0
spacy==2.0.11
graphviz==0.8.3
networkx==2.1
decorator==4.3.0
path_stage1
file contents:
["777", "7b982e54fa330a6854a0ed5397d49223fdc70645", [[1, "Compatibility", "compatibility", "NN", 1, 0], [0, "of", "of", "IN", 0, 1], [2, "systems", "system", "NNS", 1, 2], [0, "of", "of", "IN", 0, 3], [3, "linear", "linear", "JJ", 1, 4], [4, "constraints", "constraint", "NNS", 1, 5], [0, "over", "over", "IN", 0, 6], [0, "the", "the", "DT", 0, 7], [5, "set", "set", "NN", 1, 8], [0, "of", "of", "IN", 0, 9], [6, "natural", "natural", "JJ", 1, 10], [7, "numbers", "number", "NNS", 1, 11], [0, ".", ".", ".", 0, 12]]]
834
["777", "c1cd88e3fec11b0772c6be12b7504f332c0b3d8f", [[8, "Criteria", "criterion", "NNS", 1, 13], [0, "of", "of",
"IN", 0, 14], [1, "compatibility", "compatibility", "NN", 1, 15], [0, "of", "of", "IN", 0, 16], [0, "a", "a", "DT", 0, 17], [2, "system", "system", "NN", 1, 18], [0, "of", "of", "IN", 0, 19], [3, "linear", "linear", "JJ", 1, 20], [9, "Diophantine", "diophantine", "NNP", 1, 21], [10, "equations", "equation", "NNS", 1, 22], [0, ",", ",", ".", 0, 23], [11, "strict", "strict", "JJ", 1, 24], [12, "inequations", "inequation", "NNS", 1, 25], [0, ",", ",", ".", 0, 26], [0, "and", "and", "CC", 0, 27], [13, "nonstrict", "nonstrict", "NN", 1, 28], [12, "inequations", "inequation", "NNS", 1, 29], [14, "are", "be", "VBP", 1, 30], [15, "considered", "consider", "VBN", 1, 31], [0, ".", ".", ".", 0, 32]]]
1093
["777", "cb9235d7c8b21321b88462fca3a0480e29aa8ec7", [[16, "Upper", "upper", "NNP", 1, 33], [17, "bounds", "bound", "VBZ", 1, 34], [0, "for", "for", "IN", 0, 35], [18, "components", "component", "NNS", 1, 36], [0, "of", "of", "IN", 0, 37], [0, "a", "a", "DT", 0, 38], [19, "minimal", "minimal", "JJ", 1, 39], [5, "set", "set", "NN", 1, 40], [0, "of", "of", "IN", 0, 41], [20, "solutions", "solution", "NNS", 1, 42], [0, "and", "and", "CC", 0, 43], [21,
"algorithms", "algorithm", "NNS", 1, 44], [0, "of", "of", "IN", 0, 45], [22, "construction", "construction", "NN", 1, 46], [0, "of", "of", "IN", 0, 47], [19, "minimal", "minimal", "JJ", 1, 48], [23, "generating", "generating", "NN", 1, 49], [5, "sets", "set", "NNS", 1, 50], [0, "of", "of", "IN", 0, 51], [20, "solutions", "solution", "NNS", 1, 52], [0, "for", "for", "IN", 0, 53], [0, "all", "all", "DT", 0, 54], [24, "types", "type", "NNS", 1, 55], [0, "of", "of", "IN", 0, 56], [2, "systems", "system", "NNS", 1, 57], [14, "are", "be", "VBP", 1, 58], [25, "given", "give", "VBN", 1, 59], [0, ".", ".", ".", 0, 60]]]
1179
["777", "ae690a522dbc83b7c8447b5bf863ffa95d681ce0", [[0, "These", "these", "DT", 0, 61], [8, "criteria", "criterion", "NNS", 1, 62], [0, "and", "and", "CC", 0, 63], [0, "the", "the", "DT", 0, 64], [26, "corresponding", "correspond", "VBG", 1, 65], [21, "algorithms", "algorithm", "NNS", 1, 66], [0, "for", "for", "IN", 0, 67], [27, "constructing", "construct", "VBG", 1, 68], [0, "a", "a", "DT", 0, 69], [19, "minimal", "minimal", "JJ", 1, 70], [28, "supporting", "support", "VBG", 1, 71], [5, "set", "set", "NN", 1, 72], [0, "of", "of", "IN", 0, 73], [20, "solutions", "solution", "NNS", 1, 74], [0, "can", "can", "MD", 0, 75], [14, "be", "be", "VB", 1, 76], [29, "used", "use", "VBN", 1, 77], [0, "in", "in", "IN", 0, 78], [30, "solving", "solve", "VBG", 1, 79], [0, "all", "all", "PDT", 0, 80], [0, "the", "the", "DT", 0, 81], [15, "considered", "consider", "VBN", 1, 82], [24, "types", "type", "NNS", 1, 83], [2, "systems", "system", "NNS", 1, 84], [0, "and", "and", "CC", 0, 85], [2, "systems", "system", "NNS",
1, 86], [0, "of", "of", "IN", 0, 87], [31, "mixed", "mixed", "JJ", 1, 88], [24, "types", "type", "NNS", 1, 89], [0, ".", ".", ".", 0, 90]]]
Using spaCy as core NLP library for pytextrank opens up the possibility of supporting new
languages other than English.
Currently, as of spaCy 1.8.x there are four official languages supported: en, de, fr and more recently es.
I have performed an initial analysis and testing with two new languages: (1) German and (2) Spanish.
Of course, as with the English models, the user would need to run python -m spacy download de
or python -m spacy download es
.
According to my local tests executing the example notebook for German and Spanish, the following would be needed in pytextrank to support a new language:
Make lang configurable and have https://github.com/ceteri/pytextrank/blob/master/pytextrank/pytextrank.py#L187 loading the language identified by its ISO code.
[CAVEAT] If the language is available in spaCy but does not include any of the required features: (1) POS, (2) NER, and (3) noun chunking method, anything else?. pytextrank should show a warning/error message. E.g., here: https://github.com/ceteri/pytextrank/blob/master/pytextrank/pytextrank.py#L423 for noun_chunking or here https://github.com/ceteri/pytextrank/blob/master/pytextrank/pytextrank.py#L480 for NER
Out the other 3 official languages, one of them would be supported out of the box (German) and other would be supported in the next release (Spanish only lacks noun_chunking in 1.8.2, which is currently implemented on the master branch and will be in principle be included in the next release, see https://github.com/explosion/spaCy/pull/1096/commits/5b385e7d78fd955d97b59024645d2592bdbc0949)
French would need to implement NER and noun_chunking.
I would be happy to contribute code and examples if needed :-)
Dani
Currently while decoding a file, 'ascii' codec can't decode byte
error shows up. It would be good to have unicode character support. Thanks :)
RuntimeError: module compiled against API version 0xa but this version of numpy is 0x9
Kindly help me out with this issue.
Hi,
I want to produce summaries of a list of spacy doc objects.
for doc in docs:
summary = [str(sent) for sent in doc._.textrank.summary(limit_phrases=15, limit_sentences=1)]
summary = " ".join(summary).replace("\n", " ")
print(summary)
I would expect the code above to print a summary of each separate doc object. Instead, it only prints the same summary several times (the summary from the last doc in the iteration.
I can solve this bug with the following change:
for doc in docs:
intermediate_doc = nlp(doc.text)
summary = [str(sent) for sent in intermediate_doc._.textrank.summary(limit_phrases=15, limit_sentences=1)]
summary = " ".join(summary).replace("\n", " ")
print(summary)
When I run this changed code, it prints a separate summary for each doc object (the expected behaviour).
I think that this bug comes from how the summary method is attached to the doc objects
def PipelineComponent (self, doc): ... Doc.set_extension("textrank", force=True, default=self) ...
(see pytextrank.py https://github.com/DerwenAI/pytextrank/blob/master/pytextrank/pytextrank.py)
If I understand correctly, the entire "self" class might be written on the doc extension "textrank" at each iteration. This might mean that it gets overwritten everytime.
I might be wrong though, would be interested in what you think!
~/.local/lib/python3.6/site-packages/pytextrank/pytextrank.py in parse_graf(doc_id, graf_text, base_idx, spacy_nlp)
191 markup = []
192 new_base_idx = base_idx
--> 193 doc = spacy_nlp(graf_text, parse=True)
194
195 for span in doc.sents:
TypeError: __call__() got an unexpected keyword argument 'parse'
spacy.__version__ == 2.0.11
seems that spacy_nlp(graf)
does not have the option parse
(it now only has the option disable
).
Hi there, I'm working on a project extracting keywords from a german text. Is there a tutorial on how to extract keywords using pytextrank?
Best regards,
Hi Team,
It would be of great help if you could guide in this regard.
I want to rank phrases from many documents at once but spacy's 1 million character limit for nlp object is preventing me from to do so. Also if I set nlp.max_length to be higher then it stays stuck. What would be the efficient way to do this?
We're seeing problems with spaCy NER use in PyTextRank, an intermittent bug that appears to be infinite recursion.
Will temporarily disable NER use, by default -- leaving an option for people who want to override the settings in the source code.
This needs much more debugging, and probably an overhaul to use spaCy spans in lieu of the named tuples for tracking keyphrases.
So I add PyTR to the nlp pipeline as a component as shown in the example snippet but when I use like the following, the doc._.phrases
contains the phrases for the last document always.
texts = [list of strings]
docs = list(nlp.pipe(texts))
doc_terms = []
for doc in docs:
textrank_result = [p.text for p in doc._.phrases]
doc_terms.append(textrank_result)
In the for loop, doc_terms
is always the terms for the last doc object even though the doc objects are different. Any idea what I might be doing wrong here? Thanks.
Hi everyone!
It is mentioned in the project's description that enriching the lemma graph would improve TextRank's performance. I saw that showing examples of this was in the todo list of the project but I was wondering if it worked by simply adding entities to the doc before summarising? Or is it more complicated? I am particularly interested in adding hyponymy.
And what about ignoring tokens? Some tokens are ignored depending on their POS tag in your implementation. Is it possible to ignore tokens specific to our application by tagging them? With what?
Thanks in advance for your answers!!
And thank you for this project, it is great!
I'm currently working my way through the examples, and I found the documentation around the input formats to be quite unclear. Specifically, the schema isn't documented (in my understanding), and is only hinted at in the docs.
From looking at the example input, and reviewing the code, I've deduced that the input should consist of one JSON document per line, however I don't know how to structure my content to best suit this format. For example should I break a larger document into paragraphs, then include one per line in the JSON input? Or should the complete text to be summarised be included in a single 'text' element within the JSON.
Thanks,
J
In the 487th line of pytextrank.py file in the package, the unit vector is normalized by summing all its components and dividing each of the elements by it. For some reason, in my content a unit vector occurs as [0,0...0] i.e. a zero-vector. In such a case, a zero division error occurs. I think this is a rare situation, but should be handled in the code; i.e. the normalized value of the unit vector should be provided using a try-except catch. Correct me if I am wrong.
Hi there, I've been looking at your code and example for a long time and I still have no idea how to use this.
I have documents in string format, what JSON format should they have if I want to use the stages as in the examples?
I find there's a crucial piece of information missing in the documentation, which is how to use the functionality of this package with a simple document in string format (or list of strings, representing sentences). As I don't know beforehand what JSON format I have to convert my text to in order to use the stage pipeline.
Cheers
README states pytextrank can be used for three tasks 1. phrase extraction 2. summarization 3. entity linking
I see that examples and usage are available for 1 and 2 but not 3.
can someone share a reference on how it can be used, how its lemma graph can be enriched with domain knowledge, etc.
while running i got this error
Traceback (most recent call last): File "/newvolume/var/www/krackin_ai/krackin_ai/app/Scripts/stage2.py", line 17, in <module> render_ranks(graph, ranks) File "/usr/local/lib/python3.5/dist-packages/pytextrank/pytextrank.py", line 336, in render_ranks write_dot(graph, ranks, path=dot_file) File "/usr/local/lib/python3.5/dist-packages/pytextrank/pytextrank.py", line 327, in write_dot with open(path, 'w') as f: PermissionError: [Errno 13] Permission denied: 'graph.dot'
Note that the input files in the examples are all .json format. I was wondering that whether the .txt or other type input files can be used in the project?
Hi,
The explain_algo.ipynb mentioned:
Since noun chunks can be expressed in different ways (e.g., they may have articles or prepositions), we need to find a minimum span for each phrase based on combinations of lemmas...
But he code below does not do what it states.
import operator
min_phrases = {}
for compound_key, rank_tuples in phrases.items():
l = list(rank_tuples)
l.sort(key=operator.itemgetter(1), reverse=True)
phrase, rank = l[0]
count = counts[compound_key]
min_phrases[phrase] = (rank, count)
The min_phrases did not seem change the phrase at all, let alone find a minimum span for each phrase
.
Hi guys,
I'd like to use pytextrank for keywords extraction from single sentences (questions).
After the installation and downloading a language model I ran pytextrank in IPython with success.
Next, I decided to ran Stage 2 from your example.ipynb with normalize_key_phrases
feature.
After that I received KeyError: 'graf'
exception.
Details:
In [4]: graph, ranks = pytextrank.text_rank('mih.json')
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
<ipython-input-4-ce8762751470> in <module>()
----> 1 graph, ranks = pytextrank.text_rank('mih.json')
/Users/katharsis/workspace/keywords_extractor/src/pytextrank.pyc in text_rank(path)
349 run the TextRank algorithm
350 """
--> 351 graph = build_graph(json_iter(path))
352 ranks = nx.pagerank(graph)
353
/Users/katharsis/workspace/keywords_extractor/src/pytextrank.pyc in build_graph(json_iter)
299 print meta.keys()
300
--> 301 for pair in get_tiles(map(WordNode._make, meta["graf"])):
302 if DEBUG:
303 print(pair)
KeyError: 'graf'
I saw that mih.json
not contains "graf" key.
It includes "id" and "text" only.
Is it related somehow?
PS. I'm using Python 2.7 on Mac OSx.
Could an integration with VerbNet help augment the graph? In other words, identifying metanyms for verb relations, in a way similar to what's performed for nouns?
In the text rank extension, we have a summarization method; but that option will be more effective if you expose the sentence score as a touple with the sentences that come out as summarization output. It can be asked by an input parameter whether the user just wants a summary or also the sentence scores. Let me know if there is a way to get the sentence scores in current settings too.
If it is not possible, then I would like to try and add a PR for the same if allowed.
Thanks.
Have not been able to get either the long form (from wiki) or short form (from github readme) files to work successfully.
The file @ https://github.com/DerwenAI/pytextrank/blob/master/example.py throws a KeyError: 1255 when run. Output for this is below.
I have been able to get the example from the github page working but only for very small strings. Anything larger than a few words throws a KeyError with varying number depending on the length of the string.
Can't figure out the issue even using all input (txt files) from the example on the wiki page and changing the spacy version to various releases from 2.0.0 to present.
KeyError Traceback (most recent call last)
in ()
31 text = f.read()
32
---> 33 doc = nlp(text)
34
35 print("pipeline", nlp.pipe_names)
/home/pete/.local/lib/python3.5/site-packages/spacy/language.py in call(self, text, disable, component_cfg)
433 if not hasattr(proc, "call"):
434 raise ValueError(Errors.E003.format(component=type(proc), name=name))
--> 435 doc = proc(doc, **component_cfg.get(name, {}))
436 if doc is None:
437 raise ValueError(Errors.E005.format(name=name))
/usr/local/lib/python3.5/dist-packages/pytextrank/pytextrank.py in PipelineComponent(self, doc)
530 """
531 self.doc = doc
--> 532 Doc.set_extension("phrases", force=True, default=self.calc_textrank())
533 Doc.set_extension("textrank", force=True, default=self)
534
/usr/local/lib/python3.5/dist-packages/pytextrank/pytextrank.py in calc_textrank(self)
389
390 for chunk in self.doc.noun_chunks:
--> 391 self.collect_phrases(chunk)
392
393 for ent in self.doc.ents:
/usr/local/lib/python3.5/dist-packages/pytextrank/pytextrank.py in collect_phrases(self, chunk)
345 if key in self.seen_lemma:
346 node_id = list(self.seen_lemma.keys()).index(key)
--> 347 rank = self.ranks[node_id]
348 phrase.sq_sum_rank += rank
349 compound_key.add(key)
KeyError: 1255
Hi, thanks a lot for this awesome work ๐ .
I have the following pipeline
>>> nlp.pipeline
[('tagger', <spacy.pipeline.pipes.Tagger at 0x12f31ba10>),
('parser', <spacy.pipeline.pipes.DependencyParser at 0x12d2d8bb0>),
('ner', <spacy.pipeline.pipes.EntityRecognizer at 0x12ed9a8a0>),
('textcat', <spacy.pipeline.pipes.TextCategorizer at 0x12f1d9a90>),
('textrank',
<bound method TextRank.PipelineComponent of <pytextrank.pytextrank.TextRank object at 0x2dd192290>>)]
and when I run
text = "hello world"
annots = {"cats": {"POSITIVE": 1.0, "NEGATIVE": 0.0}}
doc = Doc(nlp.vocab, words=text.split(" "))
scorer = nlp.evaluate([(doc, annots)])
it just hangs.
When I remove the textrank from the pipeline, then it runs fine.
Any idea what I'm doing wrong ?
When extracting keywords,
pytextrank.normalize_key_phrases(path_stage1, ranks)
function gives the following error.
RecursionError: maximum recursion depth exceeded in comparison
Tried to install and run example as detailed on README, and encountered this error:
Traceback (most recent call last): File "stage1.py", line 5, in <module> import textrank File "/Users/mattkohl/Development/pytextrank/textrank.py", line 13, in <module> import textblob_aptagger as tag ImportError: No module named 'textblob_aptagger'
Fixed by updating textblob as detailed here
pip install -U git+https://github.com/sloria/textblob-aptagger.git@dev
I wanted to use pytextrank together with spacy_udpipe to get keywords from texts in other languages (see https://stackoverflow.com/questions/59824405/spacy-udpipe-with-pytextrank-to-extract-keywords-from-non-english-text) but I realized, that udpipe-spacy somehow "overrides" the original spacy's pipeline so the noun_chunks are not generated (btw: the noun_chunks are created in lang/en/syntax_iterators.py but it doesn't exist for all languages so even if it is called, it doesn't work e.g. for Slovak language)
Pytextrank keywords are taken from the spacy doc.noun_chunks, but if the noun_chunks are not generated, pytextrank doesn't work.
Sample code:
import spacy_udpipe, spacy, pytextrank
spacy_udpipe.download("en") # download English model
text = "Compatibility of systems of linear constraints over the set of natural numbers. Criteria of compatibility of a system of linear Diophantine equations, strict inequations, and nonstrict inequations are considered. Upper bounds for components of a minimal set of solutions and algorithms of construction of minimal generating sets of solutions for all types of systems are given. These criteria and the corresponding algorithms for constructing a minimal supporting set of solutions can be used in solving all the considered types systems and systems of mixed types."
# using spacy_udpipe
nlp_udpipe = spacy_udpipe.load("en")
tr = pytextrank.TextRank(logger=None)
nlp_udpipe.add_pipe(tr.PipelineComponent, name="textrank", last=True)
doc_udpipe = nlp_udpipe(text)
print("keywords from udpipe processing:")
for phrase in doc_udpipe._.phrases:
print("{:.4f} {:5d} {}".format(phrase.rank, phrase.count, phrase.text))
print(phrase.chunks)
# loading original spacy model
nlp_spacy = spacy.load("en_core_web_sm")
tr2 = pytextrank.TextRank(logger=None)
nlp_spacy.add_pipe(tr2.PipelineComponent, name="textrank", last=True)
doc_spacy = nlp_spacy(text)
print("keywords from spacy processing:")
for phrase in doc_spacy._.phrases:
print("{:.4f} {:5d} {}".format(phrase.rank, phrase.count, phrase.text))
print(phrase.chunks)
Would it be possible that pytextrank processes the "noun_chunks" (candidates for keywords) from a custom extension (function which uses a Matcher and the result is available e.g. as a doc._.custom_noun_chunks - see explosion/spaCy#3856 )?
I do not know why does the algorithm filter out the verbs?
def limit_keyphrases (json_iter, phrase_limit=10):
.....
for rl in lex:
if rl.pos[0] != "v":
if (used > phrase_limit) or (rl.rank < rank_thresh):
return
I'm using
pip install -U pytextrank
to install and tried to execute the stage1.py but then error appear on the execution process
doc = spacy_nlp(graf_text, parse=True)
TypeError: __call__() got an unexpected keyword argument 'parse'
I tried to remove arg parse=True but then another error raise.
is there any compatibly issue on pytextrank on scapy version 2.0?
thank you
Fixed by changing the code on pytextrank (307) from:
try:
graph.edge[pair[0]][pair[1]]["weight"] += 1.0
except KeyError:
graph.add_edge(pair[0], pair[1], weight=1.0)
to:
if "edge" in dir(graph):
graph.edge[pair[0]][pair[1]]["weight"] += 1.0
else:
graph.add_edge(pair[0], pair[1], weight=1.0)
I installed all the libraries, i installed using git and the git url provided, as i can't or on;t know how to install the requirements.txt. However installed them individually.
I then tried to run your example:-
import pytextrank
import time
import sys
path_stage0 = "in.json"
path_stage1 = "o1.json"
path_stage2 = "o2.json"
path_stage3 = "o3.json"
start=time.time()
with open(path_stage1, 'w') as f:
for graf in pytextrank.parse_doc(pytextrank.json_iter(path_stage0)):
f.write("%s\n" % pytextrank.pretty_print(graf._asdict()))
graph, ranks = pytextrank.text_rank(path_stage1)
pytextrank.render_ranks(graph, ranks)
with open(path_stage2, 'w') as f:
for rl in pytextrank.normalize_key_phrases(path_stage1, ranks):
f.write("%s\n" % pytextrank.pretty_print(rl._asdict()))
kernel = pytextrank.rank_kernel(path_stage2)
with open(path_stage3, 'w') as f:
for s in pytextrank.top_sentences(kernel, path_stage1):
f.write(pytextrank.pretty_print(s._asdict()))
f.write("\n")
phrases = ", ".join(set([p for p in pytextrank.limit_keyphrases(path_stage2, phrase_limit=3)]))
sent_iter = sorted(pytextrank.limit_sentences(path_stage3, word_limit=250), key=lambda x: x[1])
s = []
for sent_text, idx in sent_iter:
s.append(pytextrank.make_sentence(sent_text))
graf_text = " ".join(s)
#print("excerpts: %s\n\nkeywords: %s" % (graf_text, phrases,))
print('Time Taken: ',time.time()-start)
And i keep getting this error:-
AttributeError: module 'pytextrank' has no attribute 'parse_doc'
Hi, so thanks for writing this awesome package. But is there any direct way to use summarization as there are pipelines for other tasks? Only thing I have found about summarization is the explain_summ.ipynb. please mention it if there is one.
pytextrank/pytextrank/pytextrank.py
Line 300 in ef00941
As list(dict.keys())
is going to return an arbitrary list of dict keys, how could this line work?
Hello,
I'm using pytextrank with texts in Portuguese. Thanks to issue #54 I'm able to use POS information to produce some basic noun chunking, instead of syntactic information from the parser.
My question is, in this case where I'm producing chunks from POS, am I loosing something if I disable the parser and create a new pipeline component just for chunking? Are there other relevant information given by the parser used?
Hi! I'm working on a project connected with NLP and was happy to find out that there is such a tool as PyTextRank. However, I've encountered an issue at the very beginning trying to just import package to run the example code given here.
The error that I get is the following:
----> from pytextrank import json_iter, parse_doc, pretty_print
ImportError: cannot import name 'json_iter'
----> from pytextrank import parse_doc
ImportError: cannot import name 'parse_doc'
I've tried running it in iPython console and a Jupyter Notebook, both the same result. I've installed PyTextRank with pip, the python version that I have is 3.5.4., spacy 2.1.8., networkx 2.4, graphvis 0.13.2
I tried running the example associated with PYTEXT rank:-
https://github.com/DerwenAI/pytextrank/blob/master/example.ipynb
And I got the following error:-
File "C:\Users\lee.williams\AppData\Local\Continuum\anaconda3\lib\site-packages\pytextrank\pytextrank.py", line 315, in build_graph
for pair in get_tiles(map(WordNode._make, meta["graf"])):
TypeError: string indices must be integers
Any help would be appreciated. As the example output looks fine on the URL above
Regards
Lee
Hi
I am getting the following error while running pytextrank on around 300+ documents:
Traceback (most recent call last): File "........../TextRank/driver_pytxtrank.py", line 10, in <module> for graf in pytextrank.parse_doc(pytextrank.json_iter(path_stage0)): File ".........../anaconda3/lib/python3.6/site-packages/pytextrank/pytextrank.py", line 263, in parse_doc grafs, new_base_idx = parse_graf(meta["id"], graf_text, base_idx) File ".................../anaconda3/lib/python3.6/site-packages/pytextrank/pytextrank.py", line 225, in parse_graf pos_family = tok_tag.lower()[0] IndexError: string index out of range
Also I encountered a error which said that it can't handle text having more than 1000000 characters.
This was the error:
Traceback (most recent call last): File "........../TextRank/driver_pytxtrank.py", line 10, in <module> for graf in pytextrank.parse_doc(pytextrank.json_iter(path_stage0)): File "........../anaconda3/lib/python3.6/site-packages/pytextrank/pytextrank.py", line 261, in parse_doc grafs, new_base_idx = parse_graf(meta["id"], graf_text, base_idx) File ".........../anaconda3/lib/python3.6/site-packages/pytextrank/pytextrank.py", line 193, in parse_graf doc = spacy_nlp(graf_text) File ".........../anaconda3/lib/python3.6/site-packages/spacy/language.py", line 345, in __call__ max_length=self.max_length)) ValueError: [E088] Text of length 1926491 exceeds maximum of 1000000. The v2.x parser and NER models require roughly 1GB of temporary memory per 100,000 characters in the input. This means long texts may cause memory allocation errors. If you're not using the parser or NER, it's probably safe to increase the nlp.max_length limit. The limit is in number of characters, so you can check whether your inputs are too long by checking len(text).
To fix that I made the following change to pytextrank.py:
if len(graf_text) >= 999999:
graf_text = graf_text[:999999]
grafs, new_base_idx = parse_graf(meta["id"], graf_text, base_idx)
which was followed by the before mentioned errors
Windows 10
Python 3.6
running Jupyter notebook in virtualenv
installed pytextrank with pip successfully
ran -m spacy download en
...
Linking successful
c:\ml\env\lib\site-packages\en_core_web_sm -->
c:\ml\env\lib\site-packages\spacy\data\en
You can now load the model via spacy.load('en')
Trying to follow example:
with open(path_stage1, 'w') as f:
for graf in pytextrank.parse_doc(pytextrank.json_iter(path_stage0)):
...
I get:
OSError Traceback (most recent call last)
in ()
3
4 with open(path_stage1, 'w') as f:
----> 5 for graf in pytextrank.parse_doc(pytextrank.json_iter(path_stage0)):
6 f.write("%s\n" % pytextrank.pretty_print(graf._asdict()))
7 # to view output in this notebook
c:\ml\env\lib\site-packages\pytextrank\pytextrank.py in parse_doc(json_iter)
259 print("graf_text:", graf_text)
260
--> 261 grafs, new_base_idx = parse_graf(meta["id"], graf_text, base_idx)
262 base_idx = new_base_idx
263
c:\ml\env\lib\site-packages\pytextrank\pytextrank.py in parse_graf(doc_id, graf_text, base_idx, spacy_nlp)
185 if not spacy_nlp:
186 if not SPACY_NLP:
--> 187 SPACY_NLP = spacy.load("en")
188
189 spacy_nlp = SPACY_NLP
c:\ml\env\lib\site-packages\spacy_init_.py in load(name, **overrides)
19 if depr_path not in (True, False, None):
20 deprecation_warning(Warnings.W001.format(path=depr_path))
---> 21 return util.load_model(name, **overrides)
22
23
c:\ml\env\lib\site-packages\spacy\util.py in load_model(name, **overrides)
117 elif hasattr(name, 'exists'): # Path or Path-like to model data
118 return load_model_from_path(name, **overrides)
--> 119 raise IOError(Errors.E050.format(name=name))
120
121
OSError: [E050] Can't find model 'en'. It doesn't seem to be a shortcut link, a Python package or a valid path to a data directory.
Hi,
I'm getting the following error when trying to run pytextrank with my own data. Is there a way to fix this?
app_1 | Traceback (most recent call last):
app_1 | File "index.py", line 26, in
app_1 | for rl in pytextrank.normalize_key_phrases(path_stage1, ranks):
app_1 | File "/usr/local/lib/python3.6/site-packages/pytextrank/pytextrank.py", line 581, in normalize_key_phrases
app_1 | for rl in collect_entities(sent, ranks, stopwords, spacy_nlp):
app_1 | File "/usr/local/lib/python3.6/site-packages/pytextrank/pytextrank.py", line 485, in collect_entities
app_1 | w_ranks, w_ids = find_entity(sent, ranks, ent.text.split(" "), 0)
app_1 | File "/usr/local/lib/python3.6/site-packages/pytextrank/pytextrank.py", line 454, in find_entity
app_1 | return find_entity(sent, ranks, ent, i + 1)
app_1 | File "/usr/local/lib/python3.6/site-packages/pytextrank/pytextrank.py", line 454, in find_entity
app_1 | return find_entity(sent, ranks, ent, i + 1)
app_1 | File "/usr/local/lib/python3.6/site-packages/pytextrank/pytextrank.py", line 454, in find_entity
app_1 | return find_entity(sent, ranks, ent, i + 1)
app_1 | [Previous line repeated 137 more times]
app_1 | File "/usr/local/lib/python3.6/site-packages/pytextrank/pytextrank.py", line 451, in find_entity
app_1 | w = sent[i + j]
app_1 | IndexError: list index out of range
I am trying to summarize many documents and am having trouble improving performance because I can't figure out how to use the PyTextRank extension for document summarization in parallel among multiple processes. The only method for parallel processing listed in the docs shows the use of nlp.pipe()
(Language.pipe). However, you can't call PyTextRank on a document inside nlp.pipe()
, it only allows itself to be used via Language.add_pipe()
. I've looked at the code for pytextrank.pytextrank to verify this.
How do you use a Pipeline in parallel without using nlp.pipe()
? How do you make nlp.add_pipe()
parallel?
why does the project use the length of RankedLexeme to sort the key phrases?
the length are the same with each other.
def normalize_key_phrases (json_iter, ranks, stopwords=None, spacy_nlp=None, skip_ner=True):
......
for rl in sorted(phrase_lex.values(), key=lambda rl: len(rl), reverse=True):
When running pytextrank on real-world data, I often end up with keyphrases like 'm
and 've
.
This seems to be because information is lost when enumerate_chunks() runs the output of spaCy through spaCy again.
Here's an example:
(.env) vagrant@web:/vagrant/pytextrank$ cat dat/err.json
{"id":"231", "text":"im sure its great"}
(.env) vagrant@web:/vagrant/pytextrank$ ./run.sh dat/err.json
+ '[' 1 -ne 1 ']'
+ ./stage1.py dat/err.json
real 0m5.270s
user 0m3.776s
sys 0m0.992s
+ ./stage2.py o1.json
real 0m5.460s
user 0m3.984s
sys 0m0.948s
+ ./stage3.py o1.json o2.json
real 0m2.408s
user 0m1.272s
sys 0m0.512s
+ ./stage4.py o2.json o3.json
real 0m3.402s
user 0m1.680s
sys 0m0.760s
+ cat o4.md
**excerpts:** i m sure its great
**keywords:** m
(.env) vagrant@web:/vagrant/pytextrank$
I installed all the libraries, i installed using git and the git url provided, as i can't or on;t know how to install the requirements.txt. However installed them individually.
I then tried to run your example:-
import pytextrank
import time
import sys
path_stage0 = "in.json"
path_stage1 = "o1.json"
path_stage2 = "o2.json"
path_stage3 = "o3.json"
start=time.time()
with open(path_stage1, 'w') as f:
for graf in pytextrank.parse_doc(pytextrank.json_iter(path_stage0)):
f.write("%s\n" % pytextrank.pretty_print(graf._asdict()))
graph, ranks = pytextrank.text_rank(path_stage1)
pytextrank.render_ranks(graph, ranks)
with open(path_stage2, 'w') as f:
for rl in pytextrank.normalize_key_phrases(path_stage1, ranks):
f.write("%s\n" % pytextrank.pretty_print(rl._asdict()))
kernel = pytextrank.rank_kernel(path_stage2)
with open(path_stage3, 'w') as f:
for s in pytextrank.top_sentences(kernel, path_stage1):
f.write(pytextrank.pretty_print(s._asdict()))
f.write("\n")
phrases = ", ".join(set([p for p in pytextrank.limit_keyphrases(path_stage2, phrase_limit=3)]))
sent_iter = sorted(pytextrank.limit_sentences(path_stage3, word_limit=250), key=lambda x: x[1])
s = []
for sent_text, idx in sent_iter:
s.append(pytextrank.make_sentence(sent_text))
graf_text = " ".join(s)
#print("excerpts: %s\n\nkeywords: %s" % (graf_text, phrases,))
print('Time Taken: ',time.time()-start)
And i keep getting this error:-
AttributeError: module 'pytextrank' has no attribute 'parse_doc'
In the text rank extension, we have a summarization method; but that option will be more effective if you expose the sentence score as a touple with the sentences that come out as summarization output. It can be asked by an input parameter whether the user just wants a summary or also the sentence scores. Let me know if there is a way to get the sentence scores in current settings too.
If it is not possible, then I would like to try and add a PR for the same if allowed.
Thanks.
While running example, the following error occurred, any help will be appreciated.
doc = nlp(text)
Traceback (most recent call last):
File "", line 1, in
File "/home/vikas/PycharmProjects/NLP-NMT/venv/lib/python3.5/site-packages/spacy/language.py", line 435, in call
doc = proc(doc, **component_cfg.get(name, {}))
File "/home/vikas/github_projects/pytextrank/pytextrank/pytextrank.py", line 532, in PipelineComponent
Doc.set_extension("phrases", force=True, default=self.calc_textrank())
File "/home/vikas/github_projects/pytextrank/pytextrank/pytextrank.py", line 391, in calc_textrank
self.collect_phrases(chunk)
File "/home/vikas/github_projects/pytextrank/pytextrank/pytextrank.py", line 347, in collect_phrases
rank = self.ranks[node_id]
KeyError: 29
Trying to run "stage 1" of the example (Python 3.6.4), I get error "call() got an unexpected keyword argument 'parse'".
Even the following gives me the error:
path_stage0 = 'dat/mih.json' for graf in parse_doc(json_iter(path_stage0)): print(graf)
Error details:
Traceback (most recent call last): File "<stdin>", line 1, in <module> File ".../pytextrank-master/pytextrank/pytextrank.py", line 261, in parse_doc grafs, new_base_idx = parse_graf(meta["id"], graf_text, base_idx) File ".../pytextrank-master/pytextrank/pytextrank.py", line 193, in parse_graf doc = spacy_nlp(graf_text, parse=True) TypeError: __call__() got an unexpected keyword argument 'parse'
It is pretty slow when executed for a medium sized content.
Are there any tips to improve the performance? Kindly let know.
It takes around 30 seconds for a 512 KB content in both python 2 and 3.
Using latest spacy as well.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.