oudalab / biryani Goto Github PK
View Code? Open in Web Editor NEWEvent Data Processing Pipeline
Event Data Processing Pipeline
To better test and to also help other uses this code, this package should be a python package. There are many good instructions for doing this. A simple example is available on readthedocs.org. Additionally, you should also set up py.test with a simple test case that can implicitly tell a developer how this code works.
Performance of the corenlp container is decreasing after 4-5 hours of running... (it is using only one core ) one chance that causing this issue might be, the database size.
The master branch should have the code that most users will run, either all annotators
or kalman_filter_all_anno
. Ideally, both should be in the same branch with an option to turn it on or off. At a minimum, master should be a default choice of one of those two.
date field in corenlp's output article has three 0s. It requires extra processing to feed in PETRARCH otherwise PETRARCH will generate error.
Instead of using specific config files packaged with the dockerfiles such as petrach.config
and corenlp.config
. Lets define one config file name for each package. I suggest biryani.json
.
Within it we can create a json object. We should also try to use similar conventions across packages. We can a schema for each config key as we develop.
We will have a Docker container running CoreNLP natively (not the CoreNLP server) that will consume jobs from the RabbitMQ.
annotators = tokenize,ssplit,pos,lemma,parse
parse.model = edu/stanford/nlp/models/srparser/englishSR.ser.gz
CoreNLP can be downloaded here: http://nlp.stanford.edu/software/stanford-corenlp-full-2015-12-09.zip
The shift-reduce parser (faster) can be downloaded here: http://nlp.stanford.edu/software/stanford-srparser-2014-10-23-models.jar
How to use shift-reduce parser with CoreNLP: http://nlp.stanford.edu/software/srparser.shtml
And note: the job queue consumer in this container will be written in Java, while in the other processing containers it will be written in Python.
Petrarch need the date in the format (YYYYMMDD). The date in the articles is in type March 20, 2016
The consumer program is not very well structured. We want to restructure the program and also use maven
suppose say our batch size is 500 and our total documents are 756.
every time we receive a message from rabbitmq queue, we are calling a function called dowork.
what dowork does is stores the message to the blockingqueue and we wrote a logic that if size of blocking queue is 500 or ideal time with out parsing is 1 min we start parsing the documents.
but the problem is, in this case as documents size is 756, after 1st iteration we parse 500 documents and remaining document size becomes 756-500 = 256.
in second iteration we we never reach the size of 500 as the documents remaining are only 256.!
Stanford Core NLP uses several jarfiles internally. Biryani downloads these jarfiles and adds them to a local path when building its image. We need to updated these jarfiles to be downloaded from the proper location.
One that needsto be moved immediatky is https://json-simple.googlecode.com/files/json-simple-1.1.1.jar
->
https://cliftonlabs.github.io/json-simple/target/json-simple-2.1.2.jar
The CoreNLP and Petrarch2 containers write their output to a SQLite DB right now, for speed and convenience. We need code that will take this output and perform a bulk update back into Mongo for permanent storage.
Implement logging to debug easily. Look at python logging!
Rabbitmq has now a new Java API. Incorporate that into our pipeline
We need to read the SQlite DB file which has petrarch output to get the required data (nouns,verbs.. etc...)
There are some network issues which needs to be taken care of. Implementing a logging system will help to debug.
Dockerfile cannot access the parent file so it is best to have files needed in the current directory or subfolders.
As we're thinking about speed comparisons and publications, we'll need eventually to run the pipeline with all the CoreNLP annotators for comparison. We could manually edit the file to do this, but it would be nicer to have an option for specifying which annotators to run, maybe in https://github.com/oudalab/biryani/blob/1.0-phani/corenlp/corenlp.json
@cegme @PhaniJella @ahalterman one thing to notice is the different containers not only have the same pattern of memory usage also there up and down time has the same pattern, which make me thinking we can make they consume different batch size like node 1 has batch size 400 , node 2 has batch size 500 and node 3 has 600 to make their schedule different and I think the cpu should be utilized better in this way
the petrarch output is in dictionary format. We need to format it to json so that when getting data from SQlite DB we parse json string and create json objects to get the required data
@phani, so phani, I did some play of logstash with log4j tonight, and the below is what I find:
so try to use log4j2 , the version 1 is outdated, and the way logstash doing is: it is using a port to listen to, so that port need to be set up in the logstash.config and also need to be set up in the log4j with an "appender" named log4j2.xml, and elastic is just listen on the port you set up in logstash.config then you can go to localhost:elastic port to view the log info.
I pushed the set up of these two on my branch, and a lot of old set up on google i fyou search may not work , some of the way they set this up logstash not supported any more and marked as obsolete. Also if your java code compile with error, check this one to delete some related jars.
it should fix that, the problem I did not figure out is it seems also need appender to setup in the logstash not only in log4net, not find a good answer on how to tackle this down, need to do more resaerch on this:
the following is the error I got, and logstash does run, but it seems elastic did not get the result that java code sends.
What is the desired output we are looking form the corenlp? if we pass whole paragraph, right now to my understanding the output generated is sentence by sentence... like each sentence in the paragraph is being parsed not the whole paragraph.
How the structure of the output should be formated and where and how to be stored. If we can get a sample format of the output it will be helpful.
ISSUE 1:
java.lang.IllegalArgumentException
at edu.stanford.nlp.semgraph.SemanticGraph.parentPairs(SemanticGraph.java:699)
at edu.stanford.nlp.semgraph.semgrex.GraphRelation$DEPENDENT$1.advance(GraphRelation.java:324)
at edu.stanford.nlp.semgraph.semgrex.GraphRelation$SearchNodeIterator.initialize(GraphRelation.java:1102)
at edu.stanford.nlp.semgraph.semgrex.GraphRelation$SearchNodeIterator.(GraphRelation.java:1083)
at edu.stanford.nlp.semgraph.semgrex.GraphRelation$DEPENDENT$1.(GraphRelation.java:309)
at edu.stanford.nlp.semgraph.semgrex.GraphRelation$DEPENDENT.searchNodeIterator(GraphRelation.java:309)
at edu.stanford.nlp.semgraph.semgrex.NodePattern$NodeMatcher.resetChildIter(NodePattern.java:320)
at edu.stanford.nlp.semgraph.semgrex.NodePattern$NodeMatcher.(NodePattern.java:315)
at edu.stanford.nlp.semgraph.semgrex.NodePattern.matcher(NodePattern.java:276)
at edu.stanford.nlp.semgraph.semgrex.CoordinationPattern$CoordinationMatcher.(CoordinationPattern.java:147)
at edu.stanford.nlp.semgraph.semgrex.CoordinationPattern.matcher(CoordinationPattern.java:121)
at edu.stanford.nlp.semgraph.semgrex.NodePattern$NodeMatcher.resetChild(NodePattern.java:339)
at edu.stanford.nlp.semgraph.semgrex.NodePattern$NodeMatcher.goToNextNodeMatch(NodePattern.java:438)
at edu.stanford.nlp.semgraph.semgrex.NodePattern$NodeMatcher.matches(NodePattern.java:555)
at edu.stanford.nlp.semgraph.semgrex.SemgrexMatcher.find(SemgrexMatcher.java:182)
at edu.stanford.nlp.dcoref.Mention.findDependentVerb(Mention.java:1068)
at edu.stanford.nlp.dcoref.Mention.setDiscourse(Mention.java:319)
at edu.stanford.nlp.dcoref.Mention.process(Mention.java:237)
at edu.stanford.nlp.dcoref.Mention.process(Mention.java:244)
at edu.stanford.nlp.dcoref.MentionExtractor.arrange(MentionExtractor.java:215)
at edu.stanford.nlp.dcoref.MentionExtractor.arrange(MentionExtractor.java:133)
at edu.stanford.nlp.dcoref.MentionExtractor.arrange(MentionExtractor.java:108)
at edu.stanford.nlp.pipeline.DeterministicCorefAnnotator.annotate(DeterministicCorefAnnotator.java:120)
at edu.stanford.nlp.pipeline.AnnotationPipeline.annotate(AnnotationPipeline.java:71)
at edu.stanford.nlp.pipeline.StanfordCoreNLP.annotate(StanfordCoreNLP.java:499)
at edu.stanford.nlp.pipeline.AnnotationPipeline$1.lambda$next$36(AnnotationPipeline.java:148)
at edu.stanford.nlp.util.logging.Redwood$Util$1$1.run(Redwood.java:1071)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
ISSUE 2:
java.lang.IllegalArgumentException
at edu.stanford.nlp.semgraph.SemanticGraph.parentPairs(SemanticGraph.java:699)
at edu.stanford.nlp.semgraph.semgrex.GraphRelation$DEPENDENT$1.advance(GraphRelation.java:324)
at edu.stanford.nlp.semgraph.semgrex.GraphRelation$SearchNodeIterator.initialize(GraphRelation.java:1102)
at edu.stanford.nlp.semgraph.semgrex.GraphRelation$SearchNodeIterator.(GraphRelation.java:1083)
at edu.stanford.nlp.semgraph.semgrex.GraphRelation$DEPENDENT$1.(GraphRelation.java:309)
at edu.stanford.nlp.semgraph.semgrex.GraphRelation$DEPENDENT.searchNodeIterator(GraphRelation.java:309)
at edu.stanford.nlp.semgraph.semgrex.NodePattern$NodeMatcher.resetChildIter(NodePattern.java:320)
at edu.stanford.nlp.semgraph.semgrex.NodePattern$NodeMatcher.(NodePattern.java:315)
at edu.stanford.nlp.semgraph.semgrex.NodePattern.matcher(NodePattern.java:276)
at edu.stanford.nlp.semgraph.semgrex.CoordinationPattern$CoordinationMatcher.(CoordinationPattern.java:147)
at edu.stanford.nlp.semgraph.semgrex.CoordinationPattern.matcher(CoordinationPattern.java:121)
at edu.stanford.nlp.semgraph.semgrex.NodePattern$NodeMatcher.resetChild(NodePattern.java:339)
at edu.stanford.nlp.semgraph.semgrex.NodePattern$NodeMatcher.goToNextNodeMatch(NodePattern.java:438)
at edu.stanford.nlp.semgraph.semgrex.NodePattern$NodeMatcher.matches(NodePattern.java:555)
at edu.stanford.nlp.semgraph.semgrex.SemgrexMatcher.find(SemgrexMatcher.java:182)
at edu.stanford.nlp.dcoref.Mention.findDependentVerb(Mention.java:1068)
at edu.stanford.nlp.dcoref.Mention.setDiscourse(Mention.java:319)
at edu.stanford.nlp.dcoref.Mention.process(Mention.java:237)
at edu.stanford.nlp.dcoref.Mention.process(Mention.java:244)
at edu.stanford.nlp.dcoref.MentionExtractor.arrange(MentionExtractor.java:215)
at edu.stanford.nlp.dcoref.MentionExtractor.arrange(MentionExtractor.java:133)
at edu.stanford.nlp.dcoref.MentionExtractor.arrange(MentionExtractor.java:108)
at edu.stanford.nlp.pipeline.DeterministicCorefAnnotator.annotate(DeterministicCorefAnnotator.java:120)
at edu.stanford.nlp.pipeline.AnnotationPipeline.annotate(AnnotationPipeline.java:71)
at edu.stanford.nlp.pipeline.StanfordCoreNLP.annotate(StanfordCoreNLP.java:499)
at edu.stanford.nlp.pipeline.AnnotationPipeline$1.lambda$next$36(AnnotationPipeline.java:148)
at edu.stanford.nlp.util.logging.Redwood$Util$1$1.run(Redwood.java:1071)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Rather than changing hard coded options in the code, read them from a config file (e.g. like this). That'll also make it easier to combine biryani
with birdcage
. (The config file should be inter-operable between the two).
When the container restarts, there is a chance that some documents might be present in the queue which are unacknowledged. We need to find a way to know whether the container is restarted or not and if restarted we need to omit the documents from parsing which are already parsed
When we do the Petrarch2 phase, we should get the phrases out then rather than on the fly later. This will reduce complexity, speed up the coding interface, display only viable sentences, and let us get useful summary statistics about all the phrases in the corpus.
The code to get the phrases is in Petrarch2, and a wrapper for the code is here: https://github.com/ahalterman/night_ridir/blob/master/resources/extract_phrases.py
We want to change the batch size based on CPU and memory usage for the container. Hopefully this will make it more efficient, and it will definitely make things more interesting.
Integrate the logstash to allow the log messages from python
I suggest that we should keep each of the folder/package names in lower case where it is possible. This will prevent us from running into any case management issues later. This is especially important when importing modules from python or java.
For example:
biryani/CoreNlp/ --> biryani/corenlp
biryani/Petrarch/ --> biryani/petrarch
Check out the PEP8 style guide for general guidelines.
Program Details
Annotators
Program Parameters
Exception
java.lang.Error: Error: pushback value was too large
at edu.stanford.nlp.process.Morpha.zzScanError(Morpha.java)
at edu.stanford.nlp.process.Morpha.yypushback(Morpha.java)
at edu.stanford.nlp.process.Morpha.next(Morpha.java)
at edu.stanford.nlp.process.Morphology.lemmatize(Morphology.java:156)
at edu.stanford.nlp.process.Morphology.lemma(Morphology.java:110)
at edu.stanford.nlp.trees.UniversalEnglishGrammaticalStructure.correctWHAttachment(UniversalEnglishGrammaticalStructure.java:689)
at edu.stanford.nlp.trees.UniversalEnglishGrammaticalStructure.postProcessDependencies(UniversalEnglishGrammaticalStructure.java:173)
at edu.stanford.nlp.trees.GrammaticalStructure.getDeps(GrammaticalStructure.java:560)
at edu.stanford.nlp.trees.GrammaticalStructure.(GrammaticalStructure.java:215)
at edu.stanford.nlp.trees.UniversalEnglishGrammaticalStructure.(UniversalEnglishGrammaticalStructure.java:92)
at edu.stanford.nlp.trees.UniversalEnglishGrammaticalStructure.(UniversalEnglishGrammaticalStructure.java:71)
at edu.stanford.nlp.trees.UniversalEnglishGrammaticalStructureFactory.newGrammaticalStructure(UniversalEnglishGrammaticalStructureFactory.java:29)
at edu.stanford.nlp.trees.UniversalEnglishGrammaticalStructureFactory.newGrammaticalStructure(UniversalEnglishGrammaticalStructureFactory.java:5)
at edu.stanford.nlp.pipeline.ParserAnnotatorUtils.fillInParseAnnotations(ParserAnnotatorUtils.java:59)
at edu.stanford.nlp.pipeline.ParserAnnotator.finishSentence(ParserAnnotator.java:290)
at edu.stanford.nlp.pipeline.ParserAnnotator.doOneSentence(ParserAnnotator.java:260)
at edu.stanford.nlp.pipeline.SentenceAnnotator.annotate(SentenceAnnotator.java:98)
at edu.stanford.nlp.pipeline.AnnotationPipeline.annotate(AnnotationPipeline.java:71)
at edu.stanford.nlp.pipeline.StanfordCoreNLP.annotate(StanfordCoreNLP.java:499)
at edu.stanford.nlp.pipeline.AnnotationPipeline$1.lambda$next$36(AnnotationPipeline.java:148)
at edu.stanford.nlp.util.logging.Redwood$Util$1$1.run(Redwood.java:1071)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Exception 2
java.lang.StringIndexOutOfBoundsException: String index out of range: -4
at java.lang.String.(String.java:196)
at edu.stanford.nlp.process.Morpha.yytext(Morpha.java)
at edu.stanford.nlp.process.Morpha.next(Morpha.java)
at edu.stanford.nlp.process.Morphology.lemmatize(Morphology.java:157)
at edu.stanford.nlp.process.Morphology.lemma(Morphology.java:110)
at edu.stanford.nlp.trees.UniversalEnglishGrammaticalStructure.correctWHAttachment(UniversalEnglishGrammaticalStructure.java:689)
at edu.stanford.nlp.trees.UniversalEnglishGrammaticalStructure.postProcessDependencies(UniversalEnglishGrammaticalStructure.java:173)
at edu.stanford.nlp.trees.GrammaticalStructure.getDeps(GrammaticalStructure.java:560)
at edu.stanford.nlp.trees.GrammaticalStructure.(GrammaticalStructure.java:215)
at edu.stanford.nlp.trees.UniversalEnglishGrammaticalStructure.(UniversalEnglishGrammaticalStructure.java:92)
at edu.stanford.nlp.trees.UniversalEnglishGrammaticalStructure.(UniversalEnglishGrammaticalStructure.java:71)
at edu.stanford.nlp.trees.UniversalEnglishGrammaticalStructureFactory.newGrammaticalStructure(UniversalEnglishGrammaticalStructureFactory.java:29)
at edu.stanford.nlp.trees.UniversalEnglishGrammaticalStructureFactory.newGrammaticalStructure(UniversalEnglishGrammaticalStructureFactory.java:5)
at edu.stanford.nlp.pipeline.ParserAnnotatorUtils.fillInParseAnnotations(ParserAnnotatorUtils.java:61)
at edu.stanford.nlp.pipeline.ParserAnnotator.finishSentence(ParserAnnotator.java:290)
at edu.stanford.nlp.pipeline.ParserAnnotator.doOneSentence(ParserAnnotator.java:260)
at edu.stanford.nlp.pipeline.SentenceAnnotator.annotate(SentenceAnnotator.java:98)
at edu.stanford.nlp.pipeline.AnnotationPipeline.annotate(AnnotationPipeline.java:71)
at edu.stanford.nlp.pipeline.StanfordCoreNLP.annotate(StanfordCoreNLP.java:499)
at edu.stanford.nlp.pipeline.AnnotationPipeline$1.lambda$next$36(AnnotationPipeline.java:148)
at edu.stanford.nlp.util.logging.Redwood$Util$1$1.run(Redwood.java:1071)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
java.lang.ArrayIndexOutOfBoundsException
at edu.stanford.nlp.process.Morpha.zzRefill(Morpha.java)
at edu.stanford.nlp.process.Morpha.next(Morpha.java)
at edu.stanford.nlp.process.Morphology.lemmatize(Morphology.java:156)
at edu.stanford.nlp.process.Morphology.lemma(Morphology.java:110)
at edu.stanford.nlp.trees.UniversalEnglishGrammaticalStructure.correctWHAttachment(UniversalEnglishGrammaticalStructure.java:689)
at edu.stanford.nlp.trees.UniversalEnglishGrammaticalStructure.postProcessDependencies(UniversalEnglishGrammaticalStructure.java:173)
at edu.stanford.nlp.trees.GrammaticalStructure.getDeps(GrammaticalStructure.java:560)
at edu.stanford.nlp.trees.GrammaticalStructure.(GrammaticalStructure.java:215)
at edu.stanford.nlp.trees.UniversalEnglishGrammaticalStructure.(UniversalEnglishGrammaticalStructure.java:92)
at edu.stanford.nlp.trees.UniversalEnglishGrammaticalStructure.(UniversalEnglishGrammaticalStructure.java:71)
at edu.stanford.nlp.trees.UniversalEnglishGrammaticalStructureFactory.newGrammaticalStructure(UniversalEnglishGrammaticalStructureFactory.java:29)
at edu.stanford.nlp.trees.UniversalEnglishGrammaticalStructureFactory.newGrammaticalStructure(UniversalEnglishGrammaticalStructureFactory.java:5)
at edu.stanford.nlp.pipeline.ParserAnnotatorUtils.fillInParseAnnotations(ParserAnnotatorUtils.java:59)
at edu.stanford.nlp.pipeline.ParserAnnotator.finishSentence(ParserAnnotator.java:290)
at edu.stanford.nlp.pipeline.ParserAnnotator.doOneSentence(ParserAnnotator.java:260)
at edu.stanford.nlp.pipeline.SentenceAnnotator.annotate(SentenceAnnotator.java:98)
at edu.stanford.nlp.pipeline.AnnotationPipeline.annotate(AnnotationPipeline.java:71)
at edu.stanford.nlp.pipeline.StanfordCoreNLP.annotate(StanfordCoreNLP.java:499)
at edu.stanford.nlp.pipeline.AnnotationPipeline$1.lambda$next$36(AnnotationPipeline.java:148)
at edu.stanford.nlp.util.logging.Redwood$Util$1$1.run(Redwood.java:1071)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.