stoerr / grokconstructor Goto Github PK

Was messing with some grok tests. See here - https://pastebin.ca/3970948

There should be validation in the code to display a sensible error message.

I have just started using grok for logstash and I am trying to parse my log file using grok filter.
My logline is something like below

03-30-2017 13:26:13 [00089] TIMER XXX.TimerLog: entType [organization], queueType [output], memRecno = 446323718, audRecno = 2595542711, elapsed time = 998ms

I want to capture only initial date time stamp , entType [organization], and elapsed time = 998ms.
Howerver looks like I have to match pattern for every word and number in the line. Is there a way I can skip it ? I tried to look everywhere but couldn't find anything. Kindly help.

Hello,

combinedstyles.css and combinedstyles.min.css are missing after mvn clean install (they are not on war)

Thank you

Error message: com.google.apphosting.api.DeadlineExceededException: This request (0000017ceb14f022) started at 2021/11/04 13:13:10.434 UTC and was still executing at 2021/11/04 13:14:11.274 UTC.

Time: Thu Nov 04 13:14:11 UTC 2021

RequestId: null

Request Info:
ReqInfo for 6183dc4f00ff0d15efcfea2f760001737e67726f6b636f6e7374727563746f72000136000100 : {
  "_url":"/do/match",
  "grokadditional":[
    "PROBEINTERFACE \\[(%{PATH},?\\s)\\]"
  ],
  "groklibs":[
    "firewalls",
    "aws",
    "bro",
    "exim",
    "bind",
    "haproxy",
    "linux-syslog",
    "squid",
    "mcollective-patterns",
    "bacula",
    "postgresql",
    "java",
    "maven",
    "grok-patterns",
    "httpd",
    "redis",
    "nagios",
    "rails",
    "mongodb",
    "ruby",
    "mcollective",
    "junos"
  ],
  "loglines":[
    "Most recent unsynchronized folders for probe7001: []\r\nMost recent unsynchronized folders for probe2040: [[/srv/probeinterface/./logs/1636023125]]\r\nMost recent unsynchronized folders for probe0058: [[/srv/probeinterface/./logs/1636031185, /srv/probeinterface/./logs/1636031275]]"
  ],
  "pattern":[
    "\\AMost recent unsynchronized folders for %{HOSTNAME:probe}: \\[%{PROBEINTERFACE}\\]"
  ],
  "submit":[
    "Go!"
  ]
}

com.google.apphosting.api.DeadlineExceededException: This request (0000017ceb14f022) started at 2021/11/04 13:13:10.434 UTC and was still executing at 2021/11/04 13:14:11.274 UTC.

	at org.joni.ByteCodeMachine.matchAt(ByteCodeMachine.java:272)
	at org.joni.Matcher.matchCheck(Matcher.java:304)
	at org.joni.Matcher.searchInterruptible(Matcher.java:457)
	at org.joni.Matcher.search(Matcher.java:318)
	at net.stoerr.grokconstructor.JoniRegex.findIn(JoniRegex.scala:72)
	at net.stoerr.grokconstructor.matcher.MatcherEntryView$$anonfun$showResult$1.apply(MatcherEntryView.scala:73)
	at net.stoerr.grokconstructor.matcher.MatcherEntryView$$anonfun$showResult$1.apply(MatcherEntryView.scala:71)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
	at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
	at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
	at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
	at scala.collection.AbstractTraversable.map(Traversable.scala:104)
	at net.stoerr.grokconstructor.matcher.MatcherEntryView.showResult(MatcherEntryView.scala:71)
	at net.stoerr.grokconstructor.matcher.MatcherEntryView$$anonfun$result$1.apply(MatcherEntryView.scala:61)
	at net.stoerr.grokconstructor.matcher.MatcherEntryView$$anonfun$result$1.apply(MatcherEntryView.scala:61)
	at scala.Option.map(Option.scala:146)
	at net.stoerr.grokconstructor.matcher.MatcherEntryView.result(MatcherEntryView.scala:61)
	at net.stoerr.grokconstructor.webframework.WebView$class.body(WebView.scala:34)
	at net.stoerr.grokconstructor.webframework.WebViewWithHeaderAndSidebox.body(WebViewWithHeaderAndSidebox.scala:10)
	at net.stoerr.grokconstructor.webframework.WebDispatcher.doGet(WebDispatcher.scala:60)
	at net.stoerr.grokconstructor.webframework.WebDispatcher.doPost(WebDispatcher.scala:33)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780)
	at com.google.apphosting.utils.servlet.JdbcMySqlConnectionCleanupFilter.doFilter(JdbcMySqlConnectionCleanupFilter.java:60)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
	at com.google.apphosting.runtime.jetty9.ParseBlobUploadHandler.handle(ParseBlobUploadHandler.java:125)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1182)
	at com.google.apphosting.runtime.jetty9.AppEngineWebAppContext.doHandle(AppEngineWebAppContext.java:187)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at com.google.apphosting.runtime.jetty9.AppVersionHandlerMap.handle(AppVersionHandlerMap.java:293)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
	at org.eclipse.jetty.server.Server.handle(Server.java:539)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)
	at com.google.apphosting.runtime.jetty9.RpcConnection.handle(RpcConnection.java:216)
	at com.google.apphosting.runtime.jetty9.RpcConnector.serviceRequest(RpcConnector.java:81)
	at com.google.apphosting.runtime.jetty9.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:134)
	at com.google.apphosting.runtime.RequestRunner.dispatchServletRequest(RequestRunner.java:264)
	at com.google.apphosting.runtime.RequestRunner.dispatchRequest(RequestRunner.java:229)
	at com.google.apphosting.runtime.RequestRunner.run(RequestRunner.java:194)
	at com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:274)
	at java.lang.Thread.run(Thread.java:748)

Please select and provide a license file for legal references.

https://appengine.google.com/deployment?&app_id=s~grokconstructor not exist any more

Thank you

Error message: Grok pattern name NAGIOSTIME unknown at %{NAGIOSTIME}

Time: Tue Jan 23 20:18:28 UTC 2018

RequestId: null

Request Info:
ReqInfo for 5a67989400ff02fa9d279f64620001737e67726f6b636f6e7374727563746f7200013600010102 : {
"_url":"/do/automatic",
"groklibs":[
"firewalls",
"grok-patterns"
],
"loglines":[
"<166>Jan 23 2018 14:07:36: %ASA-6-302021: Teardown ICMP connection for faddr 10.29.250.193/0 gaddr 172.19.122.36/0 laddr 172.19.122.36/0 (ejacobson) type 3 code 3 "
],
"submit":[
"Go!"
]
}

Grok pattern name NAGIOSTIME unknown at %{NAGIOSTIME}
at net.stoerr.grokconstructor.GrokPatternLibrary$$anonfun$replacePatterns$1$$anonfun$apply$mcVI$sp$1.apply(GrokPatternLibrary.scala:60)
at net.stoerr.grokconstructor.GrokPatternLibrary$$anonfun$replacePatterns$1$$anonfun$apply$mcVI$sp$1.apply(GrokPatternLibrary.scala:57)
at scala.util.matching.Regex$$anonfun$replaceAllIn$1.apply(Regex.scala:473)
at scala.util.matching.Regex$$anonfun$replaceAllIn$1.apply(Regex.scala:473)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
at scala.util.matching.Regex.replaceAllIn(Regex.scala:473)
at net.stoerr.grokconstructor.GrokPatternLibrary$$anonfun$replacePatterns$1.apply$mcVI$sp(GrokPatternLibrary.scala:56)
at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160)
at net.stoerr.grokconstructor.GrokPatternLibrary$.replacePatterns(GrokPatternLibrary.scala:52)
at net.stoerr.grokconstructor.automatic.AutomaticDiscoveryView$$anonfun$namedRegexps$1.apply(AutomaticDiscoveryView.scala:23)
at net.stoerr.grokconstructor.automatic.AutomaticDiscoveryView$$anonfun$namedRegexps$1.apply(AutomaticDiscoveryView.scala:22)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:221)
at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:428)
at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:428)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at net.stoerr.grokconstructor.automatic.AutomaticDiscoveryView.namedRegexps$lzycompute(AutomaticDiscoveryView.scala:22)
at net.stoerr.grokconstructor.automatic.AutomaticDiscoveryView.namedRegexps(AutomaticDiscoveryView.scala:22)
at net.stoerr.grokconstructor.automatic.AutomaticDiscoveryView.namedRegexpsList$lzycompute(AutomaticDiscoveryView.scala:25)
at net.stoerr.grokconstructor.automatic.AutomaticDiscoveryView.namedRegexpsList(AutomaticDiscoveryView.scala:25)
at net.stoerr.grokconstructor.automatic.AutomaticDiscoveryView.matchingRegexpStructures(AutomaticDiscoveryView.scala:94)
at net.stoerr.grokconstructor.automatic.AutomaticDiscoveryView.matchingRegexpStructures(AutomaticDiscoveryView.scala:92)
at net.stoerr.grokconstructor.automatic.AutomaticDiscoveryView.result(AutomaticDiscoveryView.scala:59)
at net.stoerr.grokconstructor.webframework.WebView$class.body(WebView.scala:34)
at net.stoerr.grokconstructor.webframework.WebViewWithHeaderAndSidebox.body(WebViewWithHeaderAndSidebox.scala:10)
at net.stoerr.grokconstructor.webframework.WebDispatcher.doGet(WebDispatcher.scala:60)
at net.stoerr.grokconstructor.webframework.WebDispatcher.doPost(WebDispatcher.scala:33)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166)
at com.google.apphosting.utils.servlet.ParseBlobUploadFilter.doFilter(ParseBlobUploadFilter.java:125)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter(SaveSessionFilter.java:37)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at com.google.apphosting.utils.servlet.JdbcMySqlConnectionCleanupFilter.doFilter(JdbcMySqlConnectionCleanupFilter.java:60)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:48)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle(AppVersionHandlerMap.java:257)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:923)
at com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable(RpcRequestParser.java:76)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:146)
at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.dispatchServletRequest(JavaRuntime.java:680)
at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.dispatchRequest(JavaRuntime.java:642)
at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:612)
at com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:455)
at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:462)
at com.google.tracing.CurrentContext.runInContext(CurrentContext.java:320)
at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:321)
at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:313)
at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:459)
at com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:274)
at java.lang.Thread.run(Thread.java:745)

During construction of SGE accounting log the web page is timed out and undermentioned error shows up:

OUCH!
I'm sorry, but you have encountered a bug or missing nice display of an error message in the application. If you can't guess the problem from the error message, please contact Hans-Peter Störr (www.stoerr.net [email protected]) with a copy of this page, or open an issue on https://github.com/stoerr/GrokConstructor/issues .

Please remember that you can always press the back button to fix what was wrong - there is no state on the server, only in the page shown in the browser.

Error message: com.google.apphosting.api.DeadlineExceededException: This request (14dd6d2ba955f8fe) started at 2018/07/06 06:05:23.273 UTC and was still executing at 2018/07/06 06:06:23.193 UTC.

Time: Fri Jul 06 06:06:23 UTC 2018

RequestId: null

Request Info:
ReqInfo for 5b3f06a300ff041ebb9dd680e60001737e67726f6b636f6e7374727563746f7200013600010104 : {
"_url":"/do/match",
"groklibs":[
"firewalls",
"aws",
"bro",
"exim",
"bind",
"haproxy",
"linux-syslog",
"squid",
"mcollective-patterns",
"bacula",
"postgresql",
"java",
"maven",
"grok-patterns",
"httpd",
"redis",
"nagios",
"rails",
"mongodb",
"ruby",
"mcollective",
"junos"
],
"loglines":[
"all.q:compute-0-0.local:shanawer:shanawer:serial.sh:180:sge:0:1424235565:1424235576:1424235610:0:1:34:336.030915:0.335948:749624.000000:\r\n0:0:0:0:56445:0:0:3296.000000:35344:0:0:0:1176:1072:NONE:defaultdepartment:NONE:1:0:336.366863:7004.945530:0.014643:-q all.q:0.000000:\r\nNONE:22813732864.000000:0:0"
],
"pattern":[
"%{DATA:qname}:%{DATA:exehost}:%{DATA:group}:%{DATA:owner}:%{DATA:job_name}:%{DATA:job_number}:%{DATA:account}:%{DATA:priority}:%{INT:submission_time}:%{INT:start_time}:%{INT:end_time}:%{DATA:failed}:%{INT:exit_status}:%{INT:ru_wallclock:int}:%{DATA:ru_utime:float}:%{DATA:ru_stime:float}:%{DATA:ru_maxrss:float}:%{DATA:ru_ixrss}:%{DATA:ru_ismrss}:%{DATA:ru_idrss}:%{DATA:ru_isrss}:%{DATA:ru_minflt}:%{DATA:ru_majflt}:%{DATA:ru_nswap}:%{DATA:ru_inblock}:%{DATA:ru_oublock}:%{DATA:ru_msgsnd}:%{DATA:ru_msgrcv}:%{DATA:ru_nsignals}:%{DATA:ru_nvcsw:int}:%{DATA:ru_nivcsw:int}:%{DATA:project}:%{DATA:department}:%{DATA:granted_pe}:%{INT:slots:int}:%{DATA:task_number}:%{DATA:cpu:float}:%{DATA:mem:float}:%{DATA:io:float}:%{DATA:category}:%{DATA:iow:float}:%{DATA:pe_taskid}:%{DATA:maxvmem:int}:%{DATA:arid}:%{DATA:ar_sub_time}\r\n"
],
"submit":[
"Go!"
]
}

com.google.apphosting.api.DeadlineExceededException: This request (14dd6d2ba955f8fe) started at 2018/07/06 06:05:23.273 UTC and was still executing at 2018/07/06 06:06:23.193 UTC.
at org.joni.ByteCodeMachine.opFail(ByteCodeMachine.java:1753)
at org.joni.ByteCodeMachine.opExact1(ByteCodeMachine.java:428)
at org.joni.ByteCodeMachine.matchAt(ByteCodeMachine.java:203)
at org.joni.Matcher.matchCheck(Matcher.java:304)
at org.joni.Matcher.searchInterruptible(Matcher.java:480)
at org.joni.Matcher.search(Matcher.java:318)
at net.stoerr.grokconstructor.JoniRegex.findIn(JoniRegex.scala:72)
at net.stoerr.grokconstructor.matcher.MatcherEntryView$$anonfun$net$stoerr$grokconstructor$matcher$MatcherEntryView$$longestMatchOfRegexPrefix$1.apply(MatcherEntryView.scala:127)
at net.stoerr.grokconstructor.matcher.MatcherEntryView$$anonfun$net$stoerr$grokconstructor$matcher$MatcherEntryView$$longestMatchOfRegexPrefix$1.apply(MatcherEntryView.scala:127)
at scala.collection.immutable.Stream$$anonfun$map$1.apply(Stream.scala:418)
at scala.collection.immutable.Stream$$anonfun$map$1.apply(Stream.scala:418)
at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1233)
at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1223)
at scala.collection.immutable.Stream.filter(Stream.scala:519)
at net.stoerr.grokconstructor.matcher.MatcherEntryView.net$stoerr$grokconstructor$matcher$MatcherEntryView$$longestMatchOfRegexPrefix(MatcherEntryView.scala:127)
at net.stoerr.grokconstructor.matcher.MatcherEntryView$$anonfun$showResult$1.apply(MatcherEntryView.scala:76)
at net.stoerr.grokconstructor.matcher.MatcherEntryView$$anonfun$showResult$1.apply(MatcherEntryView.scala:71)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at net.stoerr.grokconstructor.matcher.MatcherEntryView.showResult(MatcherEntryView.scala:71)
at net.stoerr.grokconstructor.matcher.MatcherEntryView$$anonfun$result$1.apply(MatcherEntryView.scala:61)
at net.stoerr.grokconstructor.matcher.MatcherEntryView$$anonfun$result$1.apply(MatcherEntryView.scala:61)
at scala.Option.map(Option.scala:146)
at net.stoerr.grokconstructor.matcher.MatcherEntryView.result(MatcherEntryView.scala:61)
at net.stoerr.grokconstructor.webframework.WebView$class.body(WebView.scala:34)
at net.stoerr.grokconstructor.webframework.WebViewWithHeaderAndSidebox.body(WebViewWithHeaderAndSidebox.scala:10)
at net.stoerr.grokconstructor.webframework.WebDispatcher.doGet(WebDispatcher.scala:60)
at net.stoerr.grokconstructor.webframework.WebDispatcher.doPost(WebDispatcher.scala:33)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166)
at com.google.apphosting.utils.servlet.ParseBlobUploadFilter.doFilter(ParseBlobUploadFilter.java:125)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter(SaveSessionFilter.java:37)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at com.google.apphosting.utils.servlet.JdbcMySqlConnectionCleanupFilter.doFilter(JdbcMySqlConnectionCleanupFilter.java:60)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:48)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle(AppVersionHandlerMap.java:257)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:923)
at com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable(RpcRequestParser.java:76)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:146)
at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.dispatchServletRequest(JavaRuntime.java:693)
at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.dispatchRequest(JavaRuntime.java:655)
at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:625)
at com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:455)
at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:462)
at com.google.tracing.CurrentContext.runInContext(CurrentContext.java:320)
at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:321)
at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:313)
at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:459)
at com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:274)
at java.lang.Thread.run(Thread.java:745)

Any needful help or idea to build the requirements

Regards,

--jafar

I encountered an error while testing Grok pattern. The copy of error:

re: http://grokconstructor.appspot.com/do/match

First: great tool, it has helped my a lot!

It appears that Logstash's modern variable name format [foo][bar][bob] (hash entry foo->bar->bob) is not supported by GrokConstructor, the following error message is give: "invalid char in group name"

Here is an example Grok for syslog RFC3164

<%{POSINT:[syslog][pri]:int}>%{SYSLOGTIMESTAMP:[syslog][timestamp]} +(?:%{IP:[rep][address]}|%{HOSTNAME:[rep][alias]}(?: +%{IP:[rep][ip]})?) +%{USERNAME:[@metadata][sl_type]}(?:\(%{PATH:[app][name]}\))?(?:\[%{POSINT:[proc][id]:int}\]?)?:? *%{GREEDYDATA:[msg]}"

This works correctly in Logstash 5.5.1, but not with GrokConstructor.

I would simply allow [, ] and @ in a substitute variable name (maybe masking them from an underlying library?).

At the moment, I have to strip them out (manually), do my debugging, then put them back in before copying to the final config.

Have the https://travis-ci.com build for GrokConstructor automatically deploy pushes to master to GAE, and pushes to develop to a testing machine on GAE, such that the deployments are automatically triggered.

I will try to explain my need, I get the complete log of an application that is multilinhas, so I understood filebeat does the reading by document base, in this way each line it "breaks" in the visualization; I need some logs that are on separate lines to be unique in Kibana, it would not be a problem if they were in the "message" field, but I need to extract the information with a set of Fields. Ex: Log1 AND Log2 AND Log3, as it stands today the contents of this "log" are in separate fields, which does not help me when searching for the expression in kibana.

Attached I am sending you the configuration of filebeat and logstash, I have not modified anything in the structure of the two; is sending all the log to the elasticsearch.

What I really need is to take this log and extract some information from it, I'll also send you the log by attachment and explain below what I'd like to do with the reported snippet (I've just changed the contents of the fields because I'm being given stealth)

I need the fields "tef_iso" to "Bit_127" to be unique, being displayed in only one message.

logstash (1).txt
log_aplicacao (1).txt
filebeat_configuracao (1).txt

Want to match:

** WARNING ** You have enabled encryption but DISABLED certificate verification.

with:

\*\* %{LOGLEVEL:level} \*\* %{LS_MESSAGE:message1}

but receive an error.

(works on https://grokdebug.herokuapp.com/ and within Logstash)

It seams all present on your erro page:

OUCH!

I'm sorry, but you have encountered a bug or missing nice display of an error message in the application. If you can't guess the problem from the error message, please contact Hans-Peter Störr (www.stoerr.net [email protected]) with a copy of this page, or open an issue on https://github.com/stoerr/GrokConstructor/issues .

Please remember that you can always press the back button to fix what was wrong - there is no state on the server, only in the page shown in the browser.



Error message: java.util.NoSuchElementException: None.get


Request Info:
http://grokconstructor.appspot.com/do/constructionstep?null:
{"grokadditional":[""],"submit":["Continue!"],"pattern":["\\A%{TIMESTAMP_ISO8601} \\[%{USERNAME}] \\[%{UUID}] \\[%{USERNAME}] %{LOGLEVEL} %{JAVACLASS}:%{INT} - %{INT} ms; \\[%{DATA}];\\[%{USERNAME}"],"multiline":[""],"nameOfNextPart":[""],"loglines":["2015-06-30 03:09:19,190 [catalina-backend-exec-51] [762f2bed-3ac9-4e43-8516-643c42f39491] [forestry_00402_001] DEBUG ru.rlh.egais.portal.backend.jdbc.SQL.logSql:188 - 0 ms; [SELECT count(*)  FROM bo_forestry forestry  INNER JOIN bo_constituent_entity c  onstituent_entity ON constituent_entity.id = forestry.bo_const_entity_fkey  WHERE forestry.id = ? and constituent_entity.id =   ?];[G_00402,27]"],"nextPartPerHand":["]"],"groklibs":["java","grok-patterns"],"nextPart":["ksdf8wej2349j_ThisIsAMarkerForNextPartThatNextPartPerHandWasChoosen"]}


java.util.NoSuchElementException: None.get
    at scala.None$.get(Option.scala:344)
    at scala.None$.get(Option.scala:342)
    at net.stoerr.grokconstructor.incremental.IncrementalConstructionStepView$$anonfun$2.apply(IncrementalConstructionStepView.scala:26)
    at net.stoerr.grokconstructor.incremental.IncrementalConstructionStepView$$anonfun$2.apply(IncrementalConstructionStepView.scala:24)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
    at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
    at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
    at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
    at scala.collection.AbstractTraversable.map(Traversable.scala:104)
    at net.stoerr.grokconstructor.incremental.IncrementalConstructionStepView.(IncrementalConstructionStepView.scala:23)
    at net.stoerr.grokconstructor.webframework.WebDispatcher.giveView(WebDispatcher.scala:58)
    at net.stoerr.grokconstructor.webframework.WebDispatcher.doGet(WebDispatcher.scala:32)
    at net.stoerr.grokconstructor.webframework.WebDispatcher.doPost(WebDispatcher.scala:26)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
    at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166)
    at com.google.apphosting.utils.servlet.ParseBlobUploadFilter.doFilter(ParseBlobUploadFilter.java:125)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter(SaveSessionFilter.java:35)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at com.google.apphosting.utils.servlet.JdbcMySqlConnectionCleanupFilter.doFilter(JdbcMySqlConnectionCleanupFilter.java:60)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:43)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
    at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
    at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
    at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
    at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
    at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle(AppVersionHandlerMap.java:257)
    at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
    at org.mortbay.jetty.Server.handle(Server.java:326)
    at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
    at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:923)
    at com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable(RpcRequestParser.java:76)
    at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
    at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:146)
    at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:482)
    at com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:437)
    at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:444)
    at com.google.tracing.CurrentContext.runInContext(CurrentContext.java:230)
    at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:308)
    at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:300)
    at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:441)
    at com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:234)
    at java.lang.Thread.run(Thread.java:745)

To reproduce:

Log line:
30/01/2020 15:53:24.547\tYULEMATH\tGCO\SRVINTITSMDES\tARQ.AppInt.JiraItsm.BE\tIntegracion\tIntegracion\t98441013-d58f-40d3-8fab-30320c3d4590\t-\tINF\tDescripción:

Steps:
DATESTAMP
\t
WORD

Error message: java.util.NoSuchElementException: None.get

Time: Thu Jan 30 08:40:53 UTC 2020

RequestId: null

Request Info:
ReqInfo for 5e32969500ff0a62e497e38d580001737e67726f6b636f6e7374727563746f7200013600010104 : {
"_url":"/do/constructionstep",
"groklibs":[
"firewalls",
"aws",
"bro",
"exim",
"bind",
"haproxy",
"linux-syslog",
"squid",
"mcollective-patterns",
"bacula",
"postgresql",
"java",
"maven",
"grok-patterns",
"httpd",
"redis",
"nagios",
"rails",
"mongodb",
"ruby",
"mcollective",
"junos"
],
"loglines":[
"30/01/2020 15:53:24.547\tYULEMATH\tGCO\\SRVINTITSMDES\tARQ.AppInt.JiraItsm.BE\tIntegracion\tIntegracion\t98441013-d58f-40d3-8fab-30320c3d4590\t-\tINF\tDescripción: "
],
"nextPart":[
"%{HOSTNAME}"
],
"pattern":[
"\A%{DATESTAMP}\\t",
"\A%{DATESTAMP}\\t"
],
"submit":[
"Continue!"
]
}

java.util.NoSuchElementException: None.get
at scala.None$.get(Option.scala:347)
at scala.None$.get(Option.scala:345)
at net.stoerr.grokconstructor.incremental.IncrementalConstructionStepView$$anonfun$2.apply(IncrementalConstructionStepView.scala:28)
at net.stoerr.grokconstructor.incremental.IncrementalConstructionStepView$$anonfun$2.apply(IncrementalConstructionStepView.scala:26)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at net.stoerr.grokconstructor.incremental.IncrementalConstructionStepView.(IncrementalConstructionStepView.scala:25)
at net.stoerr.grokconstructor.webframework.WebDispatcher.giveView(WebDispatcher.scala:81)
at net.stoerr.grokconstructor.webframework.WebDispatcher.doGet(WebDispatcher.scala:53)
at net.stoerr.grokconstructor.webframework.WebDispatcher.doPost(WebDispatcher.scala:33)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
at com.google.apphosting.utils.servlet.JdbcMySqlConnectionCleanupFilter.doFilter(JdbcMySqlConnectionCleanupFilter.java:60)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at com.google.apphosting.runtime.jetty9.ParseBlobUploadHandler.handle(ParseBlobUploadHandler.java:119)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1182)
at com.google.apphosting.runtime.jetty9.AppEngineWebAppContext.doHandle(AppEngineWebAppContext.java:187)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at com.google.apphosting.runtime.jetty9.AppVersionHandlerMap.handle(AppVersionHandlerMap.java:293)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:539)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)
at com.google.apphosting.runtime.jetty9.RpcConnection.handle(RpcConnection.java:213)
at com.google.apphosting.runtime.jetty9.RpcConnector.serviceRequest(RpcConnector.java:81)
at com.google.apphosting.runtime.jetty9.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:134)
at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.dispatchServletRequest(JavaRuntime.java:757)
at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.dispatchRequest(JavaRuntime.java:720)
at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:690)
at com.google.apphosting.runtime.JavaRuntime$NullSandboxRequestRunnable.run(JavaRuntime.java:882)
at com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:270)
at java.lang.Thread.run(Thread.java:748)

logstash supports using a metadata semantic/identifier in grok (see https://www.elastic.co/blog/logstash-metadata "Date filter")

e.g. pattern %{TIMESTAMP_ISO8601:[@metadata][ts]} should match input 2021-03-08 23:20:36.952

but instead reports no match

Thank you.

It appears that the <article> object is automatically inheriting the max width of the page, instead of the width of the .ym-wrapper

I believe the easiest way to save this is to change .ym-gl so that it has width: inherit

I had to install java 7, maybe could be said on deployment.txt

Thank you!

There are quite a lot misusages of that field. Unclear: do people not understand what the field is for, or accidentially use the wrong field? If it's accidential, we can hide it per Javascript, so people deliberately have to make it visible. (New checkbox in the page.)

On properly-constructed forms, clicking on a radio button or checkbox's label will cause the associated element to be activated. Unfortunately, in the incremental constructor, all form elements have an id attribute of "nextPart". As a result, clicking on any radio button's label causes the first radio button to become selected. For radio buttons, the name attribute is what determines which radio buttons are linked, not the id attribute. Making each radio button have a unique id attribute would resolve this problem.

This is a great tool, but the site is sometimes slow, so it would be convenient to be able to run this in docker locally.

If you create a docker container and push it to docker hub, it makes it easier for other people to run this.

Let me know if you need help setting this up.

HI ,

Steps I have performed 👍

1. Downloaded Gork constructor on RedHat el6.x86_64
2. installed apache-maven-3.2.1
3. Exported path accordingly .
4. cd BLABLA/GrokConstructor-master
5. sh start.sh . At this point getting below errors .

[WARNING]
[WARNING] Some problems were encountered while building the effective model for net.stoerr:GrokConstructor:war:0.1.0-SNAPSHOT
[WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-war-plugin is missing. @ line 123, column 21
[WARNING]
[WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
[WARNING]
[WARNING] For this reason, future Maven versions might no longer support building such malformed projects.
[WARNING]
Downloading: http://repo.maven.apache.org/maven2/com/samaxes/maven/minify-maven-plugin/1.7.4/minify-maven-plugin-1.7.4.pom
[WARNING] Failed to retrieve plugin descriptor for com.samaxes.maven:minify-maven-plugin:1.7.4: Plugin com.samaxes.maven:minify-maven-plugin:1.7.4 or one of its dependencies could not be resolved: Failed to read artifact descriptor for com.samaxes.maven:minify-maven-plugin:jar:1.7.4
Downloading: http://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-war-plugin/2.2/maven-war-plugin-2.2.pom
[WARNING] Failed to retrieve plugin descriptor for org.apache.maven.plugins:maven-war-plugin:2.2: Plugin org.apache.maven.plugins:maven-war-plugin:2.2 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-war-plugin:jar:2.2
Downloading: http://repo.maven.apache.org/maven2/net/alchim31/maven/scala-maven-plugin/3.2.2/scala-maven-plugin-3.2.2.pom
[WARNING] Failed to retrieve plugin descriptor for net.alchim31.maven:scala-maven-plugin:3.2.2: Plugin net.alchim31.maven:scala-maven-plugin:3.2.2 or one of its dependencies could not be resolved: Failed to read artifact descriptor for net.alchim31.maven:scala-maven-plugin:jar:3.2.2
Downloading: http://repo.maven.apache.org/maven2/com/google/appengine/appengine-maven-plugin/1.9.38/appengine-maven-plugin-1.9.38.pom
[WARNING] Failed to retrieve plugin descriptor for com.google.appengine:appengine-maven-plugin:1.9.38: Plugin com.google.appengine:appengine-maven-plugin:1.9.38 or one of its dependencies could not be resolved: Failed to read artifact descriptor for com.google.appengine:appengine-maven-plugin:jar:1.9.38
Downloading: http://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-clean-plugin/2.5/maven-clean-plugin-2.5.pom
[WARNING] Failed to retrieve plugin descriptor for org.apache.maven.plugins:maven-clean-plugin:2.5: Plugin org.apache.maven.plugins:maven-clean-plugin:2.5 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-clean-plugin:jar:2.5
Downloading: http://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-resources-plugin/2.6/maven-resources-plugin-2.6.pom
[WARNING] Failed to retrieve plugin descriptor for org.apache.maven.plugins:maven-resources-plugin:2.6: Plugin org.apache.maven.plugins:maven-resources-plugin:2.6 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-resources-plugin:jar:2.6
Downloading: http://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-compiler-plugin/2.5.1/maven-compiler-plugin-2.5.1.pom
[WARNING] Failed to retrieve plugin descriptor for org.apache.maven.plugins:maven-compiler-plugin:2.5.1: Plugin org.apache.maven.plugins:maven-compiler-plugin:2.5.1 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-compiler-plugin:jar:2.5.1
Downloading: http://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-surefire-plugin/2.12.4/maven-surefire-plugin-2.12.4.pom
[WARNING] Failed to retrieve plugin descriptor for org.apache.maven.plugins:maven-surefire-plugin:2.12.4: Plugin org.apache.maven.plugins:maven-surefire-plugin:2.12.4 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-surefire-plugin:jar:2.12.4
Downloading: http://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-install-plugin/2.4/maven-install-plugin-2.4.pom
[WARNING] Failed to retrieve plugin descriptor for org.apache.maven.plugins:maven-install-plugin:2.4: Plugin org.apache.maven.plugins:maven-install-plugin:2.4 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-install-plugin:jar:2.4
Downloading: http://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-deploy-plugin/2.7/maven-deploy-plugin-2.7.pom
[WARNING] Failed to retrieve plugin descriptor for org.apache.maven.plugins:maven-deploy-plugin:2.7: Plugin org.apache.maven.plugins:maven-deploy-plugin:2.7 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-deploy-plugin:jar:2.7
Downloading: http://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-site-plugin/3.3/maven-site-plugin-3.3.pom
[WARNING] Failed to retrieve plugin descriptor for org.apache.maven.plugins:maven-site-plugin:3.3: Plugin org.apache.maven.plugins:maven-site-plugin:3.3 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-site-plugin:jar:3.3
Downloading: http://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-antrun-plugin/1.3/maven-antrun-plugin-1.3.pom
[WARNING] Failed to retrieve plugin descriptor for org.apache.maven.plugins:maven-antrun-plugin:1.3: Plugin org.apache.maven.plugins:maven-antrun-plugin:1.3 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-antrun-plugin:jar:1.3
Downloading: http://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-assembly-plugin/2.2-beta-5/maven-assembly-plugin-2.2-beta-5.pom
[WARNING] Failed to retrieve plugin descriptor for org.apache.maven.plugins:maven-assembly-plugin:2.2-beta-5: Plugin org.apache.maven.plugins:maven-assembly-plugin:2.2-beta-5 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-assembly-plugin:jar:2.2-beta-5
Downloading: http://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-dependency-plugin/2.8/maven-dependency-plugin-2.8.pom
[WARNING] Failed to retrieve plugin descriptor for org.apache.maven.plugins:maven-dependency-plugin:2.8: Plugin org.apache.maven.plugins:maven-dependency-plugin:2.8 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-dependency-plugin:jar:2.8
Downloading: http://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-release-plugin/2.3.2/maven-release-plugin-2.3.2.pom
[WARNING] Failed to retrieve plugin descriptor for org.apache.maven.plugins:maven-release-plugin:2.3.2: Plugin org.apache.maven.plugins:maven-release-plugin:2.3.2 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-release-plugin:jar:2.3.2
Downloading: http://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-metadata.xml
Downloading: http://repo.maven.apache.org/maven2/org/codehaus/mojo/maven-metadata.xml
[WARNING] Could not transfer metadata org.apache.maven.plugins/maven-metadata.xml from/to central (http://repo.maven.apache.org/maven2): repo.maven.apache.org
[WARNING] Could not transfer metadata org.codehaus.mojo/maven-metadata.xml from/to central (http://repo.maven.apache.org/maven2): repo.maven.apache.org
[WARNING] Failure to transfer org.apache.maven.plugins/maven-metadata.xml from http://repo.maven.apache.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced. Original error: Could not transfer metadata org.apache.maven.plugins/maven-metadata.xml from/to central (http://repo.maven.apache.org/maven2): repo.maven.apache.org
[WARNING] Failure to transfer org.codehaus.mojo/maven-metadata.xml from http://repo.maven.apache.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced. Original error: Could not transfer metadata org.codehaus.mojo/maven-metadata.xml from/to central (http://repo.maven.apache.org/maven2): repo.maven.apache.org
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.274 s
[INFO] Finished at: 2017-01-24T02:21:53-05:00
[INFO] Final Memory: 7M/72M
[INFO] ------------------------------------------------------------------------
[ERROR] No plugin found for prefix 'appengine' in the current project and in the plugin groups [org.apache.maven.plugins, org.codehaus.mojo] available from the repositories [local (/root/.m2/repository), central (http://repo.maven.apache.org/maven2)] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/NoPluginFoundForPrefixException

Please help . It will be really helpful if I get the steps to install the Gorkconstructor and the usage of it .
Thanks in advance .
Regards,
Pratik

OUCH!

I'm sorry, but you have encountered a bug or missing nice display of an error message in the application. If you can't guess the problem from the error message, please contact Hans-Peter Störr (www.stoerr.net [email protected]) with a copy of this page, or open an issue on https://github.com/stoerr/GrokConstructor/issues .

Please remember that you can always press the back button to fix what was wrong - there is no state on the server, only in the page shown in the browser.

Error message: java.util.NoSuchElementException: None.get

Request Info:
http://grokconstructor.appspot.com/do/constructionstep?null:
{"grokadditional":[""],"submit":["Continue!"],"pattern":["\A2015-09-28 09:50:48 [http-bio-80-exec-13] DEBUG com.weitoo.server.aspect.LogAspect -{ip:183.16.4.40,url:http://api\.weitoo\.com/server/sc/commodity/getOnlineCommodity,param:\{\"shopId\":1000001,\"needCategory\":false,\"needCommodityTotal\":false,\"searchCommodityId\":1002001}"],"multiline":[""],"nameOfNextPart":[""],"loglines":["2015-09-28 09:50:48 [http-bio-80-exec-13] DEBUG com.weitoo.server.aspect.LogAspect -{ip:183.16.4.40,url:http://api.weitoo.com/server/sc/commodity/getOnlineCommodity,param:{\"shopId\":1000001,\"needCategory\":false,\"needCommodityTotal\":false,\"searchCommodityId\":1002001},return:{\"status\":1,\"data\":{\"commodityDTO\":[{\"id\":1002001,\"shopId\":1000001,\"categoryId\":13,\"barcode\":\"6953252600013\",\"name\":\"????????\",\"price\":8.00,\"stock\":10000,\"unit\":\"\",\"picId\":61327,\"picPath\":\"/Uploads/Shoppic/2015/04/20/5534900cb9f9f.jpg\",\"status\":1,\"addType\":0}],\"commoditySize\":1,\"commodityTotal\":1,\"categoryCount\":0}},cost:3.911ms}"],"nextPartPerHand":[""],"groklibs":["firewalls","haproxy","linux-syslog","mcollective-patterns","postgresql","java","grok-patterns","redis","nagios","mongodb","ruby","mcollective","junos"],"nextPart":[",return:\\{\"status\":1,\"data\":\\{\"commodityDTO\":\\[\\{\"id\":1002001,\"shopId\":1000001,\"categoryId\":13,\"barcode\":\"6953252600013\",\"name\":\"????????\",\"price\":8\\.00,\"stock\":10000,\"unit\":\"\",\"picId\":61327,\"picPath\":\"/Uploads/Shoppic/2015/04/20/5534900cb9f9f\\.jpg\",\"status\":1,\"addType\":0}],\"commoditySize\":1,\"commodityTotal\":1,\"categoryCount\":0}}"]}

java.util.NoSuchElementException: None.get
at scala.None$.get(Option.scala:344)
at scala.None$.get(Option.scala:342)
at net.stoerr.grokconstructor.incremental.IncrementalConstructionStepView$$anonfun$2.apply(IncrementalConstructionStepView.scala:26)
at net.stoerr.grokconstructor.incremental.IncrementalConstructionStepView$$anonfun$2.apply(IncrementalConstructionStepView.scala:24)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at net.stoerr.grokconstructor.incremental.IncrementalConstructionStepView.(IncrementalConstructionStepView.scala:23)
at net.stoerr.grokconstructor.webframework.WebDispatcher.giveView(WebDispatcher.scala:58)
at net.stoerr.grokconstructor.webframework.WebDispatcher.doGet(WebDispatcher.scala:32)
at net.stoerr.grokconstructor.webframework.WebDispatcher.doPost(WebDispatcher.scala:26)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166)
at com.google.apphosting.utils.servlet.ParseBlobUploadFilter.doFilter(ParseBlobUploadFilter.java:125)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter(SaveSessionFilter.java:37)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at com.google.apphosting.utils.servlet.JdbcMySqlConnectionCleanupFilter.doFilter(JdbcMySqlConnectionCleanupFilter.java:60)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:43)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle(AppVersionHandlerMap.java:260)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:923)
at com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable(RpcRequestParser.java:78)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:147)
at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:457)
at com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:437)
at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:444)
at com.google.tracing.CurrentContext.runInContext(CurrentContext.java:230)
at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:308)
at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:300)
at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:441)
at com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:235)
at java.lang.Thread.run(Thread.java:745)

After clicking go button, though log fields are parsed correctly, all Chinese characters display incorrect. I run git clone and mvn clean install to build the standalone runnable jar file, use java -jar GrokConstructor-0.1.0-SNAPSHOT-standalone.jar to run the web application.

How to solve the encoding problem?

I'm using the matcher (http://localhost:9090/do/match#result) against a single log line.
It seems to me that pressing the "Go!" button appends a space to whats found in the pattern text area, corrupting the pattern.

Firefox 45, linux mint 17.2 in virtualbox on windows 7, commit c8bd1aa

Try incremental construction for single log line
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.115 Safari/537.36
and take longest prefix ->
org.joni.exception.SyntaxException: unmatched close parenthesis
at org.joni.ScannerSupport.newSyntaxException(ScannerSupport.java:166)
at org.joni.Parser.parseExp(Parser.java:672)
at org.joni.Parser.parseBranch(Parser.java:962)
at org.joni.Parser.parseSubExp(Parser.java:979)
at org.joni.Parser.parseRegexp(Parser.java:1012)
at org.joni.Parser.parse(Parser.java:64)
at org.joni.Analyser.compile(Analyser.java:83)
at org.joni.Regex.(Regex.java:162)
at org.joni.Regex.(Regex.java:139)
at org.joni.Regex.(Regex.java:125)
at org.joni.Regex.(Regex.java:109)
at net.stoerr.grokconstructor.JoniRegex.(JoniRegex.scala:16)
at net.stoerr.grokconstructor.incremental.IncrementalConstructionStepView.(IncrementalConstructionStepView.scala:22)
at net.stoerr.grokconstructor.webframework.WebDispatcher.giveView(WebDispatcher.scala:58)
at net.stoerr.grokconstructor.webframework.WebDispatcher.doGet(WebDispatcher.scala:32)
at net.stoerr.grokconstructor.webframework.WebDispatcher.doPost(WebDispatcher.scala:26)

Building a new pipeline via Kibana en "Dev Tools" I expect "event" to differ but it only assign "event": "Event BorrowedDenied" :

POST _ingest/pipeline/_simulate { "description": "To map license information via filebeat", "version" : 20170508, "pipeline" : { "processors" : [ { "grok" : { "field" : "message", "patterns" : ["%{DATESTAMP:datetime} - ((?<event>Event BorrowedDenied)|(?<event>Event LicenceBorrowed)|(?<event>Event LicenceGrant)|(?<event>Event LicenceRenewed)|(?<event>Event LicenceReturn)) \"%{HOSTNAME:hostname}\" \"%{USER:user}\" program=\"%{GREEDYDATA:program}\" used=%{INT:used} available=%{INT:available} \\(tc=%{INT:tc}\\)"] } } ] }, "docs" : [ { "_source" : { "message" : "04/05/2017 13:20:43 - Event LicenceBorrowed \"ITS-123456789\" \"jdoe\" program=\"Discover\" used=1 available=99 (tc=3)" } }, { "_source" : { "message" : "04/05/2017 13:20:41 - Event BorrowedDenied \"ITS-123456789\" \"jdoe\" program=\"Encom License Manager\" used=0 available=0 (tc=5)" } } ] }

Doing the same operation in Grok Constructor:
Data
04/05/2017 13:20:43 - Event LicenceBorrowed "ITS-123456789" "jdoe" program="Discover" used=1 available=99 (tc=3) 04/05/2017 13:20:41 - Event BorrowedDenied "ITS-123456789" "jdoe" program="Encom License Manager" used=0 available=0 (tc=5)

GROK
%{DATESTAMP:datetime} - ((?<event>Event BorrowedDenied)|(?<event>Event LicenceBorrowed)|(?<event>Event LicenceGrant)|(?<event>Event LicenceRenewed)|(?<event>Event LicenceReturn)) \"%{HOSTNAME:hostname}\" \"%{USER:user}\" program=\"%{GREEDYDATA:program}\" used=%{INT:used} available=%{INT:available} \(tc=%{INT:tc}\)

It parse the event into different values. I don't know if this i a bug or not but I need to start somewhere .

Cheers
Fredrik

Hi Hans-Peter,

unfortunately the page crashes every time when I'm trying to use the constructor and try to parse logfiles. Especially when it comes to HTTP Requests. The failure I'm getting is:

Error message: com.google.apphosting.api.DeadlineExceededException: This request (cf19023660eb3df5) started at 2018/04/30 07:24:49.902 UTC and was still executing at 2018/04/30 07:25:49.868 UTC.

Time: Mon Apr 30 07:25:49 UTC 2018

RequestId: null

Request Info:
ReqInfo for 5ae6c4c100ff0db951fe23c62b0001737e67726f6b636f6e7374727563746f7200013600010101 : {
"_url":"/do/constructionstep",
"groklibs":[
"firewalls",
"aws",
"bro",
"exim",
"bind",
"haproxy",
"linux-syslog",
"squid",
"mcollective-patterns",
"bacula",
"postgresql",
"java",
"maven",
"grok-patterns",
"httpd",
"redis",
"nagios",
"rails",
"mongodb",
"ruby",
"mcollective",
"junos"
],
"loglines":[
""GET ....
],
"nextPart":[
"%{WORD}"
],
"pattern":[
"\A"",
"\A""
],
"submit":[
"Continue!"
]
}

com.google.apphosting.api.DeadlineExceededException: This request (cf19023660eb3df5) started at 2018/04/30 07:24:49.902 UTC and was still executing at 2018/04/30 07:25:49.868 UTC.
at org.joni.ByteCodeMachine.matchAt(ByteCodeMachine.java:254)
at org.joni.Matcher.matchInterruptible(Matcher.java:113)
at org.joni.Matcher.match(Matcher.java:94)
at net.stoerr.grokconstructor.JoniRegex.matchStartOf(JoniRegex.scala:34)
at net.stoerr.grokconstructor.incremental.IncrementalConstructionStepView$$anonfun$5$$anonfun$6.apply(IncrementalConstructionStepView.scala:35)
at net.stoerr.grokconstructor.incremental.IncrementalConstructionStepView$$anonfun$5$$anonfun$6.apply(IncrementalConstructionStepView.scala:35)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at net.stoerr.grokconstructor.incremental.IncrementalConstructionStepView$$anonfun$5.apply(IncrementalConstructionStepView.scala:35)
at net.stoerr.grokconstructor.incremental.IncrementalConstructionStepView$$anonfun$5.apply(IncrementalConstructionStepView.scala:33)
at scala.collection.immutable.List.map(List.scala:277)
at net.stoerr.grokconstructor.incremental.IncrementalConstructionStepView.(IncrementalConstructionStepView.scala:33)
at net.stoerr.grokconstructor.webframework.WebDispatcher.giveView(WebDispatcher.scala:81)
at net.stoerr.grokconstructor.webframework.WebDispatcher.doGet(WebDispatcher.scala:53)
at net.stoerr.grokconstructor.webframework.WebDispatcher.doPost(WebDispatcher.scala:33)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166)
at com.google.apphosting.utils.servlet.ParseBlobUploadFilter.doFilter(ParseBlobUploadFilter.java:125)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter(SaveSessionFilter.java:37)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at com.google.apphosting.utils.servlet.JdbcMySqlConnectionCleanupFilter.doFilter(JdbcMySqlConnectionCleanupFilter.java:60)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:48)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle(AppVersionHandlerMap.java:257)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:923)
at com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable(RpcRequestParser.java:76)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:146)
at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.dispatchServletRequest(JavaRuntime.java:686)
at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.dispatchRequest(JavaRuntime.java:648)
at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:618)
at com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:455)
at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:462)
at com.google.tracing.CurrentContext.runInContext(CurrentContext.java:320)
at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:321)
at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:313)
at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:459)
at com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:274)
at java.lang.Thread.run(Thread.java:745)

Whenever trying to parse Snare syslog events, the constructor and matcher both fail.

Log contents:
MSWinEventLog#0115#011Security#0 Jun 29 20:53:35 2020#0115140#011Microsoft-Windows-Security-Auditing

custom patterns:
VT #011

pattern to match
MSWinEventLog%{VT}%{INT:crit}%{VT}%{WORD:event_src}%{VT}

When attempting in the inc. constructor, there is a java exception. (https://pastebin.com/k6r8kbM4)

When attempting in matcher, the pattern is missing in the results and the match fails

i was testing a big pattern, and had the whole thing in quotes, as it was coming from a logstash filter.

i kept getting all sorts of match fails, but when i simply removed the quotes from beginning and end of line, it worked perfectly.

i spent maybe 2 hours going through all sorts of dead-ends.

would it be possible to mention in the prompt to NOT contain the pattern in quotes, or could your tool simply recognize that and remove them?

thanks - cool tool.

Input logline:

This is a test.

Working Pattern:

%{DATA:My_Message}\.

Breaking Pattern:

%{DATA:My-Message}\.

This was found in the web version

Hi!
The grock contructor is agreat tool! I've been using it for the last month to check my groks.
For a few times I copied groks and the page broke down when I clicked 'Go!' - I'v just understood why:

Every time you copy a " " into the new grok expression and the delimiter between the grok name field and the regex field is not a space (if it is a tab, for example), when you click 'Go!' the entire page breaks down and loses all information.

This would be nice to fix! but I do not have the skills for it.
Thanks

Error message: org.joni.exception.SyntaxException: unmatched close parenthesis

Request Info:
http://grokconstructor.appspot.com/do/constructionstep?null:
{"grokadditional":[""],"submit":["Continue!"],"pattern":["\A%{SYSLOG5424PRINTASCII} %{LOGLEVEL} (%{BASE10NUM}"],"multiline":[""],"nameOfNextPart":[""],"loglines":["2015-07-22T08:21:27-04:00 NOTICE (5): [ACCESS GRANTED] /pusher/auth [e98fc7bf73eb62c4ad43dd4f1a1b8e75] &socket_id=47627.2019085&channel_name=presence-b703843ae8f14d53baa504db70c0b51d"],"nextPartPerHand":[""],"groklibs":["firewalls","haproxy","linux-syslog","mcollective-patterns","postgresql","java","grok-patterns","redis","nagios","mongodb","ruby","mcollective","junos"],"nextPart":["): ["]}

org.joni.exception.SyntaxException: unmatched close parenthesis
at org.joni.ScannerSupport.newSyntaxException(ScannerSupport.java:166)
at org.joni.Parser.parseExp(Parser.java:672)
at org.joni.Parser.parseBranch(Parser.java:962)
at org.joni.Parser.parseSubExp(Parser.java:979)
at org.joni.Parser.parseRegexp(Parser.java:1012)
at org.joni.Parser.parse(Parser.java:64)
at org.joni.Analyser.compile(Analyser.java:83)
at org.joni.Regex.(Regex.java:162)
at org.joni.Regex.(Regex.java:139)
at org.joni.Regex.(Regex.java:125)
at org.joni.Regex.(Regex.java:109)
at net.stoerr.grokconstructor.JoniRegex.(JoniRegex.scala:16)
at net.stoerr.grokconstructor.incremental.IncrementalConstructionStepView.(IncrementalConstructionStepView.scala:22)
at net.stoerr.grokconstructor.webframework.WebDispatcher.giveView(WebDispatcher.scala:58)
at net.stoerr.grokconstructor.webframework.WebDispatcher.doGet(WebDispatcher.scala:32)
at net.stoerr.grokconstructor.webframework.WebDispatcher.doPost(WebDispatcher.scala:26)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166)
at com.google.apphosting.utils.servlet.ParseBlobUploadFilter.doFilter(ParseBlobUploadFilter.java:125)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter(SaveSessionFilter.java:35)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at com.google.apphosting.utils.servlet.JdbcMySqlConnectionCleanupFilter.doFilter(JdbcMySqlConnectionCleanupFilter.java:60)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:43)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle(AppVersionHandlerMap.java:257)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:923)
at com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable(RpcRequestParser.java:76)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:146)
at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:482)
at com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:437)
at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:444)
at com.google.tracing.CurrentContext.runInContext(CurrentContext.java:230)
at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:308)
at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:300)
at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:441)
at com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:234)
at java.lang.Thread.run(Thread.java:745)

Test input:
19/Jul/2016:20:43:42 +0000

Test Grok String:
%{HTTPDATE:@logtimestamp}

Results:

19/Jul/2016:20:43:42 +0000
NOT MATCHED. The longest regex prefix matching the beginning of this line is as follows:
prefix  
after match:    19/Jul/2016:20:43:42 +0000

Remove '@' sign:
%{HTTPDATE:logtimestamp}

Results after Removing '@' sign:

19/Jul/2016:20:43:42 +0000
MATCHED
logtimestamp    19/Jul/2016:20:43:42·+0000

This is functionality that Logstash supports.

According to logstash-patterns-core/patterns/haproxy there is a HAPROXYHTTPBASE pattern, but testing this on http://grokconstructor.appspot.com/do/match shows an error:
"This grok pattern has an unknown name HAPROXYHTTPBASE : %{HAPROXYHTTPBASE}"
Extracting HAPROXYHTTPBASE's content and using that as a patter works fine.

Firstly, thanks for building this. It works pretty well and I think is better than others I have found as it allows you to enter multiple log lines to test against.

To recreate the issue:

-Go to http://grokconstructor.appspot.com/do/match
-(Optional) Enter a log to match, pattern, and multiline pattern.
-Check the "negate the multiline regex" checkbox.
-Click on "Go!" button near the top of the page.
-Issue: The checkbox is no longer checked when the results page loads. This means that for each subsequent test you need to check that box again if you want to negate the multiline pattern.

Description: Ubuntu 18.10
Release: 18.10
Codename: cosmic

Using the command:

docker build -t grokconstructor .

I get the following error:

Downloading: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-junit4/2.17/surefire-junit4-2.17.jar
Downloaded: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-junit4/2.17/surefire-junit4-2.17.jar (63 KB at 52.7 KB/sec)

T E S T S

Error: Could not find or load main class org.apache.maven.surefire.booter.ForkedBooter

Results :

Tests run: 0, Failures: 0, Errors: 0, Skipped: 0

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 24:27 min
[INFO] Finished at: 2019-03-25T19:21:19+00:00
[INFO] Final Memory: 16M/60M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on project GrokConstructor: Execution default-test of goal org.apache.maven.plugins:maven-surefire-plugin:2.17:test failed: The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
[ERROR] Command was /bin/sh -c cd /app && /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -jar /app/target/surefire/surefirebooter4385982829423115196.jar /app/target/surefire/surefire6109931819202034881tmp /app/target/surefire/surefire_05629268019846747388tmp
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
The command '/bin/sh -c mvn clean install' returned a non-zero code: 1

Create a version with embedded tomcat since this is easier to run as maven starting google app devserver

input lines

LogSquad: Found USQLevel Count : 21
LogSquad: Found USQLayer Count : 191

1. click go
2. use radio string matching »LogSquad:·Found·USQ«
3. click continue
4. fill in text box Optional: give name for the grok expression to retrieve it's match value with count_type
5. use radio grok pattern %{WORD}
6. click continue
7. Crash occurs

OUCH!
I'm sorry, but you have encountered a bug or missing nice display of an error message in the application. If you can't guess the problem from the error message, please contact Hans-Peter Störr (www.stoerr.net [email protected]) with a copy of this page, or open an issue on https://github.com/stoerr/GrokConstructor/issues .

Please remember that you can always press the back button to fix what was wrong - there is no state on the server, only in the page shown in the browser.

      

Error message: java.util.NoSuchElementException: None.get

Time: Sun Apr 11 18:32:31 UTC 2021

RequestId: null

Request Info:
ReqInfo for 607340bf00ff0639f41e296a820001737e67726f6b636f6e7374727563746f7200013600010104 : {
  "_url":"/do/constructionstep",
  "groklibs":[
    "firewalls",
    "aws",
    "bro",
    "exim",
    "bind",
    "haproxy",
    "linux-syslog",
    "squid",
    "mcollective-patterns",
    "bacula",
    "postgresql",
    "java",
    "maven",
    "grok-patterns",
    "httpd",
    "redis",
    "nagios",
    "rails",
    "mongodb",
    "ruby",
    "mcollective",
    "junos"
  ],
  "loglines":[
    "LogSquad: Found USQLevel Count : 21\r\nLogSquad: Found USQLayer Count : 191"
  ],
  "nameOfNextPart":[
    "count_type"
  ],
  "nextPart":[
    "%{WORD}"
  ],
  "pattern":[
    "\\ALogSquad: Found USQ",
    "\\ALogSquad: Found USQ"
  ],
  "submit":[
    "Continue!"
  ]
}


java.util.NoSuchElementException: None.get
	at scala.None$.get(Option.scala:347)
	at scala.None$.get(Option.scala:345)
	at net.stoerr.grokconstructor.incremental.IncrementalConstructionStepView$$anonfun$2.apply(IncrementalConstructionStepView.scala:28)
	at net.stoerr.grokconstructor.incremental.IncrementalConstructionStepView$$anonfun$2.apply(IncrementalConstructionStepView.scala:26)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
	at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
	at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
	at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
	at scala.collection.AbstractTraversable.map(Traversable.scala:104)
	at net.stoerr.grokconstructor.incremental.IncrementalConstructionStepView.(IncrementalConstructionStepView.scala:25)
	at net.stoerr.grokconstructor.webframework.WebDispatcher.giveView(WebDispatcher.scala:81)
	at net.stoerr.grokconstructor.webframework.WebDispatcher.doGet(WebDispatcher.scala:53)
	at net.stoerr.grokconstructor.webframework.WebDispatcher.doPost(WebDispatcher.scala:33)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780)
	at com.google.apphosting.utils.servlet.JdbcMySqlConnectionCleanupFilter.doFilter(JdbcMySqlConnectionCleanupFilter.java:60)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
	at com.google.apphosting.runtime.jetty9.ParseBlobUploadHandler.handle(ParseBlobUploadHandler.java:111)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1182)
	at com.google.apphosting.runtime.jetty9.AppEngineWebAppContext.doHandle(AppEngineWebAppContext.java:187)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at com.google.apphosting.runtime.jetty9.AppVersionHandlerMap.handle(AppVersionHandlerMap.java:293)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
	at org.eclipse.jetty.server.Server.handle(Server.java:539)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)
	at com.google.apphosting.runtime.jetty9.RpcConnection.handle(RpcConnection.java:216)
	at com.google.apphosting.runtime.jetty9.RpcConnector.serviceRequest(RpcConnector.java:81)
	at com.google.apphosting.runtime.jetty9.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:134)
	at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.dispatchServletRequest(JavaRuntime.java:782)
	at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.dispatchRequest(JavaRuntime.java:745)
	at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:715)
	at com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:261)
	at java.lang.Thread.run(Thread.java:748)

Hello Team,

need to parsing a json log using grok filter but unable to perform can any one help me to done this.

2018-08-21 09:45:03 | {"timeStamp":{"hour":9,"minute":45,"second":3,"nano":451000000},"logLevel":"INFO","user":"csrt2024","role":"[com.orange.csr.auth.Role@258371ca, com.orange.csr.auth.Role@7b343aaa]","mco":null,"type":"TECH","searchID":null,"service":null,"operation":null,"operationLevel":null,"operationDetails":"public org.springframework.http.HttpStatus com.orange.csr.service.impl.JiraServiceImpl.createJiraRating(java.lang.String,java.lang.String,java.lang.String,short)","status":true,"duration":4566,"input":["csrt2024","others","[CSRTool][User : csrt2024][Role : OFR][Rate : 2]\n\nQuestion:What went Wrong?\nAnswer:true,true,\nOthers:nullQuestion:Is this about any specific service?\nAnswer: OCPRO,\nOthers:nullQuestion:Would you like to suggest some improvement?\nAnswer:\nOthers:null",2]}

Thanks for the valuable tool. I stumbled across this issue, today, though:

This is my pattern:

(?m)^%{TIMESTAMP_ISO8601:timestamp} %{SYSLOGHOST:host} \[%{PROG:program}\]: \[%{NOTSPACE:application_thread}\] (?<loglevel>ALL|DEBUG|ERROR|FATAL|INFO|OFF|TRACE|WARN) %{JAVACLASS:class} - %{GREEDYDATA:message}?

This is some test data:

2015-04-10T06:57:41+02:00 192.168.205.14 [myapp]: [CD-DocumentServerThread-1] INFO d.c.c.d.job.StatusAwareTask - Executing something
2015-04-10T06:57:41+02:00 192.168.205.14 [myapp]: [CD-DocumentServerThread-1] DEBUG d.c.c.d.job.StatusAwareTask - Something took 0.114 seconds
2015-04-10T06:57:41+02:00 192.168.205.13 [otherapp]: [http-nio-8081-exec-28] ERROR d.c.c.r.p.m.ThrowableMapper - Something went wrong
java.lang.NullPointerException: null
2015-04-10T06:57:41+02:00 192.168.205.13 [myapp]: [CD-DocumentServer-Dispatcher] DEBUG d.c.c.d.server.Dispatcher - Received something

I noticed that when I put this into the multiline filter, with negate checked:

^%{TIMESTAMP_ISO8601}%{SPACE}

all lines get sucked into the first one, mangled into the message field.

Next I tried this to narrow down my problem:

^2015-04-10T06:57:41\+02:00 
                           ^ There is a space here

which works correctly, joining the java.lang.NullPointerException: null line into the previous one, leaving the rest intact.

Finally, this also breaks:

^2015-04-10T06:57:41\+02:00%{SPACE}

So it seems, the predefined library patterns are not working with the multiline feature. I could not find if this is not supposed to work (as per logstash documentation it seems it should), so I assume this is a bug in your implementation?