Comments (19)
lmangani
commented on July 28, 2024
Could you paste your full pastash config? Which version of Loki are you running? (or is it the one above?)
from pastash.
lmangani
commented on July 28, 2024
The grok filter is indeed no longer part of the default build - but that's a parallel issue - please post the loki errors and/or capture the socket traffic with ngrep -W byline port 3100 -d any and attach the output during the failure.
from pastash.
lmangani
commented on July 28, 2024
I also suspect your target system might lack the build tools required to compile the grok module - this might cause the 2nd failure.
from pastash.
le-ross
commented on July 28, 2024
I started to work with promtail to see if I can make it work and now I suspect it's the same error 400
promtail_1 | level=error ts=2021-01-20T20:06:26.4438654Z caller=client.go:305 component=client host=loki:3100 msg=“final error sending batch” status=400 error=“server returned HTTP status 400 Bad Request (400): entry for stream '{filename=”/var/log/AAAAAA.log", job=“varlogs”}’ has timestamp too old: 2020-12-19 12:54:30 +0000 UTC"
I dont understand why this error. It's weird that I cannot ingest old logs.
For the second error I am still looking.
from pastash.
le-ross
commented on July 28, 2024
Am I supposed to install like the output_loki? because I dont see on npm or the in repo.
$ npm install --unsafe-perm -g @pastash/filter_grok npm ERR! code E404 npm ERR! 404 Not Found - GET https://registry.npmjs.org/@pastash%2ffilter_grok - Not found npm ERR! 404 npm ERR! 404 '@pastash/filter_grok@latest' is not in the npm registry. npm ERR! 404 You should bug the author to publish it (or use the name yourself!) npm ERR! 404 npm ERR! 404 Note that you can also install from a npm ERR! 404 tarball, folder, http url, or git url. npm ERR! A complete log of this run can be found in: npm ERR! C:\Users\LeChat\AppData\Roaming\npm-cache\_logs\2021-01-20T23_24_40_297Z-debug.log
from pastash.
le-ross
commented on July 28, 2024
This is the logs when I install pastash. I dont see any reference for filters or grok
from pastash.
lmangani
commented on July 28, 2024
@le-ross I must be honest i never tried running this on windows. In either case, you do not need grok, the error is elsewhere.
Please run ngrep -W byline port 3100 -d any and while running. Try inserting data into Loki (either way). Attach the output.
from pastash.
le-ross
commented on July 28, 2024
I will need to install ubuntu becase I am not able to find ngrep for windows
I tried to wrap the package in a docker container like this
FROM node:12
RUN npm install -g @pastash/pastash @pastash/output_loki
WORKDIR /app
CMD pastash --config_file=./pastash_loki.json
It doesnt make sense that I keep having this filter_grok error also. are you sure it's not missing somewhere? I need that grok otherwise all my logs are insert with today's timestamp
I was able to insert the logs with graylog instead of loki to see that all I am missing is the grok filter
from pastash.
le-ross
commented on July 28, 2024
tried under my ubuntu session. same grok error.
[Thu, 21 Jan 2021 20:03:38 GMT] NOTICE Starting pastash 1.0.71
[Thu, 21 Jan 2021 20:03:38 GMT] INFO Max http socket 100
[Thu, 21 Jan 2021 20:03:38 GMT] INFO Loading config file : ./pastash-config.json
[Thu, 21 Jan 2021 20:03:38 GMT] INFO File loaded, 4 urls found
[Thu, 21 Jan 2021 20:03:38 GMT] INFO Loading config : 7 urls
[Thu, 21 Jan 2021 20:03:38 GMT] INFO Initializing output Gelf
[Thu, 21 Jan 2021 20:03:38 GMT] INFO Start output to Gelf localhost:12201
[Thu, 21 Jan 2021 20:03:38 GMT] INFO Initializing output file
[Thu, 21 Jan 2021 20:03:38 GMT] INFO Start output to file ./tmp/file.log
[Thu, 21 Jan 2021 20:03:38 GMT] INFO Initializing filter AddHost
[Thu, 21 Jan 2021 20:03:38 GMT] INFO Initializing filter AddTimestamp
[Thu, 21 Jan 2021 20:03:38 GMT] INFO Initializing filter AddVersion
Loading npm module... @pastash/filter_grok
[Thu, 21 Jan 2021 20:03:38 GMT] ERROR Unable to load urls from command line
[Thu, 21 Jan 2021 20:03:38 GMT] ERROR TypeError: Cannot read property 'create' of undefined
at LogstashAgent.configure (/home/rossm/.npm-global/lib/node_modules/@pastash/pastash/lib/agent.js:143:21)
at LogstashAgent.<anonymous> (/home/rossm/.npm-global/lib/node_modules/@pastash/pastash/lib/agent.js:215:10)
at /home/rossm/.npm-global/lib/node_modules/@pastash/pastash/node_modules/async/dist/async.js:3096:16
at replenish (/home/rossm/.npm-global/lib/node_modules/@pastash/pastash/node_modules/async/dist/async.js:998:17)
at iterateeCallback (/home/rossm/.npm-global/lib/node_modules/@pastash/pastash/node_modules/async/dist/async.js:983:17)
at /home/rossm/.npm-global/lib/node_modules/@pastash/pastash/node_modules/async/dist/async.js:958:16
at LogstashAgent.<anonymous> (/home/rossm/.npm-global/lib/node_modules/@pastash/pastash/lib/agent.js:265:5)
at LogstashAgent.<anonymous> (/home/rossm/.npm-global/lib/node_modules/@pastash/pastash/lib/agent.js:220:7)
at LogstashAgent.<anonymous> (/home/rossm/.npm-global/lib/node_modules/@pastash/pastash/lib/agent.js:156:9)
at FilterAddVersion.<anonymous> (/home/rossm/.npm-global/lib/node_modules/@pastash/pastash/lib/lib/base_filter.js:36:5)
from pastash.
lmangani
commented on July 28, 2024
Do you have build-essential meta package installed?
from pastash.
lmangani
commented on July 28, 2024
Btw this still does not explain the error you're receiving you're receiving:
each time I insert logs I get [Wed, 20 Jan 2021 16:34:15 GMT] ERROR error Wrong HTTP Post return code: 400
You forgot to run the ngrep command to see what we're sending out. Is the date on your server in sync with the world?
from pastash.
le-ross
commented on July 28, 2024
Oh yeah, got distracted when I saw a image ready to use. I don't have the problem of grok with that image which is version 1.0.50
here is the log with a dummy file also attached
dummy.log
interface: any
filter: ( port 3100 ) and (ip || ip6)
####################################
T 127.0.0.1:50314 -> 127.0.0.1:3100 [AP] #36
POST /loki/api/v1/push HTTP/1.1.
Content-Type: application/json.
Content-Length: 354.
Host: localhost:3100.
Connection: close.
.
{"streams":[{"labels":"{path=\"./logs/dummy.log\",type=\"undefined\",host=\"****\"}","entries":[{"ts":"2021-01-21T21:40:06.820Z","line":"2021-01-20T14:01:29: Job status 87f7c37f-e8df-427f-b810-xxxx STOPPED RUNNING 2"},{"ts":"2021-01-21T21:40:06.820Z","line":"2021-01-20T14:01:31: Job status 87f7c37f-e8df-427f-b810-xxxx STOPPED RUNNING 2"}]}]}
########
T 172.19.0.1:51500 -> 172.19.0.3:3100 [AP] #44
POST /loki/api/v1/push HTTP/1.1.
Content-Type: application/json.
Content-Length: 354.
Host: localhost:3100.
Connection: close.
.
{"streams":[{"labels":"{path=\"./logs/dummy.log\",type=\"undefined\",host=\"****\"}","entries":[{"ts":"2021-01-21T21:40:06.820Z","line":"2021-01-20T14:01:29: Job status 87f7c37f-e8df-427f-b810-xxxx STOPPED RUNNING 2"},{"ts":"2021-01-21T21:40:06.820Z","line":"2021-01-20T14:01:31: Job status 87f7c37f-e8df-427f-b810-xxxx STOPPED RUNNING 2"}]}]}
#
T 172.19.0.1:51500 -> 172.19.0.3:3100 [AP] #45
POST /loki/api/v1/push HTTP/1.1.
Content-Type: application/json.
Content-Length: 354.
Host: localhost:3100.
Connection: close.
.
{"streams":[{"labels":"{path=\"./logs/dummy.log\",type=\"undefined\",host=\"****\"}","entries":[{"ts":"2021-01-21T21:40:06.820Z","line":"2021-01-20T14:01:29: Job status 87f7c37f-e8df-427f-b810-xxxx STOPPED RUNNING 2"},{"ts":"2021-01-21T21:40:06.820Z","line":"2021-01-20T14:01:31: Job status 87f7c37f-e8df-427f-b810-xxxx STOPPED RUNNING 2"}]}]}
###
T 172.19.0.3:3100 -> 172.19.0.1:51500 [AP] #48
HTTP/1.1 204 No Content.
Date: Thu, 21 Jan 2021 21:40:08 GMT.
Connection: close.
.
#
T 172.19.0.3:3100 -> 172.19.0.1:51500 [AP] #49
HTTP/1.1 204 No Content.
Date: Thu, 21 Jan 2021 21:40:08 GMT.
Connection: close.
.
###
T 127.0.0.1:3100 -> 127.0.0.1:50314 [AP] #52
HTTP/1.1 204 No Content.
Date: Thu, 21 Jan 2021 21:40:08 GMT.
Connection: close.
.
##########
from pastash.
le-ross
commented on July 28, 2024
here is my docker-compose in case
version: "3"
networks:
loki:
services:
loki:
image: grafana/loki:latest
ports:
- 3100:3100
command: -config.file=/etc/loki/local-config.yaml
networks:
- loki
promtail:
image: grafana/promtail:latest
command: -config.file=/etc/promtail/config.yml
networks:
- loki
grafana:
image: grafana/grafana:latest
ports:
- "3000:3000"
networks:
- loki
from pastash.
lmangani
commented on July 28, 2024
204 is good. From the logs it looks as if they were accepted
HTTP/1.1 204 No Content.
Date: Thu, 21 Jan 2021 21:40:08 GMT.
Connection: close.
from pastash.
le-ross
commented on July 28, 2024
manged to get ngrep with the error 400
interface: any
filter: ( port 3100 ) and (ip || ip6)
#######
T 172.19.0.5:60510 -> 172.19.0.4:3100 [AP] #7
POST /loki/api/v1/push HTTP/1.1.
Content-Type: application/json.
Content-Length: 852.
Host: loki:3100.
Connection: close.
.
{"streams":[{"labels":"{path=\"/var/log/dummy.asdd.log\",type=\"undefined\",host=\"7d46bb4ba73c\"}","entries":[{"ts":"2021-01-22T01:38:13.725Z","line":"T01:57:49: 12 Sep 01:57:49 - [info] [udp out:26d9cdad.cf09d2] udp re-use socket: 4562 -> :8363"},{"ts":"2021-01-22T01:38:13.727Z","line":"2020-09-12T01:57:49: 12 Sep 01:57:49 - [info] [udp out:23670e42.f33822] udp re-use socket: -> 127.0.0.1:6250"},{"ts":"2021-01-22T01:38:13.727Z","line":"2020-09-12T01:57:49: 12 Sep 01:57:49 - [info] [udp out:af1ac3dc.034b7] udp re-use socket: 4562 -> :8363"},{"ts":"2021-01-22T01:38:13.727Z","line":"2020-09-12T01:57:49: 12 Sep 01:57:49 - [info] [udp out:67ac2c51.a054f4] udp re-use socket: -> localhost:4562"},{"ts":"2021-01-22T01:38:13.728Z","line":"2020-09-12T01:57:49: 12 Sep 01:57:49 - [info] [udp out:e9198db3.2a23c] udp re-use socket: 4562 -> :8363"}]}]}
#
T 172.19.0.5:60510 -> 172.19.0.4:3100 [AP] #8
POST /loki/api/v1/push HTTP/1.1.
Content-Type: application/json.
Content-Length: 852.
Host: loki:3100.
Connection: close.
.
{"streams":[{"labels":"{path=\"/var/log/dummy.asdd.log\",type=\"undefined\",host=\"7d46bb4ba73c\"}","entries":[{"ts":"2021-01-22T01:38:13.725Z","line":"T01:57:49: 12 Sep 01:57:49 - [info] [udp out:26d9cdad.cf09d2] udp re-use socket: 4562 -> :8363"},{"ts":"2021-01-22T01:38:13.727Z","line":"2020-09-12T01:57:49: 12 Sep 01:57:49 - [info] [udp out:23670e42.f33822] udp re-use socket: -> 127.0.0.1:6250"},{"ts":"2021-01-22T01:38:13.727Z","line":"2020-09-12T01:57:49: 12 Sep 01:57:49 - [info] [udp out:af1ac3dc.034b7] udp re-use socket: 4562 -> :8363"},{"ts":"2021-01-22T01:38:13.727Z","line":"2020-09-12T01:57:49: 12 Sep 01:57:49 - [info] [udp out:67ac2c51.a054f4] udp re-use socket: -> localhost:4562"},{"ts":"2021-01-22T01:38:13.728Z","line":"2020-09-12T01:57:49: 12 Sep 01:57:49 - [info] [udp out:e9198db3.2a23c] udp re-use socket: 4562 -> :8363"}]}]}
###
T 172.19.0.4:3100 -> 172.19.0.5:60510 [AP] #11
HTTP/1.1 400 Bad Request.
Content-Type: text/plain; charset=utf-8.
X-Content-Type-Options: nosniff.
Date: Fri, 22 Jan 2021 01:38:14 GMT.
Content-Length: 101.
Connection: close.
.
error parsing labels: parse error at line 1, col 2: syntax error: unexpected }, expecting IDENTIFIER
#
T 172.19.0.4:3100 -> 172.19.0.5:60510 [AP] #12
HTTP/1.1 400 Bad Request.
Content-Type: text/plain; charset=utf-8.
X-Content-Type-Options: nosniff.
Date: Fri, 22 Jan 2021 01:38:14 GMT.
Content-Length: 101.
Connection: close.
.
error parsing labels: parse error at line 1, col 2: syntax error: unexpected }, expecting IDENTIFIER
########^Cexit
from pastash.
le-ross
commented on July 28, 2024
it's enough to throw that error with that data
2020-09-12T01:57:49: 12 Sep 01:57:49 - [info] A
2020-09-12T01:57:50: 12 Sep 01:57:49 - [info] B
2020-09-12T01:57:51: 12 Sep 01:57:49 - c
2020-09-12T01:57:52: 12 Sep 01:57:49 d
2020-09-12T01:57:53: 12 Sep 01:e
2020-09-12T01:57:54: 12 Sef
2020-09-12T01:57:55: g
from pastash.
spady7
commented on July 28, 2024
I get this error too. Is this resolved?
[Wed, 02 Aug 2023 09:51:09 GMT] NOTICE Starting pastash 1.0.75 [Wed, 02 Aug 2023 09:51:09 GMT] INFO Max http socket 100 [Wed, 02 Aug 2023 09:51:09 GMT] INFO Loading config file : /opt/pastash/sonus.conf [Wed, 02 Aug 2023 09:51:09 GMT] INFO File loaded, 7 urls found [Wed, 02 Aug 2023 09:51:09 GMT] INFO Loading config : 10 urls [Wed, 02 Aug 2023 09:51:09 GMT] INFO Initializing output HEP/EEP Server [Wed, 02 Aug 2023 09:51:09 GMT] INFO Start output to HEP udp to 127.0.0.1:9060 [Wed, 02 Aug 2023 09:51:09 GMT] INFO Initializing filter AddHost [Wed, 02 Aug 2023 09:51:09 GMT] INFO Initializing filter AddTimestamp [Wed, 02 Aug 2023 09:51:09 GMT] INFO Initializing filter AddVersion [Wed, 02 Aug 2023 09:51:09 GMT] INFO Initializing filter Multiline [Wed, 02 Aug 2023 09:51:09 GMT] INFO Initialized multiline filter with start_line_regex: /^\[\d{4}-\d{2}-\d{2}/, flags: Loading npm module... @pastash/filter_grok [Wed, 02 Aug 2023 09:51:09 GMT] ERROR Unable to load urls from command line [Wed, 02 Aug 2023 09:51:09 GMT] ERROR TypeError: Cannot read properties of undefined (reading 'create') at LogstashAgent.configure (/usr/local/lib/node_modules/@pastash/pastash/lib/agent.js:143:21) at LogstashAgent.<anonymous> (/usr/local/lib/node_modules/@pastash/pastash/lib/agent.js:215:10) at /usr/local/lib/node_modules/@pastash/pastash/node_modules/async/dist/async.js:3096:16 at replenish (/usr/local/lib/node_modules/@pastash/pastash/node_modules/async/dist/async.js:998:17) at iterateeCallback (/usr/local/lib/node_modules/@pastash/pastash/node_modules/async/dist/async.js:983:17) at /usr/local/lib/node_modules/@pastash/pastash/node_modules/async/dist/async.js:958:16 at LogstashAgent.<anonymous> (/usr/local/lib/node_modules/@pastash/pastash/lib/agent.js:265:5) at LogstashAgent.<anonymous> (/usr/local/lib/node_modules/@pastash/pastash/lib/agent.js:220:7) at LogstashAgent.<anonymous> (/usr/local/lib/node_modules/@pastash/pastash/lib/agent.js:156:9) at FilterMultiline.<anonymous> (/usr/local/lib/node_modules/@pastash/pastash/lib/lib/base_filter.js:36:5)
from pastash.
Dletta
commented on July 28, 2024
Could you provide the steps you took to get to this point? What config are
you using?
What node version are you using?
…On Wed, Aug 2, 2023, 04:56 spady7 ***@***.***> wrote:
I get this error too. Is this resolved?
[Wed, 02 Aug 2023 09:51:09 GMT] NOTICE Starting pastash 1.0.75 [Wed, 02
Aug 2023 09:51:09 GMT] INFO Max http socket 100 [Wed, 02 Aug 2023 09:51:09
GMT] INFO Loading config file : /opt/pastash/sonus.conf [Wed, 02 Aug 2023
09:51:09 GMT] INFO File loaded, 7 urls found [Wed, 02 Aug 2023 09:51:09
GMT] INFO Loading config : 10 urls [Wed, 02 Aug 2023 09:51:09 GMT] INFO
Initializing output HEP/EEP Server [Wed, 02 Aug 2023 09:51:09 GMT] INFO
Start output to HEP udp to 127.0.0.1:9060 [Wed, 02 Aug 2023 09:51:09 GMT]
INFO Initializing filter AddHost [Wed, 02 Aug 2023 09:51:09 GMT] INFO
Initializing filter AddTimestamp [Wed, 02 Aug 2023 09:51:09 GMT] INFO
Initializing filter AddVersion [Wed, 02 Aug 2023 09:51:09 GMT] INFO
Initializing filter Multiline [Wed, 02 Aug 2023 09:51:09 GMT] INFO
Initialized multiline filter with start_line_regex: /^\[\d{4}-\d{2}-\d{2}/,
flags: Loading npm module... @pastash/filter_grok [Wed, 02 Aug 2023
09:51:09 GMT] ERROR Unable to load urls from command line [Wed, 02 Aug 2023
09:51:09 GMT] ERROR TypeError: Cannot read properties of undefined (reading
'create') at LogstashAgent.configure
***@***.***/pastash/lib/agent.js:143:21) at
LogstashAgent.<anonymous>
***@***.***/pastash/lib/agent.js:215:10) at
***@***.***/pastash/node_modules/async/dist/async.js:3096:16
at replenish
***@***.***/pastash/node_modules/async/dist/async.js:998:17)
at iterateeCallback
***@***.***/pastash/node_modules/async/dist/async.js:983:17)
at
***@***.***/pastash/node_modules/async/dist/async.js:958:16
at LogstashAgent.<anonymous>
***@***.***/pastash/lib/agent.js:265:5) at
LogstashAgent.<anonymous>
***@***.***/pastash/lib/agent.js:220:7) at
LogstashAgent.<anonymous>
***@***.***/pastash/lib/agent.js:156:9) at
FilterMultiline.<anonymous>
***@***.***/pastash/lib/lib/base_filter.js:36:5)
—
Reply to this email directly, view it on GitHub
<#79 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABLWWESIR5H3G5Z64T64IB3XTIP4RANCNFSM4WK6DRVA>
.
You are receiving this because you are subscribed to this thread.Message
ID: ***@***.***>
from pastash.
spady7
commented on July 28, 2024
Hi @Dletta i need to use "pastash" to parse logs from Sonus/Ribbon SBC.
i took configuration from wiki. Btw this is my conf:
`input {
file {
path => "/tmp/webui.log"
}
}
filter {
multiline {
start_line_regex => /^[\d{4}-\d{2}-\d{2}/
}
grok {
match => '(?m)[%{TIMESTAMP_ISO8601:timestamp}] %{WORD:pid} %{WORD:seq} \n%{GREEDYDATA:payload}'
}
regex {
regex => /From:.@(.):(\d+)/
fields => [srcIp,srcPort]
}
regex {
regex => /To:.@(.):(\d+)/
fields => [dstIp,dstPort]
}
regex {
regex => /Call-I.: (.)/
fields => [correlation_id]
}
}
output {
if [tags] != "_grokparsefailure" {
hep {
host => '127.0.0.1'
port => 9060
hep_id => 2222
hep_type => 1
}
}
}`
And when i try
pastash --config_file=/opt/pastash/sonus.conf
i get errors.
from pastash.
Related Issues (20)
- no Audiocodes SBC syslog seen HOT 1
- Journald Log support HOT 4
- syslog input filtering not working [podman] HOT 11
- HEP relay/forwarder not working HOT 3
- Issue with Janus Events in Loki HOT 2
- paStash installation failed
- Pastash unable to install with Node Version 16. HOT 2
- app_audiocodes Error: Cannot find module 'filters/filter_app_audiocodes' HOT 5
- Add TCP Support for output Plugin `HEP`
- Make HEP network setting configurable HOT 1
- Audiocodes SBC ERROR failed parsing Outgoing SIP. Cache on! HOT 28
- SDP issues and etc with Audiocodes filter HOT 18
- paStash docker and Audiocodes optional filter HOT 4
- UDP input/output issue HOT 2
- Audiocodes SBC syslog errors HOT 13
- Amazon MSK SASL/SCRAM auth method support HOT 2
- Installation fails on Debian 10 HOT 11
- Audiocodes Plugin: ERROR TypeError: Cannot read property '1' of null HOT 6
- app_cisco (Cisco ISR) filter plugin DNS lookup error HOT 14
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pastash.