insert host name to ip addresses about pastash HOT 20 CLOSED

Do not use pm2 for this test. Just run pastash --config_file=/path/to/pastash.conf and type the IP manually pressing enter

from pastash.

I see the issue. You don't have alignment between the select (host) and the target_field.
Update to @pastash/[email protected] which allows those being different and just picks the first result.

from pastash.

Hello @Alisher-Nabiev

this is not working well for us.

Please outline the issue in as much detail as possible. What is working, what is not, what are the expected vs. current results, etc.

from pastash.

ok, so this is our table:

this is filter config:
sqlite {
db => '/home/monogoto/userdata_qa.db'
query => 'SELECT host_name from ip_list WHERE ip_number = (?) limit 1;'
source_field => l4_src_port
target_field => host_name
}

and we don't see host_name in the index.
log:
[TAILING] Tailing last 15 lines for [0] process (change the value with --lines option)
/root/.pm2/logs/pastash-logs-error.log last 15 lines:
/root/.pm2/logs/pastash-logs-out.log last 15 lines:
0|pastash- | [Wed, 03 Jul 2024 10:37:51 GMT] INFO test query! SELECT host_name from ip_list WHERE ip_number = (?) limit 1; 8883
0|pastash- | [Wed, 03 Jul 2024 10:37:51 GMT] INFO test query! SELECT host_name from ip_list WHERE ip_number = (?) limit 1; 8883
0|pastash- | [Wed, 03 Jul 2024 10:37:51 GMT] INFO test query! SELECT host_name from ip_list WHERE ip_number = (?) limit 1; 8883
0|pastash- | [Wed, 03 Jul 2024 10:37:51 GMT] INFO test query! SELECT host_name from ip_list WHERE ip_number = (?) limit 1; 8883
0|pastash- | [Wed, 03 Jul 2024 10:37:51 GMT] INFO test query! SELECT host_name from ip_list WHERE ip_number = (?) limit 1; 8883
0|pastash- | [Wed, 03 Jul 2024 10:37:51 GMT] INFO test query! SELECT host_name from ip_list WHERE ip_number = (?) limit 1; 8883
0|pastash- | [Wed, 03 Jul 2024 10:37:51 GMT] INFO test query! SELECT host_name from ip_list WHERE ip_number = (?) limit 1; 8883
0|pastash- | [Wed, 03 Jul 2024 10:37:51 GMT] INFO test query! SELECT host_name from ip_list WHERE ip_number = (?) limit 1; 8883
0|pastash- | [Wed, 03 Jul 2024 10:37:51 GMT] INFO test query! SELECT host_name from ip_list WHERE ip_number = (?) limit 1; 8883
0|pastash- | [Wed, 03 Jul 2024 10:37:51 GMT] INFO test query! SELECT host_name from ip_list WHERE ip_number = (?) limit 1; 8883
0|pastash- | [Wed, 03 Jul 2024 10:37:51 GMT] INFO test query! SELECT host_name from ip_list WHERE ip_number = (?) limit 1; 8883
0|pastash- | [Wed, 03 Jul 2024 10:37:51 GMT] INFO test query! SELECT host_name from ip_list WHERE ip_number = (?) limit 1; 8883
0|pastash- | [Wed, 03 Jul 2024 10:37:51 GMT] INFO test query! SELECT host_name from ip_list WHERE ip_number = (?) limit 1; 8883
0|pastash- | [Wed, 03 Jul 2024 10:37:51 GMT] INFO test query! SELECT host_name from ip_list WHERE ip_number = (?) limit 1; 8883
0|pastash- | [Wed, 03 Jul 2024 10:37:51 GMT] INFO test query! SELECT host_name from ip_list WHERE ip_number = (?) limit 1; 8883

0|pastash-logs | [Wed, 03 Jul 2024 10:37:51 GMT] INFO test query! SELECT host_name from ip_list WHERE ip_number = (?) limit 1; 8883
0|pastash-logs | [Wed, 03 Jul 2024 10:37:51 GMT] INFO test query! SELECT host_name from ip_list WHERE ip_number = (?) limit 1; 8883
0|pastash-logs | [Wed, 03 Jul 2024 10:37:51 GMT] INFO test query! SELECT host_name from ip_list WHERE ip_number = (?) limit 1; 8883
0|pastash-logs | [Wed, 03 Jul 2024 10:37:51 GMT] INFO test query! SELECT host_name from ip_list WHERE ip_number = (?) limit 1; 8883
0|pastash-logs | [Wed, 03 Jul 2024 10:37:51 GMT] INFO test query! SELECT host_name from ip_list WHERE ip_number = (?) limit 1; 8883
0|pastash-logs | [Wed, 03 Jul 2024 10:37:51 GMT] INFO test query! SELECT host_name from ip_list WHERE ip_number = (?) limit 1; 8883

from pastash.

The following test succeeds for me. I created a dummy sqlite db

.open /tmp/test.db

CREATE TABLE IF NOT EXISTS lookup (
	id INTEGER PRIMARY KEY,
   	ip TEXT NOT NULL,
	host TEXT DEFAULT 0
);

INSERT INTO lookup (ip, host) VALUES('127.0.0.1', 'localhost');
INSERT INTO lookup (ip, host) VALUES('127.0.0.2', 'docker');

Then installed the sqlite plugin

npm install -g @pastash/[email protected]

And execute the following test recipe:

input {
  stdin{}
}

filter {
  sqlite {
    db => "/tmp/test.db"
    query => "SELECT host from lookup WHERE ip=(?) limit 1;"
    source_field => "message"
    target_field => "host"
  }
}
output {
  stdout{}
}

I get the expected results:

[Wed, 03 Jul 2024 14:19:38 GMT] INFO Config loaded.
127.0.0.1
[Wed, 03 Jul 2024 14:19:40 GMT] INFO 127.0.0.1 127.0.0.1 { host: 'localhost' }
[STDOUT] {
  "source": "stdin",
  "message": "127.0.0.1",
  "host": "localhost",
  "@timestamp": "2024-07-03T14:19:40.785Z",
  "@version": "1",
  "correlation_id": "nothing"
}
127.0.0.2
[Wed, 03 Jul 2024 14:19:43 GMT] INFO 127.0.0.2 127.0.0.2 { host: 'docker' }
[STDOUT] {
  "source": "stdin",
  "message": "127.0.0.2",
  "host": "docker",
  "@timestamp": "2024-07-03T14:19:43.408Z",
  "@version": "1",
  "correlation_id": "nothing"
}

from pastash.

Hi, thank you for your response.

i tried to create a new database with the same configuration as your example. I just substituted my hosts.
test:
root@ip-:/home/ubuntu# sqlite3 /home/ubuntu/qa.db "SELECT host FROM lookup WHERE ip = '8.8.8.8';"
Google-DNS

but nothing on elastic side no new field in the indecies.

log:
[TAILING] Tailing last 15 lines for [all] processes (change the value with --lines option)
/root/.pm2/pm2.log last 15 lines:
PM2 | 2024-07-03T14:37:18: PM2 log: Stopping app:pastash-logs id:0
PM2 | 2024-07-03T14:37:18: PM2 log: App [pastash-logs:0] exited with code [1] via signal [SIGINT]
PM2 | 2024-07-03T14:37:18: PM2 log: pid=1941659 msg=process killed
PM2 | 2024-07-03T14:37:18: PM2 log: App [pastash-logs:0] starting in -fork mode-
PM2 | 2024-07-03T14:37:18: PM2 log: App [pastash-logs:0] online
PM2 | 2024-07-03T14:43:41: PM2 log: Stopping app:pastash-logs id:0
PM2 | 2024-07-03T14:43:41: PM2 log: App [pastash-logs:0] exited with code [1] via signal [SIGINT]
PM2 | 2024-07-03T14:43:41: PM2 log: pid=1942553 msg=process killed
PM2 | 2024-07-03T14:43:41: PM2 log: App [pastash-logs:0] starting in -fork mode-
PM2 | 2024-07-03T14:43:41: PM2 log: App [pastash-logs:0] online
PM2 | 2024-07-03T14:58:15: PM2 log: Stopping app:pastash-logs id:0
PM2 | 2024-07-03T14:58:15: PM2 log: App [pastash-logs:0] exited with code [1] via signal [SIGINT]
PM2 | 2024-07-03T14:58:15: PM2 log: pid=1947045 msg=process killed
PM2 | 2024-07-03T14:58:15: PM2 log: App [pastash-logs:0] starting in -fork mode-
PM2 | 2024-07-03T14:58:15: PM2 log: App [pastash-logs:0] online

/root/.pm2/logs/pastash-logs-error.log last 15 lines:
/root/.pm2/logs/pastash-logs-out.log last 15 lines:
0|pastash- | [Wed, 03 Jul 2024 14:58:16 GMT] INFO Initialized Omit filter!
0|pastash- | Loading npm module... @pastash/filter_sqlite
0|pastash- | [Wed, 03 Jul 2024 14:58:16 GMT] INFO Initializing filter sqlite
0|pastash- | [Wed, 03 Jul 2024 14:58:16 GMT] INFO Initializing Filter Sqlite3: Database {
0|pastash- | name: '/home/ubuntu/qa.db',
0|pastash- | open: true,
0|pastash- | inTransaction: false,
0|pastash- | readonly: false,
0|pastash- | memory: false
0|pastash- | }
0|pastash- | [Wed, 03 Jul 2024 14:58:16 GMT] INFO Initialized Filter Sqlite3
0|pastash- | Loading npm module... @pastash/input_netflowv9
0|pastash- | [Wed, 03 Jul 2024 14:58:16 GMT] INFO Initializing input Netflowv9
0|pastash- | [Wed, 03 Jul 2024 14:58:16 GMT] INFO Opening Netflowv9 socket on port 30001
0|pastash- | [Wed, 03 Jul 2024 14:58:16 GMT] INFO Config loaded.

from pastash.

Have you installed the same version?

npm install -g @pastash/[email protected]

I do not see any useful output. You might want to add an stdout{} block to see what's happening in the logs.

output {
  stdout{}
}

from pastash.

hi, yes we install the new version.

i will share with you the stdout{} in privet

from pastash.

As you can see above my example works and its served with a fully reproducible case and each step used.
Not sure what the issue might be on your system, but since it cannot be reproduced, there's not much we can do to help.

You should rather find out why your process is exiting with errors in loop

PM2 | 2024-07-03T15:04:04: PM2 log: App [pastash-logs:0] exited with code [1] via signal [SIGINT]

from pastash.

As you can see, the error occurred yesterday, possibly because I restarted the PM2 when making new changes in the configuration file.

Also, we receive data constantly in the Elastic, and we don't observe any unexpected exits from the PM2 process.

from pastash.

I'm sorry but the issue is on your side and our example appears to be working - you need to investigate your setup, as without more information, I'm afraid there's nothing we can do. Are you even sure the pastash process has permissions to read your .db files?

from pastash.

yes i gave to the db -rwx permission and still same issue

from pastash.

Is there any reason why you did not attempt running our provided "demo" example? It might reveal the issue.

from pastash.

what do you mean by your example?

i created the same DB as you are but only with my IPs.

and i put the same filter on the config file.

if i am running the query on the instance it's retrieving the correct hostname.

but on the elastic side, i can't see it.

If you want me to test it in a different way, please let me know.

from pastash.

Our example is stand alone. It uses user input string (IP) to validate the pipeline. It has no elastic or other transformations. This is why in case of issues its useful to troubleshoot a single component as opposed to the full configuration. Try it and confirm it works before moving ahead with integration in your workflow.

from pastash.

did you reacive my privet emails with my configuration?

second;
i created a new pm2 process with your example

[TAILING] Tailing last 15 lines for [2] process (change the value with --lines option)
/root/.pm2/logs/dumy-error.log last 15 lines:
/root/.pm2/logs/dumy-out.log last 15 lines:
2|dumy | [Thu, 04 Jul 2024 08:25:36 GMT] INFO Initializing filter AddHost
2|dumy | [Thu, 04 Jul 2024 08:25:36 GMT] INFO Initializing filter AddTimestamp
2|dumy | [Thu, 04 Jul 2024 08:25:36 GMT] INFO Initializing filter AddVersion
2|dumy | Loading npm module... @pastash/filter_sqlite
2|dumy | [Thu, 04 Jul 2024 08:25:36 GMT] INFO Initializing filter sqlite
2|dumy | [Thu, 04 Jul 2024 08:25:36 GMT] INFO Initializing Filter Sqlite3: Database {
2|dumy | name: '/home/ubuntu/dumy_db.db',
2|dumy | open: true,
2|dumy | inTransaction: false,
2|dumy | readonly: false,
2|dumy | memory: false
2|dumy | }
2|dumy | [Thu, 04 Jul 2024 08:25:36 GMT] INFO Initialized Filter Sqlite3
2|dumy | [Thu, 04 Jul 2024 08:25:36 GMT] INFO Initializing input Stdin
2|dumy | [Thu, 04 Jul 2024 08:25:36 GMT] INFO Config loaded.

how did you pass the IP number to process? with input?

from pastash.

root@ip:/home/ubuntu# pastash --config_file=dumy_conf.conf
[Thu, 04 Jul 2024 08:56:49 GMT] NOTICE Starting pastash 1.0.82
[Thu, 04 Jul 2024 08:56:49 GMT] INFO Max http socket 100
[Thu, 04 Jul 2024 08:56:49 GMT] INFO Loading config file : dumy_conf.conf
[Thu, 04 Jul 2024 08:56:49 GMT] INFO File loaded, 3 urls found
[Thu, 04 Jul 2024 08:56:49 GMT] INFO Loading config : 6 urls
[Thu, 04 Jul 2024 08:56:49 GMT] INFO Initializing output Stdout
[Thu, 04 Jul 2024 08:56:49 GMT] INFO Initializing filter AddHost
[Thu, 04 Jul 2024 08:56:49 GMT] INFO Initializing filter AddTimestamp
[Thu, 04 Jul 2024 08:56:49 GMT] INFO Initializing filter AddVersion
Loading npm module... @pastash/filter_sqlite
[Thu, 04 Jul 2024 08:56:49 GMT] INFO Initializing filter sqlite
[Thu, 04 Jul 2024 08:56:49 GMT] INFO Initializing Filter Sqlite3: Database {
name: '/home/ubuntu/dumy_db.db',
open: true,
inTransaction: false,
readonly: false,
memory: false
}
[Thu, 04 Jul 2024 08:56:49 GMT] INFO Initialized Filter Sqlite3
[Thu, 04 Jul 2024 08:56:49 GMT] INFO Initializing input Stdin
[Thu, 04 Jul 2024 08:56:49 GMT] INFO Config loaded.
127.0.0.1
[STDOUT] {
"source": "stdin",
"message": "127.0.0.1",
"host": "localhost",
"@timestamp": "2024-07-04T08:56:53.164Z",
"@Version": "1"
}
127.0.0.2
[STDOUT] {
"source": "stdin",
"message": "127.0.0.2",
"host": "docker",
"@timestamp": "2024-07-04T08:56:58.756Z",
"@Version": "1"
}

from pastash.

As you can see it works. Perhaps try to use a different target header and/or experiment with the order of filters

from pastash.

in the filter "target_field" is for pass the field to the output and from there move it to elastic?
any way i can debug this process?

as you can see here:

ip_proto {
source_field => l4_dst_port
target_field => service
custom => "20:FTP,21:FTP,22:ssh,23:telnet,25:smtp,69:tftp,53:dns"
}

target field is creating "service" field and passing this to elastic:

0|pastash-logs | [STDOUT] {
0|pastash-logs | "in_bytes": 200,
0|pastash-logs | "ipv4_src_addr": "8.8.8.8",
0|pastash-logs | "ipv4_dst_addr": "",
0|pastash-logs | "protocol": 17,
0|pastash-logs | "l4_src_port": 53,
0|pastash-logs | "l4_dst_port": 45719,
0|pastash-logs | "ipv4_next_hop": "",
0|pastash-logs | "tcp_flags": 0,
0|pastash-logs | "in_dst_mac": "02dc",
0|pastash-logs | "out_src_mac": "0ac*a10",
0|pastash-logs | "postNATSourceIPv4Address": "8.8.8.8",
0|pastash-logs | "postNATDestinationIPv4Address": "*****",
0|pastash-logs | "postNAPTSourceTransportPort": 53,
0|pastash-logs | "postNAPTDestinationTransportPort": 45719,
0|pastash-logs | "fsId": 256,
0|pastash-logs | "@timestamp": "2024-07-04T06:54:14.296Z",
0|pastash-logs | "@Version": "1",
0|pastash-logs | "protocol_name": "udp",
0|pastash-logs | "service": "dns" <=========
0|pastash-logs | }

but from here not:
sqlite {
db => '/home/ubuntu/qa.db'
query => 'SELECT host from lookup WHERE ip=(?) limit 1;'
source_field => 'ipv4_dst_addr'
target_field => 'host_name'
}

from pastash.

great!! it works thanks

from pastash.