Git Product home page Git Product logo

Comments (11)

JonZeolla avatar JonZeolla commented on August 29, 2024 1

@cybersecurity99 wait you're running Zeek 5? That's definitely not supported yet. Also, correct librdkafka 1.8.2 isn't supported

If updating your versions fixes things please close this issue, otherwise let me know and we can continue to troubleshoot.

from zeek-kafka.

JonZeolla avatar JonZeolla commented on August 29, 2024

@cybersecurity99 Hmm I ran our end to end tests on Ubuntu 20.04 with Zeek 4.0.5 and it seems to be working. Are you seeing anything substantially different between your environment and what we are doing in Docker?

#38

from zeek-kafka.

cybersecurity99 avatar cybersecurity99 commented on August 29, 2024

@cybersecurity99 Hmm I ran our end to end tests on Ubuntu 20.04 with Zeek 4.0.5 and it seems to be working. Are you seeing anything substantially different between your environment and what we are doing in Docker?

#38

@JonZeolla I am not using Docker , I have tried this in Multiple VM . I am facing this in all attempts also if I install after failed test I get this when checking installation

./zeek -N Seiso::Kafka
error in /usr/local/zeek/share/zeek/base/init-bare.zeek, line 1: cannot load plugin library /usr/local/zeek/lib/zeek/plugins/packages/zeek-kafka//lib/SEISO-KAFKA.linux-x86_64.so: /usr/local/lib/librdkafka++.so.1: undefined symbol: rd_kafka_error_string
fatal error in /usr/local/zeek/share/zeek/base/init-bare.zeek, line 1: aborting after plugin errors

Is this any configuration file or logs which I can check to see real reason for error

from zeek-kafka.

JonZeolla avatar JonZeolla commented on August 29, 2024

@cybersecurity99 are you using the right version of librdkafka? We only currently support 1.4.2 or 1.4.4

from zeek-kafka.

cybersecurity99 avatar cybersecurity99 commented on August 29, 2024

@cybersecurity99 are you using the right version of librdkafka? We only currently support 1.4.2 or 1.4.4

@JonZeolla , yes I guess I downloaded from apt install librdkafka-dev.
Also
I tried to delete zeek-kafka packages and reinstall it
It installed it without any error . But it doesn't show it ran 10 test cases successfully or failed it just showed running test cases and then installed .
It is sending data to kafka But at the same time it is writing logs in zeek/logs/current folder how to stop that or is it permanent ?

from zeek-kafka.

JonZeolla avatar JonZeolla commented on August 29, 2024

@cybersecurity99 at a quick glance that may be installing the wrong version of librdkafka. May be looking at the wrong thing via my phone.

https://ubuntu.pkgs.org/20.04/ubuntu-universe-arm64/librdkafka-dev_1.2.1-1ubuntu1_arm64.deb.html

You could use a modified version of https://github.com/JonZeolla/Development/blob/76d4bc84e222544baec9e8269d71f3cfa55c0869/python/librdkafka_version_checker.py to check the version you have on disk, or more simply you could just use apt/dpkg.

I'm not sure why this package would create different logs on disk than you already had... @ottobackwards any idea on that one

from zeek-kafka.

cybersecurity99 avatar cybersecurity99 commented on August 29, 2024

@cybersecurity99 at a quick glance that may be installing the wrong version of librdkafka. May be looking at the wrong thing via my phone.

https://ubuntu.pkgs.org/20.04/ubuntu-universe-arm64/librdkafka-dev_1.2.1-1ubuntu1_arm64.deb.html

You could use a modified version of https://github.com/JonZeolla/Development/blob/76d4bc84e222544baec9e8269d71f3cfa55c0869/python/librdkafka_version_checker.py to check the version you have on disk, or more simply you could just use apt/dpkg.

I'm not sure why this package would create different logs on disk than you already had... @ottobackwards any idea on that one

Hi @JonZeolla
I tried that python script to check but got some error , is there any pre requisite to run that script

from: can't read /var/mail/ctypes
./librdkafka_version_checker.py: line 2: minver: command not found
./librdkafka_version_checker.py: line 3: syntax error near unexpected token `('
./librdkafka_version_checker.py: line 3: `dll = cdll.LoadLibrary("/usr/local/lib/librdkafka.so")'
 

Also you pointed out that 1.4.2 and 1.4.4 is supported so , we don't need to download librdkafka from apt ? Just that tar file and steps mentioned in 1st point are enough to run.
Because I have done apt install librdkafka , its shows it to be 1.8.2 .
So should I just remove them and use

$ curl -L https://github.com/edenhill/librdkafka/archive/v1.4.2.tar.gz | tar xvz
$ cd librdkafka-1.4.2/
$ ./configure --enable-sasl
$ make
$ sudo make install

After doing so and reinstalling zeek-kafka I found that data was sent to kafka and only 1 log was generated in /logs/current/ folder that is .crash-diag.out which contained following info

root@zeek-VirtualBox:/usr/local/zeek/logs/current# cat .crash-diag.out 
 
Zeek 5.0.0-dev.280
Linux 5.13.0-40-generic
 
Zeek plugins:
Seiso::Kafka - Writes logs to Kafka (dynamic, version 0.3.0)
 
==== No reporter.log
 
==== No stderr.log
 
==== No stdout.log
 
==== No .cmdline
 
==== No .env_vars
 
==== No .status
 
==== No prof.log
 
==== No packet_filter.log
 
==== No loaded_scripts.log

What it shows ?

Also how to do we differentiate data in zeek kafka topic like http.log , conn.log ,etc

Edit
When I tried by commenting zeek-kafka in local.zeek , there were no logs in current folder also
Thanks

from zeek-kafka.

cybersecurity99 avatar cybersecurity99 commented on August 29, 2024

@cybersecurity99 wait you're running Zeek 5? That's definitely not supported yet. Also, correct librdkafka 1.8.2 isn't supported

If updating your versions fixes things please close this issue, otherwise let me know and we can continue to troubleshoot.

@JonZeolla
I took this in consideration and created a fresh machine with zeek v4.0.5 , and librdkafka 1.4.2 as given in commands

Still same issue it is sending data to kafka
But at the same time writing logs in /zeek/logs/current
Can you check this does it write for you also in that folder ?

from zeek-kafka.

JonZeolla avatar JonZeolla commented on August 29, 2024

@cybersecurity99 just to be clear, it is sending logs to Kafka like you want, but it is also storing them on disk and you want to disable that? This project adds the ability to send logs to Kafka, it doesn't replace any local log writing, but you can disable that separately.

I suggest you check out https://docs.zeek.org/en/master/frameworks/logging.html#terminology and go from there

from zeek-kafka.

cybersecurity99 avatar cybersecurity99 commented on August 29, 2024

@cybersecurity99 just to be clear, it is sending logs to Kafka like you want, but it is also storing them on disk and you want to disable that? This project adds the ability to send logs to Kafka, it doesn't replace any local log writing, but you can disable that separately.

I suggest you check out https://docs.zeek.org/en/master/frameworks/logging.html#terminology and go from there

@JonZeolla okay got that part.

Just wanna know if send all logs together. How do we differentiate between http.log,conn.log ,etc at kafka end .
Any command to print specific log type using topic "zeek"

from zeek-kafka.

JonZeolla avatar JonZeolla commented on August 29, 2024

@cybersecurity99 I think you're looking for something like tag_json and then post processing the logs after receiving them from Kafka. You could also send different logs to different topics

https://github.com/SeisoLLC/zeek-kafka#tag_json

from zeek-kafka.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.