Comments (11)
@cybersecurity99 wait you're running Zeek 5? That's definitely not supported yet. Also, correct librdkafka 1.8.2 isn't supported
If updating your versions fixes things please close this issue, otherwise let me know and we can continue to troubleshoot.
from zeek-kafka.
@cybersecurity99 Hmm I ran our end to end tests on Ubuntu 20.04 with Zeek 4.0.5 and it seems to be working. Are you seeing anything substantially different between your environment and what we are doing in Docker?
from zeek-kafka.
@cybersecurity99 Hmm I ran our end to end tests on Ubuntu 20.04 with Zeek 4.0.5 and it seems to be working. Are you seeing anything substantially different between your environment and what we are doing in Docker?
@JonZeolla I am not using Docker , I have tried this in Multiple VM . I am facing this in all attempts also if I install after failed test I get this when checking installation
./zeek -N Seiso::Kafka
error in /usr/local/zeek/share/zeek/base/init-bare.zeek, line 1: cannot load plugin library /usr/local/zeek/lib/zeek/plugins/packages/zeek-kafka//lib/SEISO-KAFKA.linux-x86_64.so: /usr/local/lib/librdkafka++.so.1: undefined symbol: rd_kafka_error_string
fatal error in /usr/local/zeek/share/zeek/base/init-bare.zeek, line 1: aborting after plugin errors
Is this any configuration file or logs which I can check to see real reason for error
from zeek-kafka.
@cybersecurity99 are you using the right version of librdkafka? We only currently support 1.4.2 or 1.4.4
from zeek-kafka.
@cybersecurity99 are you using the right version of librdkafka? We only currently support 1.4.2 or 1.4.4
@JonZeolla , yes I guess I downloaded from apt install librdkafka-dev.
Also
I tried to delete zeek-kafka packages and reinstall it
It installed it without any error . But it doesn't show it ran 10 test cases successfully or failed it just showed running test cases and then installed .
It is sending data to kafka But at the same time it is writing logs in zeek/logs/current folder how to stop that or is it permanent ?
from zeek-kafka.
@cybersecurity99 at a quick glance that may be installing the wrong version of librdkafka. May be looking at the wrong thing via my phone.
https://ubuntu.pkgs.org/20.04/ubuntu-universe-arm64/librdkafka-dev_1.2.1-1ubuntu1_arm64.deb.html
You could use a modified version of https://github.com/JonZeolla/Development/blob/76d4bc84e222544baec9e8269d71f3cfa55c0869/python/librdkafka_version_checker.py to check the version you have on disk, or more simply you could just use apt/dpkg.
I'm not sure why this package would create different logs on disk than you already had... @ottobackwards any idea on that one
from zeek-kafka.
@cybersecurity99 at a quick glance that may be installing the wrong version of librdkafka. May be looking at the wrong thing via my phone.
https://ubuntu.pkgs.org/20.04/ubuntu-universe-arm64/librdkafka-dev_1.2.1-1ubuntu1_arm64.deb.html
You could use a modified version of https://github.com/JonZeolla/Development/blob/76d4bc84e222544baec9e8269d71f3cfa55c0869/python/librdkafka_version_checker.py to check the version you have on disk, or more simply you could just use apt/dpkg.
I'm not sure why this package would create different logs on disk than you already had... @ottobackwards any idea on that one
Hi @JonZeolla
I tried that python script to check but got some error , is there any pre requisite to run that script
from: can't read /var/mail/ctypes
./librdkafka_version_checker.py: line 2: minver: command not found
./librdkafka_version_checker.py: line 3: syntax error near unexpected token `('
./librdkafka_version_checker.py: line 3: `dll = cdll.LoadLibrary("/usr/local/lib/librdkafka.so")'
Also you pointed out that 1.4.2 and 1.4.4 is supported so , we don't need to download librdkafka from apt ? Just that tar file and steps mentioned in 1st point are enough to run.
Because I have done apt install librdkafka , its shows it to be 1.8.2 .
So should I just remove them and use
$ curl -L https://github.com/edenhill/librdkafka/archive/v1.4.2.tar.gz | tar xvz
$ cd librdkafka-1.4.2/
$ ./configure --enable-sasl
$ make
$ sudo make install
After doing so and reinstalling zeek-kafka I found that data was sent to kafka and only 1 log was generated in /logs/current/ folder that is .crash-diag.out which contained following info
root@zeek-VirtualBox:/usr/local/zeek/logs/current# cat .crash-diag.out
Zeek 5.0.0-dev.280
Linux 5.13.0-40-generic
Zeek plugins:
Seiso::Kafka - Writes logs to Kafka (dynamic, version 0.3.0)
==== No reporter.log
==== No stderr.log
==== No stdout.log
==== No .cmdline
==== No .env_vars
==== No .status
==== No prof.log
==== No packet_filter.log
==== No loaded_scripts.log
What it shows ?
Also how to do we differentiate data in zeek kafka topic like http.log , conn.log ,etc
Edit
When I tried by commenting zeek-kafka in local.zeek , there were no logs in current folder also
Thanks
from zeek-kafka.
@cybersecurity99 wait you're running Zeek 5? That's definitely not supported yet. Also, correct librdkafka 1.8.2 isn't supported
If updating your versions fixes things please close this issue, otherwise let me know and we can continue to troubleshoot.
@JonZeolla
I took this in consideration and created a fresh machine with zeek v4.0.5 , and librdkafka 1.4.2 as given in commands
Still same issue it is sending data to kafka
But at the same time writing logs in /zeek/logs/current
Can you check this does it write for you also in that folder ?
from zeek-kafka.
@cybersecurity99 just to be clear, it is sending logs to Kafka like you want, but it is also storing them on disk and you want to disable that? This project adds the ability to send logs to Kafka, it doesn't replace any local log writing, but you can disable that separately.
I suggest you check out https://docs.zeek.org/en/master/frameworks/logging.html#terminology and go from there
from zeek-kafka.
@cybersecurity99 just to be clear, it is sending logs to Kafka like you want, but it is also storing them on disk and you want to disable that? This project adds the ability to send logs to Kafka, it doesn't replace any local log writing, but you can disable that separately.
I suggest you check out https://docs.zeek.org/en/master/frameworks/logging.html#terminology and go from there
@JonZeolla okay got that part.
Just wanna know if send all logs together. How do we differentiate between http.log,conn.log ,etc at kafka end .
Any command to print specific log type using topic "zeek"
from zeek-kafka.
@cybersecurity99 I think you're looking for something like tag_json
and then post processing the logs after receiving them from Kafka. You could also send different logs to different topics
https://github.com/SeisoLLC/zeek-kafka#tag_json
from zeek-kafka.
Related Issues (20)
- Kakfa plugin fails under FreeBSD 13.1 HOT 7
- Zeek-Kafka not work , error in tests HOT 10
- Support for zeek 4.2 HOT 1
- support for zeek 5.0
- Fix README instructions to not include sasl HOT 1
- Update MAINTAINERS.md release instructions
- Release automation doesn't wait for PR merge
- How to add Partition in local.zeek configuration HOT 2
- SEISO-KAFKA.linux-x86_64.so can not link librdkafka.so.1 HOT 13
- zeek log send to kafka with differnet topic per module HOT 1
- e2e should have a matrix for 5x zeek builds as well as 4.2
- [Requirements] zeek-kafka supports compression
- Update documentation to show log filtering configuration for Zeek 5.x
- Update copyright to fix linting errors
- Pipfile and lock do not work HOT 1
- I had installed zeek-kafka zeek-kafka-1.1.0,but I got (dynamic, version 0.3.0) when I show version HOT 3
- The build parameter for zeek-6.0.1 does not have --with-librdkafka, so how do I install the zeek-kafka plugin? HOT 4
- @load package/zeek-kafka canβt find
- @load packages/zeek-kafka can't find HOT 5
- zeek-kafka installation causes zeek startup failure HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from zeek-kafka.