This is an output implementation of elastic beats for support Filebeat, Metricbeat, Functionbeat, Winlogbeat, Journalbeat, Auditbeat to Apache Pulsar
This output is developed and tested using Apache Pulsar Client 2.4.0 and Beats 7.3.1
mkdir -p $GOPATH/src/github.com/streamnative/
cd $GOPATH/src/github.com/streamnative/
git clone https://github.com/streamnative/pulsar-beat-output
cd pulsar-beat-output
cd filebeat
go build .
output.pulsar:
url: "pulsar://localhost:6650"
topic: my_topic
name: test123./filebeat modules enable system
./filebeat modules list
./filebeat -c filebeat.yml -e
go build -o filebeat filebeat/filebeat.go
go build -o functionbeat functionbeat/functionbeat.go
go build -o journalbeat journalbeat/journalbeat.go
go build -o winlogbeat winlogbeat/winlogbeat.go
go build -o packetbeat packetbeat/packetbeat.go
docker build -t pulsar-beat .
docker network create pulsar-beat
docker run -d -it --network pulsar-beat -p 6650:6650 -p 8080:8080 -v $PWD/data:/pulsar/data --name pulsar-beat-standalone apachepulsar/pulsar:2.7.0 bin/pulsar standalone
output.pulsar:
url: "pulsar://pulsar-beat-standalone:6650"
topic: my_topic
name: test123docker pull golang:1.12.4
docker run -it --network pulsar-beat --name filebeat golang:1.12.4 /bin/bash
mkdir -p $GOPATH/src/github.com/streamnative/
cd $GOPATH/src/github.com/streamnative/
git clone https://github.com/streamnative/pulsar-beat-output
cd pulsar-beat-output
go build -o filebeat main.go
chown -R root:root filebeat.yml test_module/modules.d/system.yml test_module/module/system
cp test_module/module/system/auth/test/test.log /var/log/messages.log
cp filebeat filebeat.yml test_module
cd test_module
./filebeat modules enable system
./filebeat -c filebeat.yml -e
docker cp pulsar-client.py pulsar-beat-standalone:/pulsar
docker exec -it pulsar-beat-standalone /bin/bash
python pulsar-client.py
Now you can see the information collected from filebeat.
| Name | Description | Default |
|---|---|---|
| url | Configure the service URL for the Pulsar service | pulsar://localhost:6650 |
| certificate_path | path of tls cert file | "" |
| private_key_path | path of tls key file | "" |
| use_tls | Whether to turn on TLS, if to start, use protocol pulsar+ssl | false |
| token | Access token information of cluster | "" |
| token_file_path | The file path where token is saved | "" |
| log_level | Setting the log level, available options(panic, fatal, error, warn, info, debug, trace) | info |
| Name | Description | Default |
|---|---|---|
| topic | Specify the topic this producer will be publishing on. | "" |
| name | Specify a name for the producer | "" |
| send_timeout | Set the send timeout | 30s |
| block_if_queue_full | Set whether the send and sendAsync operations should block when the outgoing message queue is full. | false |
| batching_max_messages | maximum number of messages in a batch | 1000 |
| batching_max_publish_delay | the batch delay | 1ms |
| message_routing_mode | the message routing mode, SinglePartition,RoundRobinPartition, CustomPartition(0,1,2) | 1 |
| hashing_schema | JavaStringHash,Murmur3_32Hash(0,1) | 0 |
| compression_type | NONE,LZ4,ZLIB,ZSTD(0,1,2,3) | 0 |
/root/go/pkg/mod/github.com/apache/[email protected]/pulsar/internal/compression/zstd_cgo.go:27:2: case-insensitive import collision: "github.com/datadog/zstd" and "github.com/DataDog/zstd"
Replace zstd_cgo.go file
cp zstd_cgo.go /root/go/pkg/mod/github.com/apache/[email protected]/pulsar/internal/compression/zstd_cgo.go
Reference https://pulsar.apache.org/docs/en/client-libraries-go/ .
If you encounter problems with dynamic libraries，please reference:https://pulsar.apache.org/docs/en/client-libraries-cpp/.
Reference elastic/beats#11054.
systemd/sd-journal.h: No such file or directory
apt-get update
apt-get install libsystemd-dev
vendor/github.com/elastic/beats/x-pack/auditbeat/module/system/package/rpm_linux.go:23:24: fatal error: rpm/rpmlib.h: No such file or directory
aapt-get install librpm-dev
Exiting: error loading config file: config file ("filebeat.yml") must be owned by the user identifier (uid=0) or root
chown -R root:root filebeat.yml
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent