lephisto / pfsense-analytics Goto Github PK

Hi ! i'm trying to setting up but i'm stuck at step 5. Cerebro.
Cerebro can't to connect to elastricsearch. I used docker stats to see if elasticsearch was running, it was actually looping. I looked at the logs : docker logs -f pfanalytics_elasticsearch_1 ; and then i saw :

[2020-10-04T18:56:56,262][INFO ][o.e.n.Node               ] [fEncsgq] starting ...
[2020-10-04T18:56:56,406][INFO ][o.e.t.TransportService   ] [fEncsgq] publish_address {172.18.0.8:9300}, bound_addresses {0.0.0.0:9300}
[2020-10-04T18:56:56,417][INFO ][o.e.b.BootstrapChecks    ] [fEncsgq] bound or publishing to a non-loopback address, enforcing bootstrap checks
ERROR: [1] bootstrap checks failed
[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65535]
[2020-10-04T18:56:56,424][INFO ][o.e.n.Node               ] [fEncsgq] stopping ...
[2020-10-04T18:56:56,458][INFO ][o.e.n.Node               ] [fEncsgq] stopped
[2020-10-04T18:56:56,458][INFO ][o.e.n.Node               ] [fEncsgq] closing ...
[2020-10-04T18:56:56,470][INFO ][o.e.n.Node               ] [fEncsgq] closed

I think that i will need to reset max file descriptors. But i'm stuck, i don't know how to do this. (i'm setting up pfense-analytics on my NAS Asustor, there is no such file /etc/security/limits.conf). Could you please help ?

tost@siem_soc:~/pfsense-analytics/Docker$ sudo docker-compose up -d
[sudo] password for tost:
Building graylog
Step 1/6 : FROM graylog/graylog:3.1
---> ca38a27808e3
Step 2/6 : USER root
---> Using cache
---> 069030ba8761
Step 3/6 : RUN mkdir -pv /etc/graylog/server/
---> Using cache
---> b85bcd69ff78
Step 4/6 : COPY ./getGeo.sh /etc/graylog/server/
---> Using cache
---> b023f4e1942e
Step 5/6 : RUN chmod +x /etc/graylog/server/getGeo.sh && /etc/graylog/server/getGeo.sh
---> Running in 02dd88419fb8
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (6) Could not resolve host: geolite.maxmind.com
tar (child): /etc/graylog/server/mm.tar.gz: Cannot open: No such file or directory
tar (child): Error is not recoverable: exiting now
tar: Child returned status 2
tar: Error is not recoverable: exiting now
ERROR: Service 'graylog' failed to build: The command '/bin/sh -c chmod +x /etc/graylog/server/getGeo.sh && /etc/graylog/server/getGeo.sh' returned a non-zero code: 2

So, I have everything setup, and I'm getting data in Grafana for both dashboards. I'm just not seeing any data in the Graylog stream.

Any ideas?

T

Hi there,

Might be a really dumb question but are you considering creating a non Docker version of this. I already have a Graylog & Elastic instance running on bare metal and would like to use that.

Hello,

Is there a way to run this solution on Synology Docker?

Thank you,

Mike

image: 'influxdb:latest' pulls InfluxDB 2.0. As a fix I set the docker compose file to pull influxdb:1.8.4-alpine.

Please can you help me I get this error in step 5/6

`Step 5/6 : RUN chmod +x /etc/graylog/server/getGeo.sh && /etc/graylog/server/getGeo.sh
---> Running in b30dbe5bac10
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 20 100 20 0 0 112 0 --:--:-- --:--:-- --:--:-- 112

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
ERROR: Service 'graylog' failed to build: The command '/bin/sh -c chmod +x /etc/graylog/server/getGeo.sh && /etc/graylog/server/getGeo.sh' returned a non-zero code: 2`

Thanks.

Docker graylog server is up but accesing port 9000 just gives me a white screen

Has anyone been able to get the NDPI dashboard to be able to show specific types of traffic per host?

I am a grafana noob and don't have the skills to make this happen but would like to be able to select a specific client ip address and then have the NDPI interface show just the traffic for that client.

Hi,

Thanks for setting this up. It's been working great in July but after 1 August, the logfile rotation has kicked in but the indices don't seem to be working correctly - so the dashboard has no new updates.

Where might I need to look to check that the indexing/search is updating correctly at month end ?

I am still getting pfSense messages in Graylog, and indices are still running.

pfsense-logs 2 indices, 55,209,052 documents, 33.9GiB default

Rotation Strategy: Index Time
Rotation Period : P1M
Index Retention Configuration - Delete Index (after Max number of indices = 12)

Per 31/12/2019 you need a registered account at MaxMind to access the GeoLite2 database, this to comply with privacy laws.
Instruction on how to do so here

Everything else is working fine in Grafana and Graylog. My only issue is that there is no map data being shown in either the Firewall Logs or DPI dashboards. I used #15 to help me refer to my licence key within getGeo.sh in order to build Graylog in the first place, but even though it was built, there is still nothing being shown on either maps in Grafana. Anyone else having this issue?

Thanks for the help

Hello,

is anyone experiencing the same issue when running the 5/6 step?

`Step 5/6 : RUN chmod +x /etc/graylog/server/getGeo.sh && /etc/graylog/server/getGeo.sh
---> Running in b30dbe5bac10
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 20 100 20 0 0 112 0 --:--:-- --:--:-- --:--:-- 112

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
ERROR: Service 'graylog' failed to build: The command '/bin/sh -c chmod +x /etc/graylog/server/getGeo.sh && /etc/graylog/server/getGeo.sh' returned a non-zero code: 2`

Thanks.

Hello

When i try to configure on NTOP the Influxdb it appear the following error:

This is what I have on System>Overview;
User 2020-05-07 18:47:00 -4:00
Your web browser 2020-05-07 18:47:00 -4:00
Graylog server 2020-05-07 22:48:00 +0:00

That is an issue with the logs. I can only see the last 8 hours of data on Grafana.

I have changed the timezone on the server without luck.
garylog.env is GRAYLOG_TIMEZONE=America/Puerto_Rico

Any ideas?

Thanks!

So, I had everything working and then had some hardware problems. After resolving them, things wouldn't start cleanly. I didn't know if it was something I did or not, so I burned all the containers and started from scratch (CENTOS 8 vm, not rebuilt, only the docker stuff was).

Now, as before, graylog can't connect to mongo. All I get is this:
raylog_1 | 2020-06-23 13:23:35,123 INFO : org.mongodb.driver.cluster - Cluster description not yet available. Waiting for 30000 ms before timing out
graylog_1 | 2020-06-23 13:23:36,172 INFO : org.mongodb.driver.cluster - Exception in monitor thread while connecting to server mongo:27017
graylog_1 | com.mongodb.MongoSocketOpenException: Exception opening socket
graylog_1 | at com.mongodb.connection.SocketStream.open(SocketStream.java:62) ~[graylog.jar:?]
graylog_1 | at com.mongodb.connection.InternalStreamConnection.open(InternalStreamConnection.java:126) ~[graylog.jar:?]
graylog_1 | at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:114) [graylog.jar:?]
graylog_1 | at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]
graylog_1 | Caused by: java.net.NoRouteToHostException: No route to host (Host unreachable)
graylog_1 | at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_232]
graylog_1 | at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_232]
graylog_1 | at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_232]
graylog_1 | at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_232]
graylog_1 | at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_232]
graylog_1 | at java.net.Socket.connect(Socket.java:607) ~[?:1.8.0_232]
graylog_1 | at com.mongodb.connection.SocketStreamHelper.initialize(SocketStreamHelper.java:59) ~[graylog.jar:?]
graylog_1 | at com.mongodb.connection.SocketStream.open(SocketStream.java:57) ~[graylog.jar:?]
graylog_1 | ... 3 more
graylog_1 | 2020-06-23 13:24:05,124 ERROR: org.graylog2.bindings.providers.MongoConnectionProvider - Error connecting to MongoDB: Timed out after 30000 ms while waiting to connect. Client view of cluster state is {type=UNKNOWN, servers=[{address=mongo:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.NoRouteToHostException: No route to host (Host unreachable)}}]
graylog_1 | 2020-06-23 13:24:05,144 INFO : org.graylog2.shared.buffers.InputBufferImpl - Message journal is enabled.

and it doesn't start right. Grafana also errors out with an message about not connecting to download the pie-chart plugin. Elasticsearch and Cerebro start fine, as far as I can tell.

Any ideas?
T.

Hi, everything is working fine in Grafana. DPI is showing correct but only i dont see any firewall Logs. Firewall Logs keeps empty.
Is there a way to fix this?

Getting error when entering the 'Node Address'

Greylog and cerebro open fine whilist I cant access graphana

Thank you very much to setting this how to up and also thank you for providing your step by step guide.

I followed your guide exactly and was able to set everything up without any error messages or other hicups.
Sadly, how ever I can not see any data in the pfsense stream on graylog and there for i also have no data in the Grafana Dashboard.

doese anybody have any idea what the reason could be that the graylog server get no data in the stream?
I'm not very Linux experienced and there for a little lost in how to find the error my self.

thanks to every one who could help here.

Best Regards

Receiving this error:

"error": "JsResultException(errors:List((,List(JsonValidationError(List('attributes' is undefined on object: {\"name\":\"vJ9IPvf\",\"transport_address\":\"172.18.0.2:9300\",\"host\":\"172.18.0.2\",\"ip\":\"172.18.0.2\",\"version\":\"6.8.5\",\"build_flavor\":\"oss\",\"build_type\":\"docker\",\"build_hash\":\"78990e9\",\"roles\":[\"master\",\"data\",\"ingest\"],\"os\":{\"refresh_interval\":\"1s\",\"refresh_interval_in_millis\":1000,\"name\":\"Linux\",\"pretty_name\":\"CentOS Linux 7 (Core)\",\"arch\":\"amd64\",\"version\":\"4.15.0-101-generic\",\"available_processors\":4,\"allocated_processors\":4},\"jvm\":{\"pid\":1,\"version\":\"13.0.1\",\"vm_name\":\"OpenJDK 64-Bit Server VM\",\"vm_version\":\"13.0.1+9\",\"vm_vendor\":\"AdoptOpenJDK\",\"start_time\":\"2020-05-26T03:05:04.547Z\",\"start_time_in_millis\":1590462304547,\"mem\":{\"heap_init\":\"1gb\",\"heap_init_in_bytes\":1073741824,\"heap_max\":\"990.7mb\",\"heap_max_in_bytes\":1038876672,\"non_heap_init\":\"7.3mb\",\"non_heap_init_in_bytes\":7667712,\"non_heap_max\":\"0b\",\"non_heap_max_in_bytes\":0,\"direct_max\":\"0b\",\"direct_max_in_bytes\":0},\"gc_collectors\":[\"ParNew\",\"ConcurrentMarkSweep\"],\"memory_pools\":[\"CodeHeap 'non-nmethods'\",\"Metaspace\",\"CodeHeap 'profiled nmethods'\",\"Compressed Class Space\",\"Par Eden Space\",\"Par Survivor Space\",\"CodeHeap 'non-profiled nmethods'\",\"CMS Old Gen\"],\"using_compressed_ordinary_object_pointers\":\"true\",\"input_arguments\":[\"-Xms1g\",\"-Xmx1g\",\"-XX:+UseConcMarkSweepGC\",\"-XX:CMSInitiatingOccupancyFraction=75\",\"-XX:+UseCMSInitiatingOccupancyOnly\",\"-Des.networkaddress.cache.ttl=60\",\"-Des.networkaddress.cache.negative.ttl=10\",\"-XX:+AlwaysPreTouch\",\"-Xss1m\",\"-Djava.awt.headless=true\",\"-Dfile.encoding=UTF-8\",\"-Djna.nosys=true\",\"-XX:-OmitStackTraceInFastThrow\",\"-Dio.netty.noUnsafe=true\",\"-Dio.netty.noKeySetOptimization=true\",\"-Dio.netty.recycler.maxCapacityPerThread=0\",\"-Dlog4j.shutdownHookEnabled=false\",\"-Dlog4j2.disable.jmx=true\",\"-Djava.io.tmpdir=/tmp/elasticsearch-125629727188914072\",\"-XX:+HeapDumpOnOutOfMemoryError\",\"-XX:HeapDumpPath=data\",\"-XX:ErrorFile=logs/hs_err_pid%p.log\",\"-Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m\",\"-Djava.locale.providers=COMPAT\",\"-XX:UseAVX=2\",\"-Des.cgroups.hierarchy.override=/\",\"-Des.path.home=/usr/share/elasticsearch\",\"-Des.path.conf=/usr/share/elasticsearch/config\",\"-Des.distribution.flavor=oss\",\"-Des.distribution.type=docker\"]}}),WrappedArray())))))"

Is anyone else seeing this error and happen to know the fix? I'm guessing it is due to the recent MaxMind sign-in requirement, but haven't dug into the script to figure out how to fix yet.

% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                               Dload  Upload   Total   Spent    Left  Speed
0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: geolite.maxmind.com
tar (child): /etc/graylog/server/mm.tar.gz: Cannot open: No such file or directory
tar (child): Error is not recoverable: exiting now
tar: Child returned status 2
tar: Error is not recoverable: exiting now
ERROR: Service 'graylog' failed to build: The command '/bin/sh -c chmod +x /etc/graylog/server/getGeo.sh && /etc/graylog/server/getGeo.sh' returned a non-zero code: 2

When looking at the dashboards I am seeing numbers that don't seem to make sense?

Some totals are in the TB totals for the last 6 hours. Most data seems wrong on the NDPI interface dashboard. Any ideas on where to start looking? I followed your instructions and all went well, just seems like something is wrong as there is no way that the data is correct.

Thanks.

Hello,is there any reason why graylog and cerebro don't show any ui whe i nevigate to
http://172.20.70.110:9000 and http://172.20.70.110:9001?

Only grafana is working

running latest stable debian 10 freenas vm.

Hi
I have set this up on a Ubuntu 18.04 and graylog is receiving the logs from the pfsense. However, I don't see anything on the Grafana dashboard. I also tested the data source pfsensefw (http://elasticsearch:9200) and it doesn't complain.

Following is the log output from the CLI and screenshots from Grafana.

tcpdump -i ens160 not port 22 | grep 5442
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens160, link-type EN10MB (Ethernet), capture size 262144 bytes
23:18:26.196359 IP xx.xx.xx.xx.syslog > pf-analytics.local.5442: SYSLOG local5.info, length: 275
23:18:26.313221 IP xx.xx.xx.xx.syslog > pf-analytics.local.5442: SYSLOG local0.info, length: 195
23:18:26.338964 IP xx.xx.xx.xx.syslog > pf-analytics.local.5442: SYSLOG local5.info, length: 276
23:18:27.190880 IP xx.xx.xx.xx.syslog > pf-analytics.local.5442: SYSLOG local5.info, length: 275
23:18:28.193809 IP xx.xx.xx.xx.syslog > pf-analytics.local.5442: SYSLOG local5.info, length: 275
23:18:29.196005 IP xx.xx.xx.xx.syslog > pf-analytics.local.5442: SYSLOG local5.info, length: 275
23:18:29.240726 IP xx.xx.xx.xx.syslog > pf-analytics.local.5442: SYSLOG local5.info, length: 311
23:18:29.332991 IP xx.xx.xx.xx.syslog > pf-analytics.local.5442: SYSLOG local0.info, length: 183
23:18:30.195597 IP xx.xx.xx.xx.syslog > pf-analytics.local.5442: SYSLOG local5.info, length: 275
23:18:31.197824 IP xx.xx.xx.xx.syslog > pf-analytics.local.5442: SYSLOG local5.info, length: 275
23:18:31.242791 IP xx.xx.xx.xx.syslog > pf-analytics.local.5442: SYSLOG local5.info, length: 301
23:18:32.195049 IP xx.xx.xx.xx.syslog > pf-analytics.local.5442: SYSLOG local5.info, length: 275
23:18:32.374036 IP xx.xx.xx.xx.syslog > pf-analytics.local.5442: SYSLOG local0.info, length: 183
23:18:33.193469 IP xx.xx.xx.xx.syslog > pf-analytics.local.5442: SYSLOG local5.info, length: 275
23:18:34.193956 IP xx.xx.xx.xx.syslog > pf-analytics.local.5442: SYSLOG local5.info, length: 275
23:18:34.252609 IP xx.xx.xx.xx.syslog > pf-analytics.local.5442: SYSLOG local5.info, length: 299
23:18:35.190906 IP xx.xx.xx.xx.syslog > pf-analytics.local.5442: SYSLOG local5.info, length: 275

change localhost:9000 to ipaddress:9000 in graylog.env and nothing happens

***secret MUST BE 16 !!!
my bad
CLOSED ISSUE

I get this strange spikes about every 30 minutes. The ip is may WAN (ppoe) IP, when the ip changes it resets and begins increasing again. It makes no sense since it keeps increasing even over my max internet bandwith (600mb).
Checking on ntopng directly there is no trace of this bandwith usage too.

Hi lephisto,

First of all, thanks for the work you are putting in this project, it makes me reconsider using pfpsense as my home firewall (used untangle up until now). Your effort is really appreciated.

Now, on the topic, I saw an issue while I am starting the containers as the kabana is exiting right away and there is no way to start it. After looking on the doker compose file, I saw you added a line to disable Kabana, I saw that this is something we can do while everything is up and running and we do not need Kabana at that point. Here is the commit, line 63 was the one causing the issue:

commit dc2fcd8

I think we should probably have instructions on the README regarding this portion, but it is better if we comment this portion for those who try to setup the whole thing.

Again, thanks for your time on this project!

Have a nice one!

Nikolay

graylog recognizes and filters ipv4 addresses correctly. but ipv6 addresses are not extracted or recognized from log messages.

Is this error known or does the error only occur during my installation ?

When I go into the new dashboard with data, and scroll down the page, apparently changes are made. It wants me to save after every time I leave and add this metadata...

{
  "__inputs": [
    {
      "name": "DS_PFS_GRAYLOG",
      "label": "PFS Graylog",
      "description": "",
      "type": "datasource",
      "pluginId": "elasticsearch",
      "pluginName": "Elasticsearch"
    }
  ],
  "__requires": [
    {
      "type": "datasource",
      "id": "elasticsearch",
      "name": "Elasticsearch",
      "version": "1.0.0"
    },
    {
      "type": "grafana",
      "id": "grafana",
      "name": "Grafana",
      "version": "6.4.3"
    },
    {
      "type": "panel",
      "id": "grafana-piechart-panel",
      "name": "Pie Chart",
      "version": "1.3.9"
    },
    {
      "type": "panel",
      "id": "grafana-worldmap-panel",
      "name": "Worldmap Panel",
      "version": "0.2.1"
    },
    {
      "type": "panel",
      "id": "graph",
      "name": "Graph",
      "version": ""
    },
    {
      "type": "panel",
      "id": "savantly-heatmap-panel",
      "name": "Heatmap",
      "version": "0.2.0"
    },
    {
      "type": "panel",
      "id": "singlestat",
      "name": "Singlestat",
      "version": ""
    },
    {
      "type": "panel",
      "id": "table",
      "name": "Table",
      "version": ""
    }
  ],

Hello
When running docker-compose up:

Step 5/6 : RUN chmod +x /etc/graylog/server/getGeo.sh && /etc/graylog/server/getGeo.sh
---> Running in 7ba489eb790e
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (6) Could not resolve host: geolite.maxmind.com

My server can resolve geolite.maxmind.com, because Maxmind is now a paid db.

How to I change that?

Here is a picture of what I am talking about.. the world map is missing some squares. I can zoom in and find detail. The entire map was there last night. Anyone else having the same issue .. see pic

After fixing the maxmind link, I now stumble across this, and it looks like becuase I already have infludb running on the machine port 8086 for things like my cable modem signal statistics and also all of my ubiquiti unifi data that I am already running in grafana - locally on the machine already.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/compose/service.py", line 625, in start_container
container.start()
File "/usr/lib/python3/dist-packages/compose/container.py", line 241, in start
return self.client.start(self.id, **options)
File "/usr/lib/python3/dist-packages/docker/utils/decorators.py", line 19, in wrapped
return f(self, resource_id, *args, **kwargs)
File "/usr/lib/python3/dist-packages/docker/api/container.py", line 1095, in start
self._raise_for_status(res)
File "/usr/lib/python3/dist-packages/docker/api/client.py", line 263, in _raise_for_status
raise create_api_error_from_http_exception(e)
File "/usr/lib/python3/dist-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
raise cls(e, response=response, explanation=explanation)
docker.errors.APIError: 500 Server Error: Internal Server Error ("b'driver failed programming external connectivity on endpoint pfanalytics_influxdb_1 (b9e0960f863784eb35a07ee1b0976e43ac6c407b060c5be250f698e38ea16e27): Error starting userland proxy: listen tcp 0.0.0.0:8086: bind: address already in use'")

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/bin/docker-compose", line 11, in
load_entry_point('docker-compose==1.25.0', 'console_scripts', 'docker-compose')()
File "/usr/lib/python3/dist-packages/compose/cli/main.py", line 72, in main
command()
File "/usr/lib/python3/dist-packages/compose/cli/main.py", line 128, in perform_command
handler(command, command_options)
File "/usr/lib/python3/dist-packages/compose/cli/main.py", line 1107, in up
to_attach = up(False)
File "/usr/lib/python3/dist-packages/compose/cli/main.py", line 1088, in up
return self.project.up(
File "/usr/lib/python3/dist-packages/compose/project.py", line 565, in up
results, errors = parallel.parallel_execute(
File "/usr/lib/python3/dist-packages/compose/parallel.py", line 112, in parallel_execute
raise error_to_reraise
File "/usr/lib/python3/dist-packages/compose/parallel.py", line 210, in producer
result = func(obj)
File "/usr/lib/python3/dist-packages/compose/project.py", line 548, in do
return service.execute_convergence_plan(
File "/usr/lib/python3/dist-packages/compose/service.py", line 545, in execute_convergence_plan
return self._execute_convergence_create(
File "/usr/lib/python3/dist-packages/compose/service.py", line 460, in _execute_convergence_create
containers, errors = parallel_execute(
File "/usr/lib/python3/dist-packages/compose/parallel.py", line 112, in parallel_execute
raise error_to_reraise
File "/usr/lib/python3/dist-packages/compose/parallel.py", line 210, in producer
result = func(obj)
File "/usr/lib/python3/dist-packages/compose/service.py", line 465, in
lambda service_name: create_and_start(self, service_name.number),
File "/usr/lib/python3/dist-packages/compose/service.py", line 457, in create_and_start
self.start_container(container)
File "/usr/lib/python3/dist-packages/compose/service.py", line 627, in start_container
if "driver failed programming external connectivity" in ex.explanation:
TypeError: a bytes-like object is required, not 'str'

It would be really helpful to have a variable drop down at the top for specific hosts, I have the variable defined but when I apply it the changes only take place on the Traffic by Host panel and not the NDPI or distribution panels. Likewise it would be neat if the already defined ndpicat variable applied to the Traffic by host panel as well.

I am a noob at this stuff but I am still trying to figure this out, if anyone has a fix to get these variables to apply to all panels please let me know.

I will include my host variable that I am using just in case it gets someone with more grafana experience started.

https://i.imgur.com/8LdsPJi.png

I am running Ubuntu Server 64bit on a Raspberry Pi 4 with 8 GB of RAM and a 128 GB SD card. When I trying to run the docker compose process I get the following error:

ubuntu@ubuntu:~/pfsense-analytics/Docker$ sudo docker-compose up -d Building graylog Step 1/6 : FROM graylog/graylog:3.1 ---> ca38a27808e3 Step 2/6 : USER root ---> Using cache ---> 08a76a36e96d Step 3/6 : RUN mkdir -pv /etc/graylog/server/ ---> Running in d7d70564f82a standard_init_linux.go:211: exec user process caused "exec format error" ERROR: Service 'graylog' failed to build: The command '/bin/sh -c mkdir -pv /etc/graylog/server/' returned a non-zero code: 1

Not sure what the issue is here. Any assistance would be very much appreciated.

There are no instructions in the readme on how to get this package to automatically start with system boot. I can never remember cd pfsense-analytics/Docker
sudo docker-compose up -d

Hello, I have reached this point where I get an error and the installation fails. Any ideas on what I can check? Thank you!

ERROR: for docker_elasticsearch_1 Cannot start service elasticsearch: b'OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:430: container init caused \"process_linux.go:359: setting rlimits for ready process caused \\\"error setting rlimit type 8: operation not permitted\\\"\"": unknown'

ERROR: for docker_mongodb_1 UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

ERROR: for elasticsearch Cannot start service elasticsearch: b'OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:430: container init caused \"process_linux.go:359: setting rlimits for ready process caused \\\"error setting rlimit type 8: operation not permitted\\\"\"": unknown'

ERROR: for mongodb UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
ERROR: An HTTP request took too long to complete. Retry with --verbose to obtain debug information.
If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 60).

This is not e issue.
Thanks for your mention. Nice job. I will work and reuse your job. I have worked on other projects such as Suricata, Zimbra, Squid and Wazuh, but they are still Graylog 2.4. The Graylog version change and no longer allowed me to decline my Content Packs. Recently a friend asked me to update everything to the current versions of Graylog and Elasticsearch. You have given me a reason to update and thus collaborate with each other. Best regards.

See screenshots

I've only have IPv4

Hello

After a long time of no problem, a restart causes this problem with Graylog server, that won't start up.

Any help?

2020-05-19 11:58:28,830 ERROR: org.graylog2.bootstrap.CmdLineTool - Invalid configuration
com.github.joschi.jadconfig.ParameterException: Couldn't convert value for parameter "root_timezone"
at com.github.joschi.jadconfig.JadConfig.processClassFields(JadConfig.java:141) ~[graylog.jar:?]
at com.github.joschi.jadconfig.JadConfig.process(JadConfig.java:99) ~[graylog.jar:?]
at org.graylog2.bootstrap.CmdLineTool.processConfiguration(CmdLineTool.java:351) [graylog.jar:?]
at org.graylog2.bootstrap.CmdLineTool.readConfiguration(CmdLineTool.java:344) [graylog.jar:?]
at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:178) [graylog.jar:?]
at org.graylog2.bootstrap.Main.main(Main.java:50) [graylog.jar:?]
Caused by: com.github.joschi.jadconfig.ParameterException: Couldn't convert value "UTC +2" to DateTimeZone.
at com.github.joschi.jadconfig.jodatime.converters.DateTimeZoneConverter.convertFrom(DateTimeZoneConverter.java:26) ~[graylog.jar:?]
atccom.github.joschi.jadconfig.jodatime.converters.DateTimeZoneConverter.convertFrom(DateTimeZoneConverter.java:12) ~[graylog.jar:?]
at com.github.joschi.jadconfig.JadConfig.convertStringValue(JadConfig.java:167) ~[graylog.jar:?]
at com.github.joschi.jadconfig.JadConfig.processClassFields(JadConfig.java:139) ~[graylog.jar:?]

org.graylog2.contentpacks.exceptions.FailedConstraintsException: Failed constraints: [PluginVersionConstraint{type=plugin-version, pluginId=org.graylog.plugins.threatintel.ThreatIntelPlugin, version=>=3.1.3}, PluginVersionConstraint{type=plugin-version, pluginId=org.graylog.plugins.threatintel.ThreatIntelPlugin, version=>=3.1.2}]

For some reason, I do not see the ThreatIntelPlugin listed in the config. Since it's not enabled and is a dependency of the content pack, the install of the content pack fails.

I'm looking through the Graylog docs to see if I missed anything.

I had to reinstall pfsense-analytics since I screwed up some backups. It was working fine until then. Now after reinstalling, the Firewall dashboard looks fine:

but the DPI dashboard is empty (the panels are all there, they just don't have any data)

So I wanted to know, in order to troubleshoot, how can verify logs are being stored in Graylog?

No connection to https://geolite.maxmind.com/
This causes the error:
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number
tar (child): /etc/graylog/server/mm.tar.gz: Cannot open: No such file or directory
tar (child): Error is not recoverable: exiting now
tar: Child returned status 2
tar: Error is not recoverable: exiting now
ERROR: Service 'graylog' failed to build: The command '/bin/sh -c chmod +x /etc/graylog/server/getGeo.sh && /etc/graylog/server/getGeo.sh' returned a non-zero code: 2

OS Debian 10
I would be glad for help or ideas on how to get around this problem.

I followed the steps in the README.md file but i got stuck when accessing the Graylog webURL.
The browser error message is "refused to connect" while the URL for Cerebro and Grafana works fine.

docker-compose.yml
GRAYLOG_HTTP_EXTERNAL_URI=http://localhost:9000/

sudo netstat -tulpn | grep 9000

sudo netstat -tulpn | grep 3000
tcp6       0      0 :::3000                 :::*                    LISTEN      14130/docker-proxy

sudo netstat -tulpn | grep 9001
tcp6       0      0 :::9001                 :::*                    LISTEN      14170/docker-proxy

lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.3 LTS
Release:	18.04
Codename:	bionic

sudo ufw status
Status: inactive

In your newest readme on that index creation instructions, your new pictured removed what it was showing to prefix the index. Isn't that step important?

I get the following error message when attempting to create an Indice under "Create index sets" in graylog:

"Please fill out this field" under the Alerts tab

I tried to set up notifications, however error persists. How do I overcome this?

Getting an error uploading the Content Pack on Graylog 3.3.4 -- from the log file:

Caused by: org.graylog2.contentpacks.exceptions.DivergingEntityConfigurationException: Expected Grok pattern for name "COMMONAPACHELOG": <%{IPORHOST:clientip} %{HTTPDUSER:ident} %{USER:auth} [%{HTTPDATE:timestamp}] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)>; actual Grok pattern: <%{IPORHOST:clientip} %{HTTPDUSER:ident} %{USER:auth} [%{HTTPDATE:timestamp;date;dd/MMM/yyyy:HH:mm:ss Z}] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)>

Looks like it has to do with this entry in the content pack:

{ "v": "1", "type": { "name": "grok_pattern", "version": "1" }, "id": "7482a5f4-868c-4ef2-839f-a22141445c5c", "data": { "name": "COMMONAPACHELOG", "pattern": "%{IPORHOST:clientip} %{HTTPDUSER:ident} %{USER:auth} \\[%{HTTPDATE:timestamp}\\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-)" }, "constraints": [ { "type": "server-version", "version": ">=3.1.2+9e96b08" } ] },

Removing this will allow the content pack to install, but breaks some of the output.

Not sure why this is happening.. Any ideas?

Hi everyone and thanks for this magnific guide.
I have just installed all and it is working except the map on grafana which shows no points.

Please someone could help me finding the problem??

I have already enabled Geo_Location Processor on Graylog configuration.

Thanks to all.

Hello,

I am not seeing data in streams messages in Graylog. I have followed the video and the guide at https://github.com/lephisto/pfsense-analytics/tree/master. I have checked for the flow of messages. what steps can I use to troubleshoot?

I have verified that data is flowing from the pfsense box to the vm port 5442. I can see the message count fluctuate under graylog/streams > pfsense, but when I click on the stream I see a blank histogram for message count as well as no messages. There is no data displayed in Grafana. The elasticsearch pfsense_0 has thousands of docs and is steadily growing. I have rebuilt a new vm since you updated the repo earlier today. Everything works up to step 6/Check Graylog.

Graylog Streams e.g. 17 messages/second. Must match all of the 2 configured stream rules.
Graylog Index e.g. pfsense-logs 1 index, 33,114 documents, 8.8MiB

pfsense filter.log extract
Feb 21 17:39:55 remote filterlog[35405]: 76,,,100000101,re0,match,pass,in ...
Feb 21 17:39:55 remote filterlog[35405]: 76,,,100000101,re0,match,pass,in ...
Feb 21 17:39:55 remote filterlog[35405]: 4,,,1000000003,re0,match,block,in ...

Jake