HAProxy dashboard / HAProxy HTTP Status Codes: field invalid about pfelk HOT 13 CLOSED

Ok I think I fixed the issues and made a PR, please let me know if it's ok for you :-)

from pfelk.

@mmartinello thank you for contributing...greatly appreciated!

from pfelk.

@mmartinello the haproxy was contributed/built by @BeNeDeLuX however, I believe they are now using nginx. I am unable to test this specific log type but believe I may have found the fix/solution.

The pfelk.grok file defines the http.response.status_code as an integer however, the index template did not contain a mapping to this field type. I've added a mapping and updated the haproxy template. Try reapplying the haproxy index template via the Templates Installation.

In the past when updating index template when data is already logged, I've experience issues and had to purge the index and data view when/if updating the index template fails to resolve the issue try the following:

1. stop sending haproxy logs or disconnect so that logs are not received

2. navigate to Stack Management>>Kibana and delete the -pfelk-haproxy data view

3. navigate to Stack Management>>Index Management>>Data Streams and delete "logs-pfelk-haproxy"

4. Add the data view back via installing the [HAProxy] Dashboard(https://github.com/pfelk/pfelk/blob/main/install/templates.md#two-dashboards)

5. enable logs to be sent (undo step 1)

6. check to see if the error persist

from pfelk.

@a3ilson I made what you told be but unfortunately this did not solved the problem:

I checked the new template and verified that status_code is now mapped as integer in the template:

but unfortunately it's still saved as keyword:

I have multiple instances of pfSense, I can make some tests and debugs.
Please tell me if they are needed, I can collaborate on this.

Thank you!

from pfelk.

That's odd... try setting up a new instance of pfelk and see if the values are still stored as "keywords" vs "int"

Alternatively, recreating the visualization would remedy the issue too.

from pfelk.

I completely deleted the instance and recreate it from scratch (I also removed Docker volumes).
I reinstalled all files using the new version but I got the same results.

I know it's odd but there should be a reason.

If you want I am still available for do some debugging.

from pfelk.

I'm debugging this.

For example, the following log entry:

Jun 26 16:30:07 fw1.dev.my.domain.com haproxy[88460]: 192.168.123.101:43540 [26/Jun/2024:16:30:07.403] Frontends.DEV-merged~ APM.DEV_ipvANY/elk.dev.my.domain.com 0/0/0/1/1 202 117 - - ---- 24/20/0/0/0 0/0 "POST /intake/v2/events HTTP/1.1"

seems to correctly match the grok pattern HAPROXY:

HAPROXY %{IP:[client][ip]}:%{INT:[client][port]} \[%{HAPROXYDATE:[haproxy][timestamp]}\] %{NOTSPACE:[haproxy][frontend_name]} %{NOTSPACE:[haproxy][backend_name]}/%{NOTSPACE:[haproxy][server_name]} %{INT:[haproxy][time_request]}/%{INT:[haproxy][time_queue]}/%{INT:[haproxy][time_backend_connect]}/%{INT:[haproxy][time_backend_response]}/%{NOTSPACE:[host][uptime]} %{INT:[http][response][status_code]} %{NOTSPACE:[haproxy][bytes_read]} %{DATA:[haproxy][http][request][captured_cookie]} %{DATA:[haproxy][http][response][captured_cookie]} %{NOTSPACE:[haproxy][termination_state]} %{INT:[haproxy][connections][active]}/%{INT:[haproxy][connections][frontend]}/%{INT:[haproxy][connections][backend]}/%{INT:[haproxy][connections][server]}/%{NOTSPACE:[haproxy][connections][retries]} %{INT:[haproxy][server_queue]}/%{INT:[haproxy][backend_queue]} (\{%{HAPROXYCAPTUREDREQUESTHEADERS}\})?( )?(\{%{HAPROXYCAPTUREDRESPONSEHEADERS}\})?( )?"(<BADREQ>|(%{WORD:[http][request][method]} (%{URIPROTO:[haproxy][mode]}://)?(?:%{USER:[user][name]}(?::[^@]*)?@)?(?:%{URIHOST:[http][request][referrer]})?(?:%{URIPATHPARAM:[http][mode]})?( HTTP/%{NUMBER:[http][version]})?))?"?

Many fields are indexed as keyword instead as int:

• %{INT:[client][port]}
• %{INT:[haproxy][time_request]}
• %{INT:[haproxy][time_backend_response]}
• %{INT:[http][response][status_code]}
• %{INT:[haproxy][client][port]}
• %{INT:[haproxy][time_request]}
• %{INT:[haproxy][time_backend_response]}

Other fields are correctly indexed as int:

• %{INT:[haproxy][time_queue]}
• %{INT:[haproxy][time_backend_connect]}
• %{INT:[haproxy][connections][active]}
• %{INT:[haproxy][connections][frontend]}
• %{INT:[haproxy][connections][backend]}
• %{INT:[haproxy][connections][server]}
• %{INT:[haproxy][server_queue]}
• %{INT:[haproxy][backend_queue]}
• %{INT:[haproxy][time_queue]}
• %{INT:[haproxy][time_backend_connect]}

I'm trying to find out what's happening, if you have any idea please let me know.

from pfelk.

I tried to change the field name in the grok pattern renaming %{INT:[http][response][status_code]} to %{INT:[http][response][status_code_new]} as follow:

HAPROXY %{IP:[client][ip]}:%{INT:[client][port]} \[%{HAPROXYDATE:[haproxy][timestamp]}\] %{NOTSPACE:[haproxy][frontend_name]} %{NOTSPACE:[haproxy][backend_name]}/%{NOTSPACE:[haproxy][server_name]} %{INT:[haproxy][time_request]}/%{INT:[haproxy][time_queue]}/%{INT:[haproxy][time_backend_connect]}/%{INT:[haproxy][time_backend_response]}/%{NOTSPACE:[host][uptime]} %{INT:[http][response][status_code_new]} %{NOTSPACE:[haproxy][bytes_read]} %{DATA:[haproxy][http][request][captured_cookie]} %{DATA:[haproxy][http][response][captured_cookie]} %{NOTSPACE:[haproxy][termination_state]} %{INT:[haproxy][connections][active]}/%{INT:[haproxy][connections][frontend]}/%{INT:[haproxy][connections][backend]}/%{INT:[haproxy][connections][server]}/%{NOTSPACE:[haproxy][connections][retries]} %{INT:[haproxy][server_queue]}/%{INT:[haproxy][backend_queue]} (\{%{HAPROXYCAPTUREDREQUESTHEADERS}\})?( )?(\{%{HAPROXYCAPTUREDRESPONSEHEADERS}\})?( )?"(<BADREQ>|(%{WORD:[http][request][method]} (%{URIPROTO:[haproxy][mode]}://)?(?:%{USER:[user][name]}(?::[^@]*)?@)?(?:%{URIHOST:[http][request][referrer]})?(?:%{URIPATHPARAM:[http][mode]})?( HTTP/%{NUMBER:[http][version]})?))?"?

Anyway I see that the field still have the same name in Elasticsearch:

For this reason, I think that that logs are not matching the HAPROXY grok pattern.

from pfelk.

Ok, now the new field name is matched:

So, I'm sure that the grok pattern matches.

Let me continue debugging.

from pfelk.

I tried to delete the data stream and index template, then I edited the grok pattern adding the :int modifier to the field:

%{INT:[http][response][status_code]:int}

I re-added the index template, anyway the field is still saved as keyword:

I really cannot understand what's not working here!

from pfelk.

@mmartinello the index_tempalte_pfelk-haproxy was updated to define several fields that did not have corresponding mappings (i.e., undefined).

This should resolve the issue but there might be additional steps on your end for it to take effect.

1. When updating template patterns where data is already stored, Elastic will likely persist with the original data type.

• You may need to purge the logs-pfelk-haproxy data stream

2. This may break the Stack Management>>Data Views>> *-pfelk-haproxy* view

• You may need to delete and recreate this view....if that's the case, please submit a PR to update the haproxy dashboard

from pfelk.

Yes, I removed the data stream after modifying the pattern.

Tomorrow I will do another try.

I saw you updated the index template and I used the latest version.
Just to be sure: did you test it? Do you have that field indexed as integer instead keyword?

from pfelk.

Ok, I made it:

We need to tell the grok pattern to convert the field to int as follow: %{INT:[http][response][status_code]:int}.

Now the question is: should it be made on all other INT�fields?

For example, port also is recognised as string:

And log.syslog.procid also:

If you confirm me that these fields should also be integer, I will send you a PR.

I'm also trying to fix the following visualization issue:

Thank you!

from pfelk.