Comments (13)
Ok I think I fixed the issues and made a PR, please let me know if it's ok for you :-)
from pfelk.
@mmartinello thank you for contributing...greatly appreciated!
from pfelk.
@mmartinello the haproxy was contributed/built by @BeNeDeLuX however, I believe they are now using nginx. I am unable to test this specific log type but believe I may have found the fix/solution.
The pfelk.grok file defines the http.response.status_code as an integer however, the index template did not contain a mapping to this field type. I've added a mapping and updated the haproxy template. Try reapplying the haproxy index template via the Templates Installation.
In the past when updating index template when data is already logged, I've experience issues and had to purge the index and data view when/if updating the index template fails to resolve the issue try the following:
-
stop sending haproxy logs or disconnect so that logs are not received
-
navigate to Stack Management>>Kibana and delete the -pfelk-haproxy data view
-
navigate to Stack Management>>Index Management>>Data Streams and delete "logs-pfelk-haproxy"
-
Add the data view back via installing the [HAProxy] Dashboard(https://github.com/pfelk/pfelk/blob/main/install/templates.md#two-dashboards)
-
enable logs to be sent (undo step 1)
-
check to see if the error persist
from pfelk.
@a3ilson I made what you told be but unfortunately this did not solved the problem:
![Screenshot 2024-06-20 at 16 45 41](https://private-user-images.githubusercontent.com/3822398/341458233-8045b595-2c8b-4eeb-a714-0f4d6e4ef5d6.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk3MzkyMzUsIm5iZiI6MTcxOTczODkzNSwicGF0aCI6Ii8zODIyMzk4LzM0MTQ1ODIzMy04MDQ1YjU5NS0yYzhiLTRlZWItYTcxNC0wZjRkNmU0ZWY1ZDYucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDYzMCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA2MzBUMDkxNTM1WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9MDBkZTZhMzMxMWZlODE3NGJkZjRmZDA1ZDIzZDczM2YyZWQxYzkzYjE5YTdhM2Y5OThjYTRmMTI0NTE2ZTM5ZiZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.waR-ceZj_C7zSI495C_HnWD9BiaU0wJzhcCP4_1GsRE)
I checked the new template and verified that status_code
is now mapped as integer in the template:
![Screenshot 2024-06-20 at 16 45 32](https://private-user-images.githubusercontent.com/3822398/341458394-4421c94a-a93a-452f-990b-8cc2dca19452.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk3MzkyMzUsIm5iZiI6MTcxOTczODkzNSwicGF0aCI6Ii8zODIyMzk4LzM0MTQ1ODM5NC00NDIxYzk0YS1hOTNhLTQ1MmYtOTkwYi04Y2MyZGNhMTk0NTIucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDYzMCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA2MzBUMDkxNTM1WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9MTBkMTI0Mjc0MjNlMzFjODQxMDIwNDk1YTMwMjE0NzdmODZmYjJjZWZjYjU0YTdmYzJiNTY2MTBhNzUzMjZhNyZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.Y5KIztb93TVscmGPQ3gEbooXPu_kk4RXM8nKns-E3W0)
but unfortunately it's still saved as keyword:
![Screenshot 2024-06-20 at 16 47 27](https://private-user-images.githubusercontent.com/3822398/341458454-e55f202d-291b-4783-8472-49e0e38631e0.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk3MzkyMzUsIm5iZiI6MTcxOTczODkzNSwicGF0aCI6Ii8zODIyMzk4LzM0MTQ1ODQ1NC1lNTVmMjAyZC0yOTFiLTQ3ODMtODQ3Mi00OWUwZTM4NjMxZTAucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDYzMCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA2MzBUMDkxNTM1WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9ZTIyZWQ2ZDRjNjU3MGNhYTkzMGYwOWQ0Yjg5Yzc1YzE1MGFlMWU1MzRkMzZmMTcxMzFlNWRjNzllMGVmNGViNCZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.RlCjAk90NbsDLQcs26f-oUDUTSEaMRwVB1DhE4ANb_s)
I have multiple instances of pfSense, I can make some tests and debugs.
Please tell me if they are needed, I can collaborate on this.
Thank you!
from pfelk.
That's odd... try setting up a new instance of pfelk and see if the values are still stored as "keywords" vs "int"
Alternatively, recreating the visualization would remedy the issue too.
from pfelk.
I completely deleted the instance and recreate it from scratch (I also removed Docker volumes).
I reinstalled all files using the new version but I got the same results.
I know it's odd but there should be a reason.
If you want I am still available for do some debugging.
from pfelk.
I'm debugging this.
For example, the following log entry:
Jun 26 16:30:07 fw1.dev.my.domain.com haproxy[88460]: 192.168.123.101:43540 [26/Jun/2024:16:30:07.403] Frontends.DEV-merged~ APM.DEV_ipvANY/elk.dev.my.domain.com 0/0/0/1/1 202 117 - - ---- 24/20/0/0/0 0/0 "POST /intake/v2/events HTTP/1.1"
seems to correctly match the grok pattern HAPROXY
:
HAPROXY %{IP:[client][ip]}:%{INT:[client][port]} \[%{HAPROXYDATE:[haproxy][timestamp]}\] %{NOTSPACE:[haproxy][frontend_name]} %{NOTSPACE:[haproxy][backend_name]}/%{NOTSPACE:[haproxy][server_name]} %{INT:[haproxy][time_request]}/%{INT:[haproxy][time_queue]}/%{INT:[haproxy][time_backend_connect]}/%{INT:[haproxy][time_backend_response]}/%{NOTSPACE:[host][uptime]} %{INT:[http][response][status_code]} %{NOTSPACE:[haproxy][bytes_read]} %{DATA:[haproxy][http][request][captured_cookie]} %{DATA:[haproxy][http][response][captured_cookie]} %{NOTSPACE:[haproxy][termination_state]} %{INT:[haproxy][connections][active]}/%{INT:[haproxy][connections][frontend]}/%{INT:[haproxy][connections][backend]}/%{INT:[haproxy][connections][server]}/%{NOTSPACE:[haproxy][connections][retries]} %{INT:[haproxy][server_queue]}/%{INT:[haproxy][backend_queue]} (\{%{HAPROXYCAPTUREDREQUESTHEADERS}\})?( )?(\{%{HAPROXYCAPTUREDRESPONSEHEADERS}\})?( )?"(<BADREQ>|(%{WORD:[http][request][method]} (%{URIPROTO:[haproxy][mode]}://)?(?:%{USER:[user][name]}(?::[^@]*)?@)?(?:%{URIHOST:[http][request][referrer]})?(?:%{URIPATHPARAM:[http][mode]})?( HTTP/%{NUMBER:[http][version]})?))?"?
Many fields are indexed as keyword
instead as int
:
%{INT:[client][port]}
%{INT:[haproxy][time_request]}
%{INT:[haproxy][time_backend_response]}
%{INT:[http][response][status_code]}
%{INT:[haproxy][client][port]}
%{INT:[haproxy][time_request]}
%{INT:[haproxy][time_backend_response]}
![Screenshot 2024-06-26 at 18 52 22](https://private-user-images.githubusercontent.com/3822398/343253496-fd79768d-8fcf-4c40-bde1-9dad6b3ee372.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk3MzkyMzUsIm5iZiI6MTcxOTczODkzNSwicGF0aCI6Ii8zODIyMzk4LzM0MzI1MzQ5Ni1mZDc5NzY4ZC04ZmNmLTRjNDAtYmRlMS05ZGFkNmIzZWUzNzIucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDYzMCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA2MzBUMDkxNTM1WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9YmFlM2MxNjgwMzI4ODgyNWE5M2RkNzAxYjUyNDhiMjA5ZDQzNGU1YmNkNDMzYmJmMjlkZTFmZWU2N2E2NzczMSZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.cZQf-_-3L1h7Vd63sh-K-9_HAdwbYZKSophCt_3Muiw)
![Screenshot 2024-06-26 at 18 52 29](https://private-user-images.githubusercontent.com/3822398/343253512-3fbbff5f-b17c-4799-9dfd-d41012461971.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk3MzkyMzUsIm5iZiI6MTcxOTczODkzNSwicGF0aCI6Ii8zODIyMzk4LzM0MzI1MzUxMi0zZmJiZmY1Zi1iMTdjLTQ3OTktOWRmZC1kNDEwMTI0NjE5NzEucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDYzMCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA2MzBUMDkxNTM1WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9YmJmMjI0ZDBkNTE0NzFkNjBmZmNkMGQ2YjU0ZDJmNGRkNjU1OWE2OGE0ZmIwMGQ3MGNiOGZjN2NkOGFlODFjOCZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.I-VavIWyzEQy5CX5xYgTgZLiUtkDvgvWTHQL0ycTA3w)
![Screenshot 2024-06-26 at 18 52 36](https://private-user-images.githubusercontent.com/3822398/343253546-4713f4f8-3330-4fa8-93ed-5b93325e95f1.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk3MzkyMzUsIm5iZiI6MTcxOTczODkzNSwicGF0aCI6Ii8zODIyMzk4LzM0MzI1MzU0Ni00NzEzZjRmOC0zMzMwLTRmYTgtOTNlZC01YjkzMzI1ZTk1ZjEucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDYzMCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA2MzBUMDkxNTM1WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9ZmQ3ZTE3YmM4ODZmNjI3YzNhNTMxY2FhZTllN2NjYWQyN2Q0MDc0NDE3YjRkM2U4Zjk2MTExYjE1NTgyNDc0NCZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.itlppOD8duhce2IuH--a_So9fNaYqkvaK1BtFGbaoWU)
![Screenshot 2024-06-26 at 18 52 44](https://private-user-images.githubusercontent.com/3822398/343253567-cf0f4a45-6983-451e-a17a-48c0e07e8d71.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk3MzkyMzUsIm5iZiI6MTcxOTczODkzNSwicGF0aCI6Ii8zODIyMzk4LzM0MzI1MzU2Ny1jZjBmNGE0NS02OTgzLTQ1MWUtYTE3YS00OGMwZTA3ZThkNzEucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDYzMCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA2MzBUMDkxNTM1WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9MjE4ZjQwZjJjM2VlMGYzZTk1OTFlYzVhMmQxYWVjNmMxOTAwNWU3OTBmYjVkZTQwMjNhMTBlY2RhNDRmNGYwYyZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.fR0OeuFz65JtMp-J1CMPH3uPDAQc5JFO4S3oKHAez2w)
![Screenshot 2024-06-26 at 18 52 52](https://private-user-images.githubusercontent.com/3822398/343253613-364369ad-45d0-4d75-a764-915e815ff1d4.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk3MzkyMzUsIm5iZiI6MTcxOTczODkzNSwicGF0aCI6Ii8zODIyMzk4LzM0MzI1MzYxMy0zNjQzNjlhZC00NWQwLTRkNzUtYTc2NC05MTVlODE1ZmYxZDQucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDYzMCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA2MzBUMDkxNTM1WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9YjZmNTFmYzMxZGNiMDM5MWMxNTdmOTI0NTc4ZWVhYjA0ODVhY2JlZWFlOTdjODY2NDc3Mzg4ZjY4ZTk1MDgxZSZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.fj4AGq1l4qgqrJrePqV6kv1i8Lg-j-4iPC0h1vpmm-o)
![Screenshot 2024-06-26 at 18 52 58](https://private-user-images.githubusercontent.com/3822398/343253634-e01eb721-9580-4432-aa3a-15747982ed2b.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk3MzkyMzUsIm5iZiI6MTcxOTczODkzNSwicGF0aCI6Ii8zODIyMzk4LzM0MzI1MzYzNC1lMDFlYjcyMS05NTgwLTQ0MzItYWEzYS0xNTc0Nzk4MmVkMmIucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDYzMCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA2MzBUMDkxNTM1WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9ZWFlYzRiMjIxZDY3NWZkMmZhZmY0YTVlMjk2ZGRlYzU3ZDZjZDQwOGRiYzc1MzEwZGRhMTFlMDU3YWEyYmFkZiZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.FRzEQSWG6TXlZkWgMyZII30suKsTe6f0_mcdKaoNaYk)
![Screenshot 2024-06-26 at 18 56 25](https://private-user-images.githubusercontent.com/3822398/343254381-9fb444a4-fce9-4f1c-b61f-02a4c0a274e7.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk3MzkyMzUsIm5iZiI6MTcxOTczODkzNSwicGF0aCI6Ii8zODIyMzk4LzM0MzI1NDM4MS05ZmI0NDRhNC1mY2U5LTRmMWMtYjYxZi0wMmE0YzBhMjc0ZTcucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDYzMCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA2MzBUMDkxNTM1WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9MTI2YTZmM2EzNWYwMzg2YmYwMzg5ZjQzNmRhN2MxYmNlM2I5ZGE4NGVmOTc1MWM3N2U2NWIyMGMzMTkxYmVmNSZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.4NXo8bN6arRqnaCOLgNWUHFuap4CXUcVUYXszlwaEQg)
Other fields are correctly indexed as int
:
%{INT:[haproxy][time_queue]}
%{INT:[haproxy][time_backend_connect]}
%{INT:[haproxy][connections][active]}
%{INT:[haproxy][connections][frontend]}
%{INT:[haproxy][connections][backend]}
%{INT:[haproxy][connections][server]}
%{INT:[haproxy][server_queue]}
%{INT:[haproxy][backend_queue]}
%{INT:[haproxy][time_queue]}
%{INT:[haproxy][time_backend_connect]}
![Screenshot 2024-06-26 at 18 56 25](https://private-user-images.githubusercontent.com/3822398/343254615-0cb3e2ef-c4b3-4e79-a539-06d6e8e588c4.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk3MzkyMzUsIm5iZiI6MTcxOTczODkzNSwicGF0aCI6Ii8zODIyMzk4LzM0MzI1NDYxNS0wY2IzZTJlZi1jNGIzLTRlNzktYTUzOS0wNmQ2ZThlNTg4YzQucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDYzMCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA2MzBUMDkxNTM1WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9MjU2Zjg2NTQ0YTBmYzI5NDQxZTY2MjdmYjI2ZmE2ZDY0MTY0NWEzZDFjODBkYTQyNmViY2RhNDUzNzQ5OGNjZCZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.iieibO2O5VqjjDdWgHF1bFxCbQNKDZBy_23pgvCdaNA)
![Screenshot 2024-06-26 at 18 56 16](https://private-user-images.githubusercontent.com/3822398/343254618-86b54d77-0566-4fad-b77a-af02978f0b6a.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk3MzkyMzUsIm5iZiI6MTcxOTczODkzNSwicGF0aCI6Ii8zODIyMzk4LzM0MzI1NDYxOC04NmI1NGQ3Ny0wNTY2LTRmYWQtYjc3YS1hZjAyOTc4ZjBiNmEucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDYzMCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA2MzBUMDkxNTM1WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9ZTAwNTkzZmFiNDZjMjk1MjRhMzEyNzQwOTkxMDA0OWM3OTUwOGQ0NzkzYjg4OWEzNTRiYTliOWJlYzRmY2QwNyZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.Lef1IoLpQEtkGbBTaxyg6A7j26FNvP2v4LKGZeMoeMA)
![Screenshot 2024-06-26 at 18 56 10](https://private-user-images.githubusercontent.com/3822398/343254623-50397dfb-6400-444a-9e46-3cc33843ce26.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk3MzkyMzUsIm5iZiI6MTcxOTczODkzNSwicGF0aCI6Ii8zODIyMzk4LzM0MzI1NDYyMy01MDM5N2RmYi02NDAwLTQ0NGEtOWU0Ni0zY2MzMzg0M2NlMjYucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDYzMCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA2MzBUMDkxNTM1WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9YzQ5ZGY4NDYwYThlNDEzNDI1YzQyZWQ3NzZiNjJjYTQ0NWJjYTczZDBhMDM3MGEwYjY1ZDRkZTZhYjFkMTM4YyZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.jwDOuaYoiigTrIIgbQWIK7WnoiW_2iKW6duD2IE7i4g)
![Screenshot 2024-06-26 at 18 56 05](https://private-user-images.githubusercontent.com/3822398/343254628-6eea4880-2635-45db-a4df-ca841634fc0b.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk3MzkyMzUsIm5iZiI6MTcxOTczODkzNSwicGF0aCI6Ii8zODIyMzk4LzM0MzI1NDYyOC02ZWVhNDg4MC0yNjM1LTQ1ZGItYTRkZi1jYTg0MTYzNGZjMGIucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDYzMCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA2MzBUMDkxNTM1WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9Yjc4ZWFkYjI2NTUyOTAzNjQ5ZTU1ZTg3M2U4NWY5NDE1OGI2ZTBmYzI2OTM2Mjk1OTRkYjJhYTFjYzIyZjI5OSZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.NIFlhCh_7fh1uZWc14k7_YA4MNjYBC1VY0UqLFFNUHQ)
![Screenshot 2024-06-26 at 18 55 59](https://private-user-images.githubusercontent.com/3822398/343254632-c008d6d5-927f-4962-ac4e-f9466e637c19.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk3MzkyMzUsIm5iZiI6MTcxOTczODkzNSwicGF0aCI6Ii8zODIyMzk4LzM0MzI1NDYzMi1jMDA4ZDZkNS05MjdmLTQ5NjItYWM0ZS1mOTQ2NmU2MzdjMTkucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDYzMCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA2MzBUMDkxNTM1WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9YzEzYTQ0ZDM0NjkwZTczZmE1NWM2NGI3NGQ0MjY2MmQzZjZiMzMyMGMyOGUwOGEzZGEwNGM2NGMxYjE2ZjAzNSZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.d12vbxTHs5PlTLHuBYMXUq3L6V53UZlWpnBvS9ReOFg)
![Screenshot 2024-06-26 at 18 55 53](https://private-user-images.githubusercontent.com/3822398/343254634-15876f27-ebe6-47d5-b104-307ea1167190.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk3MzkyMzUsIm5iZiI6MTcxOTczODkzNSwicGF0aCI6Ii8zODIyMzk4LzM0MzI1NDYzNC0xNTg3NmYyNy1lYmU2LTQ3ZDUtYjEwNC0zMDdlYTExNjcxOTAucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDYzMCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA2MzBUMDkxNTM1WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9MzNhZTYyNTY0NTBhYzExM2U5ZDUxNjI2ZWNjNjg3Mjk4MjJiMDIzMGRlNzI3YzQzZTE1ZjM0YzZhODhmYzY2OCZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.48nWTk-yIU5wOaZrvFA6HvGmsUN6sB6Bqv_rl5fJom8)
![Screenshot 2024-06-26 at 18 55 46](https://private-user-images.githubusercontent.com/3822398/343254638-5ce4274d-7bf1-4b3b-8b0b-8fb981699f04.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk3MzkyMzUsIm5iZiI6MTcxOTczODkzNSwicGF0aCI6Ii8zODIyMzk4LzM0MzI1NDYzOC01Y2U0Mjc0ZC03YmYxLTRiM2ItOGIwYi04ZmI5ODE2OTlmMDQucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDYzMCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA2MzBUMDkxNTM1WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9ZGZkZjBiMDAxMmY4NmQ2MTcwNjE4YzVjYmQxODE0MmJkZWZkNjhiYmFmODM2YzczYWFiYzE3OTQwMzNlZGZiOSZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.CHZzOFhijUTj_yFitLfEknsu_4kwlT2kJEBEC7o5Xcs)
![Screenshot 2024-06-26 at 18 55 15](https://private-user-images.githubusercontent.com/3822398/343254640-36314a70-a936-4e97-89c4-73872da1d94c.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk3MzkyMzUsIm5iZiI6MTcxOTczODkzNSwicGF0aCI6Ii8zODIyMzk4LzM0MzI1NDY0MC0zNjMxNGE3MC1hOTM2LTRlOTctODljNC03Mzg3MmRhMWQ5NGMucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDYzMCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA2MzBUMDkxNTM1WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9YmY1MTI1MWU4YTY5YTM3MmZmZjllMmI0NDQzZjIwMzg1OWI3YzNlY2ZhNzc3MTgwZDVmNTRlODY2OWE1NzA4NyZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.OTmrNo-3-CsIOhzQdP_PuQIicfvZWDGMCtizsZgKUfQ)
![Screenshot 2024-06-26 at 18 55 07](https://private-user-images.githubusercontent.com/3822398/343254645-3b046694-b711-4d46-9adf-8755c68da437.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk3MzkyMzUsIm5iZiI6MTcxOTczODkzNSwicGF0aCI6Ii8zODIyMzk4LzM0MzI1NDY0NS0zYjA0NjY5NC1iNzExLTRkNDYtOWFkZi04NzU1YzY4ZGE0MzcucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDYzMCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA2MzBUMDkxNTM1WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9MWZkYmMyMWRmOGM2N2IwYWVlODc4NmNiZDAzODIwZGI5MDczNGZiNzY2YzNhZjZmYmUxMzNmNDY4MjBjMWYxNCZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.U6oIMBpEl9xfrzcZzO2_jefVVg0OAe0BaLk25rwTfNk)
![Screenshot 2024-06-26 at 18 55 01](https://private-user-images.githubusercontent.com/3822398/343254646-8c927822-840e-4554-b807-963ab6682fe1.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk3MzkyMzUsIm5iZiI6MTcxOTczODkzNSwicGF0aCI6Ii8zODIyMzk4LzM0MzI1NDY0Ni04YzkyNzgyMi04NDBlLTQ1NTQtYjgwNy05NjNhYjY2ODJmZTEucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDYzMCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA2MzBUMDkxNTM1WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9MGMwZDJlNGE5OTlkODNlZWNmYzc5ZTk2NDc4YWY5YjNiN2E0YWQzMjljOGFmZDQzZDU5MDYyNTJiYmEzYTc1OSZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.nNVjNNz7Vm7uc01YCPrNy9N80sSvutx1Tu5nWxEjKtY)
![Screenshot 2024-06-26 at 18 54 56](https://private-user-images.githubusercontent.com/3822398/343254650-afdcdb22-ad2f-4945-afd2-ca4b8958c792.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk3MzkyMzUsIm5iZiI6MTcxOTczODkzNSwicGF0aCI6Ii8zODIyMzk4LzM0MzI1NDY1MC1hZmRjZGIyMi1hZDJmLTQ5NDUtYWZkMi1jYTRiODk1OGM3OTIucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDYzMCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA2MzBUMDkxNTM1WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9MjA0OTdmMTAyOWU1YjIyZDgxNTQyOTkzZWIzZDVjODYxNWU1ZDMzODQ3YWU3YTM1Y2ZjOGUxMjQ2MzY3NzRkZSZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.am6Pz_5xN9jfDj0PUX2Z0epMCmuNZfBNEg6qupWp8UM)
I'm trying to find out what's happening, if you have any idea please let me know.
from pfelk.
I tried to change the field name in the grok pattern renaming %{INT:[http][response][status_code]}
to %{INT:[http][response][status_code_new]}
as follow:
HAPROXY %{IP:[client][ip]}:%{INT:[client][port]} \[%{HAPROXYDATE:[haproxy][timestamp]}\] %{NOTSPACE:[haproxy][frontend_name]} %{NOTSPACE:[haproxy][backend_name]}/%{NOTSPACE:[haproxy][server_name]} %{INT:[haproxy][time_request]}/%{INT:[haproxy][time_queue]}/%{INT:[haproxy][time_backend_connect]}/%{INT:[haproxy][time_backend_response]}/%{NOTSPACE:[host][uptime]} %{INT:[http][response][status_code_new]} %{NOTSPACE:[haproxy][bytes_read]} %{DATA:[haproxy][http][request][captured_cookie]} %{DATA:[haproxy][http][response][captured_cookie]} %{NOTSPACE:[haproxy][termination_state]} %{INT:[haproxy][connections][active]}/%{INT:[haproxy][connections][frontend]}/%{INT:[haproxy][connections][backend]}/%{INT:[haproxy][connections][server]}/%{NOTSPACE:[haproxy][connections][retries]} %{INT:[haproxy][server_queue]}/%{INT:[haproxy][backend_queue]} (\{%{HAPROXYCAPTUREDREQUESTHEADERS}\})?( )?(\{%{HAPROXYCAPTUREDRESPONSEHEADERS}\})?( )?"(<BADREQ>|(%{WORD:[http][request][method]} (%{URIPROTO:[haproxy][mode]}://)?(?:%{USER:[user][name]}(?::[^@]*)?@)?(?:%{URIHOST:[http][request][referrer]})?(?:%{URIPATHPARAM:[http][mode]})?( HTTP/%{NUMBER:[http][version]})?))?"?
Anyway I see that the field still have the same name in Elasticsearch:
![Screenshot 2024-06-26 at 20 09 32](https://private-user-images.githubusercontent.com/3822398/343281951-5aac9e5d-4690-42da-89f9-83716fafb543.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk3MzkyMzUsIm5iZiI6MTcxOTczODkzNSwicGF0aCI6Ii8zODIyMzk4LzM0MzI4MTk1MS01YWFjOWU1ZC00NjkwLTQyZGEtODlmOS04MzcxNmZhZmI1NDMucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDYzMCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA2MzBUMDkxNTM1WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9NjE4ODEzNGE5MzAwY2U2ZDZiOWQwOWMwYTNmNjVhMDJhYTA5M2FmYjhlN2Q3MjFlYmE4YmFjNDRiOTEyNWEyZCZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.A9fHFKLwjD-oh_jPYMfa-aJO2IVNpEQ-Xw0MrP3Tv7c)
For this reason, I think that that logs are not matching the HAPROXY
grok pattern.
from pfelk.
Ok, now the new field name is matched:
![Screenshot 2024-06-26 at 20 34 07](https://private-user-images.githubusercontent.com/3822398/343291194-0e11addd-ef37-46bc-a53f-e808f2581ded.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk3MzkyMzUsIm5iZiI6MTcxOTczODkzNSwicGF0aCI6Ii8zODIyMzk4LzM0MzI5MTE5NC0wZTExYWRkZC1lZjM3LTQ2YmMtYTUzZi1lODA4ZjI1ODFkZWQucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDYzMCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA2MzBUMDkxNTM1WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9NmE0ZDQ0YTdiZGI4ZjQ1NjI0OTI4OGYwZWY4NGMzNzFiYjA4N2M3NWY4YWE2MGVlMDA1OGYxNmZlMjYxYmJlYyZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.A2XASKy8s9NC6xdY4EIQNP1OZ7qQFP78q377zUEkgss)
So, I'm sure that the grok pattern matches.
Let me continue debugging.
from pfelk.
I tried to delete the data stream and index template, then I edited the grok pattern adding the :int
modifier to the field:
%{INT:[http][response][status_code]:int}
I re-added the index template, anyway the field is still saved as keyword:
![Screenshot 2024-06-26 at 20 45 56](https://private-user-images.githubusercontent.com/3822398/343294503-d35967dc-32bd-4e8d-a5ec-e78feb0d5500.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk3MzkyMzUsIm5iZiI6MTcxOTczODkzNSwicGF0aCI6Ii8zODIyMzk4LzM0MzI5NDUwMy1kMzU5NjdkYy0zMmJkLTRlOGQtYTVlYy1lNzhmZWIwZDU1MDAucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDYzMCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA2MzBUMDkxNTM1WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9YzI5M2NiZDhjMWQ4NzVmMTQyNDIwZjdmNDQ0MDVlMjIwMDc2ZThkYjA3YTJhMWFjYzIxNjdiNGRmMTM0MDcwZiZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.ixZ_sfCuk-Lj4_ktSVR7L165JKTGXA3nxGFGGEccOcg)
I really cannot understand what's not working here!
from pfelk.
@mmartinello the index_tempalte_pfelk-haproxy was updated to define several fields that did not have corresponding mappings (i.e., undefined).
This should resolve the issue but there might be additional steps on your end for it to take effect.
- When updating template patterns where data is already stored, Elastic will likely persist with the original data type.
- You may need to purge the logs-pfelk-haproxy data stream
- This may break the
Stack Management>>Data Views>> *-pfelk-haproxy*
view
- You may need to delete and recreate this view....if that's the case, please submit a PR to update the haproxy dashboard
from pfelk.
Yes, I removed the data stream after modifying the pattern.
Tomorrow I will do another try.
I saw you updated the index template and I used the latest version.
Just to be sure: did you test it? Do you have that field indexed as integer instead keyword?
from pfelk.
Ok, I made it:
We need to tell the grok pattern to convert the field to int as follow: %{INT:[http][response][status_code]:int}
.
Now the question is: should it be made on all other INT
�fields?
For example, port
also is recognised as string:
And log.syslog.procid
also:
If you confirm me that these fields should also be integer, I will send you a PR.
I'm also trying to fix the following visualization issue:
Thank you!
from pfelk.
Related Issues (20)
- Firewall/Unbound dashboard issue - "Adjust autocomplete settings in kibana.yml" HOT 9
- OpenVPN+ELK Stack HOT 3
- Is there any updated documentation on starting this up in Docker? HOT 2
- Unable to create DHCP Dashboard, DHCP Saved Objects as there is a data view conflict HOT 7
- Error in logstash parse: illegal_argument_exception HOT 3
- issue when installing docker pfelk HOT 5
- Can't select firewall entry in Dashboard after uprade with Unable to fetch terms, error HOT 2
- Dashboard templates not receiving data HOT 11
- Excessive Null Values/Fields Due To "pfelk-mappings-ecs" HOT 1
- PFsense logs not being sent to PFelk HOT 1
- Data not appearing in dashboards HOT 15
- kea dhcp support HOT 15
- ngnix template error HOT 1
- Snort Dashboard - most data missing HOT 18
- Error loading Suricate template HOT 4
- Kibana will not start after changing VM (Ubuntu) IP Address HOT 1
- Cannot get data to display on NGINX dashboard HOT 16
- Cannot get data to display Suricata dashbord and Firewall dashbord HOT 8
- index_not_found_exception on logs-pfelk-haproxy HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pfelk.