Comments (2)
Решили проблему путем добавления фильтра в fluentd:
<filter **>
@type record_modifier
whitelist_keys timestamp, nsec, source, namespace, host, pod_name, container_name, stream, labels.names, labels.values, string_fields.names, string_fields.values, number_fields.names, number_fields.values, boolean_fields.names, boolean_fields.values, null_fields.names
</filter>
from loghouse.
Сделали проверку, что это за ошибка, путем модификации скрипта insert_ch.sh
Получили следующий вывод в консоль:
2017-11-08 07:12:42 +0000 [warn]: suppressed same stacktrace
Code: 117. DB::Exception: Unknown field found while parsing JSONEachRow format: _SELINUX_CONTEXT
2017-11-08 07:29:19 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-11-08 08:04:59 +0000 error_class="RuntimeError" error="command returns 29952: bash /usr/local/bin/insert_ch.sh /tmp/fluent-plugin-exec-20171108-7-11q1bjb" plugin_id="object:3f9d896aa8cc"
2017-11-08 07:29:19 +0000 [warn]: suppressed same stacktrace
// Это получили после модификации скрипта
{"_SELINUX_CONTEXT":"system_u:system_r:kernel_t:s0","_SYSTEMD_INVOCATION_ID":"dabc3bc11ebc4f8db8fc4d98fb8b235e","timestamp":1503909212,"source":"docker","host":"kubernetes-worker1","stream":"stdout","string_fields.names":["log"],"string_fields.values":["time=\"2017-08-28T08:33:32.389934701Z\" level=info msg=\"API listen on /var/run/docker.sock\""]}
Code: 117. DB::Exception: Unknown field found while parsing JSONEachRow format: _SELINUX_CONTEXT
2017-11-08 08:04:59 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-11-08 09:06:11 +0000 error_class="RuntimeError" error="command returns 29952: bash /usr/local/bin/insert_ch.sh /tmp/fluent-plugin-exec-20171108-7-1kudxtx" plugin_id="object:3f9d896aa8cc"
2017-11-08 08:04:59 +0000 [warn]: suppressed same stacktrace
Судя по всему, скрипт пытается запихнуть в БД, данные, которые не соответствуют структуре данных в таблице, поскольку содержат дополнительные ключи _SELINUX_CONTEXT, _SYSTEMD_INVOCATION_ID
Ожидаемая структура данных:
timestamp DateTime
nsec UInt32
source String
namespace String
host String
pod_name String
container_name String
stream String
labels.names Array(String)
labels.values Array(String)
string_fields.names Array(String)
string_fields.values Array(String)
number_fields.names Array(String)
number_fields.values Array(Float64)
boolean_fields.names Array(String)
boolean_fields.values Array(Float64)
null_fields.names Array(String)
Я не специалист во fluentd, но возможно, поможет модификация конфига в части обработки лог сообщения, как сделано здесь:
<filter kubelet.**>
@type record_modifier
<record>
timestamp ${time}
source "kubelet"
host ${record["_HOSTNAME"]}
stream ${record["_TRANSPORT"]}
string_fields.names ${["log"]}
string_fields.values ${[record["MESSAGE"]]}
</record>
remove_keys _TRANSPORT, PRIORITY, SYSLOG_FACILITY, _UID, _GID, _CAP_EFFECTIVE, _SYSTEMD_SLICE, _BOOT_ID, _MACHINE_ID, _HOSTNAME, SYSLOG_IDENTIFIER, _PID, _COMM, _EXE, _CMDLINE, _SYSTEMD_CGROUP, _SYSTEMD_UNIT, MESSAGE
</filter>
Как видно из конфига, запись формируется из конкретных полей лог сообщения, а невалидные ключи удаляются.
from loghouse.
Related Issues (20)
- Clickhouse doesn't respect cpu and memory limits HOT 1
- hierarchical JSON in log HOT 2
- Display multiline values with "pre" attribute HOT 1
- Helm chart: Clickhouse should be managed by StatefulSet instead of Deployment HOT 2
- Persistent volumes should be initialized
- Export in CSV file doesn't work HOT 3
- jessie-updates: 404 Not Found HOT 1
- Logs are not displayed on the dashboard
- Is there docker only (without kubernetes) installation method ? HOT 1
- Error deploying loghouse to GKE HOT 4
- Propose to add docker-compose file HOT 1
- Migrate release process to Github Actions
- Example configuration for external clickhouse? HOT 7
- Can Fluentd we swapped out for Fluent Bit by just changing the values.yaml image? HOT 1
- Helm chart is not compatible with the current cert-manager stable release HOT 1
- Unable to filter by label containing slash / symbol
- loghouse support containerd and fluent bit ?
- Logs UI
- PVC settings can't parse HOT 1
- Clickhouse still not respecting CPUShares in the latest version HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from loghouse.