tongchengopensource / ckibana Goto Github PK

欢迎体验,所有问题我们都会及时跟进(Welcome to try, we will follow up promptly if there are any questions.)

目前同程内部的日志系统已全部切换(Currently, the logging system within Tongcheng-Elong has been fully switched.)。

SET allow_experimental_object_type=1;
CREATE table github_json
(
event JSON
)
ENGINE = MergeTree ORDER BY tuple()
官方从22.3.1 版本后提供了JSON数据类型，针对半结构化数据比较友好，是否可以支持下这种数据类型结构

目前在创建索引时无法发现对应的表

在kibana 是正常的

从kibana登陆，无论用哪个用户登录，进去后都是elastic用户，是因为ckibana这块配置的原因吗？ 测试了下，没加ckibana前，用户登录是正常的，切换成ckibana后，用户登录就变成这样了。请帮忙给看下

手动填写后可以进行查询

当docker执行用户为root用户 和 文件 docker-compose/mock-filebeat.yml 用户不对应时，mock-filebeat 会启动失败。
Exiting: error loading config file: config file ("filebeat.yml") must be owned by the user identifier (uid=1000) or root
在quickstart 这种测试场景中，是不是考虑给ting001/file-beat-ck 加上 -strict.perms=false 启动参数
https://www.elastic.co/guide/en/beats/libbeat/5.3/config-file-permissions.html#_disabling_strict_permission_checks

CKibana version: v1.0.2
ES version:7.8.1
Operating system version: CentOS 7

报错如下：
com.alibaba.fastjson2.JSONException: parseLong error, value : {"value":0,"relation":"eq"}
at com.alibaba.fastjson2.JSONReader.toLong(JSONReader.java:2543)
at com.alibaba.fastjson2.JSONReader.getInt64Value(JSONReader.java:608)
at com.alibaba.fastjson2.JSONReaderUTF8.readInt64Value(JSONReaderUTF8.java:2125)
at com.alibaba.fastjson2.reader.ORG_3_3_Hits.readObject(Unknown Source)
at com.alibaba.fastjson2.reader.ORG_1_9_Response.readObject(Unknown Source)
at com.alibaba.fastjson2.JSON.parseObject(JSON.java:642)
at com.alibaba.fastjson2.JSONObject.parseObject(JSONObject.java:1942)
at com.ly.ckibana.handlers.SearchHandler.doHandle(SearchHandler.java:88)
at com.ly.ckibana.handlers.BaseHandler.handle(BaseHandler.java:76)
at com.ly.ckibana.configure.web.adpter.BaseHandlerAdapter.handle(BaseHandlerAdapter.java:39)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1067)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898)

使用clickhouse未创建字段查询，查询耗时较大。如未定义字段field1
example:
kibana查询：
field1:"xxx"

clickhouse sql:
SELECT * FROM db.table WHERE JSONExtractString(ck_assembly_extension,'field1') like '%/xxx%' ) ))

What is the problem your feature solves, or the need it fulfills?

现在一个clickhouse集群使用起来没有什么问题,但是我们有多个clickhouse集群,每个集群一套服务部署的有点多

Describe the solution you'd like

期望一套ckibana可以支持多套clickhouse集群,期望可以支持一下

example:
CREATE TABLE ops.ops_nginx_local
(
request_method LowCardinality(String),
request_uri String,
status Int64,
request_time Float64,
request_length Int64,
http_referer String,
http_user_agent String,
http_host String,
request_id String,
remote_addr String,
bytes_sent Int64,
body_bytes_sent Int64,
logtime UInt64 COMMENT 'TIMESTAMP',
ck_assembly_extension String,
INDEX timestamp_index @timestamp TYPE minmax GRANULARITY 8192
)
ENGINE = MergeTree
PARTITION BY (toYYYYMMDD(toDateTime(@timestamp / 1000, 'Asia/Shanghai')), toHour(toDateTime(@timestamp / 1000, 'Asia/Shanghai')))
ORDER BY (http_host, intHash64(@timestamp))
SAMPLE BY intHash64(@timestamp)
SETTINGS in_memory_parts_enable_wal = 0, index_granularity = 8192

In this case, we do not have a required column named '@timestamp’， ckibana knows 'logtimes' is the '@timestamp' base on comment( which include 'TIMESTAMP' substring)

CKibana info

CKibana version: v1.0.2
ClickHouse version: i.e. 23.4.1.1943
Operating system version: e.g. CentOS 7
k8s version: v1.26.4
elasticsearch kibana 8.8