infinilabs / gateway Goto Github PK
View Code? Open in Web Editor NEW🚀 A High-performance gateway designed for search scenarios. Good friend of Elasticsearch/Opensearch/Easysearch.
Home Page: https://www.infinilabs.com/en/docs/latest/gateway/
🚀 A High-performance gateway designed for search scenarios. Good friend of Elasticsearch/Opensearch/Easysearch.
Home Page: https://www.infinilabs.com/en/docs/latest/gateway/
请问一下,网关能对 _msearch的查询方法进行路由吗?根据索引名称发送到不同的es进行查询?
大大好,gateway在本地无法运行,拉取依赖时发现 https://github.com/medcl/infini-framework.git 不存在。
A description of what you expected to happen.
A description of what happens instead.
极限网关通过如下命令启动时,日志显示:unable to dial: dial tcp 127.0.0.1:9092: connetct: connection refused
nohup ./gateway-linux-amd64 -config replication_via-kafka.yml &
在replication_via-kafka.yml文件中,已经指定了远程的KAFKA_HOSTS: [ 'xx.xx.xx.xx:9092' ]
who can help me?
你好,网关缓存功能确实很有用,但是就是统计信息还不够详细,想问一下,这边有统计缓存的使用情况吗?比如使用了多少内存,缓存中主要是哪一些请求缓存了等等
尝试测试 date_range_precision_tuning,time_precision:6,预设置精度到“秒”,配置之后kibana请求报错了。
去掉改配置后正常。
flow:
- name: default_flow
filter:
- context_regex_replace:
context: "_ctx.request.uri"
pattern: "_from=[^&]*&?"
to: ""
- date_range_precision_tuning:
time_precision: 6
- get_cache:
- bulk_reshuffle:
when:
contains:
_ctx.request.path: /_bulk
elasticsearch: prod
level: node
fix_null_id: true
- elasticsearch:
elasticsearch: prod #elasticsearch configure reference name
max_connection_per_node: 1000 #max tcp connection to upstream, default for all nodes
max_response_size: -1 #default for all nodes
balancer: weight
refresh: # refresh upstream nodes list, need to enable this feature to use elasticsearch nodes auto discovery
enabled: true
interval: 60s
- set_cache:
cache_type: ristretto
min_response_size: 100
max_response_size: 1024000
cache_ttl: 30s
max_cache_items: 100000
应该正常按秒格式化kibana请求的时间
failed to parse date field [2022-08-00T05:59:32.999Z] with format [strict_date_optional_time]: [failed to parse date field [2022-08-00T05:59:32.999Z] with format [strict_date_optional_time]]
抓包结果内容:
kibana_http_400.json.zip
gateway: 1.6.0_SNAPSHOT
es: 7.8.1
kibana: 7.8.1
Build:
gateway 1.6.0_SNAPSHOT 657 2022-05-31 07:35:13 +0000 UTC 2023-12-31 10:10:10 +0000 UTC 21ec7be5de546f5454b3ccb58b2ce9183cd1ff2d
Operating system and version:
Centos 7
您好,之前在elastic社区和elastic线下活动见过您也有一些交流。新发现您开发了极限网关这个应用,我们目前有一个ES集群升级的项目想要做,想和您交流一下极限网关在使用上的可行性。
我们现在的ES集群是5版本,有3个机房,每个机房都是独立的一套集群,存储本机房数据。部署了专门的ccs节点连接这3个机房的ES集群,提供汇总数据查询功能。
目前打算将集群从5版本,升级到7版本,但是服务器有限,无法搭建和老版本等容量的集群,因此采用少量服务器先搭建ES 7集群,部分日志数据同时写入两个集群。
这时候希望借用极限网关的功能,客户端程序访问极限网关,通过判断要查询索引的名字,路由到不同的集群进行查询,架构如下:
当我们验证相关索引没问题后,通过下线ES 5的机器,扩容到ES 7 ,并再次迁移部分索引,重复相同的步骤,直到服务器和索引都放到ES 7 集群。
想和和您交流一下,这个方案是否可行?判断索引名称进行路由使用什么配置实现?这里是否有哪些需要注意的坑?或者您有什么更好的建议,希望您指点。我们希望对客户端服务是无感的,不影响业务。
非常感谢。
我们在生产环境使用极限网关同步ES数据到灾备环境,过了一段时间,发现kafka宕机了,查找问题是因为kafka硬盘被写满了,看kafka数据全是极限网关发送的监控数据,有10G左右,最后分析kafka数据,差不多每天产生1.3-1.5G的监控数据,但是开发环境并没有这么大的数据量。
通过分析kafka数据,发现生产环境10s产生18条监控数据,而开发环境只产生2条,这个就是为什么生产数据量大的原因,但是为啥每次会产生18条这么多的数据,这个是我的疑问?请各位大佬分析分析
Many ERROR log [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
systemctl start gateway
journalctl -exu gateway --no-pager
Jan 16 08:55:59 m178 gateway[3188]: [01-16 08:55:59] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:55:59 m178 gateway[3188]: [01-16 08:55:59] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:00 m178 gateway[3188]: [01-16 08:56:00] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:56:00 m178 gateway[3188]: [01-16 08:56:00] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:01 m178 gateway[3188]: [01-16 08:56:01] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:56:01 m178 gateway[3188]: [01-16 08:56:01] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:02 m178 gateway[3188]: [01-16 08:56:02] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:56:02 m178 gateway[3188]: [01-16 08:56:02] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:03 m178 gateway[3188]: [01-16 08:56:03] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:56:03 m178 gateway[3188]: [01-16 08:56:03] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:04 m178 gateway[3188]: [01-16 08:56:04] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:56:04 m178 gateway[3188]: [01-16 08:56:04] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:05 m178 gateway[3188]: [01-16 08:56:05] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:56:05 m178 gateway[3188]: [01-16 08:56:05] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:06 m178 gateway[3188]: [01-16 08:56:06] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:56:06 m178 gateway[3188]: [01-16 08:56:06] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:07 m178 gateway[3188]: [01-16 08:56:07] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:07 m178 gateway[3188]: [01-16 08:56:07] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:56:08 m178 gateway[3188]: [01-16 08:56:08] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:08 m178 gateway[3188]: [01-16 08:56:08] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:56:09 m178 gateway[3188]: [01-16 08:56:09] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:09 m178 gateway[3188]: [01-16 08:56:09] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:56:10 m178 gateway[3188]: [01-16 08:56:10] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:10 m178 gateway[3188]: [01-16 08:56:10] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:56:11 m178 gateway[3188]: [01-16 08:56:11] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:56:11 m178 gateway[3188]: [01-16 08:56:11] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:12 m178 gateway[3188]: [01-16 08:56:12] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:56:12 m178 gateway[3188]: [01-16 08:56:12] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:13 m178 gateway[3188]: [01-16 08:56:13] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:56:13 m178 gateway[3188]: [01-16 08:56:13] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:14 m178 gateway[3188]: [01-16 08:56:14] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:14 m178 gateway[3188]: [01-16 08:56:14] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:56:15 m178 gateway[3188]: [01-16 08:56:15] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:56:15 m178 gateway[3188]: [01-16 08:56:15] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:16 m178 gateway[3188]: [01-16 08:56:16] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:56:16 m178 gateway[3188]: [01-16 08:56:16] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:17 m178 gateway[3188]: [01-16 08:56:17] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:17 m178 gateway[3188]: [01-16 08:56:17] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:56:18 m178 gateway[3188]: [01-16 08:56:18] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:56:18 m178 gateway[3188]: [01-16 08:56:18] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:19 m178 gateway[3188]: [01-16 08:56:19] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:19 m178 gateway[3188]: [01-16 08:56:19] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
/lib/systemd/system/gateway.service
systemctl cat gateway
# /lib/systemd/system/gateway.service
[Unit]
Description=cubefs
After=network-online.target local-fs.target time-sync.target
Wants=network-online.target local-fs.target time-sync.target
[Service]
Type=simple
ExecStart=/usr/local/bin/gateway -config /data/es/gateway/config/gateway.yml
ExecStop=/bin/kill -s TERM $MAINPID
Restart=on-failure
RestartSec=10s
LimitNOFILE=40000
[Install]
WantedBy=multi-user.target
/data/es/gateway/config/gateway.yml
allow_multi_instance: true
path.data: /data/es/gateway/data
path.logs: /data/es/gateway/log
elasticsearch:
- name: cluster01
enabled: true
endpoint: http://192.168.6.175:9200
basic_auth:
username: elastic
password: test123
traffic_control.max_bytes_per_node: 1010485760
metadata_cache_enabled: false # Whether to cache the cluster info in memory cache
- name: cluster02
enabled: true
endpoint: http://192.168.6.176:9200
basic_auth:
username: elastic
password: test123
discovery:
enabled: false
entry:
- name: my_es_entry
enabled: true
router: my_router
max_concurrency: 200000
network:
binding: 0.0.0.0:8000
reuse_port: true
flow:
# 写请求优先发给主集群, 当主集群不可用时发给备用集群
# 当主集群数据写入成功时,记录到队列中,异步消费写入备用集群
- name: write-flow
filter:
- if:
# 当主集群可用时
cluster_available: ["cluster01"]
then:
# 先将数据写入主集群
- elasticsearch:
elasticsearch: "cluster01"
# 写入消息队列,等待 pipeline 异步消费到备用集群
- queue:
queue_name: "cluster02-queue"
else:
- elasticsearch:
elasticsearch: "cluster02"
- queue:
queue_name: "cluster01-queue"
# 读请求优先发给主集群, 当主集群不可用时发给备用集群
- name: read-flow
filter:
- if:
cluster_available: ["cluster01"]
then:
- elasticsearch:
elasticsearch: "cluster01"
else:
- elasticsearch:
elasticsearch: "cluster02"
router:
- name: my_router
# 默认路由
default_flow: write-flow
# 读请求路由
rules:
- method:
- "GET"
- "HEAD"
pattern:
- "/{any:*}"
flow:
- read-flow
- method:
- "POST"
- "GET"
pattern:
- "/_refresh"
- "/_count"
- "/_search"
- "/_msearch"
- "/_mget"
- "/{any_index}/_count"
- "/{any_index}/_search"
- "/{any_index}/_msearch"
- "/{any_index}/_mget"
flow:
- read-flow
pipeline:
- name: cluster01-consumer
auto_start: true
keep_running: true
processor:
- queue_consumer:
input_queue: "cluster01-queue"
elasticsearch: "cluster01"
when:
cluster_available: ["cluster01"] # 当集群可用时,才消费队列中的数据
- name: cluster02-consumer
auto_start: true
keep_running: true
processor:
- queue_consumer:
input_queue: "cluster02-queue"
elasticsearch: "cluster02"
when:
cluster_available: ["cluster02"]
disk_queue:
prepare_files_to_read: true
#max_bytes_per_file: 20971520
eof_retry_delay_in_ms: 500
cleanup_files_on_init: false
retention:
max_num_of_local_files: 20 # automatically cleanup consumed files
compress:
segment:
enabled: true
delete_after_compress: true # trigger cleanup after compression.
idle_threshold: 20 # max number of uncompressed consumed files to preserve.
badger:
enabled: true
single_bucket_mode: true
path: '' # defaults to {path.data}/gateway/node/{nodeID}/badger/
memory_mode: false # don't persist data to disk
sync_writes: false # flush to disk on every write
mem_table_size: 10485760
num_mem_tables: 1
# lsm tuning options
value_log_max_entries: 1000000
value_log_file_size: 536870912
value_threshold: 1048576
num_level0_tables: 1
num_level0_tables_stall: 2
No [ERR] [diskqueue_consumer.go:133] error in ?
As mentioned above
___ _ _____ __ __ __ _
/ _ \ /_\ /__ \/__\/ / /\ \ \/_\ /\_/\
/ /_\///_\\ / /\/_\ \ \/ \/ //_\\\_ _/
/ /_\\/ _ \/ / //__ \ /\ / _ \/ \
\____/\_/ \_/\/ \__/ \/ \/\_/ \_/\_/
[GATEWAY] A light-weight, powerful and high-performance search gateway.
[GATEWAY] 1.21.0#1229, 2023-12-28 08:22:36, 2024-12-31 10:10:10, 5ca75b9539303a3ceb62e37e72811b580e7e004f
gateway 1.21.0 1229 2023-12-28 08:22:36 +0000 UTC 2024-12-31 10:10:10 +0000 UTC 5ca75b9539303a3ceb62e37e72811b580e7e004f
NAME="Ubuntu"
VERSION="20.04.5 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.5 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"}],"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"},"status":400}
#!/usr/bin/env bash
# original es
es_host="http://192.168.6.175:9200"
# gateway es
es_host="http://192.168.6.177:8000"
curl \
-X POST \
-u elastic:test123 \
-H 'Content-Type: application/x-ndjson' \
${es_host}/_bulk \
-d '
{ "create" : { "_index": "test1" } }
{ "color": "create black" }
{ "create" : { "_index": "test2"} }
{ "color": "create black" }
'
sleep 3s;
curl -X GET -u elastic:test123 -H 'Content-Type: application/json' ${es_host}/test1/_search | jq
Test separately
case 1 sucess
# original es
es_host="http://192.168.6.175:9200"
case 2 no success
# gateway es
es_host="http://192.168.6.177:8000"
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"}],"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"},"status":400}
{"errors":false,"took":11,"items":[{"create":{"_index":"test1","_id":"t_-nEY0Bq-FVL_Pe6TkL","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":4,"_primary_term":1,"status":201}},{"create":{"_index":"test2","_id":"uP-nEY0Bq-FVL_Pe6TkL","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":4,"_primary_term":1,"status":201}}]}
As mentioned above
es version
elasticsearch:8.11.3
___ _ _____ __ __ __ _
/ _ \ /_\ /__ \/__\/ / /\ \ \/_\ /\_/\
/ /_\///_\\ / /\/_\ \ \/ \/ //_\\\_ _/
/ /_\\/ _ \/ / //__ \ /\ / _ \/ \
\____/\_/ \_/\/ \__/ \/ \/\_/ \_/\_/
[GATEWAY] A light-weight, powerful and high-performance search gateway.
[GATEWAY] 1.21.0#1229, 2023-12-28 08:22:36, 2024-12-31 10:10:10, 5ca75b9539303a3ceb62e37e72811b580e7e004f
gateway 1.21.0 1229 2023-12-28 08:22:36 +0000 UTC 2024-12-31 10:10:10 +0000 UTC 5ca75b9539303a3ceb62e37e72811b580e7e004f
NAME="Ubuntu"
VERSION="20.04.5 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.5 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
gateway版本:
gateway 1.7.0_SNAPSHOT 705 2022-08-16 03:13:43 +0000 UTC 2023-12-31 10:10:10 +0000 UTC 4b2bdc84d5c1d9882493adfe43486d8dfaab68bc
访问kibana返回:
{"statusCode":503,"error":"Service Unavailable","message":"Request Timeout after 30000ms"}
gateway的日志有:
[09-06 23:58:36] [ERR] [server.go:2301] error in serveConn, runtime error: invalid memory address or nil pointer dereference,runtime error: invalid memory address or nil pointer dereference
[09-07 02:40:19] [ERR] [server.go:2301] error in serveConn, runtime error: invalid memory address or nil pointer dereference,runtime error: invalid memory address or nil pointer dereference
kibana日志:
大量
{"type":"log","@timestamp":"2022-09-07T00:53:09Z","tags":["warning","plugins","licensing"],"pid":89427,"message":"License information could not be obtained from Elasticsearch due to Error: Request Timeout after 30000ms error"}
{"type":"log","@timestamp":"2022-09-07T00:53:22Z","tags":["error","plugins","taskManager","taskManager"],"pid":89427,"message":"Failed to poll for work: Error: Request Timeout after 30000ms"}
{"type":"log","@timestamp":"2022-09-07T00:53:33Z","tags":["warning","elasticsearch","data"],"pid":89427,"message":"Unable to revive connection: http://10.0.6.146:9528/"}
{"type":"log","@timestamp":"2022-09-07T00:53:33Z","tags":["warning","plugins","licensing"],"pid":89427,"message":"License information could not be obtained from Elasticsearch due to Error: Request Timeout after 30000ms error"}
{"type":"log","@timestamp":"2022-09-07T00:53:39Z","tags":["warning","plugins","licensing"],"pid":89427,"message":"License information could not be obtained from Elasticsearch due to Error: Request Timeout after 30000ms error"}
{"type":"log","@timestamp":"2022-09-07T00:53:39Z","tags":["warning","elasticsearch","data"],"pid":89427,"message":"Unable to revive connection: http://10.0.6.146:9528/"}
重启kibana后恢复正常。
设置的prefix为“fat-es5:”,remove_prefix为默认值,查询语句为GET fat-es5:article_search/_search,结果提示rticle_search索引不存在,为啥把article_search截断为rticle_search?
详细日志:
[01-12 15:26:36] [ERR] [metadata.go:254] [main], {"error":{"root_cause":[{"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [<http_request>] would be [26726095128/24.8gb], which is larger than the limit of [25501368320/23.7gb], real usage: [26726095128/24.8gb], new bytes reserved: [0/0b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=3953455036/3.6gb, model_inference=0/0b, accounting=0/0b]","bytes_wanted":26726095128,"bytes_limit":25501368320,"durability":"TRANSIENT"}],"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [<http_request>] would be [26726095128/24.8gb], which is larger than the limit of [25501368320/23.7gb], real usage: [26726095128/24.8gb], new bytes reserved: [0/0b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=3953455036/3.6gb, model_inference=0/0b, accounting=0/0b]","bytes_wanted":26726095128,"bytes_limit":25501368320,"durability":"TRANSIENT"},"status":429}
# PUT /<my-index-{now/d}>
PUT /%3Cmy-index-%7Bnow%2Fd%7D%3E
curl call put index API
curl -H "Content-Type: application/json" -u elastic:test123 \
http://192.168.6.177:8000/%3Cdemo-app-logs-%7Bnow%2Fd%7D%3E/_doc/a8dbd0ad-c965-4612-b9eb-b798d8227bd1 -XPUT -d '{"dd":11}'
{"error":"no handler found for uri [/%3Cdemo-app-logs-%7Bnow/d%7D%3E/_doc] and method [PUT]"}
A description of what you expected to happen.
curl -H "Content-Type: application/json" -u elastic:test123 \
http://192.168.6.175:9200/%3Cdemo-app-logs-%7Bnow%2Fd%7D%3E/_doc/a8dbd0ad-c965-4612-b9eb-b798d8227bd1 -XPUT -d '{"dd":11}'
{"_index":"demo-app-logs-2024.03.02","_id":"a8dbd0ad-c965-4612-b9eb-b798d8227bd1","_version":2,"result":"updated","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":1,"_primary_term":1}
elasticsearch:8.11.3
___ _ _____ __ __ __ _
/ _ \ /_\ /__ \/__\/ / /\ \ \/_\ /\_/\
/ /_\///_\\ / /\/_\ \ \/ \/ //_\\\_ _/
/ /_\\/ _ \/ / //__ \ /\ / _ \/ \
\____/\_/ \_/\/ \__/ \/ \/\_/ \_/\_/
[GATEWAY] A light-weight, powerful and high-performance search gateway.
[GATEWAY] 1.21.0#1229, 2023-12-28 08:22:36, 2024-12-31 10:10:10, 5ca75b9539303a3ceb62e37e72811b580e7e004f
gateway 1.21.0 1229 2023-12-28 08:22:36 +0000 UTC 2024-12-31 10:10:10 +0000 UTC 5ca75b9539303a3ceb62e37e72811b580e7e004f
想要通过gateway想要实现ES两个集群间的数据同步。操作系统版本为redhat linux 7.3,安装的INFINI GATEWAY 版本是gateway-1.12.0-915-linux-386.tar.gz。两个集群IP地址分别为10.0.121.16和10.0.121.47,将gateway安装在10.0.121.47的机器上。启动gateway报错截图如下:
配置的yml如下:
path.data: data
path.logs: log
elasticsearch:
name: es-cluster1
enabled: true
endpoint: http://10.0.121.47:9200
name: ecs
enabled: true
endpoint: http://10.0.121.16:9201
basic_auth:
username: elastic
password: puBlSwHy+hmk-7IREynR
entry:
flow:
router:
pipeline:
操作系统版本为redhat linux 7.3
是按https://www.infinilabs.com/docs/latest/gateway/tutorial/request-logging/这边的提供的dashboard导入,这边是说kibana7.9
我的也是是7.9.3,但是提示导入失败
看接口返回好像是kibana兼容性问题
{ "statusCode": 422, "error": "Unprocessable Entity", "message": "Document \"b1af6220-0f7a-11eb-8973-2f461ac7af4d\" has property \"visualization\" which belongs to a more recent version of Kibana [7.10.0]. The last known version is [7.9.3]" }
看提示,是要求要7.10版本的,请问是否文档写的有问题
你好重启服务器后无法启动gateway,报错clients != endpoints是咋回事呢
[reverseproxy:252]clients != endpoints,reset to 0,%!(EXTRA int=1,string =vs,int=3)
kibana链接gateway使用后,几天后kibana完全变为不可以用,发现gateway已经503显示集群不可以使用。
但检查es集群是可以用的。
检查服务的链接发现大量gateway的连接未正常释放,重启gateway后正常。
因没有源码无法确定问题出处,请检查一下是否有未正常关闭连接情况。
netstat 是重启前的记录
netstat-restart 是重启后的记录
极限网关通过如下命令启动时,日志显示:unable to dial: dial tcp 127.0.0.1:9092: connetct: connection refused
nohup ./gateway-linux-amd64 -config replication_via-kafka.yml &
在replication_via-kafka.yml文件中,已经指定了远程的KAFKA_HOSTS: [ 'xx.xx.xx.xx:9092' ]
who can help me?
大家好,遇到一个问题:通过极限网关保存ES数据的时候,提示:the request included a message larger than the max message size the server will accept. 修改极限网关配置文件:
processor:
who can help me?
gateway: 1.6.0_SNAPSHOT
es: 7.8.1
kibana: 7.8.1
尝试测试 date_range_precision_tuning,time_precision:6,预设置精度到“秒”,配置之后kibana请求报错了。
去掉改配置后正常。
flow:
- name: default_flow
filter:
- context_regex_replace:
context: "_ctx.request.uri"
pattern: "_from=[^&]*&?"
to: ""
- date_range_precision_tuning:
time_precision: 6
- get_cache:
- bulk_reshuffle:
when:
contains:
_ctx.request.path: /_bulk
elasticsearch: prod
level: node
fix_null_id: true
- elasticsearch:
elasticsearch: prod #elasticsearch configure reference name
max_connection_per_node: 1000 #max tcp connection to upstream, default for all nodes
max_response_size: -1 #default for all nodes
balancer: weight
refresh: # refresh upstream nodes list, need to enable this feature to use elasticsearch nodes auto discovery
enabled: true
interval: 60s
- set_cache:
cache_type: ristretto
min_response_size: 100
max_response_size: 1024000
cache_ttl: 30s
max_cache_items: 100000
failed to parse date field [2022-08-00T05:59:32.999Z] with format [strict_date_optional_time]: [failed to parse date field [2022-08-00T05:59:32.999Z] with format [strict_date_optional_time]]
抓包结果内容:
kibana_http_400.json.zip
Issue reporter:hebo1982
大大好,看文档说极限网关支持一键索引重建,这个功能是怎么配置呢?现在我这里想对索引进行重建,这个索引又实时有数据写入,希望能够不间断业务可以达成这个目的。谢谢啦
A description of what you expected to happen.
A description of what happens instead.
[ERR] [instance.go:70] lock file:/data/es/gateway/data/gateway/nodes/cmj34usok3lguahba9ug/.lock exists, seems instance [689] is already running, please remove it or set `allow_multi_instance` to `true`
systemctl status -l gateway --no-pager
● gateway.service - cubefs
Loaded: loaded (/lib/systemd/system/gateway.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: exit-code) since Thu 2024-01-18 12:33:24 CST; 2s ago
Process: 33852 ExecStart=/usr/local/bin/gateway -config /data/es/gateway/config/gateway.yml (code=exited, status=1/FAILURE)
Main PID: 33852 (code=exited, status=1/FAILURE)
Jan 18 12:33:54 m177 gateway[33927]: ___ _ _____ __ __ __ _
Jan 18 12:33:54 m177 gateway[33927]: / _ \ /_\ /__ \/__\/ / /\ \ \/_\ /\_/\
Jan 18 12:33:54 m177 gateway[33927]: / /_\///_\\ / /\/_\ \ \/ \/ //_\\\_ _/
Jan 18 12:33:54 m177 gateway[33927]: / /_\\/ _ \/ / //__ \ /\ / _ \/ \
Jan 18 12:33:54 m177 gateway[33927]: \____/\_/ \_/\/ \__/ \/ \/\_/ \_/\_/
Jan 18 12:33:54 m177 gateway[33927]: [GATEWAY] A light-weight, powerful and high-performance search gateway.
Jan 18 12:33:54 m177 gateway[33927]: [GATEWAY] 1.21.0#1229, 2023-12-28 08:22:36, 2024-12-31 10:10:10, 5ca75b9539303a3ceb62e37e72811b580e7e004f
Jan 18 12:33:54 m177 gateway[33927]: [01-18 12:33:54] [INF] [env.go:175] configuration auto reload enabled
Jan 18 12:33:54 m177 gateway[33927]: [01-18 12:33:54] [INF] [app.go:288] initializing gateway, pid: 33927
Jan 18 12:33:54 m177 gateway[33927]: [01-18 12:33:54] [INF] [app.go:289] using config: /data/es/gateway/config/gateway.yml
Jan 18 12:33:54 m177 gateway[33927]: [01-18 12:33:54] [ERR] [instance.go:70] lock file:/data/es/gateway/data/gateway/nodes/cmj34usok3lguahba9ug/.lock exists, seems instance [689] is already running, please remove it or set `allow_multi_instance` to `true`
Jan 18 12:33:54 m177 systemd[1]: gateway.service: Main process exited, code=exited, status=1/FAILURE
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- An ExecStart= process belonging to unit gateway.service has exited.
--
-- The process' exit code is 'exited' and its exit status is 1.
Jan 18 12:33:54 m177 systemd[1]: gateway.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- The unit gateway.service has entered the 'failed' state with result 'exit-code'.
Elasticsearch 配置里面provider ,region ,dc的含义和使用场景,我们可能存在一个es 集群在不同阿里云的分区。
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.