infinilabs / gateway Goto Github PK

View Code? Open in Web Editor NEW

26.0 3.0 3.0 5 KB

🚀 A High-performance gateway designed for search scenarios. Good friend of Elasticsearch/Opensearch/Easysearch.

Home Page: https://www.infinilabs.com/en/docs/latest/gateway/

elasticsearch proxy gateway middleware opensearch easysearch

gateway's People

Contributors

Stargazers

Watchers

Forkers

liaosy21 jingweiz300 julian10086

gateway's Issues

极限网关对对post _msearch 查询的路由

请问一下，网关能对 _msearch的查询方法进行路由吗？根据索引名称发送到不同的es进行查询？

Description

大大好，gateway在本地无法运行，拉取依赖时发现 https://github.com/medcl/infini-framework.git 不存在。

Steps to reproduce

Start INFINI Gateway
Second step
Third step

Expected behavior

A description of what you expected to happen.

Actual behavior

A description of what happens instead.

Environment

Build: [e.g. gateway 1.0.0_SNAPSHOT 2021-09-05 15:41:02 1a8f196 - type "-v" in the Command Palette]
Operating system and version: [e.g. macOS 10.14, Windows 10, Ubuntu 18.04]
[Linux] Desktop Environment and/or Window Manager: [e.g. Gnome, LXDE, i3]

极限网关启动连接kafka失败

极限网关通过如下命令启动时，日志显示：unable to dial: dial tcp 127.0.0.1:9092: connetct: connection refused

nohup ./gateway-linux-amd64 -config replication_via-kafka.yml &

在replication_via-kafka.yml文件中，已经指定了远程的KAFKA_HOSTS: [ 'xx.xx.xx.xx:9092' ]

who can help me?

缓存统计信息

你好，网关缓存功能确实很有用，但是就是统计信息还不够详细，想问一下，这边有统计缓存的使用情况吗？比如使用了多少内存，缓存中主要是哪一些请求缓存了等等

项目后期还开源吗

http://release.elasticsearch.cn/gateway/snapshot/在仓库中都能看到1.5的本版了。

date_range_precision_tuning增加后kinaba请求异常

Description

尝试测试 date_range_precision_tuning，time_precision:6，预设置精度到“秒”，配置之后kibana请求报错了。
去掉改配置后正常。

Steps to reproduce

flow:
  - name: default_flow
    filter:
      - context_regex_replace:
          context: "_ctx.request.uri"
          pattern: "_from=[^&]*&?"
          to: ""
      - date_range_precision_tuning:
          time_precision: 6
      - get_cache:
      - bulk_reshuffle:
          when:
            contains:
              _ctx.request.path: /_bulk
          elasticsearch: prod
          level: node
          fix_null_id: true
      - elasticsearch:
          elasticsearch: prod  #elasticsearch configure reference name
          max_connection_per_node: 1000 #max tcp connection to upstream, default for all nodes
          max_response_size: -1 #default for all nodes
          balancer: weight
          refresh: # refresh upstream nodes list, need to enable this feature to use elasticsearch nodes auto discovery
            enabled: true
            interval: 60s
      - set_cache:
          cache_type: ristretto
          min_response_size: 100
          max_response_size: 1024000
          cache_ttl: 30s
          max_cache_items: 100000

Expected behavior

应该正常按秒格式化kibana请求的时间

Actual behavior

failed to parse date field [2022-08-00T05:59:32.999Z] with format [strict_date_optional_time]: [failed to parse date field [2022-08-00T05:59:32.999Z] with format [strict_date_optional_time]]

抓包结果内容：
kibana_http_400.json.zip

Environment

gateway: 1.6.0_SNAPSHOT
es: 7.8.1
kibana: 7.8.1

Build:
gateway 1.6.0_SNAPSHOT 657 2022-05-31 07:35:13 +0000 UTC 2023-12-31 10:10:10 +0000 UTC 21ec7be5de546f5454b3ccb58b2ce9183cd1ff2d

Operating system and version:
Centos 7

您好，之前在elastic社区和elastic线下活动见过您也有一些交流。新发现您开发了极限网关这个应用，我们目前有一个ES集群升级的项目想要做，想和您交流一下极限网关在使用上的可行性。
我们现在的ES集群是5版本，有3个机房，每个机房都是独立的一套集群，存储本机房数据。部署了专门的ccs节点连接这3个机房的ES集群，提供汇总数据查询功能。
目前打算将集群从5版本，升级到7版本，但是服务器有限，无法搭建和老版本等容量的集群，因此采用少量服务器先搭建ES 7集群，部分日志数据同时写入两个集群。

这时候希望借用极限网关的功能，客户端程序访问极限网关，通过判断要查询索引的名字，路由到不同的集群进行查询，架构如下：

当我们验证相关索引没问题后，通过下线ES 5的机器，扩容到ES 7 ，并再次迁移部分索引，重复相同的步骤，直到服务器和索引都放到ES 7 集群。

想和和您交流一下，这个方案是否可行？判断索引名称进行路由使用什么配置实现？这里是否有哪些需要注意的坑？或者您有什么更好的建议，希望您指点。我们希望对客户端服务是无感的，不影响业务。

非常感谢。

生产环境产生大量的监控数据，导致kafka硬盘写满

问题描述

我们在生产环境使用极限网关同步ES数据到灾备环境，过了一段时间，发现kafka宕机了，查找问题是因为kafka硬盘被写满了，看kafka数据全是极限网关发送的监控数据，有10G左右，最后分析kafka数据，差不多每天产生1.3-1.5G的监控数据，但是开发环境并没有这么大的数据量。
通过分析kafka数据，发现生产环境10s产生18条监控数据，而开发环境只产生2条，这个就是为什么生产数据量大的原因，但是为啥每次会产生18条这么多的数据，这个是我的疑问？请各位大佬分析分析

Many ERROR log [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found

Description

Many ERROR log [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found

Steps to reproduce

systemctl start gateway
journalctl -exu gateway --no-pager

Jan 16 08:55:59 m178 gateway[3188]: [01-16 08:55:59] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:55:59 m178 gateway[3188]: [01-16 08:55:59] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:00 m178 gateway[3188]: [01-16 08:56:00] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:56:00 m178 gateway[3188]: [01-16 08:56:00] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:01 m178 gateway[3188]: [01-16 08:56:01] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:56:01 m178 gateway[3188]: [01-16 08:56:01] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:02 m178 gateway[3188]: [01-16 08:56:02] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:56:02 m178 gateway[3188]: [01-16 08:56:02] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:03 m178 gateway[3188]: [01-16 08:56:03] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:56:03 m178 gateway[3188]: [01-16 08:56:03] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:04 m178 gateway[3188]: [01-16 08:56:04] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:56:04 m178 gateway[3188]: [01-16 08:56:04] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:05 m178 gateway[3188]: [01-16 08:56:05] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:56:05 m178 gateway[3188]: [01-16 08:56:05] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:06 m178 gateway[3188]: [01-16 08:56:06] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:56:06 m178 gateway[3188]: [01-16 08:56:06] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:07 m178 gateway[3188]: [01-16 08:56:07] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:07 m178 gateway[3188]: [01-16 08:56:07] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:56:08 m178 gateway[3188]: [01-16 08:56:08] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:08 m178 gateway[3188]: [01-16 08:56:08] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:56:09 m178 gateway[3188]: [01-16 08:56:09] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:09 m178 gateway[3188]: [01-16 08:56:09] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:56:10 m178 gateway[3188]: [01-16 08:56:10] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:10 m178 gateway[3188]: [01-16 08:56:10] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:56:11 m178 gateway[3188]: [01-16 08:56:11] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:56:11 m178 gateway[3188]: [01-16 08:56:11] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:12 m178 gateway[3188]: [01-16 08:56:12] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:56:12 m178 gateway[3188]: [01-16 08:56:12] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:13 m178 gateway[3188]: [01-16 08:56:13] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:56:13 m178 gateway[3188]: [01-16 08:56:13] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:14 m178 gateway[3188]: [01-16 08:56:14] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:14 m178 gateway[3188]: [01-16 08:56:14] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:56:15 m178 gateway[3188]: [01-16 08:56:15] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:56:15 m178 gateway[3188]: [01-16 08:56:15] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:16 m178 gateway[3188]: [01-16 08:56:16] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:56:16 m178 gateway[3188]: [01-16 08:56:16] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:17 m178 gateway[3188]: [01-16 08:56:17] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:17 m178 gateway[3188]: [01-16 08:56:17] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:56:18 m178 gateway[3188]: [01-16 08:56:18] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:56:18 m178 gateway[3188]: [01-16 08:56:18] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:19 m178 gateway[3188]: [01-16 08:56:19] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:19 m178 gateway[3188]: [01-16 08:56:19] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found

/lib/systemd/system/gateway.service

systemctl cat gateway

# /lib/systemd/system/gateway.service
[Unit]
Description=cubefs
After=network-online.target local-fs.target time-sync.target
Wants=network-online.target local-fs.target time-sync.target

[Service]
Type=simple
ExecStart=/usr/local/bin/gateway -config /data/es/gateway/config/gateway.yml
ExecStop=/bin/kill -s TERM $MAINPID
Restart=on-failure
RestartSec=10s
LimitNOFILE=40000

[Install]
WantedBy=multi-user.target

/data/es/gateway/config/gateway.yml

allow_multi_instance: true
path.data: /data/es/gateway/data
path.logs: /data/es/gateway/log

elasticsearch:
- name: cluster01
  enabled: true
  endpoint: http://192.168.6.175:9200
  basic_auth:
    username: elastic
    password: test123
  traffic_control.max_bytes_per_node: 1010485760
  metadata_cache_enabled: false # Whether to cache the cluster info in memory cache
- name: cluster02
  enabled: true
  endpoint: http://192.168.6.176:9200
  basic_auth:
    username: elastic
    password: test123
  discovery:
    enabled: false
entry:
  - name: my_es_entry
    enabled: true
    router: my_router
    max_concurrency: 200000
    network:
      binding: 0.0.0.0:8000
      reuse_port: true

flow:
 # 写请求优先发给主集群, 当主集群不可用时发给备用集群
 # 当主集群数据写入成功时，记录到队列中，异步消费写入备用集群
  - name: write-flow
    filter:
      - if:
      # 当主集群可用时
          cluster_available: ["cluster01"]
        then:
          # 先将数据写入主集群
          - elasticsearch:
              elasticsearch: "cluster01"
            # 写入消息队列,等待 pipeline 异步消费到备用集群
          - queue:
              queue_name: "cluster02-queue"
        else:
          - elasticsearch:
              elasticsearch: "cluster02"
          - queue:
              queue_name: "cluster01-queue"
  # 读请求优先发给主集群, 当主集群不可用时发给备用集群
  - name: read-flow
    filter:
      - if:
          cluster_available: ["cluster01"]
        then:
          - elasticsearch:
              elasticsearch: "cluster01"
        else:
          - elasticsearch:
              elasticsearch: "cluster02"

router:
  - name: my_router
    # 默认路由
    default_flow: write-flow
    # 读请求路由
    rules:
      - method:
          - "GET"
          - "HEAD"
        pattern:
          - "/{any:*}"
        flow:
          - read-flow
      - method:
          - "POST"
          - "GET"
        pattern:
          - "/_refresh"
          - "/_count"
          - "/_search"
          - "/_msearch"
          - "/_mget"
          - "/{any_index}/_count"
          - "/{any_index}/_search"
          - "/{any_index}/_msearch"
          - "/{any_index}/_mget"
        flow:
          - read-flow

pipeline:
  - name: cluster01-consumer
    auto_start: true
    keep_running: true
    processor:
      - queue_consumer:
          input_queue: "cluster01-queue" 
          elasticsearch: "cluster01"
          when:
            cluster_available: ["cluster01"] # 当集群可用时，才消费队列中的数据
  - name: cluster02-consumer
    auto_start: true
    keep_running: true
    processor:
      - queue_consumer:
          input_queue: "cluster02-queue"
          elasticsearch: "cluster02"
          when:
            cluster_available: ["cluster02"]




disk_queue:
  prepare_files_to_read: true
  #max_bytes_per_file: 20971520
  eof_retry_delay_in_ms: 500
  cleanup_files_on_init: false
  retention:
    max_num_of_local_files: 20 # automatically cleanup consumed files
  compress:
    segment:
      enabled: true
    delete_after_compress: true # trigger cleanup after compression.
    idle_threshold: 20 # max number of uncompressed consumed files to preserve.


badger:
  enabled: true
  single_bucket_mode: true
  path: '' # defaults to {path.data}/gateway/node/{nodeID}/badger/
  memory_mode: false # don't persist data to disk
  sync_writes: false # flush to disk on every write
  mem_table_size: 10485760
  num_mem_tables: 1
  # lsm tuning options
  value_log_max_entries: 1000000
  value_log_file_size: 536870912
  value_threshold: 1048576
  num_level0_tables: 1
  num_level0_tables_stall: 2

Expected behavior

No [ERR] [diskqueue_consumer.go:133] error in ?

Actual behavior

As mentioned above

Environment

Build:

   ___   _   _____  __  __    __  _       
  / _ \ /_\ /__   \/__\/ / /\ \ \/_\ /\_/\
 / /_\///_\\  / /\/_\  \ \/  \/ //_\\\_ _/
/ /_\\/  _  \/ / //__   \  /\  /  _  \/ \ 
\____/\_/ \_/\/  \__/    \/  \/\_/ \_/\_/ 

[GATEWAY] A light-weight, powerful and high-performance search gateway.
[GATEWAY] 1.21.0#1229, 2023-12-28 08:22:36, 2024-12-31 10:10:10, 5ca75b9539303a3ceb62e37e72811b580e7e004f
gateway 1.21.0 1229 2023-12-28 08:22:36 +0000 UTC 2024-12-31 10:10:10 +0000 UTC 5ca75b9539303a3ceb62e37e72811b580e7e004f

Operating system and version:

NAME="Ubuntu"
VERSION="20.04.5 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.5 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

[Linux] Desktop Environment and/or Window Manager: [e.g. Gnome, LXDE, i3]

no support _builk {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"}],"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"},"status":400}

Description

{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"}],"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"},"status":400}

Steps to reproduce

Start INFINI Gateway
Second step

#!/usr/bin/env bash


# original es
es_host="http://192.168.6.175:9200"
# gateway es
es_host="http://192.168.6.177:8000"

curl \
-X POST \
-u elastic:test123 \
-H 'Content-Type: application/x-ndjson' \
${es_host}/_bulk \
-d '
{ "create" : { "_index": "test1" } }
{ "color": "create black" }
{ "create" : { "_index": "test2"} }
{ "color": "create black" }
'
sleep 3s;
curl -X GET -u elastic:test123 -H 'Content-Type: application/json' ${es_host}/test1/_search | jq

Test separately

case 1 sucess

# original es
es_host="http://192.168.6.175:9200"

case 2 no success

# gateway es
es_host="http://192.168.6.177:8000"

{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"}],"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"},"status":400}

Expected behavior

{"errors":false,"took":11,"items":[{"create":{"_index":"test1","_id":"t_-nEY0Bq-FVL_Pe6TkL","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":4,"_primary_term":1,"status":201}},{"create":{"_index":"test2","_id":"uP-nEY0Bq-FVL_Pe6TkL","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":4,"_primary_term":1,"status":201}}]}

Actual behavior

As mentioned above

Environment

es version

elasticsearch:8.11.3

Build:

   ___   _   _____  __  __    __  _       
  / _ \ /_\ /__   \/__\/ / /\ \ \/_\ /\_/\
 / /_\///_\\  / /\/_\  \ \/  \/ //_\\\_ _/
/ /_\\/  _  \/ / //__   \  /\  /  _  \/ \ 
\____/\_/ \_/\/  \__/    \/  \/\_/ \_/\_/ 

[GATEWAY] A light-weight, powerful and high-performance search gateway.
[GATEWAY] 1.21.0#1229, 2023-12-28 08:22:36, 2024-12-31 10:10:10, 5ca75b9539303a3ceb62e37e72811b580e7e004f
gateway 1.21.0 1229 2023-12-28 08:22:36 +0000 UTC 2024-12-31 10:10:10 +0000 UTC 5ca75b9539303a3ceb62e37e72811b580e7e004f

Operating system and version:

NAME="Ubuntu"
VERSION="20.04.5 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.5 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

极限网关开启后，使用不了

请求8000端口后，返回401

kibana反馈集群503不可用

gateway版本:

gateway 1.7.0_SNAPSHOT 705 2022-08-16 03:13:43 +0000 UTC 2023-12-31 10:10:10 +0000 UTC 4b2bdc84d5c1d9882493adfe43486d8dfaab68bc

访问kibana返回:

{"statusCode":503,"error":"Service Unavailable","message":"Request Timeout after 30000ms"}

gateway的日志有：

[09-06 23:58:36] [ERR] [server.go:2301] error in serveConn, runtime error: invalid memory address or nil pointer dereference,runtime error: invalid memory address or nil pointer dereference
[09-07 02:40:19] [ERR] [server.go:2301] error in serveConn, runtime error: invalid memory address or nil pointer dereference,runtime error: invalid memory address or nil pointer dereference

kibana日志:

大量
{"type":"log","@timestamp":"2022-09-07T00:53:09Z","tags":["warning","plugins","licensing"],"pid":89427,"message":"License information could not be obtained from Elasticsearch due to Error: Request Timeout after 30000ms error"}
{"type":"log","@timestamp":"2022-09-07T00:53:22Z","tags":["error","plugins","taskManager","taskManager"],"pid":89427,"message":"Failed to poll for work: Error: Request Timeout after 30000ms"}
{"type":"log","@timestamp":"2022-09-07T00:53:33Z","tags":["warning","elasticsearch","data"],"pid":89427,"message":"Unable to revive connection: http://10.0.6.146:9528/"}
{"type":"log","@timestamp":"2022-09-07T00:53:33Z","tags":["warning","plugins","licensing"],"pid":89427,"message":"License information could not be obtained from Elasticsearch due to Error: Request Timeout after 30000ms error"}
{"type":"log","@timestamp":"2022-09-07T00:53:39Z","tags":["warning","plugins","licensing"],"pid":89427,"message":"License information could not be obtained from Elasticsearch due to Error: Request Timeout after 30000ms error"}
{"type":"log","@timestamp":"2022-09-07T00:53:39Z","tags":["warning","elasticsearch","data"],"pid":89427,"message":"Unable to revive connection: http://10.0.6.146:9528/"}

重启kibana后恢复正常。

设置的prefix为“fat-es5:”，remove_prefix为默认值，查询语句为GET fat-es5:article_search/_search，结果提示rticle_search索引不存在，为啥把article_search截断为rticle_search？

设置的prefix为“fat-es5:”，remove_prefix为默认值，查询语句为GET fat-es5:article_search/_search，结果提示rticle_search索引不存在，为啥把article_search截断为rticle_search？

http_request 太大导致溢出，文档未找到yml设置位置

Description

详细日志：

[01-12 15:26:36] [ERR] [metadata.go:254] [main], {"error":{"root_cause":[{"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [<http_request>] would be [26726095128/24.8gb], which is larger than the limit of [25501368320/23.7gb], real usage: [26726095128/24.8gb], new bytes reserved: [0/0b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=3953455036/3.6gb, model_inference=0/0b, accounting=0/0b]","bytes_wanted":26726095128,"bytes_limit":25501368320,"durability":"TRANSIENT"}],"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [<http_request>] would be [26726095128/24.8gb], which is larger than the limit of [25501368320/23.7gb], real usage: [26726095128/24.8gb], new bytes reserved: [0/0b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=3953455036/3.6gb, model_inference=0/0b, accounting=0/0b]","bytes_wanted":26726095128,"bytes_limit":25501368320,"durability":"TRANSIENT"},"status":429}

not support date_math_expr like %3Cmy-index-%7Bnow%2Fd%7D%3E

Description

# PUT /<my-index-{now/d}>
PUT /%3Cmy-index-%7Bnow%2Fd%7D%3E

ref https://www.elastic.co/guide/en/elasticsearch/reference/8.12/api-conventions.html#api-date-math-index-names

Steps to reproduce

curl call put index API

Actual behavior

curl -H "Content-Type: application/json" -u elastic:test123 \
http://192.168.6.177:8000/%3Cdemo-app-logs-%7Bnow%2Fd%7D%3E/_doc/a8dbd0ad-c965-4612-b9eb-b798d8227bd1 -XPUT -d '{"dd":11}'

{"error":"no handler found for uri [/%3Cdemo-app-logs-%7Bnow/d%7D%3E/_doc] and method [PUT]"}

A description of what you expected to happen.

Expected behavior

curl -H "Content-Type: application/json" -u elastic:test123 \ 
http://192.168.6.175:9200/%3Cdemo-app-logs-%7Bnow%2Fd%7D%3E/_doc/a8dbd0ad-c965-4612-b9eb-b798d8227bd1 -XPUT -d '{"dd":11}'
{"_index":"demo-app-logs-2024.03.02","_id":"a8dbd0ad-c965-4612-b9eb-b798d8227bd1","_version":2,"result":"updated","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":1,"_primary_term":1}

Environment

elasticsearch:8.11.3

Build:

   ___   _   _____  __  __    __  _       
  / _ \ /_\ /__   \/__\/ / /\ \ \/_\ /\_/\
 / /_\///_\\  / /\/_\  \ \/  \/ //_\\\_ _/
/ /_\\/  _  \/ / //__   \  /\  /  _  \/ \ 
\____/\_/ \_/\/  \__/    \/  \/\_/ \_/\_/ 

[GATEWAY] A light-weight, powerful and high-performance search gateway.
[GATEWAY] 1.21.0#1229, 2023-12-28 08:22:36, 2024-12-31 10:10:10, 5ca75b9539303a3ceb62e37e72811b580e7e004f
gateway 1.21.0 1229 2023-12-28 08:22:36 +0000 UTC 2024-12-31 10:10:10 +0000 UTC 5ca75b9539303a3ceb62e37e72811b580e7e004f

启动时报无法分配内存

Description

想要通过gateway想要实现ES两个集群间的数据同步。操作系统版本为redhat linux 7.3，安装的INFINI GATEWAY 版本是gateway-1.12.0-915-linux-386.tar.gz。两个集群IP地址分别为10.0.121.16和10.0.121.47，将gateway安装在10.0.121.47的机器上。启动gateway报错截图如下：

配置的yml如下：
path.data: data
path.logs: log

elasticsearch:

name: es-cluster1
enabled: true
endpoint: http://10.0.121.47:9200
name: ecs
enabled: true
endpoint: http://10.0.121.16:9201
basic_auth:
username: elastic
password: puBlSwHy+hmk-7IREynR

entry:

name: my_es_entry
enabled: true
router: my_router
network:
binding: 0.0.0.0:8000

flow:

name: write-flow
filter:
- if:
  cluster_available: ["es-cluster1"]
  then:
  - elasticsearch:
    elasticsearch: "es-cluster1"
  - queue:
    queue_name: "ecs-queue"
    else:
  - elasticsearch:
    elasticsearch: "ecs"
  - queue:
    queue_name: "es-cluster1-queue"
name: read-flow
filter:
- if:
  cluster_available: ["es-cluster1"]
  then:
  - elasticsearch:
    elasticsearch: "es-cluster1"
    else:
  - elasticsearch:
    elasticsearch: "ecs"

router:

name: my_router
default_flow: write-flow
rules:
- method:
  - "GET"
  - "HEAD"
    pattern:
  - "/{any:*}"
    flow:
  - read-flow
- method:
  - "POST"
  - "GET"
    pattern:
  - "/_refresh"
  - "/_count"
  - "/_search"
  - "/_msearch"
  - "/_mget"
  - "/{any_index}/_count"
  - "/{any_index}/_search"
  - "/{any_index}/_msearch"
  - "/{any_index}/_mget"
    flow:
  - read-flow

pipeline:

name: es-cluster1-consumer
auto_start: true
keep_running: true
processor:
- queue_consumer:
  input_queue: "es-cluster1-queue"
  elasticsearch: "es-cluster1"
  when:
  cluster_available: ["es-cluster1"]
name: ecs-consumer
auto_start: true
keep_running: true
processor:
- queue_consumer:
  input_queue: "ecs-queue"
  elasticsearch: "ecs"
  when:
  cluster_available: ["ecs"]

Environment

操作系统版本为redhat linux 7.3

kibnana的dashboard导入提示错误

是按https://www.infinilabs.com/docs/latest/gateway/tutorial/request-logging/这边的提供的dashboard导入，这边是说kibana7.9

我的也是是7.9.3，但是提示导入失败

看接口返回好像是kibana兼容性问题
{ "statusCode": 422, "error": "Unprocessable Entity", "message": "Document \"b1af6220-0f7a-11eb-8973-2f461ac7af4d\" has property \"visualization\" which belongs to a more recent version of Kibana [7.10.0]. The last known version is [7.9.3]" }

看提示，是要求要7.10版本的，请问是否文档写的有问题

你好重启服务器后无法启动gateway，报错clients ！= endpoints是咋回事呢

[reverseproxy:252]clients != endpoints,reset to 0,%!(EXTRA int=1,string =vs,int=3)

connection未正常关闭

kibana链接gateway使用后，几天后kibana完全变为不可以用，发现gateway已经503显示集群不可以使用。
但检查es集群是可以用的。

检查服务的链接发现大量gateway的连接未正常释放，重启gateway后正常。
因没有源码无法确定问题出处，请检查一下是否有未正常关闭连接情况。

netstat.zip

netstat 是重启前的记录
netstat-restart 是重启后的记录

极限网关启动连接kafka失败

极限网关通过如下命令启动时，日志显示：unable to dial: dial tcp 127.0.0.1:9092: connetct: connection refused

nohup ./gateway-linux-amd64 -config replication_via-kafka.yml &

在replication_via-kafka.yml文件中，已经指定了远程的KAFKA_HOSTS: [ 'xx.xx.xx.xx:9092' ]

who can help me?

通过极限网关存储数据，提示larger than the max message size the server will accept.

大家好，遇到一个问题：通过极限网关保存ES数据的时候，提示：the request included a message larger than the max message size the server will accept. 修改极限网关配置文件：
processor:

consumer:
...
queue_selector:
labels:
type: "primary_write_ahead_log"
consumer:
fetch_max_messages: 10000000 #修改此处为1000万，保存重启网关，仍然报上面错误

who can help me?

date_range_precision_tuning增加后kinaba请求异常

gateway: 1.6.0_SNAPSHOT
es: 7.8.1
kibana: 7.8.1

尝试测试 date_range_precision_tuning，time_precision:6，预设置精度到“秒”，配置之后kibana请求报错了。
去掉改配置后正常。

flow:
  - name: default_flow
    filter:
      - context_regex_replace:
          context: "_ctx.request.uri"
          pattern: "_from=[^&]*&?"
          to: ""
      - date_range_precision_tuning:
          time_precision: 6
      - get_cache:
      - bulk_reshuffle:
          when:
            contains:
              _ctx.request.path: /_bulk
          elasticsearch: prod
          level: node
          fix_null_id: true
      - elasticsearch:
          elasticsearch: prod  #elasticsearch configure reference name
          max_connection_per_node: 1000 #max tcp connection to upstream, default for all nodes
          max_response_size: -1 #default for all nodes
          balancer: weight
          refresh: # refresh upstream nodes list, need to enable this feature to use elasticsearch nodes auto discovery
            enabled: true
            interval: 60s
      - set_cache:
          cache_type: ristretto
          min_response_size: 100
          max_response_size: 1024000
          cache_ttl: 30s
          max_cache_items: 100000

failed to parse date field [2022-08-00T05:59:32.999Z] with format [strict_date_optional_time]: [failed to parse date field [2022-08-00T05:59:32.999Z] with format [strict_date_optional_time]]

抓包结果内容：
kibana_http_400.json.zip

Issue reporter：hebo1982

极限网关支持一键索引重建

Description

大大好，看文档说极限网关支持一键索引重建，这个功能是怎么配置呢？现在我这里想对索引进行重建，这个索引又实时有数据写入，希望能够不间断业务可以达成这个目的。谢谢啦

Steps to reproduce

Start INFINI Gateway
Second step
Third step

Expected behavior

A description of what you expected to happen.

Actual behavior

A description of what happens instead.

Environment

Build: [e.g. gateway 1.0.0_SNAPSHOT 2021-09-05 15:41:02 1a8f196 - type "-v" in the Command Palette]
Operating system and version: [e.g. macOS 10.14, Windows 10, Ubuntu 18.04]
[Linux] Desktop Environment and/or Window Manager: [e.g. Gnome, LXDE, i3]

[ERR] [instance.go:70] lock file:/data/es/gateway/data/gateway/nodes/cmj34usok3lguahba9ug/.lock exists, seems instance [689] is already running, please remove it or set `allow_multi_instance` to `true`

Description

[ERR] [instance.go:70] lock file:/data/es/gateway/data/gateway/nodes/cmj34usok3lguahba9ug/.lock exists, seems instance [689] is already running, please remove it or set `allow_multi_instance` to `true`

Steps to reproduce

Start INFINI Gateway

systemctl status -l gateway --no-pager
● gateway.service - cubefs
     Loaded: loaded (/lib/systemd/system/gateway.service; enabled; vendor preset: enabled)
     Active: activating (auto-restart) (Result: exit-code) since Thu 2024-01-18 12:33:24 CST; 2s ago
    Process: 33852 ExecStart=/usr/local/bin/gateway -config /data/es/gateway/config/gateway.yml (code=exited, status=1/FAILURE)
   Main PID: 33852 (code=exited, status=1/FAILURE)

Jan 18 12:33:54 m177 gateway[33927]:    ___   _   _____  __  __    __  _
Jan 18 12:33:54 m177 gateway[33927]:   / _ \ /_\ /__   \/__\/ / /\ \ \/_\ /\_/\
Jan 18 12:33:54 m177 gateway[33927]:  / /_\///_\\  / /\/_\  \ \/  \/ //_\\\_ _/
Jan 18 12:33:54 m177 gateway[33927]: / /_\\/  _  \/ / //__   \  /\  /  _  \/ \
Jan 18 12:33:54 m177 gateway[33927]: \____/\_/ \_/\/  \__/    \/  \/\_/ \_/\_/
Jan 18 12:33:54 m177 gateway[33927]: [GATEWAY] A light-weight, powerful and high-performance search gateway.
Jan 18 12:33:54 m177 gateway[33927]: [GATEWAY] 1.21.0#1229, 2023-12-28 08:22:36, 2024-12-31 10:10:10, 5ca75b9539303a3ceb62e37e72811b580e7e004f
Jan 18 12:33:54 m177 gateway[33927]: [01-18 12:33:54] [INF] [env.go:175] configuration auto reload enabled
Jan 18 12:33:54 m177 gateway[33927]: [01-18 12:33:54] [INF] [app.go:288] initializing gateway, pid: 33927
Jan 18 12:33:54 m177 gateway[33927]: [01-18 12:33:54] [INF] [app.go:289] using config: /data/es/gateway/config/gateway.yml
Jan 18 12:33:54 m177 gateway[33927]: [01-18 12:33:54] [ERR] [instance.go:70] lock file:/data/es/gateway/data/gateway/nodes/cmj34usok3lguahba9ug/.lock exists, seems instance [689] is already running, please remove it or set `allow_multi_instance` to `true`
Jan 18 12:33:54 m177 systemd[1]: gateway.service: Main process exited, code=exited, status=1/FAILURE
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- An ExecStart= process belonging to unit gateway.service has exited.
-- 
-- The process' exit code is 'exited' and its exit status is 1.
Jan 18 12:33:54 m177 systemd[1]: gateway.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- The unit gateway.service has entered the 'failed' state with result 'exit-code'.

Elasticsearch 配置里面provider ，region ，dc的含义和使用场景

Elasticsearch 配置里面provider ，region ，dc的含义和使用场景，我们可能存在一个es 集群在不同阿里云的分区。

infinilabs / gateway Goto Github PK

gateway's People

Contributors

Stargazers

Watchers

Forkers

gateway's Issues

Description

Steps to reproduce

Expected behavior

Actual behavior

Environment

Description

Steps to reproduce

Expected behavior

Actual behavior

Environment

问题描述

Description

Steps to reproduce

Expected behavior

Actual behavior

Environment

Description

Steps to reproduce

Expected behavior

Actual behavior

Environment

Description

Description

Steps to reproduce

Actual behavior

Expected behavior

Environment

Description

Environment

Description

Steps to reproduce

Expected behavior

Actual behavior

Environment

Description

Steps to reproduce

Recommend Projects

Recommend Topics

Recommend Org