Comments (4)
Hi @jingli-wtbox , can you please attach the complete ES log?
from clearml-server.
@jkhenning , thanks. please find the full ES log below.
docker logs clearml-elastic
Apr 03, 2024 2:48:47 AM sun.util.locale.provider.LocaleProviderAdapter <clinit>
WARNING: COMPAT locale provider will be removed in a future release
{"type": "server", "timestamp": "2024-04-03T02:48:49,323Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "clearml", "node.name": "clearml", "message": "version[7.17.18], pid[7], build[default/docker/8682172c2130b9a411b1bd5ff37c9792367de6b0/2024-02-02T12:04:59.691750271Z], OS[Linux/5.15.0-1036-aws/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/21.0.2/21.0.2+13-58]" }
{"type": "server", "timestamp": "2024-04-03T02:48:49,332Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "clearml", "node.name": "clearml", "message": "JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]" }
{"type": "server", "timestamp": "2024-04-03T02:48:49,333Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "clearml", "node.name": "clearml", "message": "JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -Djava.security.manager=allow, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-9093165662722554014, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Des.cgroups.hierarchy.override=/, -Xms2g, -Xmx2g, -Dlog4j2.formatMsgNoLookups=true, -XX:MaxDirectMemorySize=1073741824, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,037Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [aggs-matrix-stats]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,038Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [analysis-common]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,038Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [constant-keyword]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,038Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [frozen-indices]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,038Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [ingest-common]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,039Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [ingest-geoip]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,039Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [ingest-user-agent]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,039Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [kibana]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,039Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [lang-expression]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,039Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [lang-mustache]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,039Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [lang-painless]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,040Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [legacy-geo]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,040Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [mapper-extras]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,040Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [mapper-version]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,040Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [parent-join]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,040Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [percolator]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,041Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [rank-eval]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,041Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [reindex]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,041Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [repositories-metering-api]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,041Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [repository-encrypted]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,042Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [repository-url]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,042Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [runtime-fields-common]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,042Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [search-business-rules]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,042Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [searchable-snapshots]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,043Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [snapshot-repo-test-kit]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,043Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [spatial]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,044Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [transform]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,044Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [transport-netty4]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,044Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [unsigned-long]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,045Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [vector-tile]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,045Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [vectors]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,045Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [wildcard]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,046Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-aggregate-metric]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,046Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-analytics]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,046Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-async]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,047Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-async-search]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,047Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-autoscaling]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,047Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-ccr]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,048Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-core]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,048Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-data-streams]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,048Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-deprecation]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,049Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-enrich]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,049Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-eql]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,049Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-fleet]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,050Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-graph]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,050Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-identity-provider]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,050Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-ilm]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,050Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-logstash]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,051Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-ml]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,051Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-monitoring]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,051Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-ql]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,052Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-rollup]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,052Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-security]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,052Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-shutdown]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,052Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-sql]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,053Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-stack]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,053Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-text-structure]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,054Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-voting-only-node]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,054Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-watcher]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,055Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "no plugins loaded" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,111Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "clearml", "node.name": "clearml", "message": "using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/root)]], net usable_space [120.4gb], net total_space [1.9tb], types [ext4]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,112Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "clearml", "node.name": "clearml", "message": "heap size [2gb], compressed ordinary object pointers [true]" }
{"type": "server", "timestamp": "2024-04-03T02:48:55,239Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "clearml", "node.name": "clearml", "message": "node name [clearml], node ID [A6LcQvX-RYyaLHo9pDNESA], cluster name [clearml], roles [transform, data_frozen, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]" }
{"type": "server", "timestamp": "2024-04-03T02:49:01,644Z", "level": "INFO", "component": "o.e.x.m.p.l.CppLogMessageHandler", "cluster.name": "clearml", "node.name": "clearml", "message": "[controller/217] [Main.cc@122] controller (64 bit): Version 7.17.18 (Build a1014b051a9d63) Copyright (c) 2024 Elasticsearch BV" }
{"type": "server", "timestamp": "2024-04-03T02:49:02,341Z", "level": "INFO", "component": "o.e.i.g.ConfigDatabases", "cluster.name": "clearml", "node.name": "clearml", "message": "initialized default databases [[GeoLite2-Country.mmdb, GeoLite2-City.mmdb, GeoLite2-ASN.mmdb]], config databases [[]] and watching [/usr/share/elasticsearch/config/ingest-geoip] for changes" }
{"type": "server", "timestamp": "2024-04-03T02:49:02,346Z", "level": "INFO", "component": "o.e.i.g.DatabaseNodeService", "cluster.name": "clearml", "node.name": "clearml", "message": "initialized database registry, using geoip-databases directory [/tmp/elasticsearch-9093165662722554014/geoip-databases/A6LcQvX-RYyaLHo9pDNESA]" }
{"type": "server", "timestamp": "2024-04-03T02:49:03,034Z", "level": "INFO", "component": "o.e.t.NettyAllocator", "cluster.name": "clearml", "node.name": "clearml", "message": "creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}]" }
{"type": "server", "timestamp": "2024-04-03T02:49:03,074Z", "level": "INFO", "component": "o.e.i.r.RecoverySettings", "cluster.name": "clearml", "node.name": "clearml", "message": "using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]" }
{"type": "server", "timestamp": "2024-04-03T02:49:03,118Z", "level": "INFO", "component": "o.e.d.DiscoveryModule", "cluster.name": "clearml", "node.name": "clearml", "message": "using discovery type [single-node] and seed hosts providers [settings]" }
{"type": "server", "timestamp": "2024-04-03T02:49:03,621Z", "level": "INFO", "component": "o.e.g.DanglingIndicesState", "cluster.name": "clearml", "node.name": "clearml", "message": "gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually" }
{"type": "server", "timestamp": "2024-04-03T02:49:04,486Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "clearml", "node.name": "clearml", "message": "initialized" }
{"type": "server", "timestamp": "2024-04-03T02:49:04,487Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "clearml", "node.name": "clearml", "message": "starting ..." }
{"type": "server", "timestamp": "2024-04-03T02:49:04,505Z", "level": "INFO", "component": "o.e.x.s.c.f.PersistentCache", "cluster.name": "clearml", "node.name": "clearml", "message": "persistent cache index loaded" }
{"type": "server", "timestamp": "2024-04-03T02:49:04,506Z", "level": "INFO", "component": "o.e.x.d.l.DeprecationIndexingComponent", "cluster.name": "clearml", "node.name": "clearml", "message": "deprecation component started" }
{"type": "server", "timestamp": "2024-04-03T02:49:04,645Z", "level": "INFO", "component": "o.e.t.TransportService", "cluster.name": "clearml", "node.name": "clearml", "message": "publish_address {192.168.16.3:9300}, bound_addresses {0.0.0.0:9300}" }
{"type": "server", "timestamp": "2024-04-03T02:49:05,021Z", "level": "INFO", "component": "o.e.c.c.Coordinator", "cluster.name": "clearml", "node.name": "clearml", "message": "cluster UUID [TTohLnsDRueenl__VV7vBw]" }
{"type": "server", "timestamp": "2024-04-03T02:49:05,173Z", "level": "INFO", "component": "o.e.c.s.MasterService", "cluster.name": "clearml", "node.name": "clearml", "message": "elected-as-master ([1] nodes joined)[{clearml}{A6LcQvX-RYyaLHo9pDNESA}{PmEz4BOmQvCTiWgtpW8GvQ}{192.168.16.3}{192.168.16.3:9300}{cdfhilmrstw} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 4, version: 75, delta: master node changed {previous [], current [{clearml}{A6LcQvX-RYyaLHo9pDNESA}{PmEz4BOmQvCTiWgtpW8GvQ}{192.168.16.3}{192.168.16.3:9300}{cdfhilmrstw}]}" }
{"type": "server", "timestamp": "2024-04-03T02:49:05,328Z", "level": "INFO", "component": "o.e.c.s.ClusterApplierService", "cluster.name": "clearml", "node.name": "clearml", "message": "master node changed {previous [], current [{clearml}{A6LcQvX-RYyaLHo9pDNESA}{PmEz4BOmQvCTiWgtpW8GvQ}{192.168.16.3}{192.168.16.3:9300}{cdfhilmrstw}]}, term: 4, version: 75, reason: Publication{term=4, version=75}" }
{"type": "server", "timestamp": "2024-04-03T02:49:05,545Z", "level": "INFO", "component": "o.e.h.AbstractHttpServerTransport", "cluster.name": "clearml", "node.name": "clearml", "message": "publish_address {192.168.16.3:9200}, bound_addresses {0.0.0.0:9200}", "cluster.uuid": "TTohLnsDRueenl__VV7vBw", "node.id": "A6LcQvX-RYyaLHo9pDNESA" }
{"type": "server", "timestamp": "2024-04-03T02:49:05,546Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "clearml", "node.name": "clearml", "message": "started", "cluster.uuid": "TTohLnsDRueenl__VV7vBw", "node.id": "A6LcQvX-RYyaLHo9pDNESA" }
{"type": "server", "timestamp": "2024-04-03T02:49:06,125Z", "level": "INFO", "component": "o.e.l.LicenseService", "cluster.name": "clearml", "node.name": "clearml", "message": "license [0f53cb6c-bba4-4c42-bec6-e47b5365624c] mode [basic] - valid", "cluster.uuid": "TTohLnsDRueenl__VV7vBw", "node.id": "A6LcQvX-RYyaLHo9pDNESA" }
{"type": "server", "timestamp": "2024-04-03T02:49:06,130Z", "level": "INFO", "component": "o.e.g.GatewayService", "cluster.name": "clearml", "node.name": "clearml", "message": "recovered [6] indices into cluster_state", "cluster.uuid": "TTohLnsDRueenl__VV7vBw", "node.id": "A6LcQvX-RYyaLHo9pDNESA" }
{"type": "server", "timestamp": "2024-04-03T02:49:06,715Z", "level": "ERROR", "component": "o.e.i.g.GeoIpDownloader", "cluster.name": "clearml", "node.name": "clearml", "message": "exception during geoip databases update", "cluster.uuid": "TTohLnsDRueenl__VV7vBw", "node.id": "A6LcQvX-RYyaLHo9pDNESA" ,
"stacktrace": ["org.elasticsearch.ElasticsearchException: not all primary shards of [.geoip_databases] index are active",
"at org.elasticsearch.ingest.geoip.GeoIpDownloader.updateDatabases(GeoIpDownloader.java:137) ~[ingest-geoip-7.17.18.jar:7.17.18]",
"at org.elasticsearch.ingest.geoip.GeoIpDownloader.runDownloader(GeoIpDownloader.java:284) [ingest-geoip-7.17.18.jar:7.17.18]",
"at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:100) [ingest-geoip-7.17.18.jar:7.17.18]",
"at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:46) [ingest-geoip-7.17.18.jar:7.17.18]",
"at org.elasticsearch.persistent.NodePersistentTasksExecutor$1.doRun(NodePersistentTasksExecutor.java:42) [elasticsearch-7.17.18.jar:7.17.18]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777) [elasticsearch-7.17.18.jar:7.17.18]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.17.18.jar:7.17.18]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]",
"at java.lang.Thread.run(Thread.java:1583) [?:?]"] }
{"type": "server", "timestamp": "2024-04-03T02:49:06,735Z", "level": "INFO", "component": "o.e.c.r.a.AllocationService", "cluster.name": "clearml", "node.name": "clearml", "message": "Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[.geoip_databases][0], [queue_metrics_d1bd92a3b039400cbafc60a7a5b1e52b_2024-04][0], [events-training_debug_image-][0], [events-plot-][0], [events-log-][0], [events-training_stats_scalar-][0]]]).", "cluster.uuid": "TTohLnsDRueenl__VV7vBw", "node.id": "A6LcQvX-RYyaLHo9pDNESA" }
{"type": "server", "timestamp": "2024-04-03T02:49:06,769Z", "level": "INFO", "component": "o.e.i.g.DatabaseNodeService", "cluster.name": "clearml", "node.name": "clearml", "message": "retrieve geoip database [GeoLite2-Country.mmdb] from [.geoip_databases] to [/tmp/elasticsearch-9093165662722554014/geoip-databases/A6LcQvX-RYyaLHo9pDNESA/GeoLite2-Country.mmdb.tmp.gz]", "cluster.uuid": "TTohLnsDRueenl__VV7vBw", "node.id": "A6LcQvX-RYyaLHo9pDNESA" }
{"type": "server", "timestamp": "2024-04-03T02:49:06,788Z", "level": "INFO", "component": "o.e.i.g.DatabaseNodeService", "cluster.name": "clearml", "node.name": "clearml", "message": "retrieve geoip database [GeoLite2-City.mmdb] from [.geoip_databases] to [/tmp/elasticsearch-9093165662722554014/geoip-databases/A6LcQvX-RYyaLHo9pDNESA/GeoLite2-City.mmdb.tmp.gz]", "cluster.uuid": "TTohLnsDRueenl__VV7vBw", "node.id": "A6LcQvX-RYyaLHo9pDNESA" }
{"type": "server", "timestamp": "2024-04-03T02:49:06,788Z", "level": "INFO", "component": "o.e.i.g.DatabaseNodeService", "cluster.name": "clearml", "node.name": "clearml", "message": "retrieve geoip database [GeoLite2-ASN.mmdb] from [.geoip_databases] to [/tmp/elasticsearch-9093165662722554014/geoip-databases/A6LcQvX-RYyaLHo9pDNESA/GeoLite2-ASN.mmdb.tmp.gz]", "cluster.uuid": "TTohLnsDRueenl__VV7vBw", "node.id": "A6LcQvX-RYyaLHo9pDNESA" }
{"type": "server", "timestamp": "2024-04-03T02:49:07,136Z", "level": "INFO", "component": "o.e.i.g.DatabaseNodeService", "cluster.name": "clearml", "node.name": "clearml", "message": "successfully reloaded changed geoip database file [/tmp/elasticsearch-9093165662722554014/geoip-databases/A6LcQvX-RYyaLHo9pDNESA/GeoLite2-Country.mmdb]", "cluster.uuid": "TTohLnsDRueenl__VV7vBw", "node.id": "A6LcQvX-RYyaLHo9pDNESA" }
{"type": "server", "timestamp": "2024-04-03T02:49:07,195Z", "level": "INFO", "component": "o.e.i.g.DatabaseNodeService", "cluster.name": "clearml", "node.name": "clearml", "message": "successfully reloaded changed geoip database file [/tmp/elasticsearch-9093165662722554014/geoip-databases/A6LcQvX-RYyaLHo9pDNESA/GeoLite2-ASN.mmdb]", "cluster.uuid": "TTohLnsDRueenl__VV7vBw", "node.id": "A6LcQvX-RYyaLHo9pDNESA" }
{"type": "server", "timestamp": "2024-04-03T02:49:07,785Z", "level": "INFO", "component": "o.e.i.g.DatabaseNodeService", "cluster.name": "clearml", "node.name": "clearml", "message": "successfully reloaded changed geoip database file [/tmp/elasticsearch-9093165662722554014/geoip-databases/A6LcQvX-RYyaLHo9pDNESA/GeoLite2-City.mmdb]", "cluster.uuid": "TTohLnsDRueenl__VV7vBw", "node.id": "A6LcQvX-RYyaLHo9pDNESA" }
from clearml-server.
i fix it by following advice from @evg-allegro
put the following setting in the docker-compose.yaml under elasticsearch->environment:
ingest.geoip.downloader.enabled: false
also i need to change elastic search folder to /var/lib/elasticsearch/data
in docker-compose.yml.
from clearml-server.
hope it will help anyone who is a beginner to clearML.
from clearml-server.
Related Issues (20)
- Nginx Not Loading Plotly.js Resource: ClearML Self-Hosted Docker HOT 7
- Failed Navigate From Overview to Experiments Details HOT 4
- Async Delete Always Failed when Removing Experiments (using Minio)
- nginx 0.6.x < 1.20.1 1-Byte Memory Overwrite RCE vulnerability HOT 2
- ElasticSearch UI and Redis UI? HOT 2
- The problem with scalars HOT 12
- Curl 7.69 < 8.4.0 Heap Buffer Overflow vulnerability HOT 2
- OpenSSL 1.1.1 < 1.1.1x Vulnerability HOT 1
- Elasticsearch image tag 7.17 does not exist HOT 4
- Git package is not installed by default in node:20-bookworm-slim HOT 1
- APP Credentials disapper in webapp HOT 21
- Scalar graphs legend is too narrow for experiments with long names HOT 9
- Update from 1.14.1 to 1.15.0 leads to several fatal issues when booting HOT 3
- AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. HOT 3
- Web server Ipv6 error
- Error Download via UI for Azure Blob Storage HOT 1
- after upgrading to 1.16.0, images dont load in web UI. HOT 15
- How to limit parallel job per queue + user permission Q HOT 2
- clearml server deletes APP credentials on server restart HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from clearml-server.