openebs-archive / jiva Goto Github PK
View Code? Open in Web Editor NEWCAS Data Engine - iSCSI Distributed Block Storage Controller built-in Go
Home Page: https://openebs.io
License: Apache License 2.0
CAS Data Engine - iSCSI Distributed Block Storage Controller built-in Go
Home Page: https://openebs.io
License: Apache License 2.0
Jiva is meant to be an wrapper project that will pull in the different binaries from other projects like openebs/longhorn, openebs/gotgt etc., and generate an docker image named "jiva-"
The 2 links mentioned in README.md file, referring to OpenEBS latest documentation aren't working. The url for the same has changed. I would like to work on this issue.
What steps did you take and what happened:
I am trying to use jiva volume with openebs hostpath backend on a IPv6 only kubernetes cluster.
Though I am able to create PVC/PV, pod is unable to mount the volume due to time out issues. Because jiva
volume is stuck in Syncing
phase which is supposed to be in Ready
. Status shown as RO
instead of RW
.
Pod events,
kubelet MountVolume.MountDevice failed for volume "pvc-8173df52-68a8-4ca9-b4f3-9721479eb6c0" : rpc error: code = FailedPrecondition desc = Max retry count exceeded, volume: {pvc-8173df52-68a8-4ca9-b4f3-9721479eb6c0} is not ready
jiva volume status
[root@k8s-master01 debug]# kubectl -n openebs get jivavolume
NAME REPLICACOUNT PHASE STATUS
pvc-8173df52-68a8-4ca9-b4f3-9721479eb6c0 Syncing RO
[root@k8s-master01 debug]#
What did you expect to happen:
jiva volume to be in Ready
phase and successfully attached to pod.
The output of the following commands will help us better understand what's going on:
(Pasting long output into a GitHub gist or other pastebin is fine.)
kubectl logs <jiva controller pod name> -n openebs
(Optional)[root@k8s-master01 ~]# k -n openebs logs pvc-8173df52-68a8-4ca9-b4f3-9721479eb6c0-jiva-ctrl-5fd8c744kcg6
Defaulted container "jiva-controller" out of: jiva-controller, maya-volume-exporter
time="2023-04-11T09:10:07Z" level=info msg="RPC_READ_TIMEOUT env not set"
time="2023-04-11T09:10:07Z" level=info msg="RPC_WRITE_TIMEOUT env not set"
time="2023-04-11T09:10:07Z" level=info msg="REPLICATION_FACTOR: 3, RPC_READ_TIMEOUT: 0s, RPC_WRITE_TIMEOUT: 0s"
time="2023-04-11T09:10:07Z" level=info msg="Starting controller with frontendIP: , and clusterIP: fd00:10:1616:2222::6144"
time="2023-04-11T09:10:07Z" level=info msg="resetting controller"
time="2023-04-11T09:10:07Z" level=info msg="Listening on :9501"
time="2023-04-11T09:10:14Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 2.038381423s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:10:14Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:10:14 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:10:14Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 2.02715368s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:10:14Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:10:14 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:10:15Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 2.029487934s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:10:15Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:10:15 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:10:19Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 7.037789788s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:10:19Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:10:19 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:10:19Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 7.026113632s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:10:19Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:10:19 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:10:20Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 7.027654537s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:10:20Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:10:20 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:10:24Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 12.036935653s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:10:24Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:10:24 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:10:24Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 12.026535002s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:10:24Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:10:24 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:10:25Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 12.028114649s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:10:25Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:10:25 +0000] "POST /v1/register HTTP/1.1" 200 0
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:10:29 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:10:29Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 17.038064107s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:10:29Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:10:29Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 17.025846103s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:10:29Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:10:29 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:10:30Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 17.0281659s State: closed Type: Backend RevisionCount: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:10:30 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:10:30Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:10:34 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:10:34Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 22.037787267s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:10:34Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:10:34Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 22.026016085s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:10:34Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:10:34 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:10:35Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 22.028069374s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:10:35Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:10:35 +0000] "POST /v1/register HTTP/1.1" 200 0
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:10:39 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:10:39Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 27.037710296s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:10:39Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:10:39Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 27.026159802s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:10:39Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:10:39 +0000] "POST /v1/register HTTP/1.1" 200 0
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:10:40 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:10:40Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 27.027734071s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:10:40Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:10:44Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 32.038739561s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:10:44Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:10:44 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:10:44Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 32.026159237s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:10:44Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:10:44 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:10:45Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 32.028531943s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:10:45Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:10:45 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:10:49Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 37.038299406s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:10:49Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:10:49 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:10:49Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 37.026251473s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:10:49Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:10:49 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:10:50Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 37.027753804s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:10:50Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:10:50 +0000] "POST /v1/register HTTP/1.1" 200 0
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:10:54 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:10:54Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 42.0376928s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:10:54Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:10:54Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 42.026162453s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:10:54Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:10:54 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:10:55Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 42.02781596s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:10:55Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:10:55 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:10:59Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 47.038438047s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:10:59Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:10:59 +0000] "POST /v1/register HTTP/1.1" 200 0
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:10:59 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:10:59Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 47.026486173s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:10:59Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:11:00 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:11:00Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 47.028286757s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:00Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:11:04 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:11:04Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 52.03781717s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:04Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:11:04Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 52.026367007s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:04Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:11:04 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:11:05Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 52.027807652s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:05Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:11:05 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:11:09Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 57.037809293s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:09Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:11:09 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:11:09Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 57.026394123s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:09Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:11:09 +0000] "POST /v1/register HTTP/1.1" 200 0
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:11:10 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:11:10Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 57.027881807s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:10Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:11:14Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 1m2.03841576s State: closed Type: Backend RevisionCount: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:11:14 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:11:14Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:11:14Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 1m2.026194299s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:14Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:11:14 +0000] "POST /v1/register HTTP/1.1" 200 0
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:11:15 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:11:15Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 1m2.028839823s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:15Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:11:19Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 1m7.037921763s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:19Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:11:19 +0000] "POST /v1/register HTTP/1.1" 200 0
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:11:19 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:11:19Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 1m7.026055585s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:19Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:11:20Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 1m7.027732459s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:20Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:11:20 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:11:24Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 1m12.037670403s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:24Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:11:24 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:11:24Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 1m12.025936605s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:24Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:11:24 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:11:25Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 1m12.028519605s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:25Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:11:25 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:11:29Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 1m17.037504881s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:29Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:11:29 +0000] "POST /v1/register HTTP/1.1" 200 0
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:11:29 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:11:29Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 1m17.026127066s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:29Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:11:30Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 1m17.02754257s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:30Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:11:30 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:11:34Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 1m22.037473733s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:34Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:11:34 +0000] "POST /v1/register HTTP/1.1" 200 0
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:11:34 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:11:34Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 1m22.026049639s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:34Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:11:35 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:11:35Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 1m22.027708656s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:35Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:11:39 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:11:39Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 1m27.037542166s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:39Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:11:39 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:11:39Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 1m27.026236535s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:39Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:11:40 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:11:40Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 1m27.027877113s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:40Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:11:44 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:11:44Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 1m32.037837041s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:44Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:11:44Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 1m32.026494435s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:44Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:11:44 +0000] "POST /v1/register HTTP/1.1" 200 0
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:11:45 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:11:45Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 1m32.027764564s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:45Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:11:49 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:11:49Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 1m37.037503691s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:49Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:11:49 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:11:49Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 1m37.026673576s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:49Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:11:50Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 1m37.031854737s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:50Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:11:50 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:11:54Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 1m42.037447622s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:54Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:11:54 +0000] "POST /v1/register HTTP/1.1" 200 0
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:11:54 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:11:54Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 1m42.026308224s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:54Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:11:55 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:11:55Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 1m42.028752873s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:55Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:11:59Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 1m47.037966348s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:59Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:11:59 +0000] "POST /v1/register HTTP/1.1" 200 0
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:11:59 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:11:59Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 1m47.026402208s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:11:59Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:12:00 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:12:00Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 1m47.029784276s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:00Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:12:04Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 1m52.037830946s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:04Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:12:04 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:12:04Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 1m52.027565843s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:04Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:12:04 +0000] "POST /v1/register HTTP/1.1" 200 0
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:12:05 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:12:05Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 1m52.029670778s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:05Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:12:09 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:12:09Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 1m57.03770945s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:09Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:12:09Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 1m57.026346501s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:09Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:12:09 +0000] "POST /v1/register HTTP/1.1" 200 0
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:12:10 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:12:10Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 1m57.027822187s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:10Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:12:14 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:12:14Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 2m2.038019684s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:14Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:12:14Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 2m2.026060834s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:14Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:12:14 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:12:15Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 2m2.028381957s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:15Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:12:15 +0000] "POST /v1/register HTTP/1.1" 200 0
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:12:19 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:12:19Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 2m7.03781419s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:19Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:12:19Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 2m7.025931586s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:19Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:12:19 +0000] "POST /v1/register HTTP/1.1" 200 0
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:12:20 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:12:20Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 2m7.028040384s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:20Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:12:24Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 2m12.03786774s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:24Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:12:24 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:12:24Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 2m12.026422844s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:24Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:12:24 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:12:25Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 2m12.028149798s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:25Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:12:25 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:12:29Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 2m17.038654957s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:29Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:12:29 +0000] "POST /v1/register HTTP/1.1" 200 0
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:12:29 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:12:29Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 2m17.026347298s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:29Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:12:30 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:12:30Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 2m17.029901277s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:30Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:12:34Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 2m22.038136584s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:34Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:12:34 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:12:34Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 2m22.026376421s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:34Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:12:34 +0000] "POST /v1/register HTTP/1.1" 200 0
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:12:35 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:12:35Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 2m22.028166348s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:35Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:12:39Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 2m27.038398724s State: closed Type: Backend RevisionCount: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:12:39 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:12:39Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:12:39Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 2m27.026729039s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:39Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:12:39 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:12:40Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 2m27.027991391s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:40Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:12:40 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:12:44Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 2m32.037518757s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:44Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:12:44 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:12:44Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 2m32.026780554s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:44Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:12:44 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:12:45Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 2m32.028121202s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:45Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:12:45 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:12:49Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 2m37.038071583s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:49Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:12:49 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:12:49Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 2m37.026273187s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:49Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:12:49 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:12:50Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 2m37.02777268s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:50Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:12:50 +0000] "POST /v1/register HTTP/1.1" 200 0
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:12:54 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:12:54Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 2m42.038401485s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:54Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:12:54Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 2m42.026322143s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:54Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:12:54 +0000] "POST /v1/register HTTP/1.1" 200 0
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:12:55 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:12:55Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 2m42.027968475s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:55Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:12:59Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 2m47.037906802s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:59Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:12:59 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:12:59Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 2m47.026308282s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:12:59Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:12:59 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:13:00Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 2m47.028735179s State: closed Type: Backend RevisionCount: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:13:00 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:13:00Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:13:04Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 2m52.037275711s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:13:04Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:13:04 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:13:04Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 2m52.026159242s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:13:04Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:13:04 +0000] "POST /v1/register HTTP/1.1" 200 0
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:13:05 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:13:05Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 2m52.028328942s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:13:05Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:13:09Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 2m57.037274849s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:13:09Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:13:09 +0000] "POST /v1/register HTTP/1.1" 200 0
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:13:09 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:13:09Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 2m57.026130557s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:13:09Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:13:10Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 2m57.027584541s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:13:10Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:13:10 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:13:14Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 3m2.038765509s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:13:14Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:13:14 +0000] "POST /v1/register HTTP/1.1" 200 0
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:13:14 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:13:14Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 3m2.026266502s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:13:14Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:13:15 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:13:15Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 3m2.031891604s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:13:15Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:13:19Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 3m7.037717213s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:13:19Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:13:19 +0000] "POST /v1/register HTTP/1.1" 200 0
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:13:19 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:13:19Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 3m7.026759958s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:13:19Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
time="2023-04-11T09:13:20Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 3m7.027741008s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:13:20Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:13:20 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:13:24Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 3m12.0378223s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:13:24Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:13:24 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:13:24Z" level=info msg="Register Replica, Address: localhost UUID: 30f446b84c2a87a75db0c17dc7b8d1a94917bfe1 Uptime: 3m12.026230865s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:13:24Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:13:24 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:13:25Z" level=info msg="Register Replica, Address: localhost UUID: ed0c2413de04a41dd2da4b39a4f52813b141ede6 Uptime: 3m12.02788935s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:13:25Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:9610:160a:42e7:bce9 - - [11/Apr/2023:09:13:25 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-04-11T09:13:29Z" level=info msg="Register Replica, Address: localhost UUID: c4d1d0f74651f2bf516814ae5663431b7479f8d2 Uptime: 3m17.03750985s State: closed Type: Backend RevisionCount: 1"
time="2023-04-11T09:13:29Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
fd00:10:1616:1111:c88b:2f46:cb28:509f - - [11/Apr/2023:09:13:29 +0000] "POST /v1/register HTTP/1.1" 200 0
fd00:10:1616:1111:6042:cb0c:28fd:7609 - - [11/Apr/2023:09:13:29 +0000] "POST /v1/register HTTP/1.1" 200 0
kubectl logs <jiva replica pod name> -n openebs
(Optional)[root@k8s-master01 ~]# k -n openebs logs pvc-8173df52-68a8-4ca9-b4f3-9721479eb6c0-jiva-rep-0
time="2023-04-11T09:17:48Z" level=info msg="MAX_CHAIN_LENGTH env not set, default value is 512"
time="2023-04-11T09:17:48Z" level=info msg="Configured logging with retentionPeriod: 180, maxLogFileSize: 100, maxBackups: 5"
time="2023-04-11T09:17:48Z" level=info msg="Creating volume openebs, size 5368709120/512"
time="2023-04-11T09:17:48Z" level=info msg="Starting replica having replicaType: , frontendIP: fd00:10:1616:2222::6144, size: 5368709120, dir: openebs"
time="2023-04-11T09:17:48Z" level=info msg="Setting replicaAddr: :9502, controlAddr: :9502, dataAddr: :9503, syncAddr: :9504"
time="2023-04-11T09:17:48Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2023-04-11T09:17:48Z" level=info msg="Listening on data :9503"
time="2023-04-11T09:17:48Z" level=info msg="Listening on control :9502"
time="2023-04-11T09:17:48Z" level=info msg="Listening on sync :9504 start: 9700 end: 9800"
time="2023-04-11T09:17:50Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2023-04-11T09:17:50Z" level=info msg="Closing replica"
time="2023-04-11T09:17:50Z" level=info msg="Skip closing replica, s.r not set"
time="2023-04-11T09:17:50Z" level=info msg="CheckAndResetFailedRebuild tcp://localhost:9502"
time="2023-04-11T09:17:50Z" level=info msg="Addreplica tcp://localhost:9502"
time="2023-04-11T09:17:50Z" level=info msg="Get Volume info from controller"
time="2023-04-11T09:17:50Z" level=info msg="Register replica at controller"
time="2023-04-11T09:17:52Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2023-04-11T09:17:54Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2023-04-11T09:17:55Z" level=info msg="Timed out waiting for response from controller, will retry"
time="2023-04-11T09:17:55Z" level=info msg="Get Volume info from controller"
time="2023-04-11T09:17:55Z" level=info msg="Register replica at controller"
time="2023-04-11T09:17:56Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2023-04-11T09:17:58Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2023-04-11T09:18:00Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2023-04-11T09:18:00Z" level=info msg="Timed out waiting for response from controller, will retry"
time="2023-04-11T09:18:00Z" level=info msg="Get Volume info from controller"
time="2023-04-11T09:18:00Z" level=info msg="Register replica at controller"
time="2023-04-11T09:18:02Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2023-04-11T09:18:04Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2023-04-11T09:18:05Z" level=info msg="Timed out waiting for response from controller, will retry"
time="2023-04-11T09:18:05Z" level=info msg="Get Volume info from controller"
time="2023-04-11T09:18:05Z" level=info msg="Register replica at controller"
time="2023-04-11T09:18:06Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2023-04-11T09:18:08Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2023-04-11T09:18:10Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2023-04-11T09:18:10Z" level=info msg="Timed out waiting for response from controller, will retry"
time="2023-04-11T09:18:10Z" level=info msg="Get Volume info from controller"
time="2023-04-11T09:18:10Z" level=info msg="Register replica at controller"
time="2023-04-11T09:18:12Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2023-04-11T09:18:14Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2023-04-11T09:18:15Z" level=info msg="Timed out waiting for response from controller, will retry"
time="2023-04-11T09:18:15Z" level=info msg="Get Volume info from controller"
time="2023-04-11T09:18:15Z" level=info msg="Register replica at controller"
time="2023-04-11T09:18:16Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2023-04-11T09:18:18Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2023-04-11T09:18:20Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2023-04-11T09:18:20Z" level=info msg="Timed out waiting for response from controller, will retry"
time="2023-04-11T09:18:20Z" level=info msg="Get Volume info from controller"
time="2023-04-11T09:18:20Z" level=info msg="Register replica at controller"
time="2023-04-11T09:18:22Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2023-04-11T09:18:24Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2023-04-11T09:18:25Z" level=info msg="Timed out waiting for response from controller, will retry"
time="2023-04-11T09:18:25Z" level=info msg="Get Volume info from controller"
time="2023-04-11T09:18:25Z" level=info msg="Register replica at controller"
time="2023-04-11T09:18:26Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2023-04-11T09:18:28Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2023-04-11T09:18:30Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2023-04-11T09:18:30Z" level=info msg="Timed out waiting for response from controller, will retry"
time="2023-04-11T09:18:30Z" level=info msg="Get Volume info from controller"
time="2023-04-11T09:18:30Z" level=info msg="Register replica at controller"
time="2023-04-11T09:18:32Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2023-04-11T09:18:34Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2023-04-11T09:18:35Z" level=info msg="Timed out waiting for response from controller, will retry"
time="2023-04-11T09:18:35Z" level=info msg="Get Volume info from controller"
time="2023-04-11T09:18:35Z" level=info msg="Register replica at controller"
time="2023-04-11T09:18:36Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2023-04-11T09:18:38Z" level=info msg="Waiting for s.Replica() to be non nil"
time="2023-04-11T09:18:40Z" level=info msg="Timed out waiting for response from controller, will retry"
kubectl exec -it <jiva controller pod name> -n openebs -- jivactl ls
[root@k8s-master01 ~]# k -n openebs exec -it pvc-8173df52-68a8-4ca9-b4f3-9721479eb6c0-jiva-ctrl-5fd8c744kcg6 -- jivactl ls
Defaulted container "jiva-controller" out of: jiva-controller, maya-volume-exporter
ADDRESS MODE CHAIN
[root@k8s-master01 ~]#
Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]
Environment:
kubectl version
): v1.24.9/etc/os-release
): CentOS Linux 7 (Core) 3.10.0-1160.81.1.el7.x86_64The CI on Jiva should cover the following:
The scope of this issue is to implement and document the CI framework for Jiva. The framework should provide example/guidelines for extending the CI framework with additional pipeline items under Functional and Non-Functional categories.
When I run the make build command. It would failded after the jiva image was build successfully.
The error showed
INFO[0000] Trash! Reading file: 'vendor.conf'
INFO[0000] Using 'github.com/openebs/jiva' as the project's root package (from vendor.conf)
INFO[0000] Preparing cache for 'github.com/Sirupsen/logrus'
INFO[0132] Checking out 'github.com/Sirupsen/logrus', commit: '446d1c146faa8ed3f4218f056fcd165f6bcfda81'
INFO[0132] Fetching latest commits from 'origin' for 'github.com/Sirupsen/logrus'
ERRO[0132] `git fetch -f -t origin` failed:
fatal: 'origin' does not appear to be a git repository
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
FATA[0132] fetch failed i={Package:github.com/Sirupsen/logrus Version:446d1c146faa8ed3f4218f056fcd165f6bcfda81 Repo: Update:false Options:{Transitive:false Staging:false}}
FATA[0135] exit status 1
make: *** [trash] Error 1
The solution from the community is Sirupsen/logrus repo has been renamed to sirupsen/logrus and we have not updated it yet. You can replace Sirupsen to sirupsen in the import paths in all the files in jiva repo and it will work.
But the repo 'github.com/aws/aws-sdk-go' still has problems.
INFO[0000] Trash! Reading file: 'vendor.conf'
INFO[0000] Using 'github.com/openebs/jiva' as the project's root package (from vendor.conf)
INFO[0000] Preparing cache for 'github.com/aws/aws-sdk-go'
INFO[0133] Checking out 'github.com/aws/aws-sdk-go', commit: 'c924893c38ecc04b18d7aab8a7aa561cb8b4c4cc'
INFO[0133] Fetching latest commits from 'origin' for 'github.com/aws/aws-sdk-go'
ERRO[0133] `git fetch -f -t origin` failed:
fatal: 'origin' does not appear to be a git repository
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
FATA[0133] fetch failed i={Package:github.com/aws/aws-sdk-go Version:c924893c38ecc04b18d7aab8a7aa561cb8b4c4cc Repo: Update:false Options:{Transitive:false Staging:false}}
FATA[0136] exit status 1
make: *** [trash] Error 1
Travis build is failing with error
The command '/bin/sh -c wget -O - https://storage.googleapis.com/golang/go1.10.3.linux-${!GOLANG_ARCH}.tar.gz | tar -xzf - -C /usr/local && go get github.com/rancher/trash && go get github.com/golang/lint/golint && go get github.com/prometheus/client_golang/prometheus/promhttp' returned a non-zero code: 1
The discovery of a Jiva iSCSI target via the Ctrl Pod/Service IP is seen to return a negative value (-1) for the target portal group tag.
karthik@MayaHost01:/var/openebs$ sudo iscsiadm -m discovery -t st -p 10.36.0.1:3260
10.102.158.249:3260,-1 iqn.2016-09.com.openebs.jiva:pvc-2845cd53-c11f-11e7-9be8-000c298ff5fc
The ansible-based integration test attempting login/volume mount is seen to fail due to the above.
Some details about this key from iSCSI RFC :
TargetPortalGroupTag=<16-bit-binary-value>
Examples:
TargetPortalGroupTag=1
The target portal group tag is a 16-bit binary-value that uniquely
identifies a portal group within an iSCSI target node. This key
carries the value of the tag of the portal group that is servicing
the Login request. The iSCSI target returns this key to the
initiator in the Login Response PDU to the first Login Request PDU
that has the C bit set to 0 when TargetName is given by the
initiator.
For the complete usage expectations of this key see Section 5.3 Login
Phase
Test error details :
TASK [Establish iSCSI session with volume] *****************************************************************************
task path: /home/ciuser/openebs/e2e/ansible/playbooks/feature/snapshots/test-snapshot.yml:67
fatal: [localhost -> None]: FAILED! => {"changed": false, "failed": true, "module_stderr": "Shared connection to 20.10.49.12 closed.\r\n", "module_stdout": "\r\nTraceback (most recent call last):\r\n File \"/tmp/ansible_8U8slY/ansible_module_open_iscsi.py\", line 390, in <module>\r\n main()\r\n File \"/tmp/ansible_8U8slY/ansible_module_open_iscsi.py\", line 333, in main\r\n target = nodes[0]\r\nIndexError: list index out of range\r\n", "msg": "MODULE FAILURE", "rc": 0}
def iscsi_get_cached_nodes(module, portal=None):
cmd = '%s --mode node' % iscsiadm_cmd
(rc, out, err) = module.run_command(cmd)
if rc == 0:
lines = out.splitlines()
nodes = []
for line in lines:
# line format is "ip:port,target_portal_group_tag targetname"
parts = line.split()
if len(parts) > 2:
module.fail_json(msg='error parsing output', cmd=cmd)
target = parts[1]
parts = parts[0].split(':')
target_portal = parts[0]
if portal is None or portal == target_portal:
nodes.append(target)
I got statistics via Jiva's stats API.
{
"Name": "pvc-30266c71-21b1-11e8-953c-060010b04334",
"ReadIOPS": "327066",
"ReplicaCounter": 0,
"RevisionCounter": 0,
"SCSIIOCount": {
"0": 6640,
"158": 454,
"160": 2,
"163": 2,
"18": 3388,
"26": 908,
"40": 327066,
"42": 258086
},
"SectorSize": "4096",
"Size": "5368709120",
"TotalReadBlockCount": "17549045760",
"TotalReadTime": "307301413134",
"TotalWriteTime": "215794984886",
"TotatWriteBlockCount": "5395550208",
"UpTime": 81611.82397587,
"UsedBlocks": "57861",
"UsedLogicalBlocks": "29075",
"WriteIOPS": "244996",
"actions": {},
"links": {
"self": "http://192.168.1.11:9501/v1/stats"
},
"type": "stats"
}
I think that these ReadIOPS/WriteIOPS values is TotalIOCount rather than IOPS.
I would like gotgt to return correct IOPS values.
Run the libiscsi tool to run iSCSI compliance tests on the jiva iSCSI target in the travis build. Refer to the following to aid this task:
What steps did you take and what happened:
[A clear and concise description of what the bug is, and what commands you ran.)
GItHub Actions CI fails Jiva builds.
What did you expect to happen:
GitHub Actions CI passes Jiva builds.
The output of the following commands will help us better understand what's going on:
(Pasting long output into a GitHub gist or other pastebin is fine.)
https://gist.github.com/niladrih/a80e950b83090696c8935aba6df2d25d
kubectl logs <jiva controller pod name> -n openebs
(Optional)kubectl logs <jiva replica pod name> -n openebs
(Optional)kubectl exec -it <jiva controller pod name> -n openebs -- jivactl ls
Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]
Environment:
kubectl version
):/etc/os-release
):The command or script should provide details on the parameters and bring out any dependencies on the infrastructure like:
WARNING: DATA RACE
Read at 0x00c4200fefc8 by goroutine 39:
github.com/openebs/jiva/controller/rest.(*Server).RegisterReplica.func1()
/go/src/github.com/openebs/jiva/controller/rest/replica.go:106 +0xfa
github.com/openebs/jiva/controller/rest.(*Server).RegisterReplica()
/go/src/github.com/openebs/jiva/controller/rest/replica.go:113 +0x42c
github.com/openebs/jiva/controller/rest.(*Server).RegisterReplica-fm()
/go/src/github.com/openebs/jiva/controller/rest/router.go:36 +0x63
github.com/openebs/jiva/replica/rest.HandleError.func1()
/go/src/github.com/openebs/jiva/replica/rest/router.go:15 +0x5e
net/http.HandlerFunc.ServeHTTP()
/usr/local/go/src/net/http/server.go:1918 +0x51
github.com/openebs/jiva/vendor/github.com/rancher/go-rancher/api.ApiHandler.func1()
/go/src/github.com/openebs/jiva/vendor/github.com/rancher/go-rancher/api/handler.go:26 +0x177
net/http.HandlerFunc.ServeHTTP()
/usr/local/go/src/net/http/server.go:1918 +0x51
github.com/openebs/jiva/vendor/github.com/gorilla/context.ClearHandler.func1()
/go/src/github.com/openebs/jiva/vendor/github.com/gorilla/context/context.go:141 +0xa2
net/http.HandlerFunc.ServeHTTP()
/usr/local/go/src/net/http/server.go:1918 +0x51
github.com/openebs/jiva/vendor/github.com/gorilla/mux.(*Router).ServeHTTP()
/go/src/github.com/openebs/jiva/vendor/github.com/gorilla/mux/mux.go:103 +0x363
github.com/openebs/jiva/vendor/github.com/gorilla/handlers.loggingHandler.ServeHTTP()
/go/src/github.com/openebs/jiva/vendor/github.com/gorilla/handlers/handlers.go:69 +0x174
github.com/openebs/jiva/vendor/github.com/gorilla/handlers.(*loggingHandler).ServeHTTP()
<autogenerated>:1 +0xa1
github.com/openebs/jiva/util.filteredLoggingHandler.ServeHTTP()
/go/src/github.com/openebs/jiva/util/util.go:88 +0x91
github.com/openebs/jiva/util.(*filteredLoggingHandler).ServeHTTP()
<autogenerated>:1 +0xa2
github.com/openebs/jiva/vendor/github.com/gorilla/handlers.ProxyHeaders.func1()
/go/src/github.com/openebs/jiva/vendor/github.com/gorilla/handlers/proxy_headers.go:54 +0xc3
net/http.HandlerFunc.ServeHTTP()
/usr/local/go/src/net/http/server.go:1918 +0x51
net/http.serverHandler.ServeHTTP()
/usr/local/go/src/net/http/server.go:2619 +0xbc
net/http.(*conn).serve()
/usr/local/go/src/net/http/server.go:1801 +0x83b
Previous write at 0x00c4200fefc8 by goroutine 37:
github.com/openebs/jiva/controller/rest.(*Server).RegisterReplica()
/go/src/github.com/openebs/jiva/controller/rest/replica.go:84 +0x123
github.com/openebs/jiva/controller/rest.(*Server).RegisterReplica-fm()
/go/src/github.com/openebs/jiva/controller/rest/router.go:36 +0x63
github.com/openebs/jiva/replica/rest.HandleError.func1()
/go/src/github.com/openebs/jiva/replica/rest/router.go:15 +0x5e
net/http.HandlerFunc.ServeHTTP()
/usr/local/go/src/net/http/server.go:1918 +0x51
github.com/openebs/jiva/vendor/github.com/rancher/go-rancher/api.ApiHandler.func1()
/go/src/github.com/openebs/jiva/vendor/github.com/rancher/go-rancher/api/handler.go:26 +0x177
net/http.HandlerFunc.ServeHTTP()
/usr/local/go/src/net/http/server.go:1918 +0x51
github.com/openebs/jiva/vendor/github.com/gorilla/context.ClearHandler.func1()
/go/src/github.com/openebs/jiva/vendor/github.com/gorilla/context/context.go:141 +0xa2
net/http.HandlerFunc.ServeHTTP()
/usr/local/go/src/net/http/server.go:1918 +0x51
github.com/openebs/jiva/vendor/github.com/gorilla/mux.(*Router).ServeHTTP()
/go/src/github.com/openebs/jiva/vendor/github.com/gorilla/mux/mux.go:103 +0x363
github.com/openebs/jiva/vendor/github.com/gorilla/handlers.loggingHandler.ServeHTTP()
/go/src/github.com/openebs/jiva/vendor/github.com/gorilla/handlers/handlers.go:69 +0x174
github.com/openebs/jiva/vendor/github.com/gorilla/handlers.(*loggingHandler).ServeHTTP()
<autogenerated>:1 +0xa1
github.com/openebs/jiva/util.filteredLoggingHandler.ServeHTTP()
/go/src/github.com/openebs/jiva/util/util.go:88 +0x91
github.com/openebs/jiva/util.(*filteredLoggingHandler).ServeHTTP()
<autogenerated>:1 +0xa2
github.com/openebs/jiva/vendor/github.com/gorilla/handlers.ProxyHeaders.func1()
/go/src/github.com/openebs/jiva/vendor/github.com/gorilla/handlers/proxy_headers.go:54 +0xc3
net/http.HandlerFunc.ServeHTTP()
/usr/local/go/src/net/http/server.go:1918 +0x51
net/http.serverHandler.ServeHTTP()
/usr/local/go/src/net/http/server.go:2619 +0xbc
net/http.(*conn).serve()
/usr/local/go/src/net/http/server.go:1801 +0x83b
Goroutine 39 (running) created at:
net/http.(*Server).Serve()
/usr/local/go/src/net/http/server.go:2720 +0x37c
net/http.(*Server).ListenAndServe()
/usr/local/go/src/net/http/server.go:2636 +0xc7
net/http.ListenAndServe()
/usr/local/go/src/net/http/server.go:2882 +0xfe
github.com/openebs/jiva/app.startController()
/go/src/github.com/openebs/jiva/app/controller.go:125 +0xc10
github.com/openebs/jiva/app.ControllerCmd.func1()
/go/src/github.com/openebs/jiva/app/controller.go:55 +0x3c
github.com/openebs/jiva/vendor/github.com/codegangsta/cli.Command.Run()
/go/src/github.com/openebs/jiva/vendor/github.com/codegangsta/cli/command.go:179 +0xcbb
github.com/openebs/jiva/vendor/github.com/codegangsta/cli.(*App).Run()
/go/src/github.com/openebs/jiva/vendor/github.com/codegangsta/cli/app.go:196 +0x1183
main.longhornCli()
/go/src/github.com/openebs/jiva/main.go:106 +0x822
main.main()
/go/src/github.com/openebs/jiva/main.go:27 +0xe4
Goroutine 37 (running) created at:
net/http.(*Server).Serve()
/usr/local/go/src/net/http/server.go:2720 +0x37c
net/http.(*Server).ListenAndServe()
/usr/local/go/src/net/http/server.go:2636 +0xc7
net/http.ListenAndServe()
/usr/local/go/src/net/http/server.go:2882 +0xfe
github.com/openebs/jiva/app.startController()
/go/src/github.com/openebs/jiva/app/controller.go:125 +0xc10
github.com/openebs/jiva/app.ControllerCmd.func1()
/go/src/github.com/openebs/jiva/app/controller.go:55 +0x3c
10.1.3.4 - - [28/Jun/2018:08:22:52 +0000] "POST /v1/register HTTP/1.1" 200 0
STATE = closed
10.1.4.4 - - [28/Jun/2018:08:22:52 +0000] "POST /v1/register HTTP/1.1" 200 0
github.com/openebs/jiva/vendor/github.com/codegangsta/cli.Command.Run()
/go/src/github.com/openebs/jiva/vendor/github.com/codegangsta/cli/command.go:179 +0xcbb
github.com/openebs/jiva/vendor/github.com/codegangsta/cli.(*App).Run()
/go/src/github.com/openebs/jiva/vendor/github.com/codegangsta/cli/app.go:196 +0x1183
main.longhornCli()
/go/src/github.com/openebs/jiva/main.go:106 +0x822
main.main()
/go/src/github.com/openebs/jiva/main.go:27 +0xe4
https://github.com/openebs/jiva/blob/06ae965ec74df6b92101024e32d36732c29bc22b/main.go#L64
I want to use the pprof and go-torch to generate the flamegraph of jiva. Where should I set the env PPROFILE to enable this function ?
We're running Kubernetes 1.9.7 on ubuntu 16.04 with ZFS.
If the storagepool is a zfs volume, e.g. root/ebs with the following definition in k8s
apiVersion: openebs.io/v1alpha1
kind: StoragePool
metadata:
name: ebs
type: hostdir
spec:
path: "/ebs"
the replica containers running
launch replica --frontendIP x.x.x.x --size 1G /openebs
fail with
time="2018-05-25T09:35:16Z" level=info msg="Creating volume /openebs, size 1073741824/512"
time="2018-05-25T09:35:16Z" level=fatal msg="Error running start replica command: open /openebs/revision.counter: invalid argument"
If we create an ext4 zvol based storagepool it works. Is this expected behaviour?
I've selected Controller.Start(addresses...string) for refactoring, which is a unit of 74 lines of code and 17 branch points. Addressing this will make our codebase more maintainable and improve Better Code Hub's Write Simple Units of Code guideline rating! ๐
Here's the gist of this guideline:
You can find more info about this guideline in Building Maintainable Software. ๐
โน๏ธ To know how many other refactoring candidates need addressing to get a guideline compliant, select some by clicking on the ๐ฒ next to them. The risk profile below the candidates signals (โ ) when it's enough! ๐
Good luck and happy coding! โจ ๐ฏ
It is possible that that container that is launching the controller/replica may be up and running, but either the communication to the replica's or from replica to underlying storage could be having issues. The client will detect such issues as IO errors. However, there needs to be a mechanism where maya should be able to check the health of VSM (controller or replica) and take necessary actions like restart, etc.,
This can be as simple as an script or docker exec command.
Describe the problem/challenge you have
Jiva uses libiscsi test suite as part of the CI. The tests are from a container called openebs/tests-libiscsi
.
The source code for the container moved to a different project as part of the litmuschoas project spin-off and then it was deleted from the main repo. the code is available here:
https://github.com/litmuschaos/test-tools/tree/d72fe9078fa344e84113f455a6832558b9fcd53d/libiscsi
Describe the solution you'd like
openebs/tests-libiscsi
Handle cases like:
What steps did you take and what happened:
[A clear and concise description of what the bug is, and what commands you ran.)
Use helm install openebs-3.3.1
helm install openebs openebs-3.3.1.tgz --set jiva.enabled=true -n openebs
install success
kubectl get pod -n openebs
NAME READY STATUS RESTARTS AGE
openebs-jiva-csi-controller-0 5/5 Running 0 116s
openebs-jiva-csi-node-5zv5g 3/3 Running 0 116s
openebs-jiva-csi-node-ccps8 3/3 Running 0 116s
openebs-jiva-csi-node-fwtzf 3/3 Running 0 116s
openebs-jiva-operator-f994f6868-hftwx 1/1 Running 0 116s
openebs-localpv-provisioner-77c9bcfd96-n2nf8 1/1 Running 0 116s
openebs-ndm-8dbzp 1/1 Running 0 116s
openebs-ndm-operator-5d55748dfd-cmv6n 1/1 Running 0 116s
openebs-ndm-rvcr8 1/1 Running 0 116s
openebs-ndm-ztwxn 1/1 Running 0 116s
Then I use this to test
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: example-jiva-csi-pvc
spec:
storageClassName: openebs-jiva-csi-default
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 4Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: busybox
labels:
app: busybox
spec:
replicas: 1
strategy:
type: RollingUpdate
selector:
matchLabels:
app: busybox
template:
metadata:
labels:
app: busybox
spec:
containers:
- resources:
limits:
cpu: 0.5
name: busybox
image: busybox
command: ['sh', '-c', 'echo Container 1 is Running ; sleep 3600']
imagePullPolicy: IfNotPresent
ports:
- containerPort: 3306
name: busybox
volumeMounts:
- mountPath: /var/lib/mysql
name: demo-vol1
volumes:
- name: demo-vol1
persistentVolumeClaim:
claimName: example-jiva-csi-pvc
pod about pvc are running
pvc-7c04c4c0-8f23-44b2-b7d9-e5cb6a371922-jiva-ctrl-6c66f965hq5x 2/2 Running 0 17m
pvc-7c04c4c0-8f23-44b2-b7d9-e5cb6a371922-jiva-rep-0 1/1 Running 0 17m
pvc-7c04c4c0-8f23-44b2-b7d9-e5cb6a371922-jiva-rep-1 1/1 Running 1 17m
pvc-7c04c4c0-8f23-44b2-b7d9-e5cb6a371922-jiva-rep-2 1/1 Running 0 17m
but pod cant't attach the volume
The jivavolume is pending not Ready
kubectl get jivavolume -n openebs
NAME REPLICACOUNT PHASE STATUS
pvc-7c04c4c0-8f23-44b2-b7d9-e5cb6a371922 Pending
And I can't find any error log in jiva-ctl and jiva-rep
What did you expect to happen:
The output of the following commands will help us better understand what's going on:
(Pasting long output into a GitHub gist or other pastebin is fine.)
this contain jiva-ctrl and jiva-rep log
kubectl logs <jiva controller pod name> -n openebs
(Optional)kubectl logs <jiva replica pod name> -n openebs
(Optional)kubectl exec -it <jiva controller pod name> -n openebs -- jivactl ls
Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]
Environment:
kubectl version
):/etc/os-release
): CentOS Linux 7 (Core)I wanted to flatten jiva pvc as much as possible, as it consisted mostly out of empty snapshots. I entered pvc controller pod and executed jivactl snapshot rm $(jivactl snapshot ls | tail -1)
a bunch of times in a loop, one after another. Initially everything looked good, but after some time mysql pod crashed (gralog logs attached).
Once I detected it I deleted the pod and rescaled it again, and automatic fsck fixed fs errors, but that ended with half of data missing. Recovery from sql backup fixed the issue.
mywql-graylog.csv.txt
mywql-graylog-pvc-ctrl.csv.txt
mywql-graylog-pvc-replica-kube2.csv.txt
mywql-graylog-pvc-replica-kube3.csv.txt
"JIVA OVERVIEW DIAGRAM" link https://github.com/openebs/openebs/blob/master/docs/images/JivaExample.png?raw=true is broken
What steps did you take and what happened:
It appears that when an issue occurs, JIVA remounts the volume, but does not remove a stale mount point, resulting in high IO latency.
What did you expect to happen:
Jiva to cleanup stale mount points periodically.
Anything else you would like to add:
Here are a list of mount points which are not cleaned up causing high load averages; and terrible performance for this storage solution.
Environment: K3s
kubectl version
): v1.23.4+k3s1/etc/os-release
): UbuntuAdd endpoint /metrics
in controller and replicas. This is required to monitor the controller and replicas by prometheus.
Add custom metrics to monitor all the endpoint response duration and request count.
/cc @kmova Any point investing efforts here?
โ go get github.com/openebs/jiva
# github.com/openebs/jiva/vendor/github.com/frostschutz/go-fibmap
../gocode/src/github.com/openebs/jiva/vendor/github.com/frostschutz/go-fibmap/fibmap.go:216:9: undefined: syscall.Fallocate
../gocode/src/github.com/openebs/jiva/vendor/github.com/frostschutz/go-fibmap/fibmap.go:221:9: undefined: syscall.Fallocate
# github.com/openebs/jiva/vendor/github.com/rancher/sparse-tools/sparse
../gocode/src/github.com/openebs/jiva/vendor/github.com/rancher/sparse-tools/sparse/fiemap.go:110:9: undefined: syscall.Fallocate
../gocode/src/github.com/openebs/jiva/vendor/github.com/rancher/sparse-tools/sparse/fiemap.go:115:9: undefined: syscall.Fallocate
../gocode/src/github.com/openebs/jiva/vendor/github.com/rancher/sparse-tools/sparse/file.go:71:33: undefined: syscall.O_DIRECT
../gocode/src/github.com/openebs/jiva/vendor/github.com/rancher/sparse-tools/sparse/file.go:75:45: undefined: syscall.O_DIRECT
# github.com/openebs/jiva/vendor/github.com/yasker/nsfilelock
../gocode/src/github.com/openebs/jiva/vendor/github.com/yasker/nsfilelock/nsfilelock.go:72:50: unknown field 'Pdeathsig' in struct literal of type syscall.SysProcAttr
# github.com/openebs/jiva/frontend/gotgt
../gocode/src/github.com/openebs/jiva/frontend/gotgt/frontend.go:37:15: undefined: port.Stats
# github.com/openebs/jiva/sync/agent
../gocode/src/github.com/openebs/jiva/sync/agent/process.go:141:12: unknown field 'Pdeathsig' in struct literal of type syscall.SysProcAttr
../gocode/src/github.com/openebs/jiva/sync/agent/process.go:194:12: unknown field 'Pdeathsig' in struct literal of type syscall.SysProcAttr
../gocode/src/github.com/openebs/jiva/sync/agent/process.go:246:12: unknown field 'Pdeathsig' in struct literal of type syscall.SysProcAttr
../gocode/src/github.com/openebs/jiva/sync/agent/process.go:281:12: unknown field 'Pdeathsig' in struct literal of type syscall.SysProcAttr
../gocode/src/github.com/openebs/jiva/sync/agent/process.go:316:12: unknown field 'Pdeathsig' in struct literal of type syscall.SysProcAttr
../gocode/src/github.com/openebs/jiva/sync/agent/process.go:351:12: unknown field 'Pdeathsig' in struct literal of type syscall.SysProcAttr
../gocode/src/github.com/openebs/jiva/sync/agent/process.go:390:12: unknown field 'Pdeathsig' in struct literal of type syscall.SysProcAttr
What steps did you take and what happened:
Running the node failure tests (openebs-archive/e2e-tests#564), where the kernel is made to crash leads to a metadata file issue as reported in one of the replica logs:
failed to read disk data, error while unmarshalling file: volume-snap-1db04a94-0aa0-4404-bbbb-3641392d587f.img.meta
What did you expect to happen:
The replicas should have come up and connected to the controller without any interruption.
Environment:
This issue tracks the creation of a placeholder for Jiva specific test suite which will run post image build.
Currently, the jiva ci includes a vdbench test run on the iscsi target. This has to be extended & standardised to include further functional cases that can be run w/o kubernetes components (i.e., non-system level), which include, but are not limited to :
Note that this should be separate in principle and wrt environment from the OpenEBS e2e which will consume Jiva and test it at a system-level (i.e., application use-cases, non-functional requirements like performance etc.,). This has been referenced in the issue #6
Travis CI tests for openebs/jiva currently include only:
The CI tests should include the tests for snapshot related functionality added as part of the OpenEBS v0.4.0 release.
What steps did you take and what happened:
I am using Dynamic NFS Provisioner on top of triple replicated Jiva volumes to make them ReadWriteMany.
After hard reboots, this seems to always happen:
It seems to me the data stored to the Jiva volumes do not go to the replicas, but instead go to the mounting pod local ephemeral storage.
I would expect the data to go to the replicas which are correctly set up, but instead Jiva stores the files to:
/var/snap/microk8s/common/var/lib/kubelet/pods/4ae2b03c-e03e-4088-92cb-dcf37ae2e118/volumes/kubernetes.io~csi/pvc-4f872aeb-385e-44e0-b0f1-9643a04ea4af/mount
Which doesn't seem to be mounted anywhere.
What did you expect to happen:
I would expect either an error in OpenEBS Jiva pvc pod start-up, or the Jiva replicas to be mounted to the mount directory using CSI/iSCSI so the data ends up on the replicas.
The output of the following commands will help us better understand what's going on:
# microk8s kubectl logs -n openebs pvc-4f872aeb-385e-44e0-b0f1-9643a04ea4af-jiva-ctrl-d74d78csgl2r
Defaulted container "jiva-controller" out of: jiva-controller, maya-volume-exportertime="2023-01-21T14:32:40Z" level=info msg="RPC_READ_TIMEOUT env not set"
time="2023-01-21T14:32:40Z" level=info msg="RPC_WRITE_TIMEOUT env not set"
time="2023-01-21T14:32:40Z" level=info msg="REPLICATION_FACTOR: 3, RPC_READ_TIMEOUT: 0s, RPC_WRITE_TIMEOUT: 0s"
time="2023-01-21T14:32:40Z" level=info msg="Starting controller with frontendIP: , and clusterIP: 10.152.183.28"
time="2023-01-21T14:32:40Z" level=info msg="resetting controller"
time="2023-01-21T14:32:40Z" level=info msg="Listening on :9501"
time="2023-01-21T14:34:24Z" level=info msg="Register Replica, Address: 10.1.109.199 UUID: 6f5a5f244254f12b437bb9cc8cd401407ee4ad92 Uptime: 25.199859908s State: dirty Type: Backend RevisionCount: 942"
time="2023-01-21T14:34:24Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
10.1.109.199 - - [21/Jan/2023:14:34:24 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-01-21T14:34:29Z" level=info msg="Register Replica, Address: 10.1.109.199 UUID: 6f5a5f244254f12b437bb9cc8cd401407ee4ad92 Uptime: 30.211059851s State: dirty Type: Backend RevisionCount: 942"
time="2023-01-21T14:34:29Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
10.1.109.199 - - [21/Jan/2023:14:34:29 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-01-21T14:34:34Z" level=info msg="Register Replica, Address: 10.1.109.199 UUID: 6f5a5f244254f12b437bb9cc8cd401407ee4ad92 Uptime: 35.131878985s State: dirty Type: Backend RevisionCount: 942"
time="2023-01-21T14:34:34Z" level=warning msg="No of yet to be registered replicas are less than 3 , No of registered replicas: 1"
10.1.109.199 - - [21/Jan/2023:14:34:34 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-01-21T14:34:36Z" level=info msg="Register Replica, Address: 10.1.140.163 UUID: 2bf646cc281748b5ed9ac9ff00b429a6fb43ba1d Uptime: 15.814225ms State: dirty Type: Backend RevisionCount: 942"
time="2023-01-21T14:34:36Z" level=info msg="Replica 10.1.109.199 signalled to start, registered replicas: map[string]types.RegReplica{\"10.1.109.199\":types.RegReplica{Address:\"10.1.109.199\", UUID:\"6f5a5f244254f12b437bb9cc8cd401407ee4ad92\", UpTime:35131878985, RevCount:942, RepType:\"Backend\", RepState:\"dirty\"}, \"10.1.140.163\":types.RegReplica{Address:\"10.1.140.163\", UUID:\"2bf646cc281748b5ed9ac9ff00b429a6fb43ba1d\", UpTime:15814225, RevCount:942, RepType:\"Backend\", RepState:\"dirty\"}}"
10.1.140.163 - - [21/Jan/2023:14:34:36 +0000] "POST /v1/register HTTP/1.1" 200 0
time="2023-01-21T14:34:36Z" level=info msg="resetting controller"
time="2023-01-21T14:34:36Z" level=info msg="Connecting to remote: 10.1.109.199:9502"
time="2023-01-21T14:34:36Z" level=info msg="Opening: 10.1.109.199:9502"
time="2023-01-21T14:34:36Z" level=info msg="check if replica tcp://10.1.109.199:9502 is already added"
time="2023-01-21T14:34:36Z" level=info msg="check if any WO replica available"
time="2023-01-21T14:34:36Z" level=info msg="Set replica mode of 10.1.109.199:9502 to : WO"
time="2023-01-21T14:34:36Z" level=info msg="Adding backend: tcp://10.1.109.199:9502"
time="2023-01-21T14:34:36Z" level=info msg="replicator reset false"
time="2023-01-21T14:34:36Z" level=info msg="buildreadwriters: prev: 0 0 cur: 1 0"
time="2023-01-21T14:34:36Z" level=info msg="Start monitoring tcp://10.1.109.199:9502"
time="2023-01-21T14:34:36Z" level=info msg="Get backend tcp://10.1.109.199:9502 clone status"
time="2023-01-21T14:34:36Z" level=info msg="Set replica mode of 10.1.109.199:9502 to : RW"
time="2023-01-21T14:34:36Z" level=info msg="Set backend tcp://10.1.109.199:9502 replica mode to RW"
time="2023-01-21T14:34:36Z" level=info msg="Set replica tcp://10.1.109.199:9502 to mode RW"
time="2023-01-21T14:34:36Z" level=info msg="addr tcp://10.1.109.199:9502 m: RW prev: WO in setmode"
time="2023-01-21T14:34:36Z" level=info msg="replicator reset false"
time="2023-01-21T14:34:36Z" level=info msg="buildreadwriters: prev: 0 0 cur: 1 1"
time="2023-01-21T14:34:36Z" level=info msg="Get backend tcp://10.1.109.199:9502 revision counter 942"
time="2023-01-21T14:34:36Z" level=info msg="sending add signal to 10.1.140.163"
time="2023-01-21T14:34:36Z" level=info msg="Create Replica for address tcp://10.1.140.163:9502"
time="2023-01-21T14:34:36Z" level=info msg="Update volume status"
time="2023-01-21T14:34:36Z" level=info msg="Previously Volume RO: true, Currently: true, Total Replicas: 1, RW replicas: 1, Total backends: 1"
time="2023-01-21T14:34:36Z" level=info msg="prevCheckpoint: , currCheckpoint: "
time="2023-01-21T14:34:36Z" level=info msg="Start SCSI target"
time="2023-01-21T14:34:36Z" level=info msg="SCSI device created"
time="2023-01-21T14:34:36Z" level=info msg="iSCSI service listening on: [::]:3260"
10.1.109.199 - - [21/Jan/2023:14:34:36 +0000] "POST /v1/volumes/cHZjLTRmODcyYWViLTM4NWUtNDRlMC1iMGYxLTk2NDNhMDRlYTRhZg==?action=start HTTP/1.1" 200 1038
time="2023-01-21T14:34:36Z" level=info msg="check if replica tcp://10.1.140.163:9502 is already added"
time="2023-01-21T14:34:36Z" level=info msg="check if any WO replica available"
time="2023-01-21T14:34:36Z" level=info msg="verify replication factor"
time="2023-01-21T14:34:36Z" level=info msg="Connecting to remote: 10.1.140.163:9502"
time="2023-01-21T14:34:36Z" level=info msg="Opening: 10.1.140.163:9502"
time="2023-01-21T14:34:36Z" level=info msg="check if replica tcp://10.1.140.163:9502 is already added"
time="2023-01-21T14:34:36Z" level=info msg="check if any WO replica available"
time="2023-01-21T14:34:36Z" level=info msg="Snapshot: 10.1.109.199:9502 23800da8-bbee-4636-8852-ead696f7d27e UserCreated false Created at 2023-01-21T14:34:36Z"
time="2023-01-21T14:34:36Z" level=info msg="successfully taken snapshots cnt 1"
time="2023-01-21T14:34:36Z" level=info msg="Snapshot: 10.1.140.163:9502 23800da8-bbee-4636-8852-ead696f7d27e UserCreated false Created at 2023-01-21T14:34:36Z"
time="2023-01-21T14:34:36Z" level=info msg="Set replica mode of 10.1.140.163:9502 to : WO"
10.1.140.163 - - [21/Jan/2023:14:34:36 +0000] "POST /v1/replicas HTTP/1.1" 200 431
time="2023-01-21T14:34:36Z" level=info msg="Adding backend: tcp://10.1.140.163:9502"
time="2023-01-21T14:34:36Z" level=info msg="replicator reset false"
time="2023-01-21T14:34:36Z" level=info msg="buildreadwriters: prev: 0 1 cur: 2 1"
time="2023-01-21T14:34:36Z" level=info msg="Previously Volume RO: true, Currently: true, Total Replicas: 2, RW replicas: 1, Total backends: 2"
time="2023-01-21T14:34:36Z" level=info msg="prevCheckpoint: , currCheckpoint: "
time="2023-01-21T14:34:36Z" level=info msg="Start monitoring tcp://10.1.140.163:9502"
10.1.140.163 - - [21/Jan/2023:14:34:36 +0000] "GET /v1/replicas/dGNwOi8vMTAuMS4xNDAuMTYzOjk1MDI= HTTP/1.1" 200 431
time="2023-01-21T14:34:36Z" level=info msg="Prepare Rebuild Replica for id tcp://10.1.140.163:9502"
time="2023-01-21T14:34:36Z" level=info msg="Synchronizing volume-head-020.img.meta@tcp://10.1.109.199:9502 to [email protected]:9700"
time="2023-01-21T14:34:37Z" level=info msg="Done synchronizing volume-head-020.img.meta to [email protected]:9700"
10.1.140.163 - - [21/Jan/2023:14:34:36 +0000] "POST /v1/replicas/dGNwOi8vMTAuMS4xNDAuMTYzOjk1MDI=?action=preparerebuild HTTP/1.1" 200 895
10.1.140.163 - - [21/Jan/2023:14:34:37 +0000] "GET /v1/replicas/dGNwOi8vMTAuMS4xNDAuMTYzOjk1MDI= HTTP/1.1" 200 431
time="2023-01-21T14:34:37Z" level=info msg="Verify Rebuild Replica for id tcp://10.1.140.163:9502"
time="2023-01-21T14:34:37Z" level=info msg="chain [volume-head-020.img volume-snap-23800da8-bbee-4636-8852-ead696f7d27e.img volume-snap-77bfa5d0-3a3a-4125-874d-64d981bf7cf7.img volume-snap-b2b1f0c5-b744-47ef-baae-b7697754898b.img volume-snap-668659c8-e7b8-4184-a216-46ef61c72420.img volume-snap-bc4a2971-3060-4d0d-9395-55a8d39167a6.img volume-snap-a9cd1a28-30e6-42e7-9175-dfc987262dc4.img volume-snap-0120e6c1-7962-4378-b090-006cf4305f50.img volume-snap-ccbd2dba-48f0-4964-abd2-bf29e30488f4.img volume-snap-b9397eee-69f0-41ca-808e-6fadba8c32c8.img volume-snap-088add84-75e5-4293-b06c-709a66220f9d.img volume-snap-8eede2b2-cb06-46d4-b862-882d3c4381bd.img volume-snap-93362d3c-51d0-4196-938a-a0455ffe463d.img volume-snap-f2d19ed0-7a0d-427f-bcba-315be4efb0f1.img] from rw replica tcp://10.1.109.199:9502, indx: 3"
time="2023-01-21T14:34:37Z" level=info msg="chain [volume-head-020.img volume-snap-23800da8-bbee-4636-8852-ead696f7d27e.img volume-snap-77bfa5d0-3a3a-4125-874d-64d981bf7cf7.img volume-snap-b2b1f0c5-b744-47ef-baae-b7697754898b.img volume-snap-668659c8-e7b8-4184-a216-46ef61c72420.img volume-snap-bc4a2971-3060-4d0d-9395-55a8d39167a6.img volume-snap-a9cd1a28-30e6-42e7-9175-dfc987262dc4.img volume-snap-0120e6c1-7962-4378-b090-006cf4305f50.img volume-snap-ccbd2dba-48f0-4964-abd2-bf29e30488f4.img volume-snap-b9397eee-69f0-41ca-808e-6fadba8c32c8.img volume-snap-088add84-75e5-4293-b06c-709a66220f9d.img volume-snap-8eede2b2-cb06-46d4-b862-882d3c4381bd.img volume-snap-93362d3c-51d0-4196-938a-a0455ffe463d.img volume-snap-f2d19ed0-7a0d-427f-bcba-315be4efb0f1.img] from wo replica tcp://10.1.140.163:9502, indx: 3"
time="2023-01-21T14:34:37Z" level=info msg="Get backend tcp://10.1.109.199:9502 revision counter 942"
time="2023-01-21T14:34:37Z" level=info msg="rw replica tcp://10.1.109.199:9502 revision counter 942"
time="2023-01-21T14:34:37Z" level=info msg="Set replica mode of 10.1.140.163:9502 to : RW"
time="2023-01-21T14:34:37Z" level=info msg="Set backend tcp://10.1.140.163:9502 replica mode to RW"
time="2023-01-21T14:34:37Z" level=info msg="Set revision counter of 10.1.140.163:9502 to : 942"
time="2023-01-21T14:34:37Z" level=info msg="Set backend tcp://10.1.140.163:9502 revision counter to 942"
time="2023-01-21T14:34:37Z" level=info msg="WO replica tcp://10.1.140.163:9502's chain verified, update replica mode to RW"
time="2023-01-21T14:34:37Z" level=info msg="Set replica tcp://10.1.140.163:9502 to mode RW"
time="2023-01-21T14:34:37Z" level=info msg="addr tcp://10.1.140.163:9502 m: RW prev: WO in setmode"
time="2023-01-21T14:34:37Z" level=info msg="replicator reset false"
time="2023-01-21T14:34:37Z" level=info msg="buildreadwriters: prev: 0 1 cur: 2 2"
time="2023-01-21T14:34:37Z" level=info msg="Previously Volume RO: true, Currently: false, Total Replicas: 2, RW replicas: 2, Total backends: 2"
time="2023-01-21T14:34:37Z" level=info msg="prevCheckpoint: , currCheckpoint: "
10.1.140.163 - - [21/Jan/2023:14:34:37 +0000] "POST /v1/replicas/dGNwOi8vMTAuMS4xNDAuMTYzOjk1MDI=?action=verifyrebuild HTTP/1.1" 200 431
time="2023-01-21T14:34:39Z" level=info msg="connection establishing at: 10.1.109.245:3260"
time="2023-01-21T14:34:39Z" level=info msg="Target is connected to initiator: 10.1.140.128:43935"
time="2023-01-21T14:34:39Z" level=error msg="read BHS failed:EOF"
time="2023-01-21T14:34:39Z" level=warning msg="iscsi connection[0] closed"
time="2023-01-21T14:34:39Z" level=info msg="connection establishing at: 10.1.109.245:3260"
time="2023-01-21T14:34:39Z" level=info msg="Target is connected to initiator: 10.1.140.128:3293"
time="2023-01-21T14:34:39Z" level=info msg="Discovery request received from initiator: iqn.2004-10.com.ubuntu:01:7e62554de3, Session type: Discovery, ISID: 0x23d000000"
time="2023-01-21T14:34:39Z" level=warning msg="unexpected connection state: full feature"
time="2023-01-21T14:34:39Z" level=error msg="read BHS failed:read tcp 10.1.109.245:3260->10.1.140.128:3293: read: connection reset by peer"
time="2023-01-21T14:34:39Z" level=warning msg="iscsi connection[0] closed"
time="2023-01-21T14:34:39Z" level=info msg="connection establishing at: 10.1.109.245:3260"
time="2023-01-21T14:34:39Z" level=info msg="Target is connected to initiator: 10.1.140.128:3075"
time="2023-01-21T14:34:40Z" level=info msg="Login request received from initiator: iqn.2004-10.com.ubuntu:01:7e62554de3, Session type: Normal, Target name:iqn.2016-09.com.openebs.jiva:pvc-4f872aeb-385e-44e0-b0f1-9643a04ea4af, ISID: 0x23d000001"
time="2023-01-21T14:34:40Z" level=error msg="rsa: 0h, sa:false not supported"
time="2023-01-21T14:34:40Z" level=warning msg="opcode: a3h err: check condition"
time="2023-01-21T14:35:23Z" level=info msg="Create Replica for address tcp://10.1.98.10:9502"
time="2023-01-21T14:35:23Z" level=info msg="check if replica tcp://10.1.98.10:9502 is already added"
time="2023-01-21T14:35:23Z" level=info msg="check if any WO replica available"
time="2023-01-21T14:35:23Z" level=info msg="verify replication factor"
time="2023-01-21T14:35:23Z" level=info msg="Connecting to remote: 10.1.98.10:9502"
time="2023-01-21T14:35:23Z" level=info msg="Opening: 10.1.98.10:9502"
time="2023-01-21T14:35:24Z" level=info msg="check if replica tcp://10.1.98.10:9502 is already added"
time="2023-01-21T14:35:24Z" level=info msg="check if any WO replica available"
time="2023-01-21T14:35:24Z" level=info msg="Snapshot: 10.1.140.163:9502 2c8fc776-68f3-4c3c-ac66-c34397b39c6e UserCreated false Created at 2023-01-21T14:35:24Z"
time="2023-01-21T14:35:24Z" level=info msg="Snapshot: 10.1.109.199:9502 2c8fc776-68f3-4c3c-ac66-c34397b39c6e UserCreated false Created at 2023-01-21T14:35:24Z"
time="2023-01-21T14:35:24Z" level=info msg="successfully taken snapshots cnt 2"
time="2023-01-21T14:35:24Z" level=info msg="Snapshot: 10.1.98.10:9502 2c8fc776-68f3-4c3c-ac66-c34397b39c6e UserCreated false Created at 2023-01-21T14:35:24Z"
time="2023-01-21T14:35:25Z" level=info msg="Set replica mode of 10.1.98.10:9502 to : WO"
time="2023-01-21T14:35:26Z" level=info msg="Adding backend: tcp://10.1.98.10:9502"
time="2023-01-21T14:35:26Z" level=info msg="replicator reset false"
time="2023-01-21T14:35:26Z" level=info msg="buildreadwriters: prev: 0 2 cur: 3 2"
time="2023-01-21T14:35:26Z" level=info msg="Previously Volume RO: false, Currently: false, Total Replicas: 3, RW replicas: 2, Total backends: 3"
time="2023-01-21T14:35:26Z" level=info msg="prevCheckpoint: , currCheckpoint: "
time="2023-01-21T14:35:26Z" level=info msg="Start monitoring tcp://10.1.98.10:9502"10.1.98.10 - - [21/Jan/2023:14:35:23 +0000] "POST /v1/replicas HTTP/1.1" 200 413
10.1.98.10 - - [21/Jan/2023:14:35:27 +0000] "GET /v1/replicas/dGNwOi8vMTAuMS45OC4xMDo5NTAy HTTP/1.1" 200 413
time="2023-01-21T14:35:27Z" level=info msg="Prepare Rebuild Replica for id tcp://10.1.98.10:9502"
time="2023-01-21T14:35:27Z" level=info msg="Synchronizing volume-head-021.img.meta@tcp://10.1.109.199:9502 to [email protected]:9700"
10.1.98.10 - - [21/Jan/2023:14:35:27 +0000] "POST /v1/replicas/dGNwOi8vMTAuMS45OC4xMDo5NTAy?action=preparerebuild HTTP/1.1" 200 946
time="2023-01-21T14:35:29Z" level=info msg="Done synchronizing volume-head-021.img.meta to [email protected]:9700"
10.1.109.199 - - [21/Jan/2023:14:35:36 +0000] "GET /v1/checkpoint HTTP/1.1" 200 10810.1.140.163 - - [21/Jan/2023:14:35:37 +0000] "GET /v1/checkpoint HTTP/1.1" 200 10810.1.98.10 - - [21/Jan/2023:14:36:10 +0000] "GET /v1/replicas/dGNwOi8vMTAuMS45OC4xMDo5NTAy HTTP/1.1" 200 413
time="2023-01-21T14:36:10Z" level=info msg="Verify Rebuild Replica for id tcp://10.1.98.10:9502"
time="2023-01-21T14:36:10Z" level=info msg="chain [volume-head-021.img volume-snap-2c8fc776-68f3-4c3c-ac66-c34397b39c6e.img volume-snap-23800da8-bbee-4636-8852-ead696f7d27e.img volume-snap-77bfa5d0-3a3a-4125-874d-64d981bf7cf7.img volume-snap-b2b1f0c5-b744-47ef-baae-b7697754898b.img volume-snap-668659c8-e7b8-4184-a216-46ef61c72420.img volume-snap-bc4a2971-3060-4d0d-9395-55a8d39167a6.img volume-snap-a9cd1a28-30e6-42e7-9175-dfc987262dc4.img volume-snap-0120e6c1-7962-4378-b090-006cf4305f50.img volume-snap-ccbd2dba-48f0-4964-abd2-bf29e30488f4.img volume-snap-b9397eee-69f0-41ca-808e-6fadba8c32c8.img volume-snap-088add84-75e5-4293-b06c-709a66220f9d.img volume-snap-8eede2b2-cb06-46d4-b862-882d3c4381bd.img volume-snap-93362d3c-51d0-4196-938a-a0455ffe463d.img volume-snap-f2d19ed0-7a0d-427f-bcba-315be4efb0f1.img] from rw replica tcp://10.1.109.199:9502, indx: 4"
time="2023-01-21T14:36:10Z" level=info msg="chain [volume-head-019.img volume-snap-2c8fc776-68f3-4c3c-ac66-c34397b39c6e.img volume-snap-23800da8-bbee-4636-8852-ead696f7d27e.img volume-snap-77bfa5d0-3a3a-4125-874d-64d981bf7cf7.img volume-snap-b2b1f0c5-b744-47ef-baae-b7697754898b.img volume-snap-668659c8-e7b8-4184-a216-46ef61c72420.img volume-snap-bc4a2971-3060-4d0d-9395-55a8d39167a6.img volume-snap-a9cd1a28-30e6-42e7-9175-dfc987262dc4.img volume-snap-0120e6c1-7962-4378-b090-006cf4305f50.img volume-snap-ccbd2dba-48f0-4964-abd2-bf29e30488f4.img volume-snap-b9397eee-69f0-41ca-808e-6fadba8c32c8.img volume-snap-088add84-75e5-4293-b06c-709a66220f9d.img volume-snap-8eede2b2-cb06-46d4-b862-882d3c4381bd.img volume-snap-93362d3c-51d0-4196-938a-a0455ffe463d.img volume-snap-f2d19ed0-7a0d-427f-bcba-315be4efb0f1.img] from wo replica tcp://10.1.98.10:9502, indx: 4"
time="2023-01-21T14:36:10Z" level=info msg="Get backend tcp://10.1.109.199:9502 revision counter 945"
time="2023-01-21T14:36:10Z" level=info msg="rw replica tcp://10.1.109.199:9502 revision counter 945"
time="2023-01-21T14:36:10Z" level=info msg="Set replica mode of 10.1.98.10:9502 to : RW"
time="2023-01-21T14:36:10Z" level=info msg="Set backend tcp://10.1.98.10:9502 replica mode to RW"
time="2023-01-21T14:36:10Z" level=info msg="Set revision counter of 10.1.98.10:9502 to : 945"
time="2023-01-21T14:36:10Z" level=info msg="Set backend tcp://10.1.98.10:9502 revision counter to 945"
time="2023-01-21T14:36:10Z" level=info msg="WO replica tcp://10.1.98.10:9502's chain verified, update replica mode to RW"
time="2023-01-21T14:36:10Z" level=info msg="Set replica tcp://10.1.98.10:9502 to mode RW"
time="2023-01-21T14:36:10Z" level=info msg="addr tcp://10.1.98.10:9502 m: RW prev: WO in setmode"
time="2023-01-21T14:36:10Z" level=info msg="replicator reset false"
time="2023-01-21T14:36:10Z" level=info msg="buildreadwriters: prev: 0 2 cur: 3 3"
time="2023-01-21T14:36:10Z" level=info msg="Previously Volume RO: false, Currently: false, Total Replicas: 3, RW replicas: 3, Total backends: 3"
10.1.98.10 - - [21/Jan/2023:14:36:10 +0000] "POST /v1/replicas/dGNwOi8vMTAuMS45OC4xMDo5NTAy?action=verifyrebuild HTTP/1.1" 200 413
time="2023-01-21T14:36:10Z" level=info msg="successfully set checkpoint cnt 3"
time="2023-01-21T14:36:10Z" level=info msg="prevCheckpoint: , currCheckpoint: volume-snap-2c8fc776-68f3-4c3c-ac66-c34397b39c6e.img"
10.1.109.199 - - [21/Jan/2023:14:36:36 +0000] "GET /v1/checkpoint HTTP/1.1" 200 16010.1.140.163 - - [21/Jan/2023:14:36:37 +0000] "GET /v1/checkpoint HTTP/1.1" 200 16010.1.98.10 - - [21/Jan/2023:14:37:11 +0000] "GET /v1/checkpoint HTTP/1.1" 200 160
# microk8s kubectl exec -it -n openebs pvc-4f872aeb-385e-44e0-b0f1-9643a04ea4af-jiva-ctrl-d74d78csgl2r -- jivactl ls
Defaulted container "jiva-controller" out of: jiva-controller, maya-volume-exporterADDRESS MODE CHAIN
tcp://10.1.109.199:9502 RW [volume-head-021.img volume-snap-2c8fc776-68f3-4c3c-ac66-c34397b39c6e.img volume-snap-23800da8-bbee-4636-8852-ead696f7d27e.img volume-snap-77bfa5d0-3a3a-4125-874d-64d981bf7cf7.img volume-snap-b2b1f0c5-b744-47ef-baae-b7697754898b.img volume-snap-668659c8-e7b8-4184-a216-46ef61c72420.img volume-snap-bc4a2971-3060-4d0d-9395-55a8d39167a6.img volume-snap-a9cd1a28-30e6-42e7-9175-dfc987262dc4.img volume-snap-0120e6c1-7962-4378-b090-006cf4305f50.img volume-snap-b9397eee-69f0-41ca-808e-6fadba8c32c8.img volume-snap-93362d3c-51d0-4196-938a-a0455ffe463d.img volume-snap-f2d19ed0-7a0d-427f-bcba-315be4efb0f1.img]
tcp://10.1.140.163:9502 RW [volume-head-021.img volume-snap-2c8fc776-68f3-4c3c-ac66-c34397b39c6e.img volume-snap-23800da8-bbee-4636-8852-ead696f7d27e.img volume-snap-77bfa5d0-3a3a-4125-874d-64d981bf7cf7.img volume-snap-b2b1f0c5-b744-47ef-baae-b7697754898b.img volume-snap-668659c8-e7b8-4184-a216-46ef61c72420.img volume-snap-bc4a2971-3060-4d0d-9395-55a8d39167a6.img volume-snap-a9cd1a28-30e6-42e7-9175-dfc987262dc4.img volume-snap-0120e6c1-7962-4378-b090-006cf4305f50.img volume-snap-b9397eee-69f0-41ca-808e-6fadba8c32c8.img volume-snap-93362d3c-51d0-4196-938a-a0455ffe463d.img volume-snap-f2d19ed0-7a0d-427f-bcba-315be4efb0f1.img]
tcp://10.1.98.10:9502 RW [volume-head-019.img volume-snap-2c8fc776-68f3-4c3c-ac66-c34397b39c6e.img volume-snap-23800da8-bbee-4636-8852-ead696f7d27e.img volume-snap-77bfa5d0-3a3a-4125-874d-64d981bf7cf7.img volume-snap-b2b1f0c5-b744-47ef-baae-b7697754898b.img volume-snap-668659c8-e7b8-4184-a216-46ef61c72420.img volume-snap-bc4a2971-3060-4d0d-9395-55a8d39167a6.img volume-snap-a9cd1a28-30e6-42e7-9175-dfc987262dc4.img volume-snap-0120e6c1-7962-4378-b090-006cf4305f50.img volume-snap-b9397eee-69f0-41ca-808e-6fadba8c32c8.img volume-snap-93362d3c-51d0-4196-938a-a0455ffe463d.img volume-snap-f2d19ed0-7a0d-427f-bcba-315be4efb0f1.img]
Environment:
kubectl version
):Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.5", GitCommit:"804d6167111f6858541cef440ccc53887fbbc96a", GitTreeState:"clean", BuildDate:"2022-12-19T15:26:36Z", GoVersion:"go1.19.4", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.5", GitCommit:"804d6167111f6858541cef440ccc53887fbbc96a", GitTreeState:"clean", BuildDate:"2022-12-19T15:27:17Z", GoVersion:"go1.19.4", Compiler:"gc", Platform:"linux/amd64"}
Using MicroK8S v1.25.5
/etc/os-release
):Ubuntu 22.04.1 LTS
The jiva CI integration tests are currently placed in a single bash script that is executed as part of the travis build process. This script needs to be modularized (separate out the individual test functions) to aid easier management and adhere to BCH standards. Increased comments/test-descriptions to specify test intent also will help.
https://github.com/openebs/jiva/blob/master/ci/start_init_test.sh
I haven't yet checked out the build script but trying to go get
on linux made me have a WTF moment :P
This depends on port.Stats
which exists only in this PR: https://github.com/openebs/gotgt/pull/9/files
Also, why is openebs/gotgt
not vendored. /cc @kmova
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.