pvc request failed to create volume with error "timeout in synchronizing txn" about glusterd2 HOT 16 CLOSED

Still the pvc status shows as pending state. When I tried to create a new pvc it is also going to pending state, not sure if is because of an already pending pvc. If this issue is different, let me know I'll file a separate issue for tracking it.

Below is the describe output for the new pvc,

[vagrant@kube1 ~]$ kubectl -n gcs describe pvc test1
Name: test1
Namespace: gcs
StorageClass: glusterfs-csi
Status: Pending
Volume:
Labels:
Annotations: volume.beta.kubernetes.io/storage-provisioner: org.gluster.glusterfs
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
Events:
Type Reason Age From Message

Normal ExternalProvisioning 13s (x5 over 36s) persistentvolume-controller waiting for a volume to be created, either by external provisioner "org.gluster.glusterfs" or manually created by system administrator
Warning ProvisioningFailed 11s (x2 over 23s) org.gluster.glusterfs_csi-provisioner-glusterfsplugin-0_e9313507-0f49-11e9-8331-0a580ae9400d failed to provision volume with StorageClass "glusterfs-csi": rpc error: code = Internal desc = failed to create volume: timeout in synchronizing txn
Normal Provisioning 8s (x3 over 36s) org.gluster.glusterfs_csi-provisioner-glusterfsplugin-0_e9313507-0f49-11e9-8331-0a580ae9400d External provisioner is provisioning volume for claim "gcs/test1"
Mounted By:

from glusterd2.

@PrasadDesala what is the size of newly created PVC and please attach the provisioner log

from glusterd2.

@PrasadDesala what is the size of newly created PVC and please attach the provisioner log

The newly created PVC size is 1GB.
Attached provisioner logs:
provsioner_logs.txt

from glusterd2.

gluster_provisoner_logs.txt

from glusterd2.

E0103 13:24:24.539942       1 controllerserver.go:269] failed to create volume: timeout in synchronizing txn
E0103 13:24:24.539996       1 utils.go:100] GRPC error: rpc error: code = Internal desc = failed to create volume: timeout in synchronizing txn
I0103 13:25:28.726550       1 utils.go:96] GRPC call: /csi.v0.Identity/GetPluginCapabilities
I0103 13:25:28.726590       1 utils.go:97] GRPC request: 
I0103 13:25:28.726657       1 identityserver.go:38] plugin capability response: capabilities:<service:<type:CONTROLLER_SERVICE > > 
I0103 13:25:28.726733       1 utils.go:102] GRPC response: capabilities:<service:<type:CONTROLLER_SERVICE > > 
I0103 13:25:28.727153       1 utils.go:96] GRPC call: /csi.v0.Controller/ControllerGetCapabilities
I0103 13:25:28.727217       1 utils.go:97] GRPC request: 
I0103 13:25:28.727297       1 utils.go:102] GRPC response: capabilities:<rpc:<type:CREATE_DELETE_VOLUME > > capabilities:<rpc:<type:LIST_VOLUMES > > capabilities:<rpc:<type:CREATE_DELETE_SNAPSHOT > > capabilities:<rpc:<type:LIST_SNAPSHOTS > > 
I0103 13:25:28.727917       1 utils.go:96] GRPC call: /csi.v0.Identity/GetPluginInfo
I0103 13:25:28.727953       1 utils.go:97] GRPC request: 
I0103 13:25:28.728032       1 identityserver.go:21] plugininfo response: name:"org.gluster.glusterfs" vendor_version:"0.0.9" 
I0103 13:25:28.728063       1 utils.go:102] GRPC response: name:"org.gluster.glusterfs" vendor_version:"0.0.9" 
I0103 13:25:28.728612       1 utils.go:96] GRPC call: /csi.v0.Controller/CreateVolume
I0103 13:25:28.728629       1 utils.go:97] GRPC request: name:"pvc-c2f5c61a-0f58-11e9-897e-525400933534" capacity_range:<required_bytes:1073741824 > volume_capabilities:<mount:<> access_mode:<mode:MULTI_NODE_MULTI_WRITER > > 
I0103 13:25:28.728685       1 controllerserver.go:115] request received name:"pvc-c2f5c61a-0f58-11e9-897e-525400933534" capacity_range:<required_bytes:1073741824 > volume_capabilities:<mount:<> access_mode:<mode:MULTI_NODE_MULTI_WRITER > > 
I0103 13:25:28.728742       1 controllerserver.go:121] creating volume with name pvc-c2f5c61a-0f58-11e9-897e-525400933534
I0103 13:25:28.760107       1 controllerserver.go:256] received request to create volume pvc-c2f5c61a-0f58-11e9-897e-525400933534 with size 1073741824
I0103 13:25:28.760146       1 controllerserver.go:266] volume create request: {Name:pvc-c2f5c61a-0f58-11e9-897e-525400933534 Transport: Subvols:[] Force:false Metadata:map[VolumeOwner:org.gluster.glusterfs] Flags:map[] Size:1073741824 DistributeCount:0 ReplicaCount:3 ArbiterCount:0 DisperseCount:0 DisperseRedundancyCount:0 DisperseDataCount:0 SnapshotEnabled:false SnapshotReserveFactor:0 LimitPeers:[] LimitZones:[] ExcludePeers:[] ExcludeZones:[] SubvolZonesOverlap:false SubvolType: VolOptionReq:{Options:map[] VolOptionFlags:{AllowAdvanced:false AllowExperimental:false AllowDeprecated:false}}}
E0103 13:25:41.408197       1 controllerserver.go:269] failed to create volume: timeout in synchronizing txn
E0103 13:25:41.408261       1 utils.go:100] GRPC error: rpc error: code = Internal desc = failed to create volume: timeout in synchronizing txn

this is the issue with glusterd2 not with csi driver

from glusterd2.

@PrasadDesala please attach glusterd2 logs

from glusterd2.

@oshankkumar PTAL

from glusterd2.

@PrasadDesala please attach glusterd2 logs

kube-3-glusterd2.log.gz
kube-2-glusterd2.log.gz
kube-1-glusterd2.log.gz

from glusterd2.

@PrasadDesala the pvc will be created but the volume will not be created. It will throw " no space available or all the devices are not registered" error.

from glusterd2.

@PrasadDesala the PVC will be created but the volume will not be created. It will throw " no space available or all the devices are not registered" error.

@rishubhjain this issue is related to E0103 13:25:41.408261 1 utils.go:100] GRPC error: rpc error: code = Internal desc = failed to create volume: timeout in synchronizing txn

from glusterd2.

pvc creation stuck at pending state when the requested size is more than the actual device size

@PrasadDesala please update issue message

from glusterd2.

@Madhu-1 @rishubhjain

1. When the device size is 10GB and requested PVC size is 50GB it returned error,
   error: code = Internal desc = failed to create volume: no space available or all the devices are not registered
2. With the above pvc is pending state, when we requested PVC size of 1GB it returned error,
   error: code = Internal desc = failed to create volume: timeout in synchronizing txn

The error codes they are returning is different. Is the second issue because of the first?

from glusterd2.

@Madhu-1 @rishubhjain

1. When the device size is 10GB and requested PVC size is 50GB it returned error,
   error: code = Internal desc = failed to create volume: no space available or all the devices are not registered

This is an expected behaviour

2. With the above pvc is pending state, when we requested PVC size of 1GB it returned error,
   error: code = Internal desc = failed to create volume: timeout in synchronizing txn

This should not fail as the request size is 1GB

The error codes they are returning is different. Is the second issue because of the first?

from glusterd2.

@Madhu-1 @rishubhjain

1. When the device size is 10GB and requested PVC size is 50GB it returned error,
   error: code = Internal desc = failed to create volume: no space available or all the devices are not registered

This is an expected behaviour

Yes, the behavior should be expected as there is no space available on the device to create pvc but here the pvc creating is stuck at pending state. Shouldn't it fail after some retry attempts?

1. With the above pvc is pending state, when we requested PVC size of 1GB it returned error,
   error: code = Internal desc = failed to create volume: timeout in synchronizing txn

This should not fail as the request size is 1GB

The error codes they are returning is different. Is the second issue because of the first?

from glusterd2.

@Madhu-1 @rishubhjain

1. When the device size is 10GB and requested PVC size is 50GB it returned error,
   error: code = Internal desc = failed to create volume: no space available or all the devices are not registered

This is an expected behaviour

Yes, the behavior should be expected as there is no space available on the device to create pvc but here the pvc creating is stuck at pending state. Shouldn't it fail after some retry attempts?

Kubernetes will typically let the resource pend indefinitely.

from glusterd2.

timeout in synchronizing transaction is fixed through #1462

from glusterd2.