gostor / gotgt Goto Github PK

Can we have multiple parallel scsi worker threads, after checking for IO conflicts?

Now, the function of NewSCSITargetService() returned single instance in the daemon, we need to add a lock or use sync.Once to make sure the thread-security.

Some SCSI commands are not supported now, such as:

ALL.CompareAndWrite
ALL.CompareAndWrite.Simple
ALL.CompareAndWrite.DpoFua
ALL.CompareAndWrite.Miscompare
ALL.CompareAndWrite.Unwritten
ALL.CompareAndWrite.InvalidDataOutSize
ALL.ExtendedCopy
ALL.ExtendedCopy.Simple
ALL.ExtendedCopy.ParamHdr
ALL.ExtendedCopy.DescrLimits
ALL.ExtendedCopy.DescrType
ALL.ExtendedCopy.ValidTgtDescr
ALL.ExtendedCopy.ValidSegDescr
ALL.GetLBAStatus
ALL.GetLBAStatus.Simple
ALL.GetLBAStatus.BeyondEol
ALL.GetLBAStatus.UnmapSingle
ALL.Inquiry
ALL.Inquiry.BlockLimits
ALL.ModeSense6
ALL.ModeSense6.AllPages
ALL.ModeSense6.Control
ALL.ModeSense6.Control-D_SENSE
ALL.ModeSense6.Control-SWP
ALL.ModeSense6.Residuals
ALL.OrWrite
ALL.OrWrite.Simple
ALL.OrWrite.BeyondEol
ALL.OrWrite.ZeroBlocks
ALL.OrWrite.Protect
ALL.OrWrite.DpoFua
ALL.OrWrite.Verify
ALL.PrinReadKeys
ALL.PrinReadKeys.Simple
ALL.PrinServiceactionRange
ALL.PrinServiceactionRange.Range
ALL.PrinReportCapabilities
ALL.PrinReportCapabilities.Simple
ALL.ProutRegister
ALL.ProutRegister.Simple
ALL.ProutReserve
ALL.ProutReserve.Simple
ALL.ProutReserve.AccessEA
ALL.ProutReserve.AccessWE
ALL.ProutReserve.AccessEARO
ALL.ProutReserve.AccessWERO
ALL.ProutReserve.AccessEAAR
ALL.ProutReserve.AccessWEAR
ALL.ProutReserve.OwnershipEA
ALL.ProutReserve.OwnershipWE
ALL.ProutReserve.OwnershipEARO
ALL.ProutReserve.OwnershipWERO
ALL.ProutReserve.OwnershipEAAR
ALL.ProutReserve.OwnershipWEAR
ALL.ProutClear
ALL.ProutClear.Simple
ALL.ProutPreempt
ALL.ProutPreempt.RemoveRegistration
ALL.ReadDefectData10
ALL.ReadDefectData10.Simple
ALL.ReadDefectData12
ALL.ReadDefectData12.Simple
ALL.ReceiveCopyResults
ALL.ReceiveCopyResults.CopyStatus
ALL.ReceiveCopyResults.OpParams
ALL.ReportSupportedOpcodes
ALL.ReportSupportedOpcodes.OneCommand
ALL.ReportSupportedOpcodes.RCTD
ALL.ReportSupportedOpcodes.SERVACTV
ALL.Reserve6
ALL.Reserve6.2Initiators
ALL.Reserve6.Logout
ALL.Reserve6.ITNexusLoss
ALL.Reserve6.TargetColdReset
ALL.Reserve6.TargetWarmReset
ALL.Reserve6.LUNReset
ALL.Sanitize
ALL.Sanitize.BlockErase
ALL.Sanitize.BlockEraseReserved
ALL.Sanitize.CryptoErase
ALL.Sanitize.CryptoEraseReserved
ALL.Sanitize.ExitFailureMode
ALL.Sanitize.InvalidServiceAction
ALL.Sanitize.Overwrite
ALL.Sanitize.OverwriteReserved
ALL.Sanitize.Readonly
ALL.Sanitize.Reservations
ALL.Sanitize.Reset
ALL.Unmap
ALL.Unmap.Simple
ALL.Unmap.VPD
ALL.Unmap.ZeroBlocks
ALL.WriteAtomic16.VPD
ALL.WriteSame10
ALL.WriteSame10.BeyondEol
ALL.WriteSame10.ZeroBlocks
ALL.WriteSame10.WriteProtect
ALL.WriteSame10.Unmap
ALL.WriteSame10.UnmapUnaligned
ALL.WriteSame10.UnmapUntilEnd
ALL.WriteSame10.UnmapVPD
ALL.WriteSame10.Check
ALL.WriteSame10.InvalidDataOutSize
ALL.WriteSame16
ALL.WriteSame16.BeyondEol
ALL.WriteSame16.ZeroBlocks
ALL.WriteSame16.WriteProtect
ALL.WriteSame16.Unmap
ALL.WriteSame16.UnmapUnaligned
ALL.WriteSame16.UnmapUntilEnd
ALL.WriteSame16.UnmapVPD
ALL.WriteSame16.Check
ALL.WriteSame16.InvalidDataOutSize
ALL.iSCSIdatasn
ALL.iSCSIdatasn.iSCSIDataSnInvalid
ALL.iSCSIResiduals
ALL.iSCSIResiduals.Read10Invalid
ALL.iSCSIResiduals.Read10Residuals
ALL.iSCSIResiduals.Read12Residuals
ALL.iSCSIResiduals.Read16Residuals
ALL.iSCSIResiduals.Write10Residuals
ALL.iSCSIResiduals.Write12Residuals
ALL.iSCSIResiduals.Write16Residuals
ALL.iSCSIResiduals.WriteVerify10Residuals
ALL.iSCSIResiduals.WriteVerify12Residuals
ALL.iSCSIResiduals.WriteVerify16Residuals
ALL.MultipathIO
ALL.MultipathIO.Simple
ALL.MultipathIO.Reset
ALL.MultipathIO.CompareAndWrite
ALL.MultipathIO.CompareAndWriteAsync

Note: the case name is from libiscsi

Currently, we didn't obey the MaxReceiveLength parameter.
We should make the SCSI IN response obey that parameter.
And we need to divided the response PDU into N (N = data size/ MaxReceiveLenght).
Current test with default initiator parameter:
Linux (PASS).
Windows (N/A)
Solaris (FAILED)
Mac(N/A)

I run "go vet ./..." for gotgt project, got some errors:

[cavan@n1 gotgt]$ go vet ./... 2>&1 | grep -v "composite literal uses unkeyed" | grep -v "vendor/" | grep -v "exit status 1"
pkg/api/client/target_create.go:30: return copies lock value: github.com/gostor/gotgt/pkg/api.SCSITarget contains sync.Mutex
pkg/api/client/target_create.go:34: return copies lock value: github.com/gostor/gotgt/pkg/api.SCSITarget contains sync.Mutex
pkg/port/iscsit/iscsid.go:182: unreachable code
pkg/port/iscsit/iscsid.go:252: unreachable code
pkg/port/iscsit/iscsid.go:755: unreachable code
pkg/port/iscsit/iscsit.go:126: literal copies lock value from *target: github.com/gostor/gotgt/pkg/api.SCSITarget contains sync.Mutex
pkg/port/iscsit/login.go:151: wrong number of args for format in Errorf call: 2 needed but 3 args
pkg/scsi/scsi.go:50: call of append copies lock value: github.com/gostor/gotgt/pkg/api.SCSITarget contains sync.Mutex
pkg/scsi/spc.go:520: pg.Size too small for shift of 8
pkg/scsi/spc.go:552: pg.Size too small for shift of 8
pkg/scsi/spc.go:630: length too small for shift of 8
pkg/scsi/target.go:100: arg lun for printf verb %s of wrong type: uint64
pkg/util/util_linux.go:1: +build comment must appear before package clause and be followed by a blank line

Target

• open source Scale-Out iSCSI cluster
• fully implemented in user space with golang
• ceph as the backend device
• controlled by kubernetes

The gotgt^TM cluster is an open source Scale-Out SAN solution offering massive scalability and performance. It uses modern cloud based technologies to provide the elasticity and agility to scale up the storage cluster simply by adding more nodes. This can be done at any time and in a truly non-disruptive manner.

The gotgt^TM cluster has a new implementation for iSCSI target in user space with golang, so that we can provide a better design than traditional iSCSI target (such as SCST and LIO). The cluster can be dockerized and controlled by kubernetes, then it will be very easy to install and operate.

The gotgt^TM is designed to provide a RESTful API to check and operate the resources (such as LUs and targets). Based on those APIs, the cluster can be operated by non-Linux administrators through its easy to use graphical interfaces.

In gotgt^TM, an iSCSI disk can have many access paths, each is identified by its virtual IP address. These IP addresses are clustered across several storage nodes. Access to the same disk is load balanced across different nodes in a symmetric Active/Active fashion.

The gotgt^TM cluster brings proven technologies, powering many of today's largest clouds to the datacenter with point and click interfaces. First, we have a self developed iSCSI target with golang in user space. It can provide better interfaces for management and has much better performance. Secondly, it uses the Ceph storage engine. Ceph is the leading edge open source SDS cloud storage solution, powering many of today's private and public clouds, some with thousands of nodes. Ceph is a fault tolerant, self healing and self adapting system with no single point of failure. Furthermore, it uses a distributed key-value storage (such as Consul or Etcd) for cloud-scale distributed metadata management .

Design details

In the above diagram, the gotgt^TM cluster has four components:

• Monitor
• iSCSI target
• Metadata storage
• Storage devices (Ceph or others)

Monitor

The client (iSCSI initiator) will connect to the monitor component to get the real login services information, the monitor service returns a message which contains a REDIRECT command. Then the client can re-login to the one or more scheduled iSCSI targets based on the user's setting.

Initial connection sequence of events:

  1. Initiator performs the discovery procedure by using executing a discovery session.

  2. The discovery session is used, the initiator opens a TCP
     connection to the discovery target portal, logs in and issues
     the "send targets" commands.  The target responds with a list
     of target names and their associated portals.  The initiator or
     user selects the portals associated with the specific target it
     is interested in establishing a session with.  The initiator
     terminates the discovery session and closes the associated TCP
     connection.

  3. Whichever discovery procedure is used, the initiator remembers
     the portals for this target as the "initial target portals".

  4. The initiator iterates through the initial target portals list
     until it succeeds in opening a TCP connection to one of them.

  5. The initiator then logs into the target, which may respond with
     a "target moved temporarily" redirect response, listing the
     redirect portal for the target.  The initiator remembers this
     as the "redirect portal."  The initiator then closes the TCP
     connection.

  6. The initiator then opens a TCP connection to the redirect
     portal and logs in.  The target accepts this login and the
     session proceeds to full feature phase.

  7. Data flow begins.

Target node or interface failure sequence of events:

  1. The initiator has an iSCSI session established and TCP
     connection open to the redirect portal.  Full feature session
     in progress.  Data is flowing.

  2. The target fails.

  3. The initiator detects the failure of the TCP connection with
     the target.

  4. The initiator iterates through the list of initial target
     portals learned in the discovery process until it succeeds in
     opening a TCP connection to one of them.

  5. If the initiator succeeds in connecting to one of the initial
     target portals, it executes steps 5 and 6 in the "Initial
     connection sequence of events" section.

  6. If the initiator fails to connect to any of the initial target
     portals, it repeats steps 1 through 6 in the "Initial
     connection sequence of events" section.

  7. Data flow resumes.

iSCSI target

The existing gotgt^TM iSCSI target library project: https://github.com/gostor/gotgt.

Metadata storage

With the help of a distributed key-value database, the metadata of cluster and iSCSI resources (targets, LUs and TPGTs) will be stored in it, so that each service can get the real time data.

Either Consul or Etcd can be used as the metadata storage.

Storage devices

Backend storage device system. We prefer to using Ceph in this implementation.

Deployment scenario and its implied deliverables

A simple deployment scenario is similar to the diagram presented earlier in this document.

The client (iSCSI initiator) applications will work transparently to take advantage of this enhanced cluster implementation of gotgt along with other docker[3] components. This will allow the application to move freely in a cloud environment.

This new gotgt cluster docker image provides multi-pathing and load balancing for the data transport and iSCSI sessions. In this way, the clustered iSCSI target library acts like an iSCSI transport gateway.

The gotgt will include plug-in based interfaces to connect to backend storage, which will include block storage like Ceph. Simple file-based storage interfaces will be provided as well.

Compatible and tested other docker images such as Ceph (backend storage) and etcd (distributed key-value store) will be provided.
For ease of management, load balancing and resilience (through replications),
kubernetes image and example cluster configuration files for pods and services will be provided.

In the longer term, there will be exported RESTful API defined for simple graphical interface implementation to replace text-based configuration files.
Also in the longer term, simple configuration graphical interface implementation will be provided for iSCSI target cluster administration.

References

[1] https://tools.ietf.org/html/draft-gilligan-iscsi-fault-tolerance-00

[2] https://www.snia.org/sites/default/education/tutorials/2011/spring/networking/HufferdJohn-IP_Storage_Protocols-iSCSI.pdf

[3] https://www.docker.com

There are any documentation/interface so we can implement our custom backing stores?

The client I'm working on needs the SecurityNegotiation stage to work, but tries to negotiate AuthMethod=none. (I also thought the negotiation stage was required in the spec?)

Would that be hard to support?

https://tools.ietf.org/html/rfc5047

Ideally, cmd package should manage targets, luns and tpgt's.

goroutine use number of CPU to run as simultaneously.
However, we didn't consider the thread safe issue.
We should simply use 1 CPU to dispatch the gorutine to avoid this issue and then use channel and mutex to enhance it.

ubuntu 18.04下无法完成编译。

apt-get install librados-dev
apt-get install librbd-dev
安装依赖后，
使用make 命令，提示

/usr/include/rados/librados.h:3272:21: note: declared here
CEPH_RADOS_API void rados_read_op_omap_get_vals(rados_read_op_t read_op,

Test Case 1:

Tried to run the below sequence of commands:

sudo dd if=/dev/urandom of=test1 bs=32k count=1
md5sum test1
sudo dd of=/dev/sdb if=test1 bs=32k count=1
sudo dd of=test2 if=/dev/sdb bs=32k count=1
md5sum test2

/dev/sdb -- Device path at initiator after login
When initiator and target are on the same host (even on two VMs on the same host), md5sum of file1 and file2 matches. But when the initiator and target are on separate hosts over physical network, the md5sum of file1 and file2 doesn't match.

Test Case 2:

mkfs.ext4 /dev/sdb was successful.
But when initiator and target were on separate hosts, mount failed with the below message:

mount: wrong fs type, bad option, bad superblock on /dev/sdb,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so

logs:

Jan 30 16:02:02 Cent10 kernel: scsi11 : iSCSI Initiator over TCP/IP
Jan 30 16:02:02 Cent10 kernel: scsi 11:0:0:0: Direct-Access     GOSTOR   GOTGT            0.1  PQ: 0 ANSI: 5
Jan 30 16:02:02 Cent10 kernel: sd 11:0:0:0: Attached scsi generic sg2 type 0
Jan 30 16:02:02 Cent10 kernel: sd 11:0:0:0: [sdb] 20971520 512-byte logical blocks: (10.7 GB/10.0 GiB)
Jan 30 16:02:02 Cent10 kernel: sd 11:0:0:0: [sdb] 4096-byte physical blocks
Jan 30 16:02:02 Cent10 kernel: sd 11:0:0:0: [sdb] Write Protect is off
Jan 30 16:02:02 Cent10 kernel: sd 11:0:0:0: [sdb] Asking for cache data failed
Jan 30 16:02:02 Cent10 kernel: sd 11:0:0:0: [sdb] Assuming drive cache: write through
Jan 30 16:02:02 Cent10 kernel: sd 11:0:0:0: [sdb] Asking for cache data failed
Jan 30 16:02:02 Cent10 kernel: sd 11:0:0:0: [sdb] Assuming drive cache: write through
Jan 30 16:02:02 Cent10 kernel: sdb: unknown partition table
Jan 30 16:02:02 Cent10 kernel: sd 11:0:0:0: [sdb] Asking for cache data failed
Jan 30 16:02:02 Cent10 kernel: sd 11:0:0:0: [sdb] Assuming drive cache: write through
Jan 30 16:02:02 Cent10 kernel: sd 11:0:0:0: [sdb] Attached SCSI disk
Jan 30 16:02:02 Cent10 iscsid: Connection9:0 to [target: iqn.2016-09.com.openebs.jiva:volx, portal: 20.10.92.2,3260] through [iface: default] is operational now
Jan 30 16:02:47 Cent10 kernel: EXT4-fs (sdb): ext4_check_descriptors: Checksum for group 0 failed (48625!=0)
Jan 30 16:02:47 Cent10 kernel: EXT4-fs (sdb): group descriptors corrupted!

After these tests:
I even attempted to assign MaxCmdSN=ExpCmdSN so that initiator doesn't send the next command until it gets a response for the last command. Still the above test cases are failing.

I figured out how to build this code, but the build instructions are incomplete.
I had to run additional commands to "go get" a bunch of packages, I discovered the missing packages by running make after getting each package that make complained on.

It would be nice if the instructions had the steps to "go get" what was needed. Better still, consider "vendoring" any needed packages.

MaxRecvDataSegmentLength not present in the list of negotiated parameters.

I created a dummy lun using null backing store and wrote 4GB using dd from /dev/zero and resident size for gotgt daemon grew to 5GB even though null backing store doesn't keep track of the writes. I think this suggests a memory leak.

I had to do the following changes to get null backing store working through config file:

bvk@iron:~/go/src/github.com/gostor/gotgt$ git diff
diff --git a/pkg/scsi/backingstore/null.go b/pkg/scsi/backingstore/null.go
index 2cbde02..56edd93 100644
--- a/pkg/scsi/backingstore/null.go
+++ b/pkg/scsi/backingstore/null.go
@@ -40,6 +40,7 @@ func newNull() (api.BackingStore, error) {
 }
 
 func (bs *NullBackingStore) Open(dev *api.SCSILu, path string) error {
+       bs.DataSize = 1024 * 1024 * 1024 * 1024
        return nil
 }
 
@@ -56,11 +57,11 @@ func (bs *NullBackingStore) Exit(dev *api.SCSILu) error {
 }
 
 func (bs *NullBackingStore) Size(dev *api.SCSILu) uint64 {
-       return 0
+       return bs.DataSize
 }
 
 func (bs *NullBackingStore) Read(offset, tl int64) ([]byte, error) {
-       return nil, nil
+       return make([]byte, tl), nil
 }
 
 func (bs *NullBackingStore) Write(wbuf []byte, offset int64) error {
bvk@iron:~/go/src/github.com/gostor/gotgt$ 

We support INQUIRY_TPGS_IMPLICIT.
How to disable this apart from sending INQUIRY_TPGS_NO in scsi inquiry?
cc: @carmark

I am testing the gotgt to use in another project that add scale-out/replication support for block volumes. When I exposed a volume and did "dd" test, I kept seeing continuous re-logins in the gotgt logs.

Steps to Reproduce:

• Start the gotgt process as described in README
• Discover and login to the LUN using iscsiadm:

 --sudo iscsiadm -m discovery -t sendtargets -p 127.0.0.1:3260
 --sudo iscsiadm -m node -l

• Get the newly attached device:

 --sudo iscsiadm -m session -P 3

• Write some data using dd command on the newly attached device:

 --sudo dd if=/dev/urandom of=/dev/sdb bs=4k count=1000

• Run the above dd command once again to get the relogin

Snippet of the logs

I1123 14:49:24.641125   12808 iscsid.go:189] rx handler: IOSTATE_RX_BHS
I1123 14:49:24.641164   12808 iscsid.go:250] got command:
Op: SCSI Data-Out (write)
Final = true
Immediate = false 
Data Segment Length = 8192
Task Tag = 63000000
AHS Length = 0
LUN = [0 0 0 0 0 0 0 0]
ExpectedDataLen = 0
CmdSN = 0
ExpStatSN = 84
Read = false 
Write = false 
CDB = 
I1123 14:49:24.641622   12808 common.go:207] io done *^@^@^@^T^@^@
^@^@^@^@^@^@^@^@
I1123 14:49:24.641878   12808 iscsid.go:588] 12
I1123 14:49:24.641905   12808 iscsid.go:189] rx handler: IOSTATE_RX_BHS
E1123 14:49:24.642398   12808 iscsid.go:779] unexpected cmd serial number: (97, 109)
I1123 14:49:37.165039   12808 iscsid.go:144] 127.0.0.1:3260
I1123 14:49:37.165098   12808 iscsid.go:145] Accepting ...
I1123 14:49:37.165146   12808 iscsid.go:150] connection is connected from 127.0.0.1:53032...
I1123 14:49:37.165178   12808 iscsid.go:138] Listening ...
I1123 14:49:37.165263   12808 iscsid.go:189] rx handler: IOSTATE_RX_BHS
I1123 14:49:37.415663   12808 iscsid.go:250] got command: 
Op: Login Request
Final = true
Immediate = true
Data Segment Length = 473 
Task Tag = 0
AHS Length = 0
ISID = 23d050000
Transit = true
Continue = false 
Current Stage = Login Operational Negotiation
Next Stage = Full Feature Phase 
I1123 14:49:37.415732   12808 iscsid.go:268] OpLoginReq
I1123 14:49:37.416104   12808 iscsid.go:572] CONN_STATE_KERNEL
I1123 14:49:37.416164   12808 iscsid.go:588] 12
I1123 14:49:37.416205   12808 iscsid.go:189] rx handler: IOSTATE_RX_BHS

Is anyone else also facing the same issue?

Length of data segment in DataIn PDUs has to be limited to MaxRecvDataSegment length and the length of the data transferred in a particular sequence of DataIn PDUs has to be limited to MaxBurstLength. (RFC 3720 Sec. 12.12 and 12.13)