Git Product home page Git Product logo

ltfs's Introduction

GH Action status BSD License

About this branch

This is the master branch of the LTFS project. At this time, this branch is used for version 2.5 development. So it wouldn't be stable a little. Please consider to follow the tree on v2.4-stable branch if you want to use stable codes.

What is the Linear Tape File System (LTFS)

The Linear Tape File System (LTFS) is a filesystem to mount a LTFS formatted tape in a tape drive. Once LTFS mounts a LTFS formatted tape as filesystem, user can access to the tape via filesystem API.

Objective of this project is being the reference implementation of the LTFS format Specifications in SNIA.

At this time, the target of this project to meet is the LTFS format specifications 2.5.

Supported Tape Drives

Vendor Drive Type Minimum F/W Level
IBM LTO5 B170
IBM LTO6 None
IBM LTO7 None
IBM LTO8 HB81
IBM LTO9 None
IBM TS1140 3694
IBM TS1150 None
IBM TS1155 None
IBM TS1160 None
HP LTO5 T.B.D.
HP LTO6 T.B.D.
HP LTO7 T.B.D.
HP LTO8 T.B.D.
HP LTO9 T.B.D.
Quantum LTO5 (Only Half Height) T.B.D.
Quantum LTO6 (Only Half Height) T.B.D.
Quantum LTO7 (Only Half Height) T.B.D.
Quantum LTO8 (Only Half Height) T.B.D.
Quantum LTO9 (Only Half Height) T.B.D.

LTFS Format Specifications

LTFS Format Specification is specified data placement, shape of index and names of extended attributes for LTFS. This specification is defined in SNIA first and then it is forwarded to ISO as ISO/IEC 20919 from version 2.2.

The table below show status of the LTFS format Specification

Version Status of SNIA Status of ISO
2.2 Published Published as 20919:2016
2.3.1 Published -
2.4 Published -
2.5.1 Published Published as 20919:2021

How to use the LTFS (Quick start)

This section is for a person who already has a machine with the LTFS installed. Instructions on how to use the LTFS is also available on Wiki.

Step1: List tape drives

# ltfs -o device_list

The output is as follows. You have 3 drives in this example and you can use "Device Name" field, like /dev/sg43 in this case, as the argument of ltfs command to mount the tape drive.

50c4 LTFS14000I LTFS starting, LTFS version 2.4.0.0 (10022), log level 2.
50c4 LTFS14058I LTFS Format Specification version 2.4.0.
50c4 LTFS14104I Launched by "/home/piste/ltfsoss/bin/ltfs -o device_list".
50c4 LTFS14105I This binary is built for Linux (x86_64).
50c4 LTFS14106I GCC version is 4.8.5 20150623 (Red Hat 4.8.5-11).
50c4 LTFS17087I Kernel version: Linux version 3.10.0-514.10.2.el7.x86_64 ([email protected]) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Mon Feb 20 02:37:52 EST 2017 i386.
50c4 LTFS17089I Distribution: NAME="Red Hat Enterprise Linux Server".
50c4 LTFS17089I Distribution: Red Hat Enterprise Linux Server release 7.3 (Maipo).
50c4 LTFS17089I Distribution: Red Hat Enterprise Linux Server release 7.3 (Maipo).
50c4 LTFS17085I Plugin: Loading "sg" tape backend.
Tape Device list:.
Device Name = /dev/sg43, Vender ID = IBM    , Product ID = ULTRIUM-TD5    , Serial Number = 9A700L0077, Product Name = [ULTRIUM-TD5] .
Device Name = /dev/sg38, Vender ID = IBM    , Product ID = ULT3580-TD6    , Serial Number = 00013B0119, Product Name = [ULT3580-TD6] .
Device Name = /dev/sg37, Vender ID = IBM    , Product ID = ULT3580-TD7    , Serial Number = 00078D00C2, Product Name = [ULT3580-TD7] .

Step2: Format a tape

As described in the LTFS format specifications, LTFS uses the partition feature of the tape drive. This means you can't use a tape just after you purchase a tape. You need format the tape before using it on LTFS.

To format a tape, you can use mkltfs command like

# mkltfs -d 9A700L0077

In this case, mkltfs tries to format a tape in the tape drive 9A700L0077. You can use the device name /dev/sg43 instead.

Step3: Mount a tape through a tape drive

After you prepared a formatted tape, you can mount it through a tape drive like

# ltfs -o devname=9A700L0077 /ltfs

In this command, the ltfs command will try to mount the tape in the tape drive 9A700L0077 to /ltfs directory. Of course, you can use a device name /dev/sg43 instead.

If the mount process is successfully done, you can access to the LTFS tape through /ltfs directory.

You must not touch any st devices while ltfs is mounting a tape.

Step4: Unmount the tape drive

You can use following command when you want to unmount the tape. The ltfs command try to write the current meta-data to the tape and close the tape cleanly.

# umount /ltfs

One thing you need to pay attention to here is, that the unmount command continues to work in the background after it returns. It just initiates a trigger to notify the the ltfs command of the unmount request. Actual unmount is completed when the ltfs command is finished.

The ltfsee_ordered_copy utility

The ltfsee_ordered_copy is a program to copy files from source to destination with LTFS order optimization.

It is written in python and it can work with both python2 and python3 (Python 2.7 or later is strongly recommended). You need to install the pyxattr module for both python2 and python3.

Building the LTFS from this GitHub project

These instructions will get a copy of the project up and running on your local machine for development and testing purposes.

Prerequisites for build

Please refer this page.

Build and install on Linux

./autogen.sh
./configure
make
make install

./configure --help shows various options for build and install.

In some systems, you might need sudo ldconfig -v after make install to load the shared libraries correctly.

Buildable Linux distributions

Dist Arch Status
RHEL 8 x86_64 OK - Not checked automatically
RHEL 8 ppc64le OK - Not checked automatically
CentOS 8 (Rocky Linux) x86_64 GH Action status
CentOS 8 (Rocky Linux) ppc64le OK - Not checked automatically
Fedora 28 x86_64 GH Action status
Ubuntu 16.04 LTS x86_64 GH Action status
Ubuntu 16.04 LTS ppc64le OK - Not checked automatically
Ubuntu 18.04 LTS x86_64 GH Action status
Ubuntu 18.04 LTS ppc64le OK - Not checked automatically
Ubuntu 20.04 LTS (Need icu-config) x86_64 GH Action status
Debian 9 x86_64 GH Action status
Debian 10 (Need icu-config) x86_64 GH Action status
ArchLinux 2018.08.01 x86_64 OK - Not checked automatically
ArchLinux 2018.12.31 (rolling) x86_64 OK - Not checked automatically

Currently, automatic build checking is working on GitHub Actions and Travis CI.

For Ubuntu20.04 and Debian10, dummy icu-config is needed in the build machine. See Issue #153.

Build and install on OSX (macOS)

Recent Homebrew system setup

Before build on macOS, you need to configure the environment like below.

export ICU_PATH="/usr/local/opt/icu4c/bin"
export LIBXML2_PATH="/usr/local/opt/libxml2/bin"
export PKG_CONFIG_PATH="/usr/local/opt/icu4c/lib/pkgconfig:/usr/local/opt/libxml2/lib/pkgconfig"
export PATH="$PATH:$ICU_PATH:$LIBXML2_PATH"

Old Homebrew system setup

Before build on OSX (macOS), some include path adjustment is required.

brew link --force icu4c
brew link --force libxml2

Building LTFS

On OSX (macOS), snmp cannot be supported, you need to disable it on configure script. And may be, you need to specify LDFLAGS while running configure script to link some required frameworks, CoreFundation and IOKit.

./autogen.sh
LDFLAGS="-framework CoreFoundation -framework IOKit" ./configure --disable-snmp
make
make install

./configure --help shows various options for build and install.

Buildable macOS systems

OS Xcode Package system Status
macOS 10.14.6 11.3 Homebrew OK - Not checked automatically
macOS 10.15 12.4 Homebrew OK - Not checked automatically
macOS 11 12.4 Homebrew OK - Not checked automatically

Build and install on FreeBSD

Note that on FreeBSD, the usual 3rd party man directory is /usr/local/man. Configure defaults to using /usr/local/share/man. So, override it on the command line to avoid having man pages put in the wrong place.

./autogen.sh
./configure --prefix=/usr/local --mandir=/usr/local/man
make
make install

Buildable versions

Version Arch Status
11 x86_64 OK - Not checked automatically
12 x86_64 OK - Not checked automatically

Build and install on NetBSD

./autogen.sh
./configure
make
make install

Buildable versions

Version Arch Status
8.1 amd64 OK - Not checked automatically
8.0 i386 OK - Not checked automatically
7.2 amd64 OK - Not checked automatically

Contributing

Please read CONTRIBUTING.md for details on our code of conduct, and the process for submitting pull requests to us.

License

This project is licensed under the BSD License - see the LICENSE file for details

ltfs's People

Contributors

amissael95 avatar dabiged avatar davidapease avatar diamant3 avatar hilocark avatar jgreen24 avatar juliocelon avatar kendmerry avatar kev009 avatar krishhhnaa avatar linnemannr avatar lshw avatar lucasvr avatar manu0401 avatar micronn avatar ochomenosocho avatar piste2750 avatar qayshp avatar retokromer avatar richard42 avatar ruixiaolee avatar rzerres avatar sardaukar avatar softloft38p-michael avatar swt2c avatar xloem avatar y-sasn avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Forkers

kendmerry piste2750 davidapease lucasvr rzerres manu0401 manu0218 kev009 hilocark neia-sas swt2c ruixiaolee jonnycrunch rofish royalmoose ryan-rao dericed veggiemike y-sasn softloft38p-michael imagineproducts woodenphone mosamu xloem eternalfs linnemannr huang96962 void-software ck-7 hedgeformac prcaaf ph0enixph0enix rebootcs piste-jp dabiged apresz m-lw rnshah9 spgill wizardtales pjbell lshw micronn dreamcodeman ochomenosocho stereo3d richard42 krishhhnaa heartofrain peterwillcn ksyuint jkim amissael95 vroad jgreen24 lopho 0800092000 banjiaojuhao diamant3 carlosodettedlcl080301 scothis alxbnct yaofei509 bonesix syaoraang abulacode helmos yzr95924 gvm- konidev20 mvinzens juliocelon

ltfs's Issues

A file which has US (0x1f) character is not accepted

Describe the bug

Cannot create a file or a symlink that has US (0x1f) character in the name.

To Reproduce

In bash,touch $'[mount_point]/aaa\x1faaa'.

LTFS returns an invalid argument error.

Expected behavior

Accept the filename and write to an index aaa%1faaa with percent encoded flag.

#57 may break the build on RHEL7 based OSs

Describe the bug
pathname.c makes compile errors when ICU6x.x is not installed.

To Reproduce
Steps to reproduce the behavior:

  1. Build the code on RHEL7 (or CentOS7)

Screenshots

pathname.c: In function '_pathname_normalize_nfc_icu':
pathname.c:671:2: warning: implicit declaration of function 'unorm_quickCheck_50' [-Wimplicit-function-declaration]
  if (unorm_quickCheck(src, -1, UNORM_NFD, &err) == UNORM_YES) {
  ^
pathname.c:671:35: error: 'UNORM_NFD' undeclared (first use in this function)
  if (unorm_quickCheck(src, -1, UNORM_NFD, &err) == UNORM_YES) {
                                   ^
pathname.c:671:35: note: each undeclared identifier is reported only once for each function it appears in
pathname.c:686:10: error: redeclaration of 'destlen' with no linkage
  int32_t destlen;
          ^
pathname.c:666:10: note: previous declaration of 'destlen' was here
  int32_t destlen;
          ^
pathname.c:687:2: warning: implicit declaration of function 'unorm_normalize_50' [-Wimplicit-function-declaration]
  destlen = unorm_normalize(src, -1, UNORM_NFC, 0, NULL, 0, &err);
  ^
pathname.c:687:40: error: 'UNORM_NFC' undeclared (first use in this function)
  destlen = unorm_normalize(src, -1, UNORM_NFC, 0, NULL, 0, &err);
                                        ^
pathname.c: In function '_pathname_normalize_nfd_icu':
pathname.c:763:35: error: 'UNORM_NFD' undeclared (first use in this function)
  if (unorm_quickCheck(src, -1, UNORM_NFD, &err) == UNORM_YES) {
                                   ^
make[3]: *** [libltfs_la-pathname.lo] Error 1

Desktop (please complete the following information):

  • OS: RHEL7

mkltfs does not work properly

Description

mkltfs -d /dev/st0 always leads to the following error message,
LTFS11261E Cannot load plugin: /usr/local/lib/ltfs/libdriver-ltotape.so: undefined symbol: fuse_opt_parse
Mount, copy Data to Tape, Restore Data from Tape, unmount is possible without any issues. (Tape was already formated via ltfs in another pc)

Environment

  • OS: Linux
  • Distribution: Ubuntu 16.04.4 LTS
  • Architecture: x86_64
  • Additional info: -

How to recreate

XML reader need to propagate more descriptive error codes

Describe the bug
Currently, XML reader may return almost one return code. But there are many symptoms to fail parsing. Need to return descriptive error codes from ltfs_read_index().

Expected behavior
LTFS need to select next action based on the return code from ltfs_read_index().

Cannot mount a LTO5 tape on a LTO7 drive

Description

ltfs command fails to mount a LTO5 tape on a LTO7 drive with following message.

LTFS14018E Cannot get read-only status of the medium.

Environment

  • OS: Linux/macOS
  • Distribution: Every
  • Architecture: Every

How to recreate

Just kick ltfs command to mount a LTO5 tape on a LTO7 drive.

Need to have some studies about what attributes are copied by ltfs_ordered_copy

Is your feature request related to a problem? Please describe.

At this time, I don't have any studies about what attribute is copied by ltfs_ordered_copy program.

It uses shutils.copy() by default and shutils.copy2() when -p, --preserve is specified. But there is no specification what attributes are copied exactly.

Describe the solution you'd like

  1. Need to study what shutils.copy() and shutils.copy2() copy to destination
  2. Support same functionality of cp -p and cp -a into ltfs_ordered_copy

Additional context

May be shutils.copy() and shutils.copy2()doesn't copy extended attribute at all. Extended attribute shall be copied when -a option is specified.

can't install on CENTOS 7 because of UUID

Hello,
I'm trying to install ltfs on centos 7 I tried v2.4.1 and master But the same proble occur:

checking for UUID_MODULE... no
configure: error: Package requirements (uuid >= 1.36) were not met:
No package 'uuid' found

Consider adjusting the PKG_CONFIG_PATH environment variable if you
installed software in a non-standard prefix.

I've installed uuid version 1.6 can't find a higher one. Thanks for your help

Value of the symlink tag shall be conformed as the name type

Summary: Value of the symlink tag shall be conformed as the name type

Description

At this time, symlink tag doesn't have "percent encoded" attribute if the value has ":" character.

Environment

  • OS: All
  • Distribution: All
  • Architecture: All
  • Additional info: None

How to recreate

  1. Create a file "[mount_point]/file:1"
  2. ln -s [mount_point]/link1 [mount_point]/file:1
  3. Inspect the index generated after unmount

lfts_ordered_copy xattr version on debian repos not matching?

I´ve tried yesterday the ltfs_ordered_copy script, first question, what is it good for? Sorting the files from small to big?

I got at first the error xattr not intsalled - so apt install python-xattr - installed version 0.9.6-1

then it startet, but produced an error telling me some methodwas not found. After a search i found out that at least the debian version of xattr has renamed the functions to getxattr(), setxattr(), listxattr(), no matter if using python2 or 3 version. The available methods are listed here:

attr.
xattr.XATTR_CREATE             xattr.getxattr(
xattr.XATTR_FINDERINFO_NAME    xattr.integer_types
xattr.XATTR_MAXNAMELEN         xattr.lib
xattr.XATTR_NOFOLLOW           xattr.listxattr(
xattr.XATTR_NOSECURITY         xattr.removexattr(
xattr.XATTR_REPLACE            xattr.setxattr(
xattr.XATTR_RESOURCEFORK_NAME  xattr.xattr(
xattr.compat

After renaming all the method the script run, but produced some errors:
Failed to copy NOTUSED_270317.sql.gz to NOTUSED_270317.sql.gz: $[Errno 22] invalid mode ('wb') or filename: _NOTUSED_270317.sql.gz'

also it throws errors on copying some "thumbs.db" from windows preview, its getting confused and says "not a directory"

Failed to copy pathpathpath/Thumbs.db: $[Errno 20] Not a directory: '/pathpathpath/Thumbs.db/Thumbs.db'
Failed to copy pathpathpath/Thumbs.db: $[Errno 20] Not a directory: 'pathpathpath/Thumbs.db/Thumbs.db'

I think thats due to the wrong xattr version? Which one to use?

EOD warning due to buffer too small

Describe the bug
I am seeing the following warning while mounting a tape:
LTFS30238W Cannot get EOD status: failed to parse the log page.

To Reproduce
Take LTO6 drive
Insert LTO6 LTFS mounted volume
Mount tape
Observe warning even though tape mounts OK.

Expected behavior
No warnings

Desktop (please complete the following information):

  • OS: CentOS 7, kernel 3.10.0-1062.4.3.el7.x86_64

Additional context
In ./src/tape_drivers/linux/sg-ibmtape/sg_ibmtape.c
sg_ibmtape_get_eod_status() checks validity of EOD capacities. It retrieves Log Sense page 0x17 and checks parameter 0x0203 "Approximate used native capacity of partition(s)"
It parses the log page and then checks the size of the parameter against the buffer size. (param_size != sizeof(buf)

The buffer is 16 bytes in size. unsigned char buf[16];
The LTO-6 drive returns 32 bytes for that parameter, as it supports up to 4 partitions and needs 8 bytes per partition. This test will only pass for LTO-5 which supports only 2 partitions.
Partial dump of the IBM Log Sense page 0x17:

01c0: 03 FF FF FF FF 02 03 43 - 20 07 00 00 00 FF FF FF   .......C  .......
01d0: FF 07 00 00 01 FF FF FF - FF 07 00 00 02 FF FF FF   ........ ........
01e0: FF 07 00 00 03 FF FF FF - FF 02 04 43 20 07 00 00   ........ ...C ...
01f0: 00 FF FF FF FF 07 00 00 - 01 FF FF FF FF 07 00 00   ........ ........
0200: 02 FF FF FF FF 07 00 00 - 03 FF FF FF FF            ........ .....

"-o eject" fails to mount the tape

Describe the bug
When i try to mount a tape with the -o eject option, the mount fails instantly.
When i remove the -o eject option, everything works fine.

see the log:

ltfs -o nonempty -o devname=/dev/sg0 /mnt/ltfs -o eject -o trace
205e LTFS14000I LTFS starting, LTFS version 2.4.2.1 (Prelim), log level 3.
205e LTFS14058I LTFS Format Specification version 2.4.0.
205e LTFS14104I Launched by "ltfs -o nonempty -o devname=/dev/sg0 /mnt/ltfs -o eject -o trace".
205e LTFS14105I This binary is built for Linux (x86_64).
205e LTFS14106I GCC version is 8.3.0.
205e LTFS17087I Kernel version: Linux version 5.4.55-1-pve (root@nora) (gcc version 8.3.0 (Debian 8.3.0-6)) #1 SMP PVE 5.4.55-1 (Mon, 10 Aug 2020 10:26:27 +0200) i386.
205e LTFS17089I Distribution: PRETTY_NAME="Debian GNU/Linux 10 (buster)".
205e LTFS14063I Sync type is "time", Sync time is 300 sec.
205e LTFS17085I Plugin: Loading "sg" tape backend.
205e LTFS17085I Plugin: Loading "unified" iosched backend.
205e LTFS30209I Opening a device through sg-ibmtape driver (/dev/sg0).
205e LTFS30250I Opened SCSI tape device 7.0.0.0 (/dev/sg0).
205e LTFS30207I Vendor ID is HP .
205e LTFS30208I Product ID is Ultrium 5-SCSI .
205e LTFS30214I Firmware revision is Z6ED.
205e LTFS30215I Drive serial is xxx.
205e LTFS30285I Reserved buffer size of /dev/sg0 is 1048576.
205e LTFS12023D Reserving device.
205e LTFS30392D Backend reserve (PRO) xxxx.
205e LTFS12028D Unlocking medium.
205e LTFS30392D Backend allow medium removal xxx.
205e LTFS30392D Backend read block limits xxx.
205e LTFS17160I Maximum device block size is 1048576.
205e LTFS11330I Loading cartridge.
205e LTFS30392D Backend load xxx.
205e LTFS30398D Backend readpos: (0, 0) FM = 0 xxx.
205e LTFS12026D Locking medium in the drive.
205e LTFS30392D Backend prevent medium removal xx.
205e LTFS30398D Backend readpos: (0, 0) FM = 0 xxx.
205e LTFS30392D Backend read block limits xxx.
205e LTFS11332I Load successful.
205e LTFS17157I Changing the drive setting to append-only mode.
205e LTFS30392D Backend load xxxx.
205e LTFS30398D Backend readpos: (0, 0) FM = 0 xxxx.
205e LTFS30201D CDB check condition: sense = 052600, Invalid Field in Parameter List.
205e LTFS30205I MODESELECT (0x55) returns -20500.
205e LTFS30263I MODESELECT returns Invalid Field in Parameter List (-20500) /dev/sg0.
205e LTFS17155E Cannot set WRITE MODE: Mode Select for Device Configuration Extension failed (-20500).
205e LTFS14075E Cannot set up tape drive.
205e LTFS12028D Unlocking medium.
205e LTFS30392D Backend allow medium removal xxxx.
205e LTFS12025D Releasing device.
205e LTFS30392D Backend release (PRO) xxxx.
205e LTFS30393D Backend LBP Enable: 0 .
205e LTFS30393D Backend LBP Method: 1 .
205e LTFS30201D CDB check condition: sense = 052600, Invalid Field in Parameter List.
205e LTFS30205I MODESELECT (0x55) returns -20500.
205e LTFS30263I MODESELECT returns Invalid Field in Parameter List (-20500) /dev/sg0.

I thought the cartrige was bad, but i extensively tested it with the HP tools, everything good. If i mount the device without -o eject it works fine:

ltfs -o nonempty -o devname=/dev/sg0 /mnt/ltfs -o trace
4869 LTFS14000I LTFS starting, LTFS version 2.4.2.1 (Prelim), log level 3.
4869 LTFS14058I LTFS Format Specification version 2.4.0.
4869 LTFS14104I Launched by "ltfs -o nonempty -o devname=/dev/sg0 /mnt/ltfs -o trace".
4869 LTFS14105I This binary is built for Linux (x86_64).
4869 LTFS14106I GCC version is 8.3.0.
4869 LTFS17087I Kernel version: Linux version 5.4.55-1-pve (root@nora) (gcc version 8.3.0 (Debian 8.3.0-6)) #1 SMP PVE 5.4.55-1 (Mon, 10 Aug 2020 10:26:27 +0200) i386.
4869 LTFS17089I Distribution: PRETTY_NAME="Debian GNU/Linux 10 (buster)".
4869 LTFS14063I Sync type is "time", Sync time is 300 sec.
4869 LTFS17085I Plugin: Loading "sg" tape backend.
4869 LTFS17085I Plugin: Loading "unified" iosched backend.
4869 LTFS14095I Set the tape device write-anywhere mode to avoid cartridge ejection.
4869 LTFS30209I Opening a device through sg-ibmtape driver (/dev/sg0).
4869 LTFS30250I Opened SCSI tape device 7.0.0.0 (/dev/sg0).
4869 LTFS30207I Vendor ID is HP .
4869 LTFS30208I Product ID is Ultrium 5-SCSI .
4869 LTFS30214I Firmware revision is Z6ED.
4869 LTFS30215I Drive serial is xxxx.
4869 LTFS30285I Reserved buffer size of /dev/sg0 is 1048576.
4869 LTFS12023D Reserving device.
4869 LTFS30392D Backend reserve (PRO) xxxx.
4869 LTFS12028D Unlocking medium.
4869 LTFS30392D Backend allow medium removal xxxx.
4869 LTFS30392D Backend read block limits xxxx.
4869 LTFS17160I Maximum device block size is 1048576.
4869 LTFS11330I Loading cartridge.
4869 LTFS30392D Backend load xxxx.
4869 LTFS30398D Backend readpos: (0, 0) FM = 0 xxxx.
4869 LTFS12026D Locking medium in the drive.
4869 LTFS30392D Backend prevent medium removal xxxx.
4869 LTFS30398D Backend readpos: (0, 0) FM = 0 xxxx.
4869 LTFS30392D Backend read block limits xxxx.
4869 LTFS11332I Load successful.
4869 LTFS17157I Changing the drive setting to write-anywhere mode.
4869 LTFS11005I Mounting the volume.
4869 LTFS11012D Loading the tape.
4869 LTFS30392D Backend load xxxx.
4869 LTFS30398D Backend readpos: (0, 0) FM = 0 xxxx.
4869 LTFS30398D Backend readpos: (0, 0) FM = 0 xxxx.
4869 LTFS30392D Backend read block limits xxxx.
4869 LTFS30397D Backend locate: (0, 0) xxxx.
4869 LTFS30398D Backend readpos: (0, 0) FM = 0 xxxx.
4869 LTFS11007D Tape is loaded.
4869 LTFS11008D Reading partition labels.
4869 LTFS30392D Backend read block limits xxxx.
4869 LTFS30397D Backend locate: (0, 0) xxxx.
4869 LTFS30398D Backend readpos: (0, 0) FM = 0 xxxx.
4869 LTFS30201D CDB check condition: sense = 000001, Filemark Detected.
4869 LTFS30204D READ (0x08) expected error -20004.
4869 LTFS30219D Read block: file mark detected.
4869 LTFS30201D CDB check condition: sense = 000001, Filemark Detected.
4869 LTFS30204D READ (0x08) expected error -20004.
4869 LTFS30219D Read block: file mark detected.
4869 LTFS30392D Backend read block limits xxxx.
4869 LTFS30397D Backend locate: (1, 0) xxxx.
4869 LTFS30398D Backend readpos: (1, 0) FM = 0 xxxx.
4869 LTFS30201D CDB check condition: sense = 000001, Filemark Detected.
4869 LTFS30204D READ (0x08) expected error -20004.
4869 LTFS30219D Read block: file mark detected.
4869 LTFS30201D CDB check condition: sense = 000001, Filemark Detected.
4869 LTFS30204D READ (0x08) expected error -20004.
4869 LTFS30219D Read block: file mark detected.
4869 LTFS30392D Backend read block limits xxxx.
4869 LTFS11013D Partition labels are valid.
4869 LTFS11014D Reading MAM parameters.
4869 LTFS11339D Read volume lock status (0).
4869 LTFS30201D CDB check condition: sense = 052400, Invalid Field in CDB.
4869 LTFS30205I MODESENSE (0x5a) returns -20501.
4869 LTFS30263I MODESENSE returns Invalid Field in CDB (-20501) /dev/sg0.
4869 LTFS11018D Done reading MAM parameters.
4869 LTFS11019D Checking volume consistency.
4869 LTFS30397D Backend locate: (0, 5) xxxx.
4869 LTFS30398D Backend readpos: (0, 5) FM = 3 xxxx.
4869 LTFS30201D CDB check condition: sense = 000000, No Additional Sense Information.
4869 LTFS30204D READ (0x08) expected error -20000.
4869 LTFS30218D Read block: underrun in illegal length. residual = 521786, actual = 2502.
4869 LTFS30201D CDB check condition: sense = 000001, Filemark Detected.
4869 LTFS30204D READ (0x08) expected error -20004.
4869 LTFS30219D Read block: file mark detected.
4869 LTFS30396D Backend space back file marks: 1 xxxx.
4869 LTFS30398D Backend readpos: (0, 6) FM = 3 xxxx.
4869 LTFS30396D Backend space forward file marks: 1 xxxx.
4869 LTFS30398D Backend readpos: (0, 7) FM = 4 xxxx.
4869 LTFS11025D Volume is consistent.
4869 LTFS11028D Consistency check finished.
4869 LTFS17227I Tape attribute: Vendor = IBM .
4869 LTFS17227I Tape attribute: Application Name = LTFS .
4869 LTFS17227I Tape attribute: Application Version = 2.4.2.1 .
4869 LTFS17227I Tape attribute: Medium Label = .
4869 LTFS17228I Tape attribute: Text Localization ID = 0x81.
4869 LTFS17227I Tape attribute: Barcode = .
4869 LTFS17227I Tape attribute: Application Format Version = 2.4.0 .
4869 LTFS17228I Tape attribute: Volume Lock Status = 0x00.
4869 LTFS17227I Tape attribute: Media Pool name = .
4869 LTFS14111I Initial setup completed successfully.
4869 LTFS14112I Invoke 'mount' command to check the result of final setup.
4869 LTFS14113I Specified mount point is listed if succeeded.

I´m using the latest build of the software, can i post anything else to help you locate that bug?

Monitoring TapeAlert while ltfs is active

Is your feature request related to a problem? Please describe.
No

Describe the solution you'd like
I'd like the ability for external software to monitor some of the tape drive's status while it's under control of LTFS. Currently this is impossible as LTFS opens the device with O_EXCL access, blocking secondary access. An API to read status of the drive without interfering with LTFS would be most elegant. I would like to be able to read the TapeAlert flags, to monitor the Eject button.

Describe alternatives you've considered
I reached my goal by changing the _raw_dev_open() function in sg-ibmtape.c to drop the O_EXCL flag. That allows me to open the device a second time and issue Log Sense commands from my software while LTFS is running. LTFS is unaware of this and seems happy, but I do not know for sure if this might interfere anywhere. Potential risk right now is that LTFS or other software could be activated on the same device, so some other means of preventing double access would be needed.

Additional context
I am writing software that uses LTFS tapes as a repository for multiple legacy virtual tape images (www.consolitape.com). The software includes a daemon that monitors the tape drive for tapes to be inserted. When a new tape arrives, the daemon checks if it is an LTFS tape in the MAM chip, and if so, sends a Prevent Media Removal, then calls LTFS to mount the tape. If the mount succeeds, it scans the mounted tape for virtual tape images, and automatically spins up a matching Virtual tape library with Virtual drives and slots that allow read-only access to those images as if they were a real library and drives. The Virtual library is shared through iSCSI and can be accessed by preserved legacy backup applications without requiring the legacy tape hardware or tapes.

To make operator interaction easy, I monitor the TapeAlert flags for an Eject button pushed event. When detected, the Virtual library and drives are automatically taken off-line, the LTFS tape is dismounted, an Allow Media Removal is sent, and the tape is ejected. The operator does need any GUI or command line access.

`ltfs` may crash in printing logs

Description

ltfs may crash when it tries to print dentry name into log. This is the side effect of commit #14
(Issue #4).

Environment

  • OS: All
  • Distribution: All
  • Architecture: All
  • Additional info: None

How to recreate

Try to print LTFS17221W.

The message of this ID includes dentry name. But I changed name structure from char * to struct ltfs_name. So ltfs crashes when it tries to print.

Can't mount a tape

Describe the bug

Index parser returns failure when symlink target tag is longer than 255 characters.

To Reproduce

  1. Create a symlink that the target name is longer than 255 bytes
  2. unmount the tape
  3. mount the tape again

Expected behavior

Do not check the length of symlink that the target. Check only for name.

Can't mount tape.

I installed ltfs software on my PC with IBM LTO-6 drive (ULT3580-TD6).

OS: CentOS Linux release 7.3.1611 (Core)
Kernel: Linux version 3.10.0-514.el7.x86_64

In my case, LTO drive is assigned to /dev/sg2 as blow.

 >ltfs -o device_list
 9a9 LTFS14000I LTFS starting, LTFS version 2.4.1.0 (Prelim), log level 2.
 9a9 LTFS14058I LTFS Format Specification version 2.4.0.
 9a9 LTFS14104I Launched by "ltfs -o device_list".
 9a9 LTFS14105I This binary is built for Linux (x86_64).
 9a9 LTFS14106I GCC version is 4.8.5 20150623 (Red Hat 4.8.5-28).
 9a9 LTFS17087I Kernel version: Linux version 3.10.0-514.el7.x86_64 ([email protected]) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Tue Nov 22 16:42:41 UTC 2016 i386.
 9a9 LTFS17089I Distribution: CentOS Linux release 7.3.1611 (Core) .
 9a9 LTFS17089I Distribution: NAME="CentOS Linux".
 9a9 LTFS17089I Distribution: CentOS Linux release 7.3.1611 (Core) .
 9a9 LTFS17089I Distribution: CentOS Linux release 7.3.1611 (Core) .
 9a9 LTFS17085I Plugin: Loading "sg" tape backend.
 Tape Device list:.
 Device Name = /dev/sg2, Vender ID = IBM     , Product ID = ULT3580-TD6     , Serial Number = 1068007767, Product Name =[ULT3580-TD6].

I could format the LTO tape successfully, but mounting the tape was failed as following.

[[ Tape Format ]]

 >mkltfs -f -d /dev/sg2
 LTFS15000I Starting mkltfs, LTFS version 2.4.1.0 (Prelim), log level 2.
 LTFS15041I Launched by "mkltfs -f -d /dev/sg2".
 LTFS15042I This binary is built for Linux (x86_64).
 LTFS15043I GCC version is 4.8.5 20150623 (Red Hat 4.8.5-28).
 LTFS17087I Kernel version: Linux version 3.10.0-514.el7.x86_64 ([email protected]) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Tue Nov 22 16:42:41 UTC 2016 i386.
 LTFS17089I Distribution: CentOS Linux release 7.3.1611 (Core) .
 LTFS17089I Distribution: NAME="CentOS Linux".
 LTFS17089I Distribution: CentOS Linux release 7.3.1611 (Core) .
 LTFS17089I Distribution: CentOS Linux release 7.3.1611 (Core) .
 LTFS15003I Formatting device '/dev/sg2'.
 LTFS15004I LTFS volume blocksize: 524288.
 LTFS15005I Index partition placement policy: None.
 LTFS11337I Update index-dirty flag (1) - NO_BARCODE (0x0x257cdd0).
 LTFS17085I Plugin: Loading "sg" tape backend.
 LTFS30209I Opening a device through sg-ibmtape driver (/dev/sg2).
 LTFS30207I Vendor ID is IBM     .
 LTFS30208I Product ID is ULT3580-TD6     .
 LTFS30214I Firmware revision is C9T4.
 LTFS30215I Drive serial is 1068007767.
 LTFS17160I Maximum device block size is 1048576.
 LTFS11330I Loading cartridge.
 LTFS30252I Logical block protection is disabled.
 LTFS11332I Load successful.
 LTFS17157I Changing the drive setting to write-anywhere mode.
 LTFS15049I Checking the medium (load).
 LTFS30252I Logical block protection is disabled.
 LTFS15010I Creating data partition b on SCSI partition 1.
 LTFS15011I Creating index partition a on SCSI partition 0.
 LTFS17165I Resetting the medium's capacity proportion.
 LTFS11097I Partitioning the medium.
 LTFS30263I MODESELECT returns Mode Parameters Rounded (-20101) /dev/sg2.
 LTFS11100I Writing label to partition b.
 LTFS11278I Writing index to partition b.
 LTFS17235I Writing index of NO_BARCODE to b (Reason: Format, 0 files) 1068007767.
 LTFS17236I Wrote index of NO_BARCODE (b, 1068007767).
 LTFS11337I Update index-dirty flag (0) - NO_BARCODE (0x0x257cdd0).
 LTFS11100I Writing label to partition a.
 LTFS11278I Writing index to partition a.
 LTFS17235I Writing index of NO_BARCODE to a (Reason: Format, 0 files) 1068007767.
 LTFS17236I Wrote index of NO_BARCODE (a, 1068007767).
 LTFS15013I Volume UUID is: 93c85bf5-c9a3-4a81-9e37-7601699c4311.
 LTFS15019I Volume capacity is 2408 GB.
 LTFS30252I Logical block protection is disabled.
 LTFS15024I Medium formatted successfully.

[[ Mount ]]

 ltfs -o devname=/dev/sg2 /mnt/ltfs6
 9ad LTFS14000I LTFS starting, LTFS version 2.4.1.0 (Prelim), log level 2.
 9ad LTFS14058I LTFS Format Specification version 2.4.0.
 9ad LTFS14104I Launched by "ltfs -o devname=/dev/sg2 /mnt/ltfs6".
 9ad LTFS14105I This binary is built for Linux (x86_64).
 9ad LTFS14106I GCC version is 4.8.5 20150623 (Red Hat 4.8.5-28).
 9ad LTFS17087I Kernel version: Linux version 3.10.0-514.el7.x86_64 ([email protected]) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Tue Nov 22 16:42:41 UTC 2016 i386.
 9ad LTFS17089I Distribution: CentOS Linux release 7.3.1611 (Core) .
 9ad LTFS17089I Distribution: NAME="CentOS Linux".
 9ad LTFS17089I Distribution: CentOS Linux release 7.3.1611 (Core) .
 9ad LTFS17089I Distribution: CentOS Linux release 7.3.1611 (Core) .
 9ad LTFS14063I Sync type is "time", Sync time is 300 sec.
 9ad LTFS17085I Plugin: Loading "sg" tape backend.
 9ad LTFS17085I Plugin: Loading "unified" iosched backend.
 9ad LTFS14095I Set the tape device write-anywhere mode to avoid cartridge ejection.
 9ad LTFS30209I Opening a device through sg-ibmtape driver (/dev/sg2).
 9ad LTFS30207I Vendor ID is IBM     .
 9ad LTFS30208I Product ID is ULT3580-TD6     .
 9ad LTFS30214I Firmware revision is C9T4.
 9ad LTFS30215I Drive serial is 1068007767.
 9ad LTFS17160I Maximum device block size is 1048576.
 9ad LTFS11330I Loading cartridge.
 9ad LTFS30252I Logical block protection is disabled.
 9ad LTFS11332I Load successful.
 9ad LTFS17157I Changing the drive setting to write-anywhere mode.
 9ad LTFS11005I Mounting the volume.
 9ad LTFS30252I Logical block protection is disabled.
 9ad LTFS30216W Length mismatch is detected. (Act = 4096, resid = 0, resid_sense = 4016).
 9ad LTFS12049E Cannot read: backend call failed (-21716).
 9ad LTFS11174E Cannot read ANSI label: read failed (-21716).
 9ad LTFS11170E Failed to read label (-21716) from partition 0.
 9ad LTFS11009E Cannot read volume: failed to read partition labels.
 9ad LTFS14013E Cannot mount the volume.
 9ad LTFS30252I Logical block protection is disabled.

What is “Cannot read: backend call failed (-21716).”? I feel something occurs in tape driver, but I do not know.
I appreciate very much if somebody could help and advise me.

best regards

ltfs_write_index() does not let the caller know that the volume's tc_coherency structs are stale

After writing the index XML to the tape, failure to read the VCR MAM attribute or write the VCI MAM attribute prevents the ip_coh and dp_coh from being updated, but the errors are not propagated up so the caller has no idea that the these structs are stale and should not be trusted. Inside the LTFS fuse application this isn't a large problem, but for libltfs users it can be. It would be better if the return value for ltfs_write_index() could contain a flag indicating the coherency structs are stale or communicate through the coherency struct or some other channel that the data is stale and should not be used.

Problems mounting tapes

All of a sudden, I'm having trouble mounting both of my tapes (which mounted before successfully). This is what I'm seeing:

Sep  9 07:34:15 alice2 ltfs[25588]: 63f4 LTFS14000I LTFS starting, LTFS version 2.4.1.1 (10227), log level 3.
Sep  9 07:34:15 alice2 ltfs[25588]: 63f4 LTFS14058I LTFS Format Specification version 2.4.0.
Sep  9 07:34:15 alice2 ltfs[25588]: 63f4 LTFS14104I Launched by "ltfs -o devname=/dev/sg3 /opt/VS/vsMounts/000025L7 -o trace".
Sep  9 07:34:15 alice2 ltfs[25588]: 63f4 LTFS14105I This binary is built for Linux (x86_64).
Sep  9 07:34:15 alice2 ltfs[25588]: 63f4 LTFS14106I GCC version is 6.3.0 20170516.
Sep  9 07:34:15 alice2 ltfs[25588]: 63f4 LTFS17087I Kernel version: Linux version 4.20.11 (root@debian) (gcc version 6.3.0 20170516 (Debian 6.3.0-18+deb9u1)) #1 SMP Thu Feb 21 09:13:14 EST 2019 i386.
Sep  9 07:34:15 alice2 ltfs[25588]: 63f4 LTFS17089I Distribution: PRETTY_NAME="Debian GNU/Linux 9 (stretch)".
Sep  9 07:34:15 alice2 ltfs[25588]: 63f4 LTFS14063I Sync type is "time", Sync time is 300 sec.
Sep  9 07:34:15 alice2 ltfs[25588]: 63f4 LTFS17085I Plugin: Loading "sg" tape backend.
Sep  9 07:34:15 alice2 ltfs[25588]: 63f4 LTFS17085I Plugin: Loading "unified" iosched backend.
Sep  9 07:34:15 alice2 ltfs[25588]: 63f4 LTFS14095I Set the tape device write-anywhere mode to avoid cartridge ejection.
Sep  9 07:34:15 alice2 ltfs[25588]: 63f4 LTFS30209I Opening a device through sg-ibmtape driver (/dev/sg3).
Sep  9 07:34:15 alice2 ltfs[25588]: 63f4 LTFS30207I Vendor ID is IBM     .
Sep  9 07:34:15 alice2 ltfs[25588]: 63f4 LTFS30208I Product ID is ULTRIUM-HH7     .
Sep  9 07:34:15 alice2 ltfs[25588]: 63f4 LTFS30214I Firmware revision is JAY1.
Sep  9 07:34:15 alice2 ltfs[25588]: 63f4 LTFS30215I Drive serial is 10WT074468.
Sep  9 07:34:15 alice2 ltfs[25588]: 63f4 LTFS17160I Maximum device block size is 1048576.
Sep  9 07:34:15 alice2 ltfs[25588]: 63f4 LTFS11330I Loading cartridge.
Sep  9 07:34:15 alice2 ltfs[25588]: 63f4 LTFS30252I Logical block protection is disabled.
Sep  9 07:34:15 alice2 ltfs[25588]: 63f4 LTFS11332I Load successful.
Sep  9 07:34:15 alice2 ltfs[25588]: 63f4 LTFS17157I Changing the drive setting to write-anywhere mode.
Sep  9 07:34:15 alice2 ltfs[25588]: 63f4 LTFS11005I Mounting the volume.
Sep  9 07:34:15 alice2 ltfs[25588]: 63f4 LTFS30252I Logical block protection is disabled.
Sep  9 07:34:21 alice2 ltfs[25588]: 63f4 LTFS30216W Length mismatch is detected. (Act = 4096, resid = 0, resid_sense = -520192).
Sep  9 07:34:21 alice2 ltfs[25588]: 63f4 LTFS30216W Length mismatch is detected. (Act = 4096, resid = 0, resid_sense = -520192).
Sep  9 07:34:21 alice2 ltfs[25588]: 63f4 LTFS12049E Cannot read: backend call failed (-21716).
Sep  9 07:34:21 alice2 ltfs[25588]: 63f4 LTFS11174E Cannot read ANSI label: read failed (-21716).
Sep  9 07:34:21 alice2 ltfs[25588]: 63f4 LTFS11171E Failed to read label (-21716) from partition 1.
Sep  9 07:34:21 alice2 ltfs[25588]: 63f4 LTFS11009E Cannot read volume: failed to read partition labels.
Sep  9 07:34:21 alice2 ltfs[25588]: 63f4 LTFS14013E Cannot mount the volume.
Sep  9 07:34:21 alice2 ltfs[25588]: 63f4 LTFS30252I Logical block protection is disabled.

It doesn't look like --enable-buggy-ifs would help in this case as it seems to be an overrun.

HBA is:

06:08.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1068 PCI-X Fusion-MPT SAS (rev 01)

How to trigger the finishd umount process?

Is your feature request related to a problem? Please describe.
I have a script like umount_ltfs.sh:
umount /mnt/ltfs
mt -f /dev/st0 eject

which will fail to eject the tape on big data sets written because the umount command returns but the backend is not finishd, just like you describe it in the wiki "One thing you need to pay attention here is it is not a unmount completion when umount command is returned."

Describe the solution you'd like
"Actual unmount is completed when the ltfs command is finished."
How to watch for this? Some process is alive? A lockfile? Do the sg driver tell when it´s finishd?

Additional context
System is Debian 10 with a HP LTO5 Drive.

Need to clip preior FW level in IBM LTO8 drive

Description

IBM LTO8 drive initial FW does not support M8 tape completely.

The problem is the initial FW does change the density code to L7 when M8 tape is fomrtted by LTFS. It is not a LTFS problem but a FW problem and IBM LTO8 drive team will new FW level to fix this problem.

For preserve M8 tape density, I decided to clip the initial level of FW in LTO8.

Environment

  • OS: All
  • Distribution: All
  • Architecture: All
  • Additional info: None

How to recreate

  1. Prepare M8 formatted tape
  2. Prepare IBM's first firmware for LTO8
  3. Format the tape with mkltfs
  4. confirm the capacity

is there any ltfs software that works for Ubuntu-18.04?

I am having trouble formatting tape on an ULTRIUM-HH8 tape drive attached to a computer having UBUNTU-18.04. Error is as follows:

sudo mkltfs -d /dev/sg1
LTFS15000I Starting mkltfs, LTFS version 2.4.2.0 (Prelim), log level 2.
LTFS15041I Launched by "mkltfs -d /dev/sg1".
LTFS15042I This binary is built for Linux (x86_64).
LTFS15043I GCC version is 7.4.0.
LTFS17087I Kernel version: Linux version 4.15.0-88-generic (buildd@lgw01-amd64-036) (gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1)) #88-Ubuntu SMP Tue Feb 11 20:11:34 UTC 2020 i386.
LTFS17089I Distribution: NAME="Ubuntu".
LTFS17089I Distribution: DISTRIB_ID=Ubuntu.
LTFS15003I Formatting device '/dev/sg1'.
LTFS15004I LTFS volume blocksize: 524288.
LTFS15005I Index partition placement policy: None.

LTFS11337I Update index-dirty flag (1) - NO_BARCODE (0x0x561e60511c10).
LTFS17085I Plugin: Loading "sg" tape backend.
LTFS30209I Opening a device through sg-ibmtape driver (/dev/sg1).
LTFS30213I Unsupported Drive 'ULTRIUM-HH8 '.
Segmentation fault

mkltfs fail if lock status is unavailable

Describe the bug
If lock status attribute is not available from the tape, mkltfs fill fail with error 11341E

To Reproduce
mkltfs /dev/nst0
I do not know why this attribute happens to be missing at mine, though.

Expected behavior
Format should succeed whatever prior content exists on tape.

Screenshots
n/a

Desktop (please complete the following information):
n/a

Smartphone (please complete the following information):
n/a

Additional context
That bug is worked around in HPE LTFS by using a fake lock status during mkltfs operation. A pull request is coming soon to fix this issue.

Nonlinear performance scaling when writing to multiple drives

Over the past several years I have developed software for my employer for the purposes of high-performance tar and LTFS tape reading and writing for LTO tape drives. This software runs primarily under MacOS but we also support some Linux installations as well. This software is used in large-scale media production, and it's not uncommon for us to write 10-20 terabytes of source content per day on LTO 6/7/8 tapes for periods of a month or more. We have observed a non-linear performance scaling behavior when writing to many tape drives at the same time and would like to understand and eliminate this phenomenon to maximize our tape write performance. We speculate that there may be some underlying architectural feature of the LTFS driver which is causing the performance degradation, because all of the other pieces of our system should be able to support the total aggregate bandwidth.

I have attached a spreadsheet which shows some performance numbers from a recent test series. These were taken on a system with 4 LTO6 drives. We typically write to mirrored (duplicate) tape sets. Our software works by having a Reader thread which reads data from a filesystem and places buffers of this input data into separate queues, one per tape drive. There is also one Writer thread per tape drive, which dequeues these buffers and writes the data to the LTFS filesystem via POSIX C open/write functions. OS filesystem caching is disabled (on both the read and write sides) to avoid polluting the RAM with unnecessary disk cache data.

It is understandable that the performance will go down when comparing the 1-drive case to the 2-drive (mirrored set) case. Each individual drive will naturally have a varying write speed, due to the physical characteristics of the drive and defects in the tape medium. When writing a mirrored set, we are coupling the drives together because we only read the data once (and the Writer queue size is finite), causing the the overall streaming speed to be limited to the minimum of the speeds of the 2 drives (within a certain window). It's like you have a 2-car train; one car cannot go faster than the other. To decouple the drive performance as much as possible and prevent "rubber-banding", our Writer queue size is very large (1GB per drive). The attached spreadsheet shows that we are achieving 150 MB/s average with a single drive, but only 133 MB/s with 2 drives in a mirrored configuration. This is expected.

The unexpected result is the further performance decrease when we write to 4 drives. In this case, we are writing 2 mirrored sets, and each mirrored set is a totally independent operation with its own threads. Our Mac Pro machines are 12-core 2.8 GHz with 64GB of RAM, and the filesystems from which we're reading the data are capable of 600 - 1500 MB/s of read bandwidth, so the input side should not be a problem. You can see from the results that one mirrored set (Group 02) is writing at a nearly nominal rate (115-155 MB/s) but the other is much more degraded (85 - 111 MB/s). After Group 02 finishes, the slow Group 01 goes much faster (the last 2 packages are 151-159 MB/s).

Can you think of any shared resources in the LTFS or FUSE drivers could explain this performance issue?
Single_Double_Quad_Tape_Test.xlsx

LTO 7 mounting sporadically not possible

Description

After formating the Tape, mounting is not possible, i always get the following error ,
6a0 LTFS11332I Load successful.
6a0 LTFS17157I Changing the drive setting to write-anywhere mode.
6a0 LTFS11005I Mounting the volume.
6a0 LTFS30252I Logical block protection is disabled.
6a0 LTFS30200I Failed to execute SG_IO ioctl, opcode = 08 (22).
6a0 LTFS30263I READ returns ioctl error (-21700) /dev/st0.
6a0 LTFS12049E Cannot read: backend call failed (-21700).
6a0 LTFS17041E XML parser: read failed while looking for a file mark (-21700).
6a0 LTFS17037E XML parser: failed to read from XML stream.
6a0 LTFS17016E Cannot parse index direct from medium.
6a0 LTFS11194W Cannot read index: failed to read and parse XML data (-1014).
6a0 LTFS11024E Cannot mount volume: read index failed on the index partition.
6a0 LTFS14013E Cannot mount the volume.
6a0 LTFS30252I Logical block protection is disabled.

upd: To ensure that the Hardware isn't broken i checked it under Windows 2012 Server R2, everything works very well.

Environment

  • OS: Linux
  • Distribution: Ubuntu 16.04 LTS
  • Architecture: x86_64 HP-Microserver Gen8 & HP H222 Controller
  • Additional info: -

How to recreate

LBP capabilities not detected for LTO8

When attempting to set the LBP for an LTO8 drive:

sg_ibmtape.c _set_lbp method

it checks for capability and determines that it doesn't support CRC32C, so it defaults to Reed Solomon CRC. But it does support CRC32C...

Cannot write more than 524288 bytes: LTFS30200I Failed to execute SG_IO ioctl, opcode = 0a (22)

After formatting and mounting tape, I can only write the first 524288 bytes block of the first file. Then the write fails.

ltfs was compiled with ./configure --enable-buggy-ifs.

# ltfs -o device_list
737e LTFS14000I LTFS starting, LTFS version 2.4.2.0 (Prelim), log level 2.
737e LTFS14058I LTFS Format Specification version 2.4.0.
737e LTFS14104I Launched by "ltfs -o device_list".
737e LTFS14105I This binary is built for Linux (x86_64).
737e LTFS14106I GCC version is 4.8.5 20150623 (Red Hat 4.8.5-36).
737e LTFS17087I Kernel version: Linux version 3.10.0-957.21.3.el7.x86_64 ([email protected]) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-36) (GCC) ) #1 SMP Tue Jun 18 16:35:19 UTC 2019 i386.
737e LTFS17089I Distribution: CentOS Linux release 7.6.1810 (Core) .
737e LTFS17089I Distribution: NAME="CentOS Linux".
737e LTFS17089I Distribution: CentOS Linux release 7.6.1810 (Core) .
737e LTFS17089I Distribution: CentOS Linux release 7.6.1810 (Core) .
737e LTFS17085I Plugin: Loading "sg" tape backend.
Tape Device list:.
Device Name = /dev/sg5, Vendor ID = IBM     , Product ID = ULTRIUM-HH7     , Serial Number = 10WT064673, Product Name =[ULTRIUM-HH7].

Formatting works:

# mkltfs -d /dev/sg5 -s 18L013 -n 18L013L6
LTFS15000I Starting mkltfs, LTFS version 2.4.2.0 (Prelim), log level 2.
LTFS15041I Launched by "mkltfs -d /dev/sg5 -s 18L013 -n 18L013L6".
LTFS15042I This binary is built for Linux (x86_64).
LTFS15043I GCC version is 4.8.5 20150623 (Red Hat 4.8.5-36).
LTFS17087I Kernel version: Linux version 3.10.0-957.21.3.el7.x86_64 ([email protected]) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-36) (GCC) ) #1 SMP Tue Jun 18 16:35:19 UTC 2019 i386.
LTFS17089I Distribution: CentOS Linux release 7.6.1810 (Core) .
LTFS17089I Distribution: NAME="CentOS Linux".
LTFS17089I Distribution: CentOS Linux release 7.6.1810 (Core) .
LTFS17089I Distribution: CentOS Linux release 7.6.1810 (Core) .
LTFS15003I Formatting device '/dev/sg5'.
LTFS15004I LTFS volume blocksize: 524288.
LTFS15005I Index partition placement policy: None.

LTFS11337I Update index-dirty flag (1) - 18L013 (0x0x1804020).
LTFS17085I Plugin: Loading "sg" tape backend.
LTFS30209I Opening a device through sg-ibmtape driver (/dev/sg5).
LTFS30207I Vendor ID is IBM     .
LTFS30208I Product ID is ULTRIUM-HH7     .
LTFS30214I Firmware revision is J4D1.
LTFS30215I Drive serial is 10WT064673.
LTFS30285I Reserved buffer size of /dev/sg5 is 524288.
LTFS30205I TEST_UNIT_READY (0x00) returns -20601.
LTFS17160I Maximum device block size is 1048576.
LTFS11330I Loading cartridge.
LTFS30252I Logical block protection is disabled.
LTFS11332I Load successful.
LTFS17157I Changing the drive setting to write-anywhere mode.
LTFS15049I Checking the medium (mount).
LTFS30252I Logical block protection is disabled.
LTFS15010I Creating data partition b on SCSI partition 1.
LTFS15011I Creating index partition a on SCSI partition 0.
LTFS17165I Resetting the medium's capacity proportion.
LTFS11097I Partitioning the medium.
LTFS30263I MODESELECT returns Mode Parameters Rounded (-20101) /dev/sg5.
LTFS11100I Writing label to partition b.
LTFS11278I Writing index to partition b.
LTFS30205I READ_ATTR (0x8c) returns -20501.
LTFS30263I READ_ATTR returns Invalid Field in CDB (-20501) /dev/sg5.
LTFS30233I Cannot read attribute (-20501).
LTFS11336I The attribute does not exist. Ignore the expected error.
LTFS17235I Writing index of 18L013 to b (Reason: Format, 0 files) 10WT064673.
LTFS17236I Wrote index of 18L013 (b, 10WT064673).
LTFS11337I Update index-dirty flag (0) - 18L013 (0x0x1804020).
LTFS11100I Writing label to partition a.
LTFS11278I Writing index to partition a.
LTFS30205I READ_ATTR (0x8c) returns -20501.
LTFS30263I READ_ATTR returns Invalid Field in CDB (-20501) /dev/sg5.
LTFS30233I Cannot read attribute (-20501).
LTFS11336I The attribute does not exist. Ignore the expected error.
LTFS17235I Writing index of 18L013 to a (Reason: Format, 0 files) 10WT064673.
LTFS17236I Wrote index of 18L013 (a, 10WT064673).
LTFS15013I Volume UUID is: 9e1c2519-76a8-40df-bf7d-bf03fa27e431.

LTFS15019I Volume capacity is 2408 GB.
LTFS30252I Logical block protection is disabled.
LTFS15024I Medium formatted successfully.

Mounting works (well, sometimes it doesn't, but maybe that should be a separate issue). Successful mount log:

# ltfs -o devname=/dev/sg5 /mnt/ltfs
7bc5 LTFS14000I LTFS starting, LTFS version 2.4.2.0 (Prelim), log level 2.
7bc5 LTFS14058I LTFS Format Specification version 2.4.0.
7bc5 LTFS14104I Launched by "ltfs -o devname=/dev/sg5 /mnt/ltfs".
7bc5 LTFS14105I This binary is built for Linux (x86_64).
7bc5 LTFS14106I GCC version is 4.8.5 20150623 (Red Hat 4.8.5-36).
7bc5 LTFS17087I Kernel version: Linux version 3.10.0-957.21.3.el7.x86_64 ([email protected]) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-36) (GCC) ) #1 SMP Tue Jun 18 16:35:19 UTC 2019 i386.
7bc5 LTFS17089I Distribution: CentOS Linux release 7.6.1810 (Core) .
7bc5 LTFS17089I Distribution: NAME="CentOS Linux".
7bc5 LTFS17089I Distribution: CentOS Linux release 7.6.1810 (Core) .
7bc5 LTFS17089I Distribution: CentOS Linux release 7.6.1810 (Core) .
7bc5 LTFS14063I Sync type is "time", Sync time is 300 sec.
7bc5 LTFS17085I Plugin: Loading "sg" tape backend.
7bc5 LTFS17085I Plugin: Loading "unified" iosched backend.
7bc5 LTFS14095I Set the tape device write-anywhere mode to avoid cartridge ejection.
7bc5 LTFS30209I Opening a device through sg-ibmtape driver (/dev/sg5).
7bc5 LTFS30207I Vendor ID is IBM     .
7bc5 LTFS30208I Product ID is ULTRIUM-HH7     .
7bc5 LTFS30214I Firmware revision is J4D1.
7bc5 LTFS30215I Drive serial is 10WT064673.
7bc5 LTFS30285I Reserved buffer size of /dev/sg5 is 524288.
7bc5 LTFS17160I Maximum device block size is 1048576.
7bc5 LTFS11330I Loading cartridge.
7bc5 LTFS30252I Logical block protection is disabled.
7bc5 LTFS11332I Load successful.
7bc5 LTFS17157I Changing the drive setting to write-anywhere mode.
7bc5 LTFS11005I Mounting the volume.
7bc5 LTFS30252I Logical block protection is disabled.
7bc5 LTFS17227I Tape attribute: Vendor = IBM     .
7bc5 LTFS17227I Tape attribute: Application Name = LTFS                            .
7bc5 LTFS17227I Tape attribute: Application Version = 2.4.2.0 .
7bc5 LTFS17227I Tape attribute: Medium Label = 18L013L6.
7bc5 LTFS17228I Tape attribute: Text Localization ID = 0x81.
7bc5 LTFS17227I Tape attribute: Barcode = 18L013                          .
7bc5 LTFS17227I Tape attribute: Application Format Version = 2.4.0           .
7bc5 LTFS17228I Tape attribute: Volume Lock Status = 0x00.
7bc5 LTFS17227I Tape attribute: Media Pool name = .
7bc5 LTFS14111I Initial setup completed successfully.
7bc5 LTFS14112I Invoke 'mount' command to check the result of final setup.
7bc5 LTFS14113I Specified mount point is listed if succeeded.

# mount | grep ltfs
ltfs:/dev/sg5 on /mnt/ltfs type fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other)

But copying to the tape fails after 524288 bytes. /var/log/ltfs.log shows this:

Jul  9 11:51:36 zukini ltfs[31718]: 7be7 LTFS14029I Ready to receive file system requests.
Jul  9 11:54:46 zukini ltfs[31718]: 7be7 LTFS11337I Update index-dirty flag (1) - 18L013 (0x0x2208de0).
Jul  9 11:54:52 zukini ltfs[31718]: 7bea LTFS30200I Failed to execute SG_IO ioctl, opcode = 0a (22).
Jul  9 11:54:52 zukini ltfs[31718]: 7bea LTFS30263I WRITE returns ioctl error (-21700) /dev/sg5.
Jul  9 11:54:52 zukini ltfs[31718]: 7bea LTFS12045E Cannot write block: backend call failed (-21700). Dropping to read-only mode.
Jul  9 11:54:52 zukini ltfs[31718]: 7bea LTFS11072E Cannot write blocks: failed to write to the medium (-21700).
Jul  9 11:54:52 zukini ltfs[31718]: 7bea LTFS11077E Cannot write: failed to write blocks to the medium (-21700).
Jul  9 11:54:52 zukini ltfs[31718]: 7bea LTFS13014W Data partition writer: failed to write data to the tape (-21700).
Jul  9 11:55:06 zukini ltfs[31718]: 7be7 LTFS10020E Error on chmod: /Dynamit_Audio_51/Audio_Final_51/VonMagnetix_51_DE_190322 (16877, 0).
Jul  9 11:55:06 zukini ltfs[31718]: 7ca6 LTFS10020E Error on utimens: /Dynamit_Audio_51/Audio_Final_51/VonMagnetix_51_DE_190322 (0, 0).
Jul  9 11:55:06 zukini ltfs[31718]: 7be8 LTFS10020E Error on chmod: /Dynamit_Audio_51/Audio_Final_51 (16877, 0).
Jul  9 11:55:06 zukini ltfs[31718]: 7be7 LTFS10020E Error on utimens: /Dynamit_Audio_51/Audio_Final_51 (0, 0).
Jul  9 11:56:36 zukini ltfs[31718]: 7beb LTFS11030I Failed to sync volume (-1126). Stop the periodic sync thread.

Use of deprecated ICU functions

Description

LTFS uses deprecated ICU functions for path normalization checks. For instance:

pathname.c: In function ‘_pathname_normalize_nfc_icu’:
pathname.c:661:2: warning: ‘unorm_quickCheck_60’ is deprecated [-Wdeprecated-declarations]
  if (unorm_quickCheck(src, -1, UNORM_NFC, &err) == UNORM_YES) {
  ^~
In file included from /usr/local/include/unicode/platform.h:25:0,
                 from /usr/local/include/unicode/ptypes.h:52,
                 from /usr/local/include/unicode/umachine.h:46,
                 from /usr/local/include/unicode/utypes.h:38,
                 from /usr/local/include/unicode/uchar.h:28,
                 from pathname.c:62:
/usr/local/include/unicode/unorm.h:240:1: note: declared here
 unorm_quickCheck(const UChar *source, int32_t sourcelength,
 ^

Using the alternative functions declared in unorm2.h seems to be the best approach moving forward.

Environment

  • OS: Linux
  • Distribution: GoboLinux 016.01
  • Architecture: x86_64
  • Additional info: LibICU4C 60.2

How to recreate

Compile LTFS with a recent version of LibICU4C (such as 60.2).

Unable to mount the tape when Logical Block Protection is enabled

Describe the bug
LTFS cannot mount a tape when -o scsi_lbprotect=on is specified under the sg backend.

To Reproduce

Start LTFS with -o scsi_lbprotect=on option

Additional context

2019-06-26T16:54:07.651992+09:00 ltfs204 ltfs[16040]: 591b LTFS14814I Mounting cartridge MB0348JE.
2019-06-26T16:54:07.652662+09:00 ltfs204 ltfs[16040]: 591b LTFS14812I Moving cartridge MB0348JE from 1053 to 262 - 0000078AA1260407 .
2019-06-26T16:54:07.652826+09:00 ltfs204 ltfs[16040]: 591b LTFS11629I Aqcuireing MoveLock - 0000078AA1260407.
2019-06-26T16:54:07.652980+09:00 ltfs204 ltfs[16040]: 591b LTFS11630I Aqcuired Move Lock (17) - 0000078AA1260407.
2019-06-26T16:54:12.155137+09:00 ltfs204 ltfs[16040]: 591b LTFS14813I Moved cartridge MB0348JE to 262 successfully - 0000078AA1260407.
2019-06-26T16:54:12.155380+09:00 ltfs204 ltfs[16040]: 591b LTFS11635I Link the drive 0000078PG16B to the cartridge MB0348JE.
2019-06-26T16:54:12.155740+09:00 ltfs204 ltfs[16040]: 591b LTFS11635I Link the drive 0000078PG16B to the cartridge MB0348JE.
2019-06-26T16:54:12.155933+09:00 ltfs204 ltfs[16040]: 591b LTFS11330I Loading cartridge.
2019-06-26T16:54:22.806317+09:00 ltfs204 ltfs[16040]: 591b LTFS30205I LOAD_UNLOAD (0x1b) returns -20601.
2019-06-26T16:54:22.816513+09:00 ltfs204 ltfs[16040]: 591b LTFS30251I Logical block protection is enabled.
2019-06-26T16:54:22.835110+09:00 ltfs204 ltfs[16040]: 591b LTFS11332I Load successful.
2019-06-26T16:54:22.838368+09:00 ltfs204 ltfs[16040]: 591b LTFS11746I Mounting MB0348JE on 0000078PG16B.
2019-06-26T16:54:22.838583+09:00 ltfs204 ltfs[16040]: 591b LTFS11330I Loading cartridge.
2019-06-26T16:54:22.851163+09:00 ltfs204 ltfs[16040]: 591b LTFS30251I Logical block protection is enabled.
2019-06-26T16:54:22.861295+09:00 ltfs204 ltfs[16040]: 591b LTFS11332I Load successful.
2019-06-26T16:54:22.872540+09:00 ltfs204 ltfs[16040]: 591b LTFS17157I Changing the drive setting to append-only mode.
2019-06-26T16:54:27.076548+09:00 ltfs204 ltfs[16040]: 591b LTFS11508I Mounting the volume from cache index.
2019-06-26T16:54:31.295102+09:00 ltfs204 ltfs[16040]: 591b LTFS30251I Logical block protection is enabled.
2019-06-26T16:54:31.319850+09:00 ltfs204 ltfs[16040]: 591b LTFS11743I Reading index of MB0348JE from volume cache (801 files).
2019-06-26T16:54:31.335991+09:00 ltfs204 ltfs[16040]: 591b LTFS11745I Read index of MB0348JE from volume cache.
2019-06-26T16:54:31.336178+09:00 ltfs204 ltfs[16040]: 591b LTFS11509I Volume mounted successfully from cache index.
2019-06-26T16:54:32.705652+09:00 ltfs204 ltfs[16040]: 591b LTFS39803E CRC check failed: Len = 4096, Actual CRC = 0f74734c, Expected CRC = 0080a069.
2019-06-26T16:54:32.705866+09:00 ltfs204 ltfs[16040]: 591b LTFS30221E Logical block protection Error on read.
2019-06-26T16:54:32.706093+09:00 ltfs204 ltfs[16040]: 591b LTFS30262I Forcing drive dump.
2019-06-26T16:54:51.808055+09:00 ltfs204 ltfs[16040]: 591b LTFS30253I Saving drive dump to /tmp/ltfs_0000078PG16B_2019_0626_165432_f.dmp.
2019-06-26T16:54:52.181495+09:00 ltfs204 ltfs[16040]: 591b LTFS12049E Cannot read: backend call failed (-20402).
2019-06-26T16:54:52.181718+09:00 ltfs204 ltfs[16040]: 591b LTFS11174E Cannot read ANSI label: read failed (-20402).
2019-06-26T16:54:52.183584+09:00 ltfs204 ltfs[16040]: 591b LTFS11575I Drive 0x106 detects an error and becomes unavailable.
2019-06-26T16:54:52.185139+09:00 ltfs204 ltfs[16040]: 591b LTFS12022I Unloading medium.
2019-06-26T16:55:23.810222+09:00 ltfs204 ltfs[16040]: 591b LTFS30205I TEST_UNIT_READY (0x00) returns -20209.
2019-06-26T16:55:23.810479+09:00 ltfs204 ltfs[16040]: 591b LTFS30205I TEST_UNIT_READY (0x00) returns -20209.
2019-06-26T16:55:23.810874+09:00 ltfs204 ltfs[16040]: 591b LTFS30205I TEST_UNIT_READY (0x00) returns -20209.
2019-06-26T16:55:23.823892+09:00 ltfs204 ltfs[16040]: 591b LTFS14816E Failed to mount cartridge MB0348JE (-20402).
2019-06-26T16:55:23.824123+09:00 ltfs204 ltfs[16040]: 591b LTFSD0402E Logical Block Protection read error.

I'm seeing a similar error message ("Failed to execute SG_IO ioctl, opcode = 0a (22)."), but I'm not sure if I'm seeing the same root problem or not.

I'm seeing a similar error message ("Failed to execute SG_IO ioctl, opcode = 0a (22)."), but I'm not sure if I'm seeing the same root problem or not.

Sep  3 11:48:44 alice2 ltfs[25370]: 631a LTFS14000I LTFS starting, LTFS version 2.4.1.1 (10227), log level 3.
Sep  3 11:48:44 alice2 ltfs[25370]: 631a LTFS14058I LTFS Format Specification version 2.4.0.
Sep  3 11:48:44 alice2 ltfs[25370]: 631a LTFS14104I Launched by "ltfs -o devname=/dev/st0 /opt/VS/vsMounts/000025L7 -o trace".
Sep  3 11:48:44 alice2 ltfs[25370]: 631a LTFS14105I This binary is built for Linux (x86_64).
Sep  3 11:48:44 alice2 ltfs[25370]: 631a LTFS14106I GCC version is 6.3.0 20170516.
Sep  3 11:48:44 alice2 ltfs[25370]: 631a LTFS17087I Kernel version: Linux version 4.20.11 (root@debian) (gcc version 6.3.0 20170516 (Debian 6.3.0-18+deb9u1)) #1 SMP Thu Feb 21 09:13:14 EST 2019 i386.
Sep  3 11:48:44 alice2 ltfs[25370]: 631a LTFS17089I Distribution: PRETTY_NAME="Debian GNU/Linux 9 (stretch)".
Sep  3 11:48:44 alice2 ltfs[25370]: 631a LTFS14063I Sync type is "time", Sync time is 300 sec.
Sep  3 11:48:44 alice2 ltfs[25370]: 631a LTFS17085I Plugin: Loading "sg" tape backend.
Sep  3 11:48:44 alice2 ltfs[25370]: 631a LTFS17085I Plugin: Loading "unified" iosched backend.
Sep  3 11:48:44 alice2 ltfs[25370]: 631a LTFS14095I Set the tape device write-anywhere mode to avoid cartridge ejection.
Sep  3 11:48:44 alice2 ltfs[25370]: 631a LTFS30209I Opening a device through sg-ibmtape driver (/dev/st0).
Sep  3 11:48:44 alice2 ltfs[25370]: 631a LTFS30207I Vendor ID is IBM     .
Sep  3 11:48:44 alice2 ltfs[25370]: 631a LTFS30208I Product ID is ULTRIUM-HH7     .
Sep  3 11:48:44 alice2 ltfs[25370]: 631a LTFS30214I Firmware revision is JAY1.
Sep  3 11:48:44 alice2 ltfs[25370]: 631a LTFS30215I Drive serial is 10WT074468.
Sep  3 11:48:44 alice2 ltfs[25370]: 631a LTFS17160I Maximum device block size is 1048576.
Sep  3 11:48:44 alice2 ltfs[25370]: 631a LTFS11330I Loading cartridge.
Sep  3 11:48:47 alice2 ltfs[25370]: 631a LTFS30252I Logical block protection is disabled.
Sep  3 11:48:47 alice2 ltfs[25370]: 631a LTFS11332I Load successful.
Sep  3 11:48:47 alice2 ltfs[25370]: 631a LTFS17157I Changing the drive setting to write-anywhere mode.
Sep  3 11:48:47 alice2 ltfs[25370]: 631a LTFS11005I Mounting the volume.
Sep  3 11:48:47 alice2 ltfs[25370]: 631a LTFS30252I Logical block protection is disabled.
Sep  3 11:49:01 alice2 ltfs[25370]: 631a LTFS17227I Tape attribute: Vendor = IBM     .
Sep  3 11:49:01 alice2 ltfs[25370]: 631a LTFS17227I Tape attribute: Application Name = LTFS                            .
Sep  3 11:49:01 alice2 ltfs[25370]: 631a LTFS17227I Tape attribute: Application Version = 2.4.1.1 .
Sep  3 11:49:01 alice2 ltfs[25370]: 631a LTFS17227I Tape attribute: Medium Label = .
Sep  3 11:49:01 alice2 ltfs[25370]: 631a LTFS17228I Tape attribute: Text Localization ID = 0x81.
Sep  3 11:49:01 alice2 ltfs[25370]: 631a LTFS17227I Tape attribute: Barcode = 000025L7                        .
Sep  3 11:49:01 alice2 ltfs[25370]: 631a LTFS17227I Tape attribute: Application Format Version = 2.4.0           .
Sep  3 11:49:01 alice2 ltfs[25370]: 631a LTFS17228I Tape attribute: Volume Lock Status = 0x00.
Sep  3 11:49:01 alice2 ltfs[25370]: 631a LTFS17227I Tape attribute: Media Pool name = .
Sep  3 11:49:01 alice2 ltfs[25370]: 631a LTFS14111I Initial setup completed successfully.
Sep  3 11:49:01 alice2 ltfs[25370]: 631a LTFS14112I Invoke 'mount' command to check the result of final setup.
Sep  3 11:49:01 alice2 ltfs[25370]: 631a LTFS14113I Specified mount point is listed if succeeded.
Sep  3 11:49:01 alice2 ltfs[25412]: 6345 LTFS14029I Ready to receive file system requests.
Sep  3 11:49:03 alice2 ltfs[25412]: 6346 LTFS11337I Update index-dirty flag (1) - NO_BARCODE (0x0x5579c4466410).
Sep  3 11:49:41 alice2 ltfs[25412]: 6347 LTFS30200I Failed to execute SG_IO ioctl, opcode = 0a (22).
Sep  3 11:49:41 alice2 ltfs[25412]: 6347 LTFS30263I WRITE returns ioctl error (-21700) /dev/st0.
Sep  3 11:49:41 alice2 ltfs[25412]: 6347 LTFS12045E Cannot write block: backend call failed (-21700). Dropping to read-only mode.
Sep  3 11:49:41 alice2 ltfs[25412]: 6347 LTFS11072E Cannot write blocks: failed to write to the medium (-21700).
Sep  3 11:49:41 alice2 ltfs[25412]: 6347 LTFS11077E Cannot write: failed to write blocks to the medium (-21700).
Sep  3 11:49:41 alice2 ltfs[25412]: 6347 LTFS13014W Data partition writer: failed to write data to the tape (-21700).
Sep  3 11:49:41 alice2 ltfs[25412]: 6345 LTFS14022E Cannot flush file data to the medium. Data might be lost (ltfs_fuse_flush).
Sep  3 11:49:41 alice2 ltfs[25412]: 6346 LTFS10020E Error on utimens: /Preview+Archive.24/2019-08-31/h264/0720_06_59.mkv (0, 0).
Sep  3 11:49:45 alice2 ltfs[25412]: 6344 LTFS11033E Cannot unmount: failed to write an index.
Sep  3 11:49:45 alice2 ltfs[25412]: 6344 LTFS30252I Logical block protection is disabled.

Originally posted by @swt2c in #144 (comment)

unsupported drive with space padded

Describe the bug

When trying to format an LTO8 tape in a ULTRIUM-HH8 deck, I get this error

LTFS15000I Starting mkltfs, LTFS version 2.4.1.2 (10254), log level 2.
LTFS15041I Launched by "mkltfs -c --device=0 --tape-serial=LT0001 --volume-name=LT0001".
LTFS15042I This binary is built for Mac OS X .
LTFS15043I GCC version is 4.2.1 Compatible Apple LLVM 10.0.1 (clang-1001.0.46.4).
LTFS17087I Kernel version: Darwin Kernel Version 18.7.0: Sun Dec  1 18:59:03 PST 2019; root:xnu-4903.278.19~1/RELEASE_X86_64.
LTFS15003I Formatting device '0'.
LTFS15004I LTFS volume blocksize: 524288.
LTFS15005I Index partition placement policy: None.

LTFS11337I Update index-dirty flag (1) - LT0001 (0x0x7fa681407e70).
LTFS17085I Plugin: Loading "iokit" tape backend.
LTFS30810I Opening a device through iokit driver (0).
LTFS30813I Unsupported drive 'ULTRIUM-HH8     '.

I see in

ltfs/src/tape_drivers/ibm_tape.c

Line 291 in 0a8c1b8

TAPEDRIVE( IBM_VENDOR_ID, "ULTRIUM-HH8", DRIVE_LTO8_HH, "[ULTRIUM-HH8]" ), /* IBM Ultrium Gen 8 Half-High */
that ULTRIUM-HH8 is supported. Just guessing but is this an issue of 'ULTRIUM-HH8 ' != 'ULTRIUM-HH8', related to #18.

To Reproduce
Steps to reproduce the behavior:

mkltfs -c --device=0 --tape-serial=LT0001 --volume-name=LT0001

Expected behavior
mkltfs formats an LTO8 tape

Cannot mount a tape when filename has `BOM` of UTF-8

Describe the bug

LTFS rejects a filename that has BOM of UTF-8

To Reproduce

  1. Format a tape by IBM Spectrum Archive SDE 2.2 or IBM Spectrum Archive LE 2.1
  2. Mount the tape by IBM Spectrum Archive SDE 2.2 or IBM Spectrum Archive LE 2.1
  3. Create a file that has BOM
  4. Unmount the tape
  5. Mount the tape by LTFS 2.4.x.x -> Mount fail

Expected behavior

May be LTFS shall not reject the BOM in file anme.

RFC: split write over multiple tapes

The Problem: Howto control/handle write operations, that consume multiple tapes?

I need to split up a btrfs-send/btrfs-receive pipe, since data will not fit on a single tape.

Solution proposel

LTFS should offer an api, advertising e.g.

  • "no space left",
  • " space left"

This will enable the callee to interrupt a write process without preprocessing the available free LTFS partition space.
The machanism how the EOT message is exchanged need to be discussed.

Callee needs to handle following tasks:

  • Write a mark/struct/file describing the active slice/parent/sibbling

  • LTO loader functions / manual operator intervention must be issued to unmount + unload the given tape

  • LTO loader functions / manual operator intervention must be issued to load + mount a new tape.

  • Write a mark/struct/file describing the active slice/parent/sibbling

  • continue to save the data pipe

this is just a rough starter. I haven't read any available solution on how to implement this featurerequest.

Any feedback is quite welcome.

Linking problem in ArchLinux

Can't link compiled code

I did successfully compile the code in ArchLinux (configure, make)
When linking, process stops with following error message. (make install)

Compile environment

gcc (GCC) 8.1.1 20180531
GNU ld (GNU Binutils) 2.30
makedepends=('git' 'make' 'fuse' 'libxml2' 'icu' 'libutil-linux' 'net-snmp')

./autogen.sh
DEFAULT_TAPE=sg ./configure
--enable-buggy-ifs
--prefix=/usr
retun: ok
make DESTDIR="/tmp/" install

Compile/Linker error

libtool: relink: gcc -shared -fPIC -DPIC .libs/libtape_file_la-filedebug_tc.o .libs/libtape_file_la-filedebug_conf_tc.o .libs/libtape_file_la-ibm_tape.o -L/usr/lib -L../../../../messages/ -ltape_generic_file_dat -L/data/development/ltfs_ibm/pkg/usr/lib -lltfs -L../../messages -lfuse -luuid -lxml2 -licui18n -licuuc -licudata -lnetsnmpmibs -lpci -ldl -lnetsnmpagent -lnetsnmp -lcrypto -lnl-3 -lm -llibltfs_dat -linternal_error_dat -ltape_common_dat -lrt -lpthread -march=x86-64 -mtune=generic -O2 -fstack-protector-strong -fstack-protector-strong -march=x86-64 -mtune=generic -O2 -fstack-protector-strong -O2 -g -Wl,--no-undefined -Wl,--as-needed -pthread -Wl,-O1 -Wl,--sort-common -Wl,--as-needed -Wl,-z -Wl,relro -Wl,-z -Wl,now -Wl,-E -Wl,-rpath -Wl,/usr/lib/perl5/5.26/core_perl/CORE -Wl,-O1 -Wl,--sort-common -Wl,--as-needed -Wl,-z -Wl,relro -Wl,-z -Wl,now -Wl,-O1 -Wl,--sort-common -Wl,--as-needed -Wl,-z -Wl,relro -Wl,-z -Wl,now -pthread -Wl,-soname -Wl,libtape-file.so -o .libs/libtape-file.so
.libs/libtape_file_la-filedebug_conf_tc.o: In function _filedebug_parser_init': /data/development/ltfs_ibm/src/ltfs/src/tape_drivers/generic/file/filedebug_conf_tc.c:191: undefined reference to xml_next_tag'
.libs/libtape_file_la-filedebug_conf_tc.o: In function _filedebug_tc_parse_schema': /data/development/ltfs_ibm/src/ltfs/src/tape_drivers/generic/file/filedebug_conf_tc.c:223: undefined reference to xml_next_tag'

LTFS Cartridge Fault

Hi,

Any ideas what might be happening here?

Oct 30 20:30:16 bobby ltfs[10971]: 2adb LTFS14000I LTFS starting, LTFS version 2.4.1.1 (10227), log level 3.
Oct 30 20:30:16 bobby ltfs[10971]: 2adb LTFS14058I LTFS Format Specification version 2.4.0.
Oct 30 20:30:16 bobby ltfs[10971]: 2adb LTFS14104I Launched by "ltfs -o devname=/dev/sg2 /opt/VS/vsMounts/000024L7 -o trace".
Oct 30 20:30:16 bobby ltfs[10971]: 2adb LTFS14105I This binary is built for Linux (x86_64).
Oct 30 20:30:16 bobby ltfs[10971]: 2adb LTFS14106I GCC version is 6.3.0 20170516.
Oct 30 20:30:16 bobby ltfs[10971]: 2adb LTFS17087I Kernel version: Linux version 4.20.11 (root@debian) (gcc version 6.3.0 20170516 (Debian 6.3.0-18+deb9u1)) #1 SMP Thu Feb 21 09:13:14 EST 2019 i386.
Oct 30 20:30:16 bobby ltfs[10971]: 2adb LTFS17089I Distribution: PRETTY_NAME="Debian GNU/Linux 9 (stretch)".
Oct 30 20:30:16 bobby ltfs[10971]: 2adb LTFS14063I Sync type is "time", Sync time is 300 sec.
Oct 30 20:30:16 bobby ltfs[10971]: 2adb LTFS17085I Plugin: Loading "sg" tape backend.
Oct 30 20:30:16 bobby ltfs[10971]: 2adb LTFS17085I Plugin: Loading "unified" iosched backend.
Oct 30 20:30:16 bobby ltfs[10971]: 2adb LTFS14095I Set the tape device write-anywhere mode to avoid cartridge ejection.
Oct 30 20:30:16 bobby ltfs[10971]: 2adb LTFS30209I Opening a device through sg-ibmtape driver (/dev/sg2).
Oct 30 20:30:16 bobby ltfs[10971]: 2adb LTFS30207I Vendor ID is IBM     .
Oct 30 20:30:16 bobby ltfs[10971]: 2adb LTFS30208I Product ID is ULTRIUM-HH7     .
Oct 30 20:30:16 bobby ltfs[10971]: 2adb LTFS30214I Firmware revision is JAY1.
Oct 30 20:30:16 bobby ltfs[10971]: 2adb LTFS30215I Drive serial is 10WT074468.
Oct 30 20:30:16 bobby ltfs[10971]: 2adb LTFS17160I Maximum device block size is 1048576.
Oct 30 20:30:16 bobby ltfs[10971]: 2adb LTFS11330I Loading cartridge.
Oct 30 20:30:18 bobby ltfs[10971]: 2adb LTFS30252I Logical block protection is disabled.
Oct 30 20:30:18 bobby ltfs[10971]: 2adb LTFS11332I Load successful.
Oct 30 20:30:18 bobby ltfs[10971]: 2adb LTFS17157I Changing the drive setting to write-anywhere mode.
Oct 30 20:30:18 bobby ltfs[10971]: 2adb LTFS11005I Mounting the volume.
Oct 30 20:30:18 bobby ltfs[10971]: 2adb LTFS30252I Logical block protection is disabled.
Oct 30 20:32:40 bobby ltfs[10971]: 2adb LTFS30205I READ (0x08) returns -20301.
Oct 30 20:32:40 bobby ltfs[10971]: 2adb LTFS30263I READ returns Cartridge Fault (-20301) /dev/sg2.
Oct 30 20:32:40 bobby ltfs[10971]: 2adb LTFS30261I Taking drive dump in buffer.
Oct 30 20:32:40 bobby ltfs[10971]: 2adb LTFS30253I Saving drive dump to /tmp/ltfs_10WT074468_2019_1030_123240.dmp.
Oct 30 20:32:59 bobby ltfs[10971]: 2adb LTFS30262I Forcing drive dump.
Oct 30 20:32:59 bobby ltfs[10971]: 2adb LTFS30253I Saving drive dump to /tmp/ltfs_10WT074468_2019_1030_123240_f.dmp.
Oct 30 20:33:12 bobby ltfs[10971]: 2adb LTFS12049E Cannot read: backend call failed (-20301).
Oct 30 20:33:12 bobby ltfs[10971]: 2adb LTFS11174E Cannot read ANSI label: read failed (-20301).
Oct 30 20:33:12 bobby ltfs[10971]: 2adb LTFS11171E Failed to read label (-20301) from partition 1.
Oct 30 20:33:12 bobby ltfs[10971]: 2adb LTFS11009E Cannot read volume: failed to read partition labels.
Oct 30 20:33:12 bobby ltfs[10971]: 2adb LTFS14013E Cannot mount the volume.
Oct 30 20:33:12 bobby ltfs[10971]: 2adb LTFS30252I Logical block protection is disabled.

Issue: Can't mount tape on Ubuntu (SG backend)

Mounting a formatted tape via sg backend breaks

using mkltfs i can format a given tape via the sg backend (sg-ibmtape driver).
Partitions (a and b) are created. Volume UUID is assigned. Volumes capacity is shown.
As stated from the binary the mkltfs call seems completely wrokout.

a following ltfs call terminates with an read block fauillure message (-21719). Driver encounters a length mismatch detection (LTFS30216W) and therefore can't read the label from partition 0. Mounting is aborted.

What did i miss?
Do not hesitate, if you need more/precise information. Thanks

Steps to reproduce the behavior:

  1. running as root (uid=0)
  2. configured /etc/ltfs.conf (namely: plugin tape sg /usr/lib/x86_64-linux-gnu/ltfs/libtape-sg-ibmtape.so, default tape sg)
  3. configured /etc/ltfs.conf.local (namely: option work_directory /run/user/0/ltfs)
    same effect, if i do keep the systemd default (/tmp/ltfs)
  4. calling mkltfs
    mkltfs -d /dev/sg9 -f -n tape009 (exit code 0)
  5. create mountpoint (/media/tape)
    mkdir -p /media/tape
  6. calling ltfs
    ltfs -o devname=/dev/sg9,verbose=2 /media/tape

Expected behavior

the formatted tape should be mounted at /media/tape.

Loggin information

  1. mkltfs call
    mkltfs -d /dev/sg9 -f -n tape009
    LTFS15000I Starting mkltfs, LTFS version 2.4.0.1 (10062), log level 2.
    LTFS15041I Launched by "mkltfs -d /dev/sg9 -f -n tape009".
    LTFS15042I This binary is built for Linux (x86_64).
    LTFS15043I GCC version is 7.3.0.
    LTFS17087I Kernel version: Linux version 4.11.0-14-generic (buildd@lcy01-08) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu116.04.4) ) #2016.04.1-Ubuntu SMP Wed Aug 9 09:06:22 UTC 2017 i386.
    LTFS17089I Distribution: NAME="Ubuntu".
    LTFS17089I Distribution: DISTRIB_ID=Ubuntu.
    LTFS15003I Formatting device '/dev/sg9'.
    LTFS15004I LTFS volume blocksize: 524288.
    LTFS15005I Index partition placement policy: None.
    LTFS11337I Update index-dirty flag (1) - NO_BARCODE (0x0x36ad5a1ac0).
    LTFS17085I Plugin: Loading "sg" tape backend.
    LTFS30209I Opening a device through sg-ibmtape driver (/dev/sg9).
    LTFS30207I Vendor ID is IBM .
    LTFS30208I Product ID is ULTRIUM-HH7 .
    LTFS30214I Firmware revision is FA17.
    LTFS30215I Drive serial is 10WT030737.
    LTFS17160I Maximum device block size is 1048576.
    LTFS11330I Loading cartridge.
    LTFS30252I Logical block protection is disabled.
    LTFS11332I Load successful.
    LTFS17157I Changing the drive setting to write-anywhere mode.
    LTFS15049I Checking the medium (load).
    LTFS30252I Logical block protection is disabled.
    LTFS15010I Creating data partition b on SCSI partition 1.
    LTFS15011I Creating index partition a on SCSI partition 0.
    LTFS17165I Resetting the medium's capacity proportion.
    LTFS11097I Partitioning the medium.
    LTFS30263I MODESELECT returns Mode Parameters Rounded (-20101) /dev/sg9.
    LTFS11100I Writing label to partition b.
    LTFS11278I Writing index to partition b.
    LTFS17235I Writing index of NO_BARCODE to b (Reason: Format, 0 files) 10WT030737.
    LTFS17236I Wrote index of NO_BARCODE (b, 10WT030737).
    LTFS11337I Update index-dirty flag (0) - NO_BARCODE (0x0x36ad5a1ac0).
    LTFS11100I Writing label to partition a.
    LTFS11278I Writing index to partition a.
    LTFS17235I Writing index of NO_BARCODE to a (Reason: Format, 0 files) 10WT030737.
    LTFS17236I Wrote index of NO_BARCODE (a, 10WT030737).
    LTFS15013I Volume UUID is: dbcbe63b-6785-4e2a-8594-0d6565c0aba2.
    LTFS15019I Volume capacity is 5732 GB.
    LTFS30252I Logical block protection is disabled.
    LTFS15024I Medium formatted successfully.

  2. ltfs call
    ltfs -o devname=/dev/sg9,verbose=2 /media/tape
    104e LTFS14000I LTFS starting, LTFS version 2.4.0.1 (10062), log level 2.
    104e LTFS14058I LTFS Format Specification version 2.4.0.
    104e LTFS14104I Launched by "ltfs -o devname=/dev/sg9,verbose=2 /media/tape".
    104e LTFS14105I This binary is built for Linux (x86_64).
    104e LTFS14106I GCC version is 7.3.0.
    104e LTFS17087I Kernel version: Linux version 4.11.0-14-generic (buildd@lcy01-08) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu116.04.4) ) #2016.04.1-Ubuntu SMP Wed Aug 9 09:06:22 UTC 2017 i386.
    104e LTFS17089I Distribution: NAME="Ubuntu".
    104e LTFS17089I Distribution: DISTRIB_ID=Ubuntu.
    104e LTFS14063I Sync type is "time", Sync time is 300 sec.
    104e LTFS17085I Plugin: Loading "sg" tape backend.
    104e LTFS17085I Plugin: Loading "unified" iosched backend.
    104e LTFS14095I Set the tape device write-anywhere mode to avoid cartridge ejection.
    104e LTFS30209I Opening a device through sg-ibmtape driver (/dev/sg9).
    104e LTFS30207I Vendor ID is IBM .
    104e LTFS30208I Product ID is ULTRIUM-HH7 .
    104e LTFS30214I Firmware revision is FA17.
    104e LTFS30215I Drive serial is 10WT030737.
    104e LTFS17160I Maximum device block size is 1048576.
    104e LTFS11330I Loading cartridge.
    104e LTFS30252I Logical block protection is disabled.
    104e LTFS11332I Load successful.
    104e LTFS17157I Changing the drive setting to write-anywhere mode.
    104e LTFS11005I Mounting the volume.
    104e LTFS30252I Logical block protection is disabled.
    104e LTFS30216W Length mismatch is detected. (Act = 4096, resid = 0, resid_sense = 4016).
    104e LTFS12049E Cannot read: backend call failed (-21716).
    104e LTFS11174E Cannot read ANSI label: read failed (-21716).
    104e LTFS11170E Failed to read label (-21716) from partition 0.
    104e LTFS11009E Cannot read volume: failed to read partition labels.
    104e LTFS14013E Cannot mount the volume.
    104e LTFS30252I Logical block protection is disabled.

How to mount hp Ultrium lto5 tape dirver

Is your feature request related to a problem? Please describe.

Does this project support HP EH957/EH958.
I install hp ltt and ltfs on archlinux using yaourt:
aur/ltfs 2.4.0.2-1
aur/hpltt 4.27-1

but ltfs shows:

39f5 LTFS14000I LTFS starting, LTFS version 2.4.0.2, log level 2.
39f5 LTFS14058I LTFS Format Specification version 2.4.0.
39f5 LTFS14104I Launched by "ltfs".
39f5 LTFS14105I This binary is built for Linux (x86_64).
39f5 LTFS14106I GCC version is 8.3.0.
39f5 LTFS17087I Kernel version: Linux version 5.1.5-arch1-2-ARCH (builduser@heftig-23132) (gcc version 8.3.0 (GCC)) #1 SMP PREEMPT Mon May 27 03:37:39 UTC 2019 i386.
39f5 LTFS17089I Distribution: NAME="Arch Linux".
39f5 LTFS17089I Distribution: .
39f5 LTFS14063I Sync type is "time", Sync time is 300 sec.
39f5 LTFS17085I Plugin: Loading "sg" tape backend.
39f5 LTFS17085I Plugin: Loading "unified" iosched backend.
39f5 LTFS14095I Set the tape device write-anywhere mode to avoid cartridge ejection.
39f5 LTFS30209I Opening a device through sg-ibmtape driver (/dev/sg0).
39f5 LTFS30213I Unsupported Drive 'Ultrium 5-SCSI '.
39f5 LTFS12012E Cannot open device: failed backend open call.
39f5 LTFS10004E Cannot open device '/dev/sg0'.

Describe the solution you'd like

Should I install hp spefic driver? or ltfs not support hp lto5 tape driver.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Documentation for KMI Backend

Howto use the KMI backend

The codebase states this feature experimental. Beside the source code, there is no documentation on how to use the feature yet.

Manpage / Doc file

could you please give some short hints, on the intended usage?
I'd be glad to help out writing a manpage or markup text.

Relation / relevance

In Europe, we have to address the "Datenschutz-Grundverordnung" (EU-DSGVO) which have been requested law since 25.05.2018. Probably quite a lot of IT professionals have to address the need to protect personal data. And tapes beeing outside the secured office environment are an active target in terms of the EU-DSGVO. Thanks

"Datenschutverordnung since I'd like to use

A low level tool to capture indexes on a tape

Background

Sometimes we got a problem related into dirty index. But we don't have any effective way to capture it. Because current code can capture a good index after the code did parse it.

Describe the solution you'd like

I would like to have a tool to capture indexes in low level (without any XML check at all)

Describe alternatives you've considered

Or it is helpful if LTFS can capture BAD index on-the-fly as special name for FA.

MRSW lock can deadlock after reader count reaches zero

Summary

The MRSW lock uses a single reader mutex that is locked by the first reader thread to acquire the read lock and released by the last thread to release the read lock. If the first acquiring and last releasing threads are not the same threads, the mutex will be left locked by the first thread and never unlocked. The next attempt to acquire the read lock will deadlock the calling thread since it will try to lock the mutex, which is still locked.

Description

Example:

  • Thread 1 acquires the read MRSW lock on a volume, and with the reader count < 1 locks the reader mutex
  • Thread 2 acquires the read MRSW lock on the same volume, and only ticks up the reader count
  • Thread 1 releases the read MRSW lock on the volume, and does not unlock the reader mutex
  • Thread 2 releases the read MRSW lock on the volume, and erroneously unlocks the reader mutex, but does not detect that the unlock failed due to the thread not owning the mutex lock in the first place
  • Thread N acquires the read MRSW lock on the volume, and blocks to lock the reader mutex that thread 1 left locked
  • Thread N is now deadlocked

Environment

  • OS: Linux/FreeBSD, probably others
  • Distribution: All
  • Architecture: All
  • Additional info:

How to recreate

This will not show up in the FUSE driver, since it is inherently single threaded. You will need to write a test program that links to libltfs and spawns threads that hold the reader lock on a volume in overlapping rather than nested windows. You could guarantee the timing by coordinating the threads with condvars.

I've gotten around this with a local delta by replacing the innards of the mrsw implementation with pthread_rwlock_t for our FreeBSD build, but that is not a portable solution.

Verbose mode for ltfs_ordered_copy

The only thing holding me back with the aging ltfscp is the lack of verbose mode for ltfs_ordered_copy. A --log option would be great also.

homebrew formula for ltfs

Is your feature request related to a problem? Please describe.
No

Describe the solution you'd like
A hope for brew install ltfs or brew install some-tap/ltfs to work.

Describe alternatives you've considered
Following the installation instructions of the readme.

Additional context
It seems like the installation instructions put a lot of consideration to homebrew environments but I don't see a formula for ltfs itself. I've started a draft of that work at amiaopensource/homebrew-amiaos#220 but am stuck on an installation issue. I wanted to share that here in case any others are interested in contributing towards having an ltfs brew formula.

Format is failed because of "Cleaning required"

Describe the bug
Format is failed when drive reports "Cleaning required" when LTFS set up the "append only mode".

To Reproduce
It is very hard to reproduce. Because "Cleaning required" sense is reported based on the condition of tape and drive.

Expected behavior
"Cleaning required"

Screenshots

2020-03-10T09:19:02.444046-07:00 kanda ltfs[23044]: 115f LTFS14837I Formatting cartridge JD2406JD (0x56, 0x56).
2020-03-10T09:19:02.444220-07:00 kanda ltfs[23044]: 115f LTFS11613I Moving cartridge JD2406JD to drive 0000078PG255.
2020-03-10T09:19:02.444348-07:00 kanda ltfs[23044]: 115f LTFS11626I Cartridge JD2406JD is already in drive 0000078PG255.
2020-03-10T09:19:02.445681-07:00 kanda ltfs[23044]: 115f LTFS11330I Loading cartridge.
2020-03-10T09:19:24.545468-07:00 kanda ltfs[23044]: 115f LTFS30252I Logical block protection is disabled.
2020-03-10T09:19:24.555075-07:00 kanda ltfs[23044]: 115f LTFS11332I Load successful.
2020-03-10T09:19:24.555757-07:00 kanda ltfs[23044]: 115f LTFS11337I Update index-dirty flag (1) - JD2406 (0x0x1a81800).
2020-03-10T09:19:24.561391-07:00 kanda ltfs[23044]: 115f LTFS17157I Changing the drive setting to write-anywhere mode.
2020-03-10T09:19:24.562267-07:00 kanda ltfs[23044]: 115f LTFS17157I Changing the drive setting from append-only mode to write-anywhere mode.
2020-03-10T09:19:55.427930-07:00 kanda ltfs[23044]: 115f LTFS30263I LOAD_UNLOAD returns Drive Needs Cleaning (-20098) /dev/sg5.
2020-03-10T09:19:55.481085-07:00 kanda ltfs[23044]: 115f LTFS17151E Cannot set WRITE MODE: Failed to unload medium (-20098).
2020-03-10T09:19:55.481244-07:00 kanda ltfs[23044]: 115f LTFS14838I Formatted cartridge JD2406JD (0x56).
2020-03-10T09:19:55.482802-07:00 kanda ltfs[23044]: 115f LTFS14789E Failed to format cartridge JD2406JD (-20098).
2020-03-10T09:19:55.482975-07:00 kanda ltfs[23044]: 115f LTFSD0098E (IBM LTO 00/8282) Drive requests cleaning.
2020-03-10T09:19:55.532019-07:00 kanda ltfs[23044]: 1666 LTFS14802I Moving cartridge JD2406JD.
2020-03-10T09:19:55.534826-07:00 kanda ltfs[23044]: 1666 LTFS30205I TEST_UNIT_READY (0x00) returns -20209.
2020-03-10T09:19:55.535000-07:00 kanda ltfs[23044]: 1666 LTFS12029E Device is not ready (-20209).
2020-03-10T09:19:55.535160-07:00 kanda ltfs[23044]: 1666 LTFS14812I Moving cartridge JD2406JD from 258 to 1059 - 0000013FA0020410 .
2020-03-10T09:19:55.535282-07:00 kanda ltfs[23044]: 1666 LTFS11629I Aqcuireing MoveLock - 0000013FA0020410.
2020-03-10T09:19:55.536236-07:00 kanda ltfs[23044]: 1666 LTFS11630I Aqcuired Move Lock (11) - 0000013FA0020410.
2020-03-10T09:20:02.073660-07:00 kanda ltfs[23044]: 1666 LTFS11636I Unlink the drive 0000078PG255 from the cartridge JD2406JD.
2020-03-10T09:20:02.073805-07:00 kanda ltfs[23044]: 1666 LTFS14813I Moved cartridge JD2406JD to 1059 successfully - 0000013FA0020410.
2020-03-10T09:20:02.075403-07:00 kanda ltfs[23044]: 1666 LTFS14803I Cartridge JD2406JD is moved successfully.
2020-03-10T09:20:02.135238-07:00 kanda ltfs[23044]: 16af LTFS14784I Removing cartridge JD2406JD.
2020-03-10T09:20:02.135411-07:00 kanda ltfs[23044]: 16af LTFS11638I Cartridge JD2406JD was unassigned because there is no libvol.
2020-03-10T09:20:02.136473-07:00 kanda ltfs[23044]: 16af LTFS14785I Cartridge JD2406JD is removed successfully.

Read out tape-serial / volume-name after ltfs-tape is mounted

I'd like to check tape identification id's after a given tape is loaded and mounted

When using a media-changer device, a given backup-policy will introduce requirements that handle media usage.

Usualy Media are grouped in MediaPools. Media will be accepted as a targets of a given media-pool by referencing a unique identifier. Pool association could rely on the volume-name (= VolumeTag), maybe combined with the tape-serial.

An external handler/wrapper will verify user selectable policy options (append to tape, calander based media changing, always overwrite loaded media, etc.).

Thus the handler needs to know and verify the unique identifier of the active/mounted ltfs tape, to take care of the required procedures, e.g:

  • mount/unmount a needed tape
  • load/undload a given tape
  • advice a media-changer move a tape (from slot to device, from device to slot)

mtx is a good candidate to handle the media-changer.

mkltfs
When you create/partition a new ltfs, mkltfs will offer all needed options.
You can wipe an already prepared tape, or create a new one assigning he needed attributes.

ltfs
the given device will be mounted via ltfs. Depending on the choosen verbosity level, you will see media identification information.

Once mounted, i can't find a way to read out media information from a mounted device.

Practical solution
Possible I have overseen an already implemented method. If you can point out how to use it? If there is no solution yet I'd like to propose following ideas:

  • /proc/mounts
    Since fuse will append and set other options via /proc/mounts, it might be an option to add tape-name /and/or/ volume-name. That can be parsed quit easy and will be very cheap

  • ltfs option
    It might be an option to extend ltfs to write out selectable device options.

Build fails on Fedora 28

When installing LTFS on an updated Fedora 28 server, build fails in the utils directory

Steps to reproduce the behavior:

  • Install Fedora 28 and update (yum -y update), reboot to activate latest kernel

  • Install prerequisite packages as described on homepage, plus net-snmp-devel

  • Download ltfs head branch (git pull https://github.com/LinearTapeFileSystem/ltfs)

  • Use default options to build (./autogen ; ./configure ; make)

  • When in the utils directory, the build error below occurs:
    make[3]: Entering directory '/root/ltfs/src/utils'
    gcc -DHAVE_CONFIG_H -I. -I../.. -D_GNU_SOURCE -I../../src -DLTFS_CONFIG_FILE='"/usr/local/etc/ltASE_DIR='"NONE"' -I .. -Wall -Wsign-compare -fsigned-char -I/usr/include/fuse -D_FILE_OFFSET_BITe/uuid -I/usr/include/libxml2 -I/usr/include -D ENABLE_SNMP -I. -I/usr/include -fkeep-inline-func -O2 -D_FORTIFY_SOURCE=0 -O0 -ggdb -MT mkltfs-mkltfs.o -MD -MP -MF .deps/mkltfs-mkltfs.Tpo -c -o est -f 'mkltfs.c' || echo './'`mkltfs.c
    mv -f .deps/mkltfs-mkltfs.Tpo .deps/mkltfs-mkltfs.Po
    /bin/sh ../../libtool --tag=CC --mode=link gcc -Wall -Wsign-compare -fsigned-char -I/usr/incluFSET_BITS=64 -I/usr/include/uuid -I/usr/include/libxml2 -I/usr/include -D ENABLE_SNMP -I. -I/usr/ine-functions -rdynamic -g -O2 -D_FORTIFY_SOURCE=0 -O0 -ggdb -Wl,--no-undefined,--as-needed -lfus -lxml2 -L/usr/lib64 -licui18n -licuuc -licudata -Wl,-z,relro -Wl,-z,now -L/usr/lib64 -lnets-ldl -lrpm -lrpmio -lnetsnmpagent -Wl,--enable-new-dtags -Wl,-z,relro -Wl,-z,now -specs=/usr/lib/hardened-ld -lnetsnmp -lssl -lssl -lcrypto -lm 2> /dev/null -Wl,-z,relro -Wl,-z,now -L/usr/lib64 ssl -lcrypto -lm -L../../messages -lbin_mkltfs_dat -o mkltfs mkltfs-mkltfs.o ../libltfs/libltfs.
    libtool: link: gcc -Wall -Wsign-compare -fsigned-char -I/usr/include/fuse -D_FILE_OFFSET_BITS=64 d -I/usr/include/libxml2 -I/usr/include -D ENABLE_SNMP -I. -I/usr/include -fkeep-inline-functions-D_FORTIFY_SOURCE=0 -O0 -ggdb -Wl,--no-undefined -Wl,--as-needed -pthread -Wl,-z -Wl,relro -Wl,-zble-new-dtags -Wl,-z -Wl,relro -Wl,-z -Wl,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -Wl,- -Wl,now -o .libs/mkltfs mkltfs-mkltfs.o -L/usr/lib64 -L../../messages -lbin_mkltfs_dat ../liblto -lfuse -luuid -lxml2 -licui18n -licuuc -licudata -lnetsnmpmibs -lsensors -ldl -lrpm -lrpmio -lnnmp -lssl -lcrypto -lm -llibltfs_dat -linternal_error_dat -ltape_common_dat -lrt -lpthread -pthre/usr/local/lib
    make[3]: *** [Makefile:466: mkltfs] Error 1
    make[3]: Leaving directory '/root/ltfs/src/utils'
    make[2]: *** [Makefile:682: all-recursive] Error 1
    make[2]: Leaving directory '/root/ltfs/src'
    make[1]: *** [Makefile:524: all-recursive] Error 1
    make[1]: Leaving directory '/root/ltfs'
    make: *** [Makefile:430: all] Error 2

  • OS: Fedora 28 Server (Linux vtape.consolitape.local 4.17.12-200.fc28.x86_64 #1 SMP Fri Aug 3 15:01:13 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux)

Linux kernel driver lin_tape fails on CentOS 7 systems with StorNext SAN

We have encountered a particularly frustrating system problem on all of our machines running CentOS 7 with IBM tape drives (LTO7 / LTO8) and Quantum's StorNext SAN technology. I believe it is a system problem related to the Linux VFS. I have directly observed this problem on machines with 16GB of RAM and 128GB of RAM. Here is a piece of the kernel log which gets dumped out when the problem occurs:

Sep 18 13:22:38 dfsnl2 kernel: ltfs: page allocation failure: order:7, mode:0xd0
Sep 18 13:22:38 dfsnl2 kernel: CPU: 6 PID: 3010 Comm: ltfs Tainted: P           OE  ------------   3.10.0-693.17.1.el7.x86_64 #1
Sep 18 13:22:38 dfsnl2 kernel: Hardware name: Supermicro X10DRH/X10DRH-i, BIOS 3.1 06/18/2018
Sep 18 13:22:38 dfsnl2 kernel: Call Trace:
Sep 18 13:22:38 dfsnl2 kernel: [<ffffffff816a6071>] dump_stack+0x19/0x1b
Sep 18 13:22:38 dfsnl2 kernel: [<ffffffff8118a6f0>] warn_alloc_failed+0x110/0x180
Sep 18 13:22:38 dfsnl2 kernel: [<ffffffff816a204a>] __alloc_pages_slowpath+0x6b6/0x724
Sep 18 13:22:38 dfsnl2 kernel: [<ffffffff8118ec85>] __alloc_pages_nodemask+0x405/0x420
Sep 18 13:22:38 dfsnl2 kernel: [<ffffffff811d3218>] alloc_pages_current+0x98/0x110
Sep 18 13:22:38 dfsnl2 kernel: [<ffffffffc0992bb8>] lin_tape_blk_rq+0x1d8/0x520 [lin_tape]
Sep 18 13:22:38 dfsnl2 kernel: [<ffffffffc0992f48>] tape_execute_scsi_command+0x48/0xc0 [lin_tape]
Sep 18 13:22:38 dfsnl2 kernel: [<ffffffffc09930a0>] tape_send_rw_indirect+0xe0/0x220 [lin_tape]
Sep 18 13:22:38 dfsnl2 kernel: [<ffffffffc0992902>] ? tape_log_cdb+0x22/0x100 [lin_tape]
Sep 18 13:22:38 dfsnl2 kernel: [<ffffffffc09a5ac1>] tape_send_scsi_io+0x341/0x4d0 [lin_tape]
Sep 18 13:22:38 dfsnl2 kernel: [<ffffffff810f7cf5>] ? futex_wait+0x195/0x280
Sep 18 13:22:38 dfsnl2 kernel: [<ffffffffc099babf>] tape_send_scsi_cmd+0x7f/0x310 [lin_tape]
Sep 18 13:22:38 dfsnl2 kernel: [<ffffffffc098f02a>] lin_tape_perform_write+0x11a/0x260 [lin_tape]
Sep 18 13:22:38 dfsnl2 kernel: [<ffffffffc098f3cf>] lin_tape_drive_write+0x25f/0x350 [lin_tape]
Sep 18 13:22:38 dfsnl2 kernel: [<ffffffffc097112b>] lin_tape_write+0x15b/0x2e0 [lin_tape]
Sep 18 13:22:38 dfsnl2 kernel: [<ffffffff81202ced>] vfs_write+0xbd/0x1e0
Sep 18 13:22:38 dfsnl2 kernel: [<ffffffff81203aff>] SyS_write+0x7f/0xe0
Sep 18 13:22:38 dfsnl2 kernel: [<ffffffff816b89fd>] system_call_fastpath+0x16/0x1b
Sep 18 13:22:38 dfsnl2 kernel: Mem-Info:
Sep 18 13:22:38 dfsnl2 kernel: active_anon:381997 inactive_anon:133600 isolated_anon:0#012 active_file:86055 inactive_file:30654233 isolated_file:0#012 unevictable:2732 dirty:533918 writeback:0 unstable:6#012 slab_reclaimable:535681 slab_unreclaimable:45135#012 mapped:43312 shmem:4933 pagetables:10320 bounce:0#012 free:726908 free_pcp:615 free_cma:0
Sep 18 13:22:38 dfsnl2 kernel: Node 0 DMA free:9332kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15924kB managed:15840kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:120kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Sep 18 13:22:38 dfsnl2 kernel: lowmem_reserve[]: 0 1673 128641 128641
Sep 18 13:22:38 dfsnl2 kernel: Node 0 DMA32 free:509220kB min:876kB low:1092kB high:1312kB active_anon:34252kB inactive_anon:36352kB active_file:6836kB inactive_file:595116kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1967316kB managed:1715128kB mlocked:0kB dirty:23632kB writeback:0kB mapped:3160kB shmem:12kB slab_reclaimable:492120kB slab_unreclaimable:17816kB kernel_stack:1152kB pagetables:912kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Sep 18 13:22:38 dfsnl2 kernel: lowmem_reserve[]: 0 0 126968 126968
Sep 18 13:22:38 dfsnl2 kernel: Node 0 Normal free:2389080kB min:66696kB low:83368kB high:100044kB active_anon:1493736kB inactive_anon:498048kB active_file:337384kB inactive_file:122021468kB unevictable:10928kB isolated(anon):0kB isolated(file):0kB present:132120576kB managed:130015732kB mlocked:10928kB dirty:2112040kB writeback:0kB mapped:170088kB shmem:19720kB slab_reclaimable:1650604kB slab_unreclaimable:162604kB kernel_stack:19376kB pagetables:40368kB unstable:24kB bounce:0kB free_pcp:2516kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Sep 18 13:22:38 dfsnl2 kernel: lowmem_reserve[]: 0 0 0 0
Sep 18 13:22:38 dfsnl2 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 1*32kB (U) 1*64kB (U) 0*128kB 0*256kB 0*512kB 1*1024kB (U) 2*2048kB (UM) 1*4096kB (M) = 9332kB
Sep 18 13:22:38 dfsnl2 kernel: Node 0 DMA32: 171*4kB (UM) 13008*8kB (UEM) 9669*16kB (UEM) 2836*32kB (UEM) 337*64kB (UM) 30*128kB (UEM) 230*256kB (UEM) 116*512kB (UE) 15*1024kB (UE) 0*2048kB 0*4096kB = 509244kB
Sep 18 13:22:38 dfsnl2 kernel: Node 0 Normal: 185564*4kB (UEM) 102081*8kB (UEM) 41566*16kB (UEM) 5087*32kB (UEM) 57*64kB (UEM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2390392kB
Sep 18 13:22:38 dfsnl2 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Sep 18 13:22:38 dfsnl2 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Sep 18 13:22:38 dfsnl2 kernel: 30746346 total pagecache pages
Sep 18 13:22:38 dfsnl2 kernel: 0 pages in swap cache
Sep 18 13:22:38 dfsnl2 kernel: Swap cache stats: add 0, delete 0, find 0/0
Sep 18 13:22:38 dfsnl2 kernel: Free swap  = 32964604kB
Sep 18 13:22:38 dfsnl2 kernel: Total swap = 32964604kB
Sep 18 13:22:38 dfsnl2 kernel: 33525954 pages RAM
Sep 18 13:22:38 dfsnl2 kernel: 0 pages HighMem/MovableOnly
Sep 18 13:22:38 dfsnl2 kernel: 589279 pages reserved

When this happens, the machine typically has a lot of free memory (as reported by 'top'), but all of the physical RAM is being consumed by cached disk pages. This is expected because the machine is under heavy I/O load, and by default all files read or written are stored in the kernel's VFS page cache. However the kernel doesn't seem to be releasing memory pages aggressively enough, and the lin_tape driver makes a call to get 512KB of kernel RAM for a SCSI read/write operation, using the alloc_pages() kernel function, and it fails. This eventually bubbles up to the application which sees it as a failed read or write call on a file in a mounted LTFS filesystem. This makes it impossible for us to use IBM LTO tape drives with LTFS on systems which are direct attached to our SANs.

For some reason, we do not observe this problem when we have network filesystems mounted with CIFS or NFS; I do not know why it only occurs when using the StorNext SAN. We escalated this issue with Quantum several months ago and worked with them for a bit but they were unable to resolve the problem.

I fixed this problem on our systems by making a small modification to the lin_tape driver (based on v3.0.39.1). I changed the alloc_pages() function call inside of lin_tape_ioctl_tape.c to use the __GFP_NOFAIL flag in addition to GFP_KERNEL, and it seems to have completely resolved this serious problem for us.

Is there any way that I can submit this patch to a dev team at IBM for inclusion with future versions of the lin_tape driver?

Thanks!

HP LTO5 Modesens invalid field

Hello,

I´ve compiled the multivendor package, runs like a charm!
But I get an error at the mount command, I dont know what that means but mabe it is interesting for you?
"3cad LTFS30263I MODESENSE returns Invalid Field in CDB (-20501) /dev/sg0."

3cad LTFS14000I LTFS starting, LTFS version 2.4.2.1 (Prelim), log level 2.
3cad LTFS14058I LTFS Format Specification version 2.4.0.
3cad LTFS14104I Launched by "ltfs -o nonempty -o devname=/dev/sg0 /mnt/ltfs".
3cad LTFS14105I This binary is built for Linux (x86_64).
3cad LTFS14106I GCC version is 8.3.0.
3cad LTFS17087I Kernel version: Linux version 5.4.44-1-pve (root@nora) (gcc version 8.3.0 (Debian 8.3.0-6)) #1 SMP PVE 5.4.44-1 (Fri, 12 Jun 2020 08:18:46 +0200) i386.
3cad LTFS17089I Distribution: PRETTY_NAME="Debian GNU/Linux 10 (buster)".
3cad LTFS14063I Sync type is "time", Sync time is 300 sec.
3cad LTFS17085I Plugin: Loading "sg" tape backend.
3cad LTFS17085I Plugin: Loading "unified" iosched backend.
3cad LTFS14095I Set the tape device write-anywhere mode to avoid cartridge ejection.
3cad LTFS30209I Opening a device through sg-ibmtape driver (/dev/sg0).
3cad LTFS30250I Opened SCSI tape device 6.0.0.0 (/dev/sg0).
3cad LTFS30207I Vendor ID is HP .
3cad LTFS30208I Product ID is Ultrium 5-SCSI .
3cad LTFS30214I Firmware revision is Z6ED.
3cad LTFS30215I Drive serial is xxxxxxxx.
3cad LTFS30285I Reserved buffer size of /dev/sg0 is 1048576.
3cad LTFS17160I Maximum device block size is 1048576.
3cad LTFS11330I Loading cartridge.
3cad LTFS11332I Load successful.
3cad LTFS17157I Changing the drive setting to write-anywhere mode.
3cad LTFS11005I Mounting the volume.

3cad LTFS30205I MODESENSE (0x5a) returns -20501.
3cad LTFS30263I MODESENSE returns Invalid Field in CDB (-20501) /dev/sg0.

3cad LTFS11026I Performing a full medium consistency check.
3cad LTFS11233I Updating MAM coherency data.
3cad LTFS17227I Tape attribute: Vendor = IBM .
3cad LTFS17227I Tape attribute: Application Name = LTFS .
3cad LTFS17227I Tape attribute: Application Version = 2.4.2.1 .
3cad LTFS17227I Tape attribute: Medium Label = .
3cad LTFS17228I Tape attribute: Text Localization ID = 0x81.
3cad LTFS17227I Tape attribute: Barcode = .
3cad LTFS17227I Tape attribute: Application Format Version = 2.4.0 .
3cad LTFS17228I Tape attribute: Volume Lock Status = 0x00.
3cad LTFS17227I Tape attribute: Media Pool name = .
3cad LTFS14111I Initial setup completed successfully.
3cad LTFS14112I Invoke 'mount' command to check the result of final setup.
3cad LTFS14113I Specified mount point is listed if succeeded.

I can share further debug infos if you tell me what you need to fix that.

greetings from Austria

WORM support is corrupted

Describe the bug

WORM support is corrupted

To Reproduce

  1. Format worm tape on IBM enterprise tape
  2. Mount and write some files
  3. Unmount (fail because of append position error)

Expected behavior

Succesfully unmounted

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.