user
user is a CLI renter for Sia. It is an alternative to siad's renter module.
The biggest difference is that user is a program that you invoke to perform
specific actions, whereas siad is a daemon that runs continuously in the
background. The other major difference is that siad manages your contracts for
you by select good hosts, maintaining a pool of usuable contracts, and
automatically renewing contracts when necessary. user, by contrast, offloads
these responsibilities to a muse
server.
user provides some functionality that siad does not. It allows you to upload
and download with having to run a full node; it efficiently stores small files;
it makes it easy to share files with your friends; and more. On the other hand,
since user does not run in the background, it cannot automatically repair your
files like siad does.
Setup
First you'll need to install user by checking out this repository and running
make. Run user version to confirm that it's installed.
user cannot operate on its own; it needs contracts, which it gets from a
muse server. You can specify the
address of this server using a CLI flag, but it's more convenient to add it to
your config file.
Uploading and Downloading Files
user stores and retrieves files using metafiles, which are small files
containing the metadata necessary to retrieve and modify a file stored on a
host. Uploading a file creates a metafile, and downloading a metafile creates a
file. Metafiles can be downloaded by anyone possessing contracts with the file's
hosts. You can share a metafile simply by sending it; to share multiple files,
bundle their corresponding metafiles in an archive such as a .tar or .zip.
Note that metafiles represent "snapshots" of a file at a particular time; if you share a metafile, and then modify your copy, the recipient will not see your modifications. Likewise, if the recipient modifies their copy, it will not affect your own.
The upload and download commands are straightforward:
$ user upload [file] [metafile]
$ user download [metafile] [file]
file is the path of the file to be read (during upload) or written (during
download), and metafile is the path where the file metadata will be written.
The extension for metafiles is .usa (a for "archive"). If you omit the final
argument, the name of the file or metafile is chosen automatically (by either
removing or appending the .usa extension).
When uploading, you must specify the desired redundancy of the file, which you
can do by passing the -m flag or by setting the min_shards value in your
config file. This value refers to the minimum number of hosts
that must be reachable for you to download the file. For example, if you have 10
contracts, and you upload with -m 5, you will be able to download as long as
any 5 hosts are reachable. The redundancy of the file in this example is 2x.
The upload command erasure-encodes file into "shards," encrypts each shard
with a different key, and uploads one shard to each host. The download command
is the inverse: it downloads shards from each host, decrypts them, and joins the
erasure-encoded shards back together, writing the result to file.
Uploads and downloads are resumable. If metafile already exists when starting
an upload, or if file is smaller than the target filesize when starting a
download, then these commands will pick up where they left off.
You can also upload or download multiple files by specifying a directory path
for both file and metafile. The directory structure of the metafiles will
mirror the structure of the files. This variant is strongly recommended when
uploading many small files, because it allows user to pack multiple files
into a single 4MB sector, which saves lots of bandwidth and money. (Normally,
each uploaded file must be padded to 4MB.)
It is also possible to redirect a download command:
$ user download [metafile] | wc -l
This means you can pipe downloaded files directly into other commands without creating a temporary file.
Migrating Files
When metafiles have a redundancy greater than 1x, they can still be downloaded
even if some of their hosts are unreachable. But if too many hosts become
unreachable, the metafile will be lost. For this reason, it is prudent to
re-upload your metafiles to better hosts if they are at risk of being lost.
In us, this process is called "migration."
If you have a local copy of the original file, you can re-upload it to the new
hosts immediately. If you don't have a local copy, you must download the file
first, then re-upload it to the new hosts. In user, these options are called
file and remote, respectively.
Let's assume that you uploaded a file to three hosts with min_shards = 2, and
one of them is now unresponsive. You would like to repair the missing redundancy
by migrating the shard on the unresponsive host to a new host. If you had a copy
of the original file, you could run:
$ user migrate -file=[file] [metafile]
Unfortunately, in this example, you do not have the original file. However, there are still two good hosts available, so you can download their shards and use them to reconstruct the third shard by running:
$ user migrate -remote [metafile]
Note that in a remote migration, the file is not actually downloaded to disk; it is processed piecewise in RAM. You don't need any free disk space to perform a migration.
Like uploads and downloads, migrations can be resumed if interrupted, and can also be applied to directories.
Configuration
user can be configured via a file named ~/.config/user/config.toml:
# API address of muse server.
# REQUIRED.
muse_addr = "muse.lukechampine.com/<my-muse-id>"
# API address of SHARD server.
# OPTIONAL. If not provided, the muse server will be used instead.
shard_addr = "shard.lukechampine.com"
# Minimum number of hosts required to download a file. Also controls
# file redundancy: uploading to 40 hosts with min_shards = 10 results
# in 4x redundancy.
# REQUIRED (unless the -m flag is passed to user).
min_shards = 10Extras
Uploading and Downloading with FUSE
FUSE is a technology that allows you to mount a "virtual filesystem" on your
computer. The user FUSE filesystem behaves like a normal folder, but behind
the scenes, it is transferring data to and from Sia hosts. You can upload a file
simply by copying it into the folder, or download by opening a file within the
folder.
The command to mount the virtual filesystem is:
$ user mount [metadir] [mnt]
metadir is the directory where metafiles will be written and read. Each such
metafile will correspond to a virtual file in the mnt directory. For example,
if you create bar/foo.txt in mnt, then bar/foo.txt.usa will appear in
metadir.
Unlike most user commands, mount will remain running until you stop it with
Ctrl-C. Don't kill it suddenly (e.g. by turning off your computer) or you will
almost certainly lose data. If you do experience an unclean shutdown, you may
encounter errors accessing the folder later. To fix this, run fusermount -u
on the mnt directory to forcibly unmount it.
Downloading over HTTP
user can serve a directory of metafiles over HTTP with the serve command:
$ user serve [metadir]
You can then browse to http://localhost:8080 to view the files in your web browser.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent