Utility for working with files and folders stored on Google Drive.
Note: skicka is not an official Google product!
skicka makes it easy to copy files to and from Google Drive and to work with files stored on Google Drive.
For example, skicka upload ~/Pictures /Pictures copies the entire
contents of the local ~/Pictures directory to a folder Pictures in
Google Drive. If you then run skicka download /Pictures ~/Pictures2,
then the contents of your ~/Pictures2 directory will match the contents
of ~/Pictures.
More generally, skicka makes it easy to list the files in Google Drive folders, compute the space used by Drive folders, and copy files between your computer to Google Drive. If you'd like to encrypt your files before uploading them, skicka supports AES-256 encryption.
skicka is not a general solution for automatically synchronizing files stored in Google Drive across multiple machines. In particular, it doesn't have logic to reconcile concurrent changes to the same file on different machines.
Furthermore, although skicka has been robust in usage so far and has no known data corruption bugs, it should for now be treated as "alpha" software. Bug reports are welcome.
- You must have a Go compiler installed.
- Download and build skicka:
go get github.com/google/skicka - Either copy the
skickaexecutable in$GOPATH/binto a directory in yourPATH, or add$GOPATH/binto yourPATH. - Run
skicka initto create a skeleton~/.skicka.configfile. Various options can be set in this file to control skicka's operation; see comments in the file for more information. - Authorize skicka to access your Google Drive files: run
skicka ls, and skicka will attempt to open an authorization page in your web browser. Click the "Accept" button to authorize skicka. You only need to perform this step once.- Alternatively, you can authorize skicka by running
skicka -no-browser-auth lsand skicka will ask you to visit a URL: go to the URL in a browser, log into your Google account (if needed), and give permission for the application to access your Google Drive files. After you click 'accept', copy the code from your browser window to the terminal with the "Enter verification code" prompt from skicka.
- Alternatively, you can authorize skicka by running
- After skicka is authorized, it will download and locally cache information about the files you have stored in Google Drive. This may take a while if you have many files on Drive; a progress indicator will try to keep you posted about how this is going. Once this data is stored locally, future runs of skicka will get going much more quickly.
- If you're going store encrypted files in Google Drive, create an
encryption key: set the environment variable
SKICKA_PASSPHRASEto hold your passphrase and runskicka genkey. Copy the lines of text that are printed to the[encryption]section of your~/.skicka.configfile. - Try it out: run
skicka ls -l /. A list of the files and folders in the root directory of your Google Drive should be printed. - Join the mailing list!
For a general overview of skicka's commands, run skicka help.
To copy a local directory hierarchy to Google Drive, use the upload
command. As the upload progresses, skicka periodically reports how
much data has been uploaded and how much time has elapsed.
% skicka upload ~/Pictures /Pictures
11 / 11 [=====================================================] 100.00 %
skicka: preparation time 1s, sync time 10s
skicka: updated 0 Drive files, 8 local files
skicka: 4.52 MiB read from disk, 0 B written to disk
skicka: 4.52 MiB uploaded (482 kiB/s), 0 B downloaded (0 B/s)
skicka: 32.76 MiB peak memory used
%
If you'd like to encrypt the files, create an encryption key as described
above in "Getting Started" and then use the -encrypt command-line
option. skicka looks for your encryption passphrase in the
SKICKA_PASSPHRASE environment variable; thus, you might use:
% env SKICKA_PASSPHRASE=mySecretPassphrase skicka upload -encrypt ~/Pictures /EncryptedPictures
To download a directory hierarchy from Google Drive to your local disk, use
download. As the download progresses, status is periodically reported:
% skicka download /Pictures/2014 ~/Pictures.copy/2014
10 / 10 [=====================================================] 100.00 %
skicka: preparation time 1s, sync time 6s
skicka: updated 0 Drive files, 10 local files
skicka: 0 B read from disk, 16.18 MiB written to disk
skicka: 0 B uploaded (0 B/s), 16.18 MiB downloaded (2.33 MiB/s)
skicka: 50.23 MiB peak memory used
%
The ls command can be used to list files and directories in Google
Drive. For example, after uploading your ~/Pictures directory, you might
run:
% skicka ls /Pictures
2012
2013
2014
To see more detail, ls -l can be used:
% skicka ls -l /Pictures/2013
-rw-r--r-- 2.62 MiB Sun Mar 10 14:41:08 2013 IMG_1127.JPG
-rw-r--r-- 2.63 MiB Sun Mar 10 14:41:09 2013 IMG_1128.JPG
-rw-r--r-- 2.32 MiB Sun Mar 10 14:41:16 2013 IMG_1129.JPG
-rw-r--r-- 2.47 MiB Sun Mar 10 14:43:16 2013 IMG_1130.JPG
[...]
In addition to the filename, we see the file size, the file permissions, and the local modification time that the file on Google Drive is synced to. The file permissions are based on the permissions of the file when it was uploaded to Google Drive.
(For even more detail, ls -ll can be used, which also prints the MD5
checksums of the files on Google Drive.)
The contents of a single file can be downloaded using cat:
% skicka cat /Pictures/2013/IMG_1129.JPG > img.jpg
Google Drive folders can be created with mkdir. (As with the Unix
command, the -p option can be specified to indicate that the intermediate
directories in the path should be created).
% skicka mkdir /Pictures/2015
The amount of space used by the children of a given Google
Drive folder can be reported by du:
% skicka du /Pictures
81.56 MiB /Pictures/2012
92.14 MiB /Pictures/2013
121.02 MiB /Pictures/2014
294.72 MiB /Pictures
You can remove a file with rm; by default, files are moved to the Trash
in Google Drive:
% skicka rm /Pictures/2014/embarassing.jpg
If you'd like to delete the file immediately with no chance to recover it,
use the -s command-line option to rm. To remove a folder and
everything inside it, use -r.
Finally, there is a fsck command that checks the file system on Google
Drive for problems and verifies that the local cache of file metadata is
in-sync with the files stored on Drive.
Yes. The only limitations are that regular Google Drive files don't store
the Unix permissions or the local modification of the original file when it
was uploaded. Therefore, in this case skicka download uses 644
permissions for files it creates and 755 permissions for directories.
skicka ls -l indicates that a file has world-readable permissions on Google Drive; does this mean anyone can access it?
No. Those permissions are only used to set the local file permissions when
the file is downloaded with skicka download. The access permissions for
the files stored on Drive are handled with Drive's regular mechanisms.
There's a fixed per-file overhead for each file uploaded to Google Drive that limits skicka to creating roughly five files a second; if files are relatively small, this overhead will be more of a limit than the time spent transferring the contents of the files.
If the uploaded files don't need to be accessed individually, creating a
tar or zip archive of them before uploading may help in this case.
A variety of transient errors can happen when using RESTful APIs like the Google Drive API. When these errors are encountered, skicka makes a number of attempts to retry the operation before giving up.
It may be that skicka should make more attempts before giving up or that there are better error handling strategies; one trade-off is that if there is a serious error (like the internet connection is lost), then it's useful for the user to know this sooner rather than later.
If you do see these errors, re-run the operation you were performing; any files that weren't transferred the first time should be taken care of with a second run.
Yes. By default, skicka doesn't try to limit its bandwidth usage. However,
if you add a line bytes-per-second-limit=... to the [upload] or
[download] section of your .skicka.config file, you can specify a
maximum number of bytes per second to transfer for uploads or downloads,
respectively.
If this line isn't present (or has a value of zero), then bandwidth won't be limited.
Yes--just set the HTTP_PROXY environment variable appropriately.
Swedish, "to send".
When a directory hierarchy is uploaded, Google Drive file is created for each local file and a Google Drive folder is created for each local directory. skicka stores the time the local copy file was last modified in the "modifiedDate" Google Drive File resource. The Unix file permissions of the file or directory are stored in using a custom "Permissions" file property, stored as a string with the octal file permissions.
See the discussion of encryption below for details about how encrypted files are represented.
When deciding if a local file needs to be uploaded to Google Drive, skicka performs the following checks.
- If there is no corresponding file on Drive, the local file will be uploaded.
- Otherwise, if there is a corresponding file on Drive and the sizes of the files are different, the local file will be uploaded.
- Otherwise, if the local file's modification time is after the modification time of
the file the last time it was synced to Drive, then an MD5 checksum of the
file contents is computed. If this checksum differs from the checksum of
the file stored on Google Drive, the file will be uploaded. (Thus, if
touchis run on a file to update its modification time but the file's contents aren't modified, skicka won't unnecessarily re-upload the file.)
Note that skicka trusts that file modification times are meaningful: if a
file's contents are modified leaving the file size is unchanged and if the
modification time is set to be in the past, then skicka won't compute an
MD5 checksum and won't know that the file should be uploaded. To override
this behavior, run skicka upload with the -ignore-times flag; if this
flag is provided, then the MD5 checksum check in the third step will be
applied regardless of the file modification time.
Note also that this algorithm is an algorithm to efficiently mirror the
contents of a set of local files on Google Drive; it's not a general
bidirectional synchronization algorithm. For example, if a file is
modified both on Drive and on the local filesystem, a skicka upload run
will clobber the file contents on Drive. In other words, the assumption is
that the source directory hierarchy is by definition the canonical one and
the destination directory's role is to perfectly reflect the source.
When downloading from Google Drive, skicka follows the same general
approach: only files that don't yet exist, have different sizes, or
different MD5 checksums from the corresponding local file will be
downloaded. The -ignore-times option can also be used to bypass the file
modification time check and to force a comparison of file contents to
decide whether to download.
If the -encrypt flag is provided to the upload command, skicka will
encrypt the contents of each file before uploading it to Google Drive.
Conversely, the download and cat commands transparently decrypt
encrypted files when downloading them (if the encryption key and passphrase
are available!)
Encryption requires both an encryption key and a passphrase; your
passphrase should be stored in the SKICKA_PASSPHRASE environment
variable. To generate an encryption key, run skicka genkey. For example:
% env SKICKA_PASSPHRASE=mySecretPassphrase skicka genkey
[...]
skicka will generate a few lines of output that are your encrypted
encryption key; to save your key, edit your ~/.skicka.config file and add
the printed text to the [encryption] section of the file. You can now
upload and encrypt files:
% env SKICKA_PASSPHRASE=mySecretPassphrase skicka upload -encrypt ~/backups /backups
If you lose your encryption key or forget your passphrase, your encrypted data is lost forever. If you use encryption, please be very careful with this information.
skicka only encrypts file contents: it doesn't encrypt filenames or hide
file sizes, for example. If this is problematic for your usage, you should
probably use tar or zip to put the files you want to encrypt into a
single file before uploading it.
When skicka genkey is executed, an encryption key is generated as
follows:
- skicka generates a random 32-byte salt using Go's
rand.Reader, which returns
cryptographically secure pseudo-random numbers. The hex-encoded salt is
printed out, and should be recorded in the
saltfield of the[encryption]section of the config file. - The user's passphrase, read from the
SKICKA_PASSPHRASEenvironment variable, is run through the PBKDF2 key derivation function, using 65536 iterations and the SHA-256 hash function to derive a 64-byte hash. - The first 32 bytes of the hash are hex encoded and printed out; they
should be copied to the
passphrase-hashfield of the[encryption]section of the config file. These bytes are later used only to validate that the user has provided the correct passphrase on subsequent runs of skicka. - A random 32-byte encryption key is generated (again with rand.Reader). This is the key that will actually be used for encryption and decryption of file contents.
- A random 16-byte initialization vector is generated with rand.Reader. It
is hex encoded and printed out, and should be copied to the
encrypted-key-ivconfiguration file field. - The encryption key from #4 is encrypted using the initialization vector
from #5, using the second 32 bytes of the hash computed in #2 as the
encryption key. The result should be copied to the
encrypted-keyfield of the config file.
Upon subsequent runs of skicka, the salt is loaded from the config file so that PBKDF2 can be used as in #2 above to hash the user's passphrase. If the first 32 bytes of the passphrase hash match the stored bytes, then the second 32 bytes of the hash and the stored IV are used to decrypt the encrypted encryption key.
Given the encryption key, when a file is to be encrypted before being uploaded to Google Drive, skicka uses the key along with a fresh 16-byte initialization vector for each file to encrypt the file using AES-256. The initialization vector is prepended to the file contents before upload. (Thus, encrypted files are 16 bytes larger on Google Drive than they are locally.)
The initialization vector is also stored hex-encoded as a Google Drive file Property with the name "IV". We store the initialization vector redundantly so that if one downloads the encrypted file contents, it's possible to decrypt the file using the file contents (and the key!) alone. Conversely, also having the IV available in a property makes it possible to encrypt the contents of a local version of a file without needing to download any of the contents of the corresponding file from Google Drive.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent