madslundt / cloud-media-scripts Goto Github PK

Upload and stream media from the cloud with or without encryption. Cache all new and recently streamed media locally to access quickly and reduce API calls

License: MIT License

Shell 100.00%

rclone cloud google-drive crypt mount stream plex plex-media-server plexdrive cloud-storage

cloud-media-scripts's Introduction

These scripts are created to have your media synced between your cloud- and local store. All media is always encrypted before being uploaded. This also means if you loose your encryption keys you can't read your media.

Plexdrive version 4.0.0 and Rclone version 1.39 is used.

The config right now is configured to have atleast 1 TB for caching and a decent internet connection. If you have a smaller drive or just want to optimize it click here.

Cloud-media-scripts included in a docker image. Check it out here

Easy install

git, curl and bash is needed to run easy install.

sudo apt-get install git-core curl bash -y

Now run:

bash <( curl -Ls https://github.com/madslundt/cloud-media-scripts/raw/master/INSTALL )

By default this will place cloud-media-scripts in the directory ./cloud-media-scripts. An extra argument can be added to change this

bash <( curl -Ls https://github.com/madslundt/cloud-media-scripts/raw/master/INSTALL ) [PATH]

This has only been tested on Ubuntu 16.04+. Please create an issue if you have any problems.

Upgrading

If you want to make sure you are using the latest version of cloud-media-scripts or needs to update it just go in to your cloud-media-scripts root folder and run INSTALL again bash INSTALL. This will ask you if you want to upgrade and will NOT modify or change your config.

Your config, cloud folders, local folders, rclone config and plexdrive will NOT be overwritten.

Content

How this works?
Installation without easy install
- Setup
- Setup cronjobs
My setup
Optimize configuration WIP
Upgrade
Donate

How this works?

Following services are used to sync, encrypt/decrypt and mount media:

Plexdrive
Rclone
UnionFS/MergerFS

This gives us a total of 5 directories:

Cloud encrypt dir: Containing encrypted data from the cloud provider (Mounted with Plexdrive)
Cloud decrypt dir: Containing decrypted data by decrypting the cloud encrypt dir (Mounted with Rclone). If encryption is turned off this will be mounted directly with Plexdrive.
Local decrypt dir: Containing local data stored locally on the hard drive.
Plexdrive temp dir: Containing temp/cache data from Plexdrive. This prevents Plexdrive from redownloading files everytime they are accessed.
Local media dir: Containing all data from cloud provider and data stored locally (Mounted with Union-FS).

Cloud data is mounted to a local folder (cloud_encrypt_dir). This folder is then decrypted and mounted to a local folder (cloud_decrypt_dir).

A local folder (local_decrypt_dir) is created to contain media stored locally. The local folder (local_decrypt_dir) and cloud folder (cloud_decrypt_dir) are then mounted to a third folder (local_media_dir) with certain permissions - local folder with Read/Write permissions. The cloud folder is set to Read-only permissions.

Everytime new media is retrieved it should be added to local_media_dir or local_decrypt_dir. By adding new data to local_media_dir it will automatically write it to local_decrypt_dir because of the permissions and the pooling priority setup. At this moment the media has not been uploaded to the cloud yet but only appears locally.

When running cloudupload it makes sure to upload the files from local_decrypt_dir to the cloud. This will only upload to the cloud and the file appears both locally and on the cloud. However, in local_media_dir it only appears as one file. If your move_ind is set to 1 it will move the file to the cloud instead of copying.

Later media is going to be removed locally from local_decrypt_dir. This is done by running rmlocal which depends on remove_files_based_on setting. remove_files_based_on can be set to space, time or instant. This command makes sure to move files to cloud and then afterwards remove them locally. Media is then only removed from local_decrypt_dir and still appears in local_media_dir because it is still accessable from the cloud.

If remove_files_based_on is set to space it will only move data to the cloud (if local media size has exceeded remove_files_when_space_exceeds GB) starting from the oldest accessed file and will only free up atleast freeup_atleast GB. If time is set it will only move files older than remove_files_older_than to the cloud. If instant is set it will make sure to move all media to the cloud for then afterwards removing it locally.

Media is always uploaded to cloud before removing it locally.

Plexdrive

Plexdrive is used to mount Google Drive to a local folder (cloud_encrypt_dir).

Plexdrive version 4.0.0 requires a running MongoDB server. This is not included in the scripts but can either be installed from .deb packages or in a Docker container.

Plexdrive create two files: config.json and token.json. This is used to get access to Google Drive. These can either be set up via Plexdrive or by using the templates located in the plexdrive directory (copy the files, name them config.json and token.json and insert your Google API details).

Rclone

Rclone is used to encrypt, decrypt and upload files to the cloud. Rclone is used to mount and decrypt Plexdrive to a different folder (cloud_decrypt_dir). Rclone encrypts and uploads from a local folder (local_decrypt_dir) to the cloud.

Rclone creates a config file: config.json. This is used to get access to the cloud provider and encryption/decryption keys. This can either be set up via Rclone or by using the templates located in the rclone directory (just copy the file and name it rclone.conf).

Some have reported permission issues with Rclone directory. If that occurs it can be fixed by setting uid/gid variables under the # Mount user Id section in config.

FS-Pooling

To pool the data together in your target media folder, either UnionFS or MergerFS is used to mount both cloud and local media to a local folder (local_media_dir).

This option can be set within config by changing pool_choice.

0 = UnionFS, this option is 100% read only.
1 = MergerFS, this option will set the local media to RO, however, RO still allows deletions /w MergerFS.
Cloud media is mounted with Read-only permissions.
Local media is mounted with Read/Write permissions.

The reason for these permissions are that when writing to the local folder (local_media_dir) it will not try to write it directly to the cloud folder, but instead to the local media (local_decrypt_dir). Later this will be encrypted and uploaded to the cloud by Rclone.

Installation without easy install

Change config to match your settings.
Change paths to config in all script files.
Run bash setup.sh and follow the instructions*.
Run ./mount.remote to mount plexdrive and decrypt by using rclone.

To unmount run ./umount.remote

*If this doesn't work, follow the manual setup instructions here.

Rclone setup

Most of the configuration to set up is done through Rclone. Read their documentation here.

3 remotes are needed:

Endpoint to your cloud storage.
- Create new remote [Press N]
- Give it a name (default in config gd)
- Choose Google Drive [Press 8]
- If you have a client id paste it here or leave it blank
- Choose headless machine [Press N]
- Open the url in your browser and enter the verification code
Encryption and decryption for your cloud storage.
- Create new remote [Press N]
- Give it the same name as the variable rclone_cloud_endpoint specified in the config but without colon (:) (default in config gd-crypt)
- Choose Encrypt/Decrypt a remote [Press 5]
- Enter the name of the endpoint created in cloud-storage appended with a colon (:) and the subfolder on your cloud. Example gd:/Media or just gd: if you have your files in root in the cloud.
- Choose how to encrypt filenames. I prefer option 2 Encrypt the filenames
- Choose to either generate your own or random password. I prefer to enter my own.
- Choose to enter pass phrase for the salt or leave it blank. I prefer to enter my own.
Encryption and decryption for your local storage.
- Create new remote [Press N]
- Give it the same name as specified in the environment variable rclone_local_endpoint but without colon (:) (default in config local-crypt)
- Choose Encrypt/Decrypt a remote [Press 5]
- Enter the encrypted folder specified as cloud_encrypt_dir in config. If you are using subdirectory on Google Drive append it to it. Example /.cloud-encrypt/Media
- Choose the same filename encrypted as you did with the cloud storage.
- Enter the same password as you did with the cloud storage.
- Enter the same pass phrase as you did with the cloud storage.

Rclone documentation if needed click here

View my example for an rclone configuration here.

Good idea to backup your Rclone configuration and Plexdrive configuration and cache for easier setup next time.

Manually

To install the necessary stuff manually do the following:

Install unionfs-fuse.
Install bc.
Install GNU screen.
Install Rclone 1.39.
Install Plexdrive 4.0.0.
Create the folders pointing, in the config file, to local_decrypt_dir and plexdrive_temp_dir.
Run rclone bin, installed in step 4, with the parameter --config=RCLONE_CONFIG config where RCLONE_CONFIG is the variable set in the config file.
Set up Google Drive remote, Crypt for Google Drive remove (rclone_cloud_endpoint) and crypt for local directory (rclone_local_endpoint).
Run plexdrive bin, installed in step 5, with the parameters --config=PLEXDRIVE_DIR --mongo-database=MONGO_DATABASE --mongo-host=MONGO_HOST --mongo-user=MONGO_USER --mongo-password=MONGO_PASSWORD. Remember to match the parameters with the variables in the config file.
Enter authorization to your Google Drive.
Cancel Plexdrive by pressing CTRL+C. Run PlexDrive with GNU screen: screen -dmS plexdrive PLEXDRIVE_BIN --config=PLEXDRIVE_DIR --mongo-database=MONGO_DATABASE --mongo-host=MONGO_HOST --mongo-user=MONGO_USER --mongo-password=MONGO_PASSWORD PLEXDRIVE_OPTIONS CLOUD_ENCRYPT_DIR.
Exit screen session by pressing CTRL+A then D.

Setup cronjobs

My suggestions for cronjobs is in the file cron. These should be inserted into crontab -e. I suggest to wait a minute to start Plex to make sure the mount is up and running.

Mount 20 seconds after boot.
Checks if mount is up 40 seconds after boot (if not it makes sure to remount).
Upload to cloud daily at 03:30 AM.
Check to remove local content every tuesday after upload has finished (this only remove files depending on the option 'space', 'time' or 'instant').
Check hourly that mount is up or else remounting.
Check every 12th hour if mount is up and running (if not it makes sure to remount). If mount is up and running, it will empty Plex trash.

If you have a small local disk you may change upload and remove local content to do it more often.

*If 'space' is set it will only move content to cloud, starting from the oldest accessed file, if media size has exceeded remove_files_when_space_exceeds, and will free up atleast freeup_atleast. If 'time' is set it will only move files to cloud older than remove_files_older_than. If 'instant' is set it will move all files to cloud when running. Only when file has been successfully moved to cloud it will be deleted locally.

Media is never deleted locally before being uploaded successfully to the cloud.

OBS: mountcheck is used to check if mount is up. I've had some problems where either Plexdrive or Rclone stops the mount. mountcheck will make sure to mount your stuff again if something like this happens.

My setup

My setup with this is quite simple.

I've an Intel NUC with only 128GB ssd. This is connected to a 4TB external hard drive that contains local media recently downloaded (local_decrypt_dir) and recently streamed media (plexdrive cache plexdrive_temp_dir).

I'm running this with up to 1TB plexdrive cache (--clear-chunk-max-size=1000G) and removing files based on space (remove_files_based_on="space") when local-decrypt-dir exceeds 2TB (remove_files_when_space_exceeds=2000) and frees up atleast 1TB (freeup_atleast=1000).

Optimize configuration WIP

Space

Right now the config is set for atleast 1 TB drive.

To use these scripts on a smaller drive, make these changes to the config:

Plexdrive

clear-chunk-max-size the allowed space for plexdrive cache. After the space of this has exceeded, the older blocks will be overwritten.
clear-chunk-age the expiration of plexdrive cache. After expiration the blocks will be deleted (this is only used when clear-chunk-max-size is removed).

Misc. config

remove_files_based_on can either be time, space or instant.
- Time will move the files to the cloud after remove_files_older_than days and afterwards remove them locally.
- Space will move the files to the cloud, starting from the oldest, when space exceeds remove_files_when_space_exceeds and free up atleast freeup_atleast GB. Afterwards these files are removed locally.
- Instant will move all the files to the cloud and afterwards remove them locally.

Internet connection

Depending on your internet connection, you can optimize when plexdrive download chunks.

Plexdrive

chunk-size the size of the chunks downloaded by Plexdrive. For faster connections increase this.

Upgrade

You can easily upgrade those scripts with the following command

git pull origin master

Donate

If you want to support the project or just buy me a beer I accept Paypal and bitcoins.

BitCoin address: 18fXu7Ty9RB4prZCpD8CDD1AyhHaRS1ef3

cloud-media-scripts's People

Contributors

Stargazers

Watchers

Forkers

the-omg axyjo antexa phiberoptick adamperyman vcctr da7oomhtpc xbl3 ronioncloud ximliu sumonst21

cloud-media-scripts's Issues

Still seeing permission issue?

I'm still seeing issues.. just tried to save a file in the local-media folder and it won't allow it. All users are uid 1000, including the dockers. Could it be because root runs the scripts? Is there another workaround? I cant find one so far.

recated@mediaserver:/local-media/Media/Shows$ echo 123 > test
-bash: test: Permission denied

Couldn't import episode /downloads/complete/tv/*redacted*/*redacted*.mkv: Access to the path "/tv/*redacted*/Season 3" is denied.

/tv for Sonarr is mapped to /local-media/Media/Shows.

Config:


#!/bin/bash
# shellcheck disable=SC2034
###############################################################################
# DIRECTORIES
###############################################################################
media_dir="/opt/cloud"

# Cloud directories
cloud_encrypt_dir="${media_dir}/.cloud-encrypt"
cloud_decrypt_dir="${media_dir}/.cloud-decrypt"

# Local directory
local_decrypt_dir="/local-decrypt"
plexdrive_temp_dir="/chunks"

# Media directory (FINAL)
local_media_dir="/local-media"


# Encrypt media (0 means no encryption)
encrypt_media=1


###############################################################################
# UNION-FS
###############################################################################
ufs_bin="/usr/bin/unionfs"

ufs_options="-o cow,allow_other,direct_io,nonempty,auto_cache,sync_read"


###############################################################################
# PLEXDRIVE
###############################################################################
plexdrive_dir="${media_dir}/plexdrive"
plexdrive_bin="${plexdrive_dir}/plexdrive-linux-amd64"

mongo_database="plexdrive"
mongo_host="localhost"
mongo_user=""
mongo_password=""

plexdrive_options="--temp=${plexdrive_temp_dir} -o allow_other,read_only --clear-chunk-max-size=300G --clear-chunk-age=24h --chunk-size=10M"


###############################################################################
# RCLONE
###############################################################################
rclone_dir="${media_dir}/rclone"
rclone_bin="${rclone_dir}/rclone"

rclone_config="${rclone_dir}/rclone.conf"
rclone_options="--buffer-size 500M --checkers 16"
rclone_mount_options="${rclone_options} --read-only --allow-non-empty --allow-other --max-read-ahead 30G"

# Rclone endpoints
rclone_cloud_endpoint="gd-crypt:"
rclone_local_endpoint="local-crypt:"

google_drive_media_directory="Media" # Empty means media files are in root


###############################################################################
# MISC. CONFIG
###############################################################################
date_format="+%F@%T"

# Select to remove media files based on 'time', 'space' or 'instant'
remove_files_based_on="instant"
#remove_files_based_on="time"
#remove_files_based_on="space"


# Remove media files based on 'time'
remove_files_older_than=60 # In days

# Remove media files based on 'space'
remove_files_when_space_exceeds=500 # In Gigabytes
freeup_atleast=300  # In Gigabytes

Not uploading to Google Drive

So I've been trying to configure everything, but no matter what it seems that the files are never encrypted and uploaded to Google Drive. Seems to never encrypt anything either. Here's the log from cloudupload.


[ 2017-10-14@12:53:07 ] Transfering file -> /mnt/Library/Plex-Servers/PleXXX/local-decrypt/PleXXX/PleXXX/TV Shows/Halo 4 - Forward Unto Dawn/season-specials.tbn to Google Drive in /PleXXX/TV Shows/Halo 4 - Forward Unto Dawn.
[ 2017-10-14@12:53:07 ] Transfering file -> /mnt/Library/Plex-Servers/PleXXX/local-decrypt/PleXXX/PleXXX/TV Shows/Halo 4 - Forward Unto Dawn/season01-banner.jpg to Google Drive in /PleXXX/TV Shows/Halo 4 - Forward Unto Dawn.
[ 2017-10-14@12:53:08 ] Transfering file -> /mnt/Library/Plex-Servers/PleXXX/local-decrypt/PleXXX/PleXXX/TV Shows/Halo 4 - Forward Unto Dawn/season-specials-banner.jpg to Google Drive in /PleXXX/TV Shows/Halo 4 - Forward Unto Dawn.
[ 2017-10-14@12:53:09 ] Transfering file -> /mnt/Library/Plex-Servers/PleXXX/local-decrypt/PleXXX/PleXXX/TV Shows/Halo 4 - Forward Unto Dawn/.actors/Tom_Green.tbn to Google Drive in /PleXXX/TV Shows/Halo 4 - Forward Unto Dawn/.actors.
[ 2017-10-14@12:53:09 ] Transfering file -> /mnt/Library/Plex-Servers/PleXXX/local-decrypt/PleXXX/PleXXX/TV Shows/Halo 4 - Forward Unto Dawn/.actors/Anna_Popplewell.tbn to Google Drive in /PleXXX/TV Shows/Halo 4 - Forward Unto Dawn/.actors.
[ 2017-10-14@12:53:10 ] Transfering file -> /mnt/Library/Plex-Servers/PleXXX/local-decrypt/PleXXX/PleXXX/TV Shows/Halo 4 - Forward Unto Dawn/tvshow.nfo to Google Drive in /PleXXX/TV Shows/Halo 4 - Forward Unto Dawn.
[ 2017-10-14@12:53:11 ] Transfering file -> /mnt/Library/Plex-Servers/PleXXX/local-decrypt/PleXXX/PleXXX/TV Shows/Halo 4 - Forward Unto Dawn/folder.jpg to Google Drive in /PleXXX/TV Shows/Halo 4 - Forward Unto Dawn.
[ 2017-10-14@12:53:12 ] Transfering file -> /mnt/Library/Plex-Servers/PleXXX/local-decrypt/PleXXX/PleXXX/TV Shows/Halo 4 - Forward Unto Dawn/poster.jpg to Google Drive in /PleXXX/TV Shows/Halo 4 - Forward Unto Dawn.
[ 2017-10-14@12:53:12 ] Transfering file -> /mnt/Library/Plex-Servers/PleXXX/local-decrypt/PleXXX/PleXXX/TV Shows/Halo 4 - Forward Unto Dawn/season01.tbn to Google Drive in /PleXXX/TV Shows/Halo 4 - Forward Unto Dawn.
root@PleXXX:/mnt/Library/Plex-Servers/PleXXX#

Not an issue

This is not an issue. Apologizes if this isn't the place for this.

Seeing how you're creating a cloud of content from ex: homeserver. Could there be a script you can run lets say on a Windows desktop at an other location and mount GDrive locally?

I have already used the current script to mount on linux. This is aimed towards Windows.

[MergerFS] Input/output error

I recently upgraded to the version using MergerFS. It turns out that I have the following error when copying a file :

root@1696351a9d23:/input cp /input/aFile.test /output/aFile.test
cp: error writing ‘/output/aFile.test’: Input/output error
cp: failed to extend ‘/output/aFile.test’: Input/output error
cp: failed to close ‘/output/aFile.test’: Input/output error
root@1696351a9d23:/input#

@Dulanic Looks like there's a connection between my mistake and the issue you reported on the MergerFS github.

trapexit/mergerfs#466

OAuth issues?

So I am now seeing a new issue with this and I'm pretty sure it's a rclone issue but I don't know why it keeps happening? Every single day I start seeing oauth errors. Any reasoning I migth be running into this? My main rclone is no problem, so I could remove the --config and just use that, but I'd like to drill down to why it keeps happening.

2018/01/12 13:39:27 Failed to create file system for "gd-crypt:/Media/Shows/Black Mirror/Season 4": failed to make remote "gd:Media/k27ti189mhh81aujfcjfe7ct90/34dic7kr6v2e6sg2dh2pq9g8ak/su4dqm6ldtrsi9c7pk8tp1houo/r19nfjmb1pmclsidsakqeibn2k" to wrap: couldn't read info about Drive: Get https://www.googleapis.com/drive/v2/about?alt=json: oauth2: cannot fetch token: 401 Unauthorized
Response: {
  "error" : "invalid_client",
  "error_description" : "The OAuth client was not found."
}

Rclone.conf issue /w setup.sh

Hey madslundt. So when I was playing around with my fork I noticed that the rclone.conf was not properly setting the local-crypt and was missing the GD folder after the local-crypt even though it shows it in the template. I tried to look through to see how you set that, but didn't figure it out off hand. Want to take a look at that?

[local-crypt]
type = crypt
remote = /opt/cloud-media-scripts/.cloud-encrypt
filename_encryption = standard

remote = <ENCRYPTED_FOLDER>/**<GOOGLE_DRIVE_MEDIA_DIRECTORY>**

FYI, my googledrive in config is Media.

How can I make /root/cloud-media-scripts/.cloud-decrypt sync simultaneously？

I have only 20g space on vps. so I would like to stream on /root/cloud-media-scripts/.cloud-decrypt dir. My idea is download file to /root/cloud-media-scripts/.local-decrypt and upload it right after download completed and remove the file. and stream the file on /root/cloud-media-scripts/.cloud-decrypt dir.
is this the correct way to use this script?
every time I execute /root/cloud-media-scripts/scripts/cloudupload the script will upload file from /root/cloud-media-scripts/.local-decrypt. However after upload completed. I still can not find the newly uploaded file under /root/cloud-media-scripts/.cloud-decrypt. Only I can found it when I reboot the server.

How can I make the file appear in /root/cloud-media-scripts/.cloud-decrypt immediately right after I upload the file?

Thanks.

Missing Directories

So at some point I noticed in plex I only had like 2 shows and I wasnt sure why. I ran the mount check and it came back as everything successfully mounted. Obviously something dropped but wasn't catching what dropped. I am assuming it had to be rclone since it was all my gdrive stuff that wasn't showing.

This also caused radarr/sonarr to go hey missing files, lets go find them again. Any other logs I could pull or things I could look into to see why it didnt identify it as dropped? My plex has been scanning for hours trying to rebuild the metadata lol

I'm not sure what else to check for the root cause? I tried to umount.remote and then mount.remote but it just hang on .cloud-decrypt mounting. But when I reboot the server it mounts no problem.

Status.log:
[ 2018-01-07@20:15:01 ] /local-media is mounted properly [ 2018-01-07@20:15:01 ] /opt/cloud/.cloud-encrypt is mounted properly [ 2018-01-07@20:15:01 ] /opt/cloud/.cloud-decrypt is mounted properly [ 2018-01-07@20:15:01 ] /chunks is mounted properly [ 2018-01-07@20:20:01 ] /local-media is mounted properly [ 2018-01-07@20:20:01 ] /opt/cloud/.cloud-encrypt is mounted properly [ 2018-01-07@20:20:01 ] /opt/cloud/.cloud-decrypt is mounted properly [ 2018-01-07@20:20:01 ] /chunks is mounted properly [ 2018-01-07@20:30:01 ] /local-media is mounted properly [ 2018-01-07@20:30:01 ] /opt/cloud/.cloud-encrypt is mounted properly [ 2018-01-07@20:30:01 ] /opt/cloud/.cloud-decrypt is mounted properly [ 2018-01-07@20:30:01 ] /chunks is mounted properly [ 2018-01-07@20:35:01 ] /local-media is mounted properly [ 2018-01-07@20:35:01 ] /opt/cloud/.cloud-encrypt is mounted properly [ 2018-01-07@20:35:01 ] /opt/cloud/.cloud-decrypt is mounted properly [ 2018-01-07@20:35:01 ] /chunks is mounted properly [ 2018-01-07@20:40:01 ] /local-media is mounted properly [ 2018-01-07@20:40:01 ] /opt/cloud/.cloud-encrypt is mounted properly [ 2018-01-07@20:40:01 ] /opt/cloud/.cloud-decrypt is mounted properly [ 2018-01-07@20:40:01 ] /chunks is mounted properly

Mount.log - You can see the last time it mounted 2 days ago and it dropped something today, but not sure when.
[ 2018-01-05@17:26:26 ] ###### Mounted successfully ###### [ 2018-01-05@18:38:11 ] ###### Start Mount ###### [ 2018-01-05@18:38:16 ] Mounting Google Drive with Plexdrive in mountpoint: /opt/cloud/.cloud-encrypt [ 2018-01-05@18:38:16 ] Mounted successfully with Plexdrive [ 2018-01-05@18:38:21 ] Mounting decrypted Google Drive with Rclone in mountpoint: /opt/cloud/.cloud-decrypt [ 2018-01-05@18:38:21 ] Mounted successfully with Rclone [ 2018-01-05@18:38:26 ] Mounting final library with UnionFS in mountpoint: /local-media [ 2018-01-05@18:38:26 ] Mounted successfully with UnionFS [ 2018-01-05@18:38:26 ] ###### Mounted successfully ###### [ 2018-01-07@20:28:59 ] ###### Start Mount ###### [ 2018-01-07@20:29:04 ] Mounting Google Drive with Plexdrive in mountpoint: /opt/cloud/.cloud-encrypt [ 2018-01-07@20:29:04 ] Mounted successfully with Plexdrive [ 2018-01-07@20:29:09 ] Mounting decrypted Google Drive with Rclone in mountpoint: /opt/cloud/.cloud-decrypt [ 2018-01-07@20:29:09 ] Mounted successfully with Rclone [ 2018-01-07@20:29:14 ] Mounting final library with UnionFS in mountpoint: /local-media [ 2018-01-07@20:29:14 ] Mounted successfully with UnionFS [ 2018-01-07@20:29:14 ] ###### Mounted successfully ######

Error /dev/fd/63: No such file or directory

So my mount stopped mounting. So I reran the curl command and now I'm getting this...

redacted@mediaserver:/opt$ sudo /bin/bash <( curl -Ls https://github.com/madslundt/cloud-media-scripts/raw/master/INSTALL ) /opt/cloud
/bin/bash: /dev/fd/63: No such file or directory
redacted@mediaserver:/opt$ (23) Failed writing body

Unknown character in config file

Hey,
Still somewhat new with git stuff, and don't really know how to pull and make changes and commit em (haven't spent much time looking it either). But anyways, there seems to be a weird character in the config file. I noticed it created a folder named "plexdrive?" and it's not usable. Here is what it looks like:

# Local directory
local_decrypt_dir="${media_dir}/.local-decrypt/Media"
plexdrive_temp_dir="${media_dir}/.local-decrypt/plexdrive"â€‹

Changing to this seems to be fine, not sure where it comes from tho.

# Local directory
local_decrypt_dir="${media_dir}/.local-decrypt/Media"
plexdrive_temp_dir="${media_dir}/.local-decrypt/plexdrive"

It isn't visible in nano when opening the script on debian, but there is definitely some kind of character there as i can delete the "blank" characters within nano. Either way, just wanted to help out.

Plexdrive

Hi
Can I use this with plexdrive 5 ?
I already have version 5 setup and I have no root access.

Buffering issue

I've experienced several buffering issues when using google drive as storage. Plexdrive has an option named 'chunk-load-ahead'. Could you expose it ?

By the way, I've seen that your merged a PR allowing deletion on the cloud mount. Would that work with google drive ? At the moment, when radarr/sonarr replaces content, I get a duplicate file on google drive as it can't delete the old. Would that PR fix this ?

Issues with updated script

Disregard, I'm an idiot.

Request to update docker container with latest version of plexdrive (5.0) and latest version of rclone

Could not find decrypted mountpoint

I'm sure this may be an easy fix. Also their may be a possiblity I did something wrong. However I've spent an entire day attempting to figure out the issue.

[ 2018-02-10@11:15:21 ] ###### Start Mount ###### [ 2018-02-10@11:15:26 ] Mounting Google Drive with Plexdrive in mountpoint: /opt/cloud-media-scripts/.cloud-encrypt [ 2018-02-10@11:15:26 ] Mounted successfully with Plexdrive [ 2018-02-10@11:15:31 ] Mounting decrypted Google Drive with Rclone in mountpoint: /opt/cloud-media-scripts/.cloud-decrypt [ 2018-02-10@11:15:31 ] Mounted successfully with Rclone [ 2018-02-10@11:15:36 ] Could not find decrypted mountpoint: /opt/cloud-media-scripts/.cloud-decrypt. Retrying in 10 seconds. [ 2018-02-10@11:15:46 ] Could not find decrypted mountpoint: /opt/cloud-media-scripts/.cloud-decrypt. Retrying in 10 seconds. [ 2018-02-10@11:15:56 ] Could not find decrypted mountpoint: /opt/cloud-media-scripts/.cloud-decrypt. Retrying in 10 seconds. [ 2018-02-10@11:16:06 ] Could not find decrypted mountpoint: /opt/cloud-media-scripts/.cloud-decrypt. Retrying in 10 seconds. [ 2018-02-10@11:16:16 ] Could not find decrypted mountpoint: /opt/cloud-media-scripts/.cloud-decrypt. Retrying in 10 seconds. [ 2018-02-10@11:16:26 ] Could not read mountpoint: /opt/cloud-media-scripts/.cloud-decrypt [ 2018-02-10@11:16:26 ] Mount failed

What else can I provide to get this up and running?

Error in cloudupload

Hey mads, can you take a look at this, I'm just realizing that cloudupload isn't working... I converted the find to a echo and ir errors outs.

dulanic@mediaserver:/opt/cloud-media-scripts/scripts$ ./cloudupload
[ 2018-03-16@14:35:12 ] ###### Start cloud upload ######
stat: missing operand
Try 'stat --help' for more information.
[ 2018-03-16@14:35:12 ] ###### Cloud upload ended successfully ######

I copied it to test and edited to echo it....

dulanic@mediaserver:/opt/cloud-media-scripts/scripts$ ./test
[ 2018-03-16@14:36:24 ] ###### Start cloud upload ######
find /local-decrypt/Media -type f -print0 | xargs -0 stat --format '%Y :%y %n' | sort -nr | cut -d: -f2- | awk '{===; print ./test}'
dulanic@mediaserver:/opt/cloud-media-scripts/scripts$ find /local-decrypt/Media -type f -print0 | xargs -0 stat --format '%Y :%y %n' | sort -nr | cut -d: -f2- | awk '{===; print ./test}'
awk: line 1: syntax error at or near ==
awk: 1: unexpected character '.'
stat: missing operand
Try 'stat --help' for more information.

mount.remote - Could not find encrypted mountpoint: /home/services/cloud-media-scripts/.cloud-decrypt

When running mount.remote, after using setup.sh, the first mount (.cloud-encrypt) and second mount (.cloud-decrypt) succeed, then .cloud-decrypt cannot be found.

Fresh Ubuntu 16.04 install, running from root for testing.

[ 2017-10-19@03:46:45 ] ###### Start Mount ######
[ 2017-10-19@03:46:50 ] Mounting Google Drive with Plexdrive in mountpoint: /home/services/cloud-media-scripts/.cloud-encrypt
[ 2017-10-19@03:46:50 ] Mounted successfully with Plexdrive
[ 2017-10-19@03:46:55 ] Mounting decrypted Google Drive with Rclone in mountpoint: /home/services/cloud-media-scripts/.cloud-decrypt
[ 2017-10-19@03:46:55 ] Mounted successfully with Rclone

Could not find decrypted mountpoint

I follow the manual untill last step. I even reinstall the system. it awyas hang here.
Do you want to start mounting now [Y/n]? y

This may take a while because Plexdrive needs to cache your files
[ 2018-02-26@13:37:39 ] ###### Start Mount ######
[ 2018-02-26@13:37:44 ] Mounting Google Drive with Plexdrive in mountpoint: /root/cloud-media-scripts/.cloud-encrypt
[ 2018-02-26@13:37:44 ] Mounted successfully with Plexdrive
[ 2018-02-26@13:37:49 ] Mounting decrypted Google Drive with Rclone in mountpoint: /root/cloud-media-scripts/.cloud-decrypt
[ 2018-02-26@13:37:49 ] Mounted successfully with Rclone
[ 2018-02-26@13:37:54 ] Could not find decrypted mountpoint: /root/cloud-media-scripts/.cloud-decrypt. Retrying in 10 seconds.
[ 2018-02-26@13:38:04 ] Could not find decrypted mountpoint: /root/cloud-media-scripts/.cloud-decrypt. Retrying in 10 seconds.
[ 2018-02-26@13:38:14 ] Could not find decrypted mountpoint: /root/cloud-media-scripts/.cloud-decrypt. Retrying in 10 seconds.
[ 2018-02-26@13:38:24 ] Could not find decrypted mountpoint: /root/cloud-media-scripts/.cloud-decrypt. Retrying in 10 seconds.
[ 2018-02-26@13:38:34 ] Could not find decrypted mountpoint: /root/cloud-media-scripts/.cloud-decrypt. Retrying in 10 seconds.
[ 2018-02-26@13:38:44 ] Could not read mountpoint: /root/cloud-media-scripts/.cloud-decrypt
[ 2018-02-26@13:38:44 ] Mount failed

how clould I fix this?

Thanks

setup.sh failure

Hello,

I am attempting to use this version to see if it works around the docker issue I opened. However,m when I run setup.sh it fails /w this:

No Rclone config exists
setup.sh: 69: setup.sh: Bad substitution

I am running ubuntu 16.04.

Config change, and suggestion

I would recommend changing a few things in your main Config file and then in your setup.sh. I would do this myself, but honestly don't use github enough to know how to properly do pulls and commits.

I would recommend adding a "install_folder" variable that will contain only cloud-media-scripts contents so it is separate from any "Media."

I modified it and have it working on my end, but I run my VM on a different device than my storage. I would also recommend putting in a mount option for your media as well and have that built into the "mount.remote" script. This mount would allow you to mount your media directories directly into your local-decrypt folder.

#!/bin/bash
# shellcheck disable=SC2034
###############################################################################
# DIRECTORIES
###############################################################################
media_dir="/mnt/Library"
install_dir="/opt/cloud-media-scripts"

# Cloud directories
cloud_encrypt_dir="${media_dir}/.cloud-encrypt"
cloud_decrypt_dir="${media_dir}/.cloud-decrypt"

# Local directory
local_decrypt_dir="${media_dir}/.local-decrypt/Media"
plexdrive_temp_dir="${media_dir}/.local-decrypt/plexdrive"

# Media directory (FINAL)
local_media_dir="${media_dir}/media"


# Encrypt media (0 means no encryption)
encrypt_media=0

I then replaced all the ${media_dir} with ${install_dir} so that rclone/plexdrive install into the scripts folder rather than having it install into the "Media" folder as well. To me its cleaner, and you can move things around without breaking stuff. The only issue you run into with this is using fstab to mount your directories. You will need to have your rclone/plexdrive mounted before you can mount your Media into local-decrypt.

EDIT - Not all instances of ${media_dir} were changed, only the ones that were relevant.

Example of setup.sh attached.
setup.zip

My current setup is like this:

/mnt/Library/.local-decrypt/
/mnt/Library/.cloud-decrypt/
/mnt/Library/media/

Then after mount.remote is run, I have my media mounted into /mnt/Library/.local-decrypt/

This way I have my already setup media being moved into the cloud without killing my currently running plex. All my media is still in the same location on my NAS, and when it finishes uploading I can just switch my library locations very easily.

This is how I set this up for me, so you may or may not see any of this as helpful.

Mongodb issue?

I am just running into problem after problem and I'm sure a lot is due to me. However, now I am seeing this when attempting to mount?

[OPT/CLOUD/PLEXDRIVE/PLEXDRIVE-LINUX-AMD64] [2017-11-03 09:06] ERROR : Could not open mongo db connection

Yet if I type mongo i get the cli interface and if I do show dbs I see plexdrive?

redacted@mediaserver:/var/lib/mongodb$ mongo
MongoDB shell version v3.4.10
connecting to: mongodb://127.0.0.1:27017
MongoDB server version: 3.4.10
Welcome to the MongoDB shell.
For interactive help, type "help".
For more comprehensive documentation, see
        http://docs.mongodb.org/
Questions? Try the support group
        http://groups.google.com/group/mongodb-user
Server has startup warnings:
2017-11-03T09:02:38.550-0500 I CONTROL  [initandlisten]
2017-11-03T09:02:38.550-0500 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2017-11-03T09:02:38.550-0500 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2017-11-03T09:02:38.550-0500 I CONTROL  [initandlisten]
2017-11-03T09:02:38.550-0500 I CONTROL  [initandlisten]
2017-11-03T09:02:38.550-0500 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2017-11-03T09:02:38.550-0500 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2017-11-03T09:02:38.550-0500 I CONTROL  [initandlisten]
> show dbs
local      0.078GB
plexdrive  0.078GB
>

Issue removing files with rmlocal.

So I got all my stuff uploaded to google drive now, but when the rmlocal script runs I receive the following error:

du: cannot access ‘'   /mnt/Library/Plex-Servers/PleXXX/local-decrypt/PleXXX/[P]/Misc/Season 02/S0002E0006.mp4'’: No such file or directory
/mnt/Library/Plex-Servers/PleXXX/scripts/rmlocal: line 126: 0 + : syntax error: operand expected (error token is "+ ")

Looking at the scripts, I think it has to do with the escape characters not existing when using $n as a parameter for fileSize=$(du -sb "$n" | awk '{print $1}') on line 124.

	find "${local_decrypt_dir}" -type f -exec stat -c "%x %n" {} \; | sort | awk '{$1=$2=$3=""; print $0}' |
		while IFS= read -r n; do
			if [ "$addedSpace" -gt "$freeup" ]; then
					spaceInGb=$(($addedSpace / 1000 / 1000 / 1000))

					spaceLeft=$(($(du -sb "$local_decrypt_dir" | awk '{print $1}') / 1000 / 1000 / 1000))
					echo "[ $(date ${date_format}) ] Removed ${spaceInGb} GB. Media in total ${spaceLeft} GB."
					break
			fi

Permissions issue?

I am running into a permission issue /w local-media folder. For some reason radarr keeps getting permission denials with directory. Radarr is run in docker.
Couldn't import episode /downloads/complete/tv/redacted: Access to the path "/tv/redacted is denied.

All source files in /downloads are owned by user 1000. /tv is a docker folder that is mapped to /local-media/Media/Shows. It seems like there is a write issue to local-media for some reason? That should write it to my local-decrypt?

Log files not saving

Submitted commit to fix. logs folder was not created in the script used by curl.

#23

Add requirements section

Is this possible to add a requirements section to the readme

I run into a loop trouble with the mongoDB, and we didn't get log to get the behavior quickly, maybe a good point to add log on plexdrive and rclone

Good work @madslundt 👍

Can't mount after update

So this was running for me as mentioned previously. I purged all of my current install for this and reran the scripts. I am finding again that it won't mount now. Below is what I am seeing and not sure what the issue is....

CLI:

dulanic@mediaserver:/opt/cloud/scripts$ ./mount.remote
[ 2018-01-08@10:05:50 ] ###### Start Mount ######
[ 2018-01-08@10:05:55 ] Mounting Google Drive with Plexdrive in mountpoint: /opt/cloud/.cloud-encrypt
[ 2018-01-08@10:05:55 ] Mounted successfully with Plexdrive
[ 2018-01-08@10:06:00 ] Could not find encrypted mountpoint: /opt/cloud/.cloud-encrypt. Retrying in 10 seconds.
[ 2018-01-08@10:06:10 ] Could not find encrypted mountpoint: /opt/cloud/.cloud-encrypt. Retrying in 10 seconds.
[ 2018-01-08@10:06:20 ] Could not find encrypted mountpoint: /opt/cloud/.cloud-encrypt. Retrying in 10 seconds.
[ 2018-01-08@10:06:30 ] Could not find encrypted mountpoint: /opt/cloud/.cloud-encrypt. Retrying in 10 seconds.
[ 2018-01-08@10:06:40 ] Could not find encrypted mountpoint: /opt/cloud/.cloud-encrypt. Retrying in 10 seconds.
[ 2018-01-08@10:06:50 ] Could not read mountpoint: /opt/cloud/.cloud-encrypt
[ 2018-01-08@10:06:50 ] Mount failed

My rclone is good....

dulanic@mediaserver:/opt/cloud/rclone$ rclone --config rclone.conf ls gd-crypt:
     1024 .fstab.swp
        4 Media/test
      306 Media/Books/.driveinfo.calibre
        2 Media/Books/.metadata.calibre

My config:

#!/bin/bash
# shellcheck disable=SC2034
###############################################################################
# DIRECTORIES
###############################################################################
media_dir="/opt/cloud"

# Cloud directories
cloud_encrypt_dir="${media_dir}/.cloud-encrypt"
cloud_decrypt_dir="${media_dir}/.cloud-decrypt"

# Local directory
local_decrypt_dir="${media_dir}/.local-decrypt/Media"
plexdrive_temp_dir="/chunks"

# Media directory (FINAL)
local_media_dir="/local-media"


# Encrypt media (0 means no encryption)
encrypt_media=1


###############################################################################
# UNION-FS
###############################################################################
ufs_bin="/usr/bin/unionfs"

# Mount user Id's. Replace $(id with a different ID if you need to override.
uid=1000
gid=1000

ufs_options="-o cow,allow_other,direct_io,nonempty,auto_cache,sync_read,uid=$uid,gid=$gid"


###############################################################################
# PLEXDRIVE
###############################################################################
plexdrive_dir="${media_dir}/plexdrive"
plexdrive_bin="${plexdrive_dir}/plexdrive-linux-amd64"

mongo_database="plexdrive"
mongo_host="192.168.2.155"
mongo_user=""
mongo_password=""

plexdrive_options="--temp=${plexdrive_temp_dir} -o allow_other,read_only --clear-chunk-max-size=300G --clear-chunk-age=24h --chunk-size=10M"


###############################################################################
# RCLONE
###############################################################################
rclone_dir="${media_dir}/rclone"
rclone_bin="${rclone_dir}/rclone"

rclone_config="${rclone_dir}/rclone.conf"
rclone_options="--buffer-size 500M --checkers 16"
rclone_mount_options="${rclone_options} --read-only --allow-non-empty --allow-other --max-read-ahead 30G"

# Rclone endpoints
rclone_cloud_endpoint="gd-crypt:"
rclone_local_endpoint="local-crypt:"

google_drive_media_directory="Media" # Empty means media files are in root


###############################################################################
# Plex (empty trash)
###############################################################################
plex_url="http://192.168.2.155:32400"
plex_token="redacted"


###############################################################################
# MISC. CONFIG
###############################################################################
date_format="+%F@%T"

# Select to remove media files based on 'time', 'space' or 'instant'
#remove_files_based_on="instant"
#remove_files_based_on="time"
remove_files_based_on="space"


# Remove media files based on 'time'
remove_files_older_than=60 # In days

# Remove media files based on 'space'
remove_files_when_space_exceeds=500 # In Gigabytes
freeup_atleast=300  # In Gigabytes

Possible Enhancement - Allow Deletion

So I know last year plexidrive added the ability to delete files. I have edited my config etc... to remove ready only and change RO to RW. So far it's working great,

Am I correct that UFS should write to local-decrypt 1st as UFS lists the local decrypt dir as 1st level? Seems to be working so far. This would allow us to delete in plex and have it remove the files from GD.

My changes currently:
Per testing it is properly loading files to decrypt when downloading.

ufs_options="-o cow,allow_other,direct_io,nonempty,auto_cache,sync_read,uid=$uid,gid=$gid"
plexdrive_options="--temp=${plexdrive_temp_dir} -o allow_other --clear-chunk-max-size=300G --clear-chunk-age=24h --chunk-size=10M"
ufs_mounts="${local_decrypt_dir}=RW:${cloud_decrypt_dir}=RW"

If you think it's a decent idea to add, I can help code it in the config/mount.remote.

Mount dropped and won't remount

So this morning I noticed my mount dropped and I tried to run mountcheck and this is all I see...

The only fix was for me to reboot. Mongodb is installed and running properly? Not sure why rebooting fixes it, but the script wont?


[ 2018-01-16@11:03:12 ] Mount has dropped. Remount in progress.
[ 2018-01-16@11:03:12 ] Unmounting...
[ 2018-01-16@11:03:12 ] Unmounting decrypted cloud mountpoint -> /opt/cloud/.cloud-decrypt
fusermount: failed to unmount /opt/cloud/.cloud-decrypt: Invalid argument
[ 2018-01-16@11:03:12 ] Unmounting encrypted cloud mountpoint -> /opt/cloud/.cloud-encrypt
fusermount: failed to unmount /opt/cloud/.cloud-encrypt: Invalid argument
[ 2018-01-16@11:03:12 ] Unmounting final library mountpoint -> /local-media
fusermount: failed to unmount /local-media: Invalid argument
[ 2018-01-16@11:03:12 ] Unmounted successfully
[ 2018-01-16@11:03:12 ] Mounting...
[ 2018-01-16@11:03:12 ] ###### Start Mount ######
[ 2018-01-16@11:03:17 ] Could not connect to Mongodb. Retrying in 5 seconds.
[ 2018-01-16@11:03:22 ] Could not connect to Mongodb. Retrying in 5 seconds.
[ 2018-01-16@11:03:27 ] Could not connect to Mongodb. Retrying in 5 seconds.
[ 2018-01-16@11:03:32 ] Could not connect to Mongodb. Retrying in 5 seconds.
[ 2018-01-16@11:03:37 ] Could not connect to Mongodb. Retrying in 5 seconds.
[ 2018-01-16@11:03:42 ] Could not connect to Mongodb. Please make sure Mongodb is running before mounting
[ 2018-01-16@11:03:42 ] Mount failed
[ 2018-01-16@11:03:42 ] Mounted successfully

madslundt / cloud-media-scripts Goto Github PK

cloud-media-scripts's Introduction

Cloud-media-scripts included in a docker image. Check it out here

Easy install

Upgrading

Content

How this works?

Plexdrive

Rclone

FS-Pooling

Installation without easy install

Rclone setup

Manually

Setup cronjobs

My setup

Optimize configuration WIP

Space

Internet connection

Upgrade

Donate

cloud-media-scripts's People

Contributors

Stargazers

Watchers

Forkers

cloud-media-scripts's Issues

Recommend Projects

Recommend Topics

Recommend Org