Git Product home page Git Product logo

shuxiaokai / multi-cloud-transfer-tool Goto Github PK

View Code? Open in Web Editor NEW

This project forked from thecaduceus/multi-cloud-transfer-tool

0.0 1.0 0.0 1.23 MB

The most Advanced yet simple Multi Cloud tool to transfer/manage your Data from any cloud to any cloud remotely based on Rclone and other engines remotely.⚡

Home Page: https://www.caduceus.ml/Multi-Cloud-Transfer-Tool/

HTML 23.39% Jupyter Notebook 76.61%

multi-cloud-transfer-tool's Introduction

Multi Cloud Transfer (Advanced) 🔥

1.Setup and Start using Rclone on Google Colab and Create/Edit/View and delete your Rclone config file and keep them always with you using this! No matters which device you are on! Use Anywhere and Anytime and perform all Rclone task and immediately save it!

2.Implementation of Multi Cloud Transfer tool! it means you can easily transfer your all account data to any Cloud service which you like! And since it will work on Google Server, so it will be lightning fast⚡and will not consume your Internet or System Resources.😉

3.Ability to pack or extract Files or Folder in variety of formats with or without Password Protection while moving the data from One Cloud to another Cloud.

4.Various amazing Add-ons which will make your task or work simpler. It combines the power of Rclone with other engines which make a combination that can do your work by taking less efforts and giving more output.

📑 INDEX

Easily navigate through out the guide and use MCT Tool effectively and make your work simple!

⚡MCT Tool

🆕What's New!🆕

⛔NOTICE

🔌Before you Start!

⚙️ How to use?

📑Creating Config File for More Clouds

🪝Multi Cloud Transfer

🪀Using Service Accounts

🪄Special Add-ons

->📥Direct Download Links

->📃Download Links from txt file

->🎥YouTube videos/playlist & other sites

->🧲Magnet Links

->🗃️Torrent Files

->📤Upload Files to Anonfiles

->🪞Mirror GDTOT Links

->🧿Mirror HubDrive/DriveHub links

->Ⓜ️Download MEGA Links

📦Packing & Extracting Files/Folder

⚠️Error Messages

🚅Modes of Transfer

🎯Points to be Remembered

🗃️Supported Command List

🤔Why use this?

🔐 Safe or Not? ✅

⛑Contact Us!

❤️Credits & Thanks

🆕What's New!🆕

1.Package Upgrades:

->Rclone upgraded to v1.59.0

->yt-dlp upgraded to v2022.06.29

->7-Zip upgraded to v22.06.15

2.New features:

->Combine multiple remotes in one directory tree

->MCT Configuration data can be imported or exported on/from custom paths.

->Error Mitigators to easily mitigate error while retriveing/exporting configuration files.

->New test makefile mode to create a file with random content of a particular size.

->Multiple Backends to prevent any error during setup.

->Speed up hash checking by aborting the other hash if first returns nothing.

->Added --header-filename to honor the HTTP header filename directive.

->Implemented --disable-http-keep-alives to disable HTTP Keep Alives.

->Warn about inaccurate results when objects with unknown size.

3.Bug fixes:

->Addressed error where Notebook is unable to download setup when connection failed to back-end.

->Mitigated error causing 7z extraction in case of some specific files.

->Made --drive-shared-with-me work with shared drives. (Google Drive)

->Fixed 404 errors on copy/server side copy objects from public folder. (Google Drive)

->Solved No free space left error. (pCloud)

Show Update History

⛔ NOTICE

1.Project uses Rclone v1.59.0

2.For auto-config always choose No(n) because we are going to run the Rclone not on our own local device! Otherwise, login will not work.

3.Using this project for processing any illegal content (as per Google Policy) can terminate your access to Google Colab

🔌Before you Start!

1.Please Connect your Google Account, so that Rclone can easily save its progress automatically.

2.Fetch your previous saved MCT Tool config from Google Drive->My Drive->MCT Config folder to stop wasting your time.

Options:

Check_Root_MyDrive: Scan Root directory of My Drive for usable MCT Configuration files.

Custom_Import_Path: Scan a custom path (or directory) give by user for usable MCT Configuration files.

Custom_Export_Path: Export all your MCT Configuration files to a gives path.

3.Run the auto-reconnector to prevent Google Colab from getting timeout before your Transfer got Completed.

4.You can easily use the Service Accounts (if you want to) using the given code, please read "Using Service Accounts" Section to learn how to.

Options:

Upload_Service_Accounts: Upload new Service Accounts.

Save_Service_Accounts_MyDrive: Save new/existing Service Accounts to Google Drive-> My Drive

Using SA

5.Ability to easily delete your Files/Folder/Directory from the given code.

Options:

Delete: This Mode will delete the content of folder or single file.

Purge: To delete whole Directory and leaving nothing behind.

rmdir: Completely Delete the given path.

rmdirs: Remove any empty directories under the path.

Purge Options

6.As stated, you can use various add-ons and can make your hard work into effective and easy task.

7.Quickly pack or extract files or folders between any cloud with large variety of format support.

Packing & Extraction

⚙️ How to use?

1.First Open the Code on Google Colab: Open Code

2.Now, before performing any task, Download & Setup Rclone and other dependencies by running the code as shown in the Image. You can also choose different flavours of Rclone which is described as follows:

Stable: For best Stability.

Beta: For Earlier testing of newly released features in Rclone.

Force_Setup: Forcefully install Rclone by excluding all other dependencies and download them separately. Useful if any dependency (except Rclone) having temporary setup problem.

1

3.To Create/Edit/Delete/View config file/remote just run the code as shown in the Image. You can also save your Rclone config file for using it in future without doing same thing repeatedly.

Rclone config Saving options:

Upload_Config_File: Upload your already made Rclone config file to use it.

Save_Config_MyDrive: To save created Rclone Config File in Google Drive's My Drive.

Download_Config: Download Created Rclone Config file in your Device.

2

4.After Running the Code! Checkout the available options (shown in image) and type an alphabet and hit Enter key to continue, but if input filed is not visible then just click on the blinking line.

3

5.Type n and hit Enter to create a new Config file or remote and give name, so that you can identify it in future.

4

6.Now after Entering the name! You have to choose the cloud for which you have to make config file or remote (by typing its list number), like if you need to make config file or remote of Google Drive then type number 16 and hit Enter

5

7.If you don't know about Client ID and Client Secret, then just hit Enter without entering anything! Otherwise, this can be used to bypass login into your Google Account.

6

8.After that, it will ask you to grant which level of access to your Rclone config file, to choose it just type and hit enter the list number! I recommend using number 1 which is for full access.

7

9.If you want to use "Computers" folder than you can provide its root_folder_id otherwise just hit Enter to ignore.

8

10.Again! If you want to use Service Accounts, then you can fill the shown field otherwise leave it and hit Enter!

9

11.Also, for doing advanced configuration you can type y or just hit Enter to ignore it.

10

12.This is an important step! For Auto-Config write n and hit Enter because here we are using a Remote Machine which is not our Device! So always choose n otherwise login will not work. 11

13.Now Rclone will ask you to type something like rclone authorize "drive" XXXXXXXXXXX in your own Computer or Mobile Phone! Please read the following steps carefully!

For 📲Mobile Phones:

1.First Download Termux App in your Phone: Android | iOS

2.Now Run the Following Commands one by one in it!

->pkg install rclone
->Command given by Rclone in STEP 13!

3.After it! Rclone will open Browser for login into your Account!

4.Once you Successfully login! Return back to Termux App and copy the code which will appear there and paste it into the Google Colab to continue!

For 🖥️PC:

1.Download Rclone, as per the Operating-System in your PC: Download Now

For Windows:

2.After Downloading Rclone zip for Windows! Extract it and open the Folder as shown in the Image.

3.Now type CMD in the Address bar and hit Enter to configure CMD in that folder and to use Rclone.

4.It will open CMD, Now Enter the command given by Rclone in the STEP 13 written above. It will open the Browser, so just login into your account return back to CMD Window

5.Once you login successfully! a code will be visible. Just copy the code and paste it in Google Colab to continue.

For Linux:

Read Here: Rclone on Linux

For MacOS:

Read Here: Rclone on MacOS

14.Now, it will ask you if you want to use Shared Drive or not? y for Yes and n for No.

15.If you want to use Shared Drive, then it will ask you to choose the list number of Shared Drive from the list shown (of available Shared Drive in your Account.)

14

16.Finally! after above steps, it will show you the final code of Rclone config file, you can copy paste it in any Text Editor and after that type y and hit Enter to confirm and then type q to quite the configuration of file.

15 16

17.After Typing q and hitting Enter! The program will automatically save the rclone.conf file in your My Drive.

17

18.You can execute or run any Rclone command in this project! Just click on the Drop-down arrow of Advanced Task and that sit! Enjoy

18

📑Creating Config File for More Clouds

You can learn creating config file for all Cloud Services: Learn Here

🪝Multi Cloud Transfer

Let's move Data from any cloud to any cloud as you want! Since we are using Colab, so it will not consume your Internet or System Resource.

1.Before we procced! I recommend to either create/upload or fetch your Rclone Config file to Google Colab using above three methods! Creating and fetching will automatically place the file in correct location!

Options:

Check_Root_MyDrive: Scan Root directory of My Drive for usable MCT Configuration files.

Custom_Import_Path: Scan a custom path (or directory) give by user for usable MCT Configuration files.

Custom_Export_Path: Export all your MCT Configuration files to a gives path.

2.Once you done creating or fetching the config files! Just run the Code named Connect to Cloud Accounts and it will automatically detect your config file and will show you dropdown list to mount multiple remotes as shown in the image. Now select remoted and click mount to connect to those remotes between which you have to transfer your Data.

3.After mounting those remotes or Cloud Accounts, Now click on the File icon as shown in the image and then you will see two folder named drives open the second folder named drives and you will see the names of Remotes or Cloud Account which you mounted previously!

4.Now simply open that Folder and copy the file or folder path which needs to be copied and then right click on the file or folder and select Copy Path option from the list

5.After that! Paste the path of file or folder in Source field and again go to File Icon->drives and open the Cloud Account into which you want to transfer files/folders and copy its path! You can either directly copy the path of Cloud Account or of any folder inside it and paste the path in Destination field.

6.Choose the Transfer mode trough which you want to transfer your Data! Please refer to "🚅Modes of Transfer" below to learn about all of them.

7.If you want to get notified! when your transfer got completed then check the Email_notification option and provide Gmail Email ID as well as its password and the Receiver Email ID on which you want to receive the notification! (optional)

Email_notification: Select this option to enable following options.

logs: Check this option to also get log in Mail.

emailID: Enter your Gmail ID.

password: Enter Password of given Gmail ID.

Receiver_ID: The Email ID on which you want to receive the Notification.

8.Dry Run option is just to check the Source as well as destination path without copying anything. (Optional)

9.Finally! Run the code by pressing Play button to start transferring. Enjoy!🙃

🪀Using Service Accounts

Service Accounts are just like normal Google Account and thus have same Upload or Download limits as Google Account which is 750GB Upload and 10TB Download. They are used to act on behalf of a Google Account and hence we can use them to prevent hitting Google Drive limits by creating them in a bulk amount.

1.Don't have Service Accounts? Create Here

2.After creating! make the zip of accounts folder which have 100 Json files and keep its name as accounts.zip.

3.Now back to Google Colab screen and select saving options for Service Accounts to use them without uploading them again next time.

Options:

Upload_Service_Accounts: Upload new Service Accounts.

Save_Service_Accounts_MyDrive: Save new/existing Service Accounts to My Drive->MCT Config folder

4.After it! Click the file icon, then open accounts folder and right click on anyone Json file and select Copy path.

5.Run the Create/Edit/Delete Rclone Config File and paste the path in service_account_file during creating remote for Google Drive.

🪄Special Add-ons

This project comes with various amazing Add-ons which will make your task or work simpler. It combines the power of Rclone with other engines which make a combination that can do your work by taking less efforts and giving more output.

📥Direct Download Links:

This feature enables you to download files or data from any direct download to any cloud. It uses Aria engine to deliver you best and fastest performance possible. You can also download two or more links at same time with this.

Just put your direct download links in Source field and then enter the Destination path to where you want to download data. You can enter multiple Direct Download links in the source field by separating them using space between each link in the following format:

Link1 Link2

📃Download Links from txt file:

Download Links from txt file feature enables you download all links written in a txt file. This feature is useful when you have too many links and it will take too much time separate each link with space.

You just have to upload the txt file containing the download links to any cloud which you connected then have to provide its path in Source field and Destination path in Destination field to where you want to download the data. I will recommend to arrange the links in txt files in the below given format:

Link1

Link2

Link3

🎥YouTube videos/playlist & other sites

This feature gives you the flexibility to download YouTube Videos or Playlist and videos from other supported sites directly to any Cloud. It uses yt-dlp which downloads everything is maximum quality. You can download two or more videos which are not in same playlist or even two separate playlist at once. Refer to yt-dlp documentation for more information.

Just provide the YouTube link of video or playlist or link to videos located on other website (Non-DRM Protected) in Source field and Destination path in Destination field to where you want to download it. You can separate the link of videos or playlist by providing space between each in following format:

VideoLink1 VideoLink2

PlaylistLink1 PlaylistLink2

VideoLink1 PlaylistLink1

🧲Magnet Links:

Magnet Links feature allow you to download torrent using magnet link directly into any cloud with high speed. It uses LibTorrent which is a lightweight powerful BitTorrent client. You can download multiple magnet links at once, you don't have to write magnet link in Source Field.

Just provide the Destination path in Destination field where you have to download the torrent and run the code. You don't have to put magnet link in Source field because when you run the code then you will see input box where you have to put magnet link, you can provide as many magnet link as you can but only 1 magnet link per input box and hit enter! once you done adding magnet links, just type Exit and it will start downloading it.

🗃️Torrent Files:

This feature gives you ability to download torrent from .torrent files, you can provide as many torrent files as much you want. It also uses LibTorrent to deliver fast download speed.

You have to provide Destination path to where you want to download torrent data and torrent file through Choose file option. Once you provided torrent file you want to download then click Cancel Upload then it will start downloading torrent for you.

📤Upload Files to Anonfiles

Anonfiles is a anonymous file sharing platform that allow you to upload files upto 20GB without any sign up. MCT Tool enables you to upload files to Anonfiles from any cloud and in return it will give you the file download link.

1.For this! you just need to provide the Source path is Source field and run code, and once your file got uploaded then program will automatically return you the file download link.

🪞Mirror GDTOT Links

This feature automatically bypasses and clone the GDTOT Links directly into your Google Drive's My Drive, so that you can easily move the file to any cloud. You just have to provide your crypt cookie of GDTOT to use this feature.

To use this feature, you need a GDTOT link and GDTOT Crypt cookie. Here is how to get your Crypt cookie for GDTOT:

1.First open any browser and go to GDTOT Website then login with your Google Account.

2.Now open Developer Tools by pressing CTRL + SHIFT + I and go to application, cookies option is now visible as shown in the image and by opening it you will see a cookie named crypt just press on it and its value will be visible! Remove % from end and words written next to it, now you can use the final value with this tool.

3.You have to enter GDTOT Link in Source field and crypt in the input box which you will get after running the code.

🧿Mirror HubDrive/DriveHub links

Mirror HubDrive/DriveHub mode bypasees and clone the HubDrive or DriveHub links to your Google Drive's My Drive to make it possible for you to move its data to any cloud.

To use this feature, you need a DriveHub link and HubDrive Crypt cookie. Here is how to get your Crypt cookie for HubDrive:

1.First open any browser and go to HubDrive Website and then login with your Google Account.

2.Now open Developer Tools by pressing CTRL + SHIFT + I and go to application, cookies option is now visible as shown in the image and by opening it you will see a cookie named crypt just press on it and its value will be visible! Remove % from end and words written next to it, now you can use the final value with this tool.

3.You have to enter HubDrive or DriveHub Link in Source field and crypt in the input box which you will get after running the code.

Ⓜ️Download MEGA Links

This feature enables you to download MEGA Link with or without your MEGA Account incase if you want to use your MEGA Premium account directly to any cloud.

1.As stated above! you can use your MEGA Account if you want and this tool will save it automtically for future use. Enter MEGA Link in Source field and Destination in Destination field where you have to download the MEGA Files.

2.If destination path is not provided then downloaded files will be moved to downloades folder in connected runtime (Temporary storage).

📦Packing & Extracting Files/Folder

This Project comes with the ability to pack or extract of Files or Folder in variety of formats with or without Password Protection while moving the data from One Cloud to another Cloud. What you need is just Source Path which needs to be packed or extracted and Destination Path where you need the final result, so let's learn how to use this feature.😆

1.First connect the Cloud Accounts between which you need to transfer the packed or extracted data. Please read STEP 2 of "🪝Multi Cloud Transfer" section to learn how to do that!

2.Now Get both Source Path & Destination Path, Refer to STEP 3 of "🪝Multi Cloud Transfer" section to learn how to get both Path.

3.After it! Choose the action which needs to be done with the Source Path and then enter following Details:

Packed_Output_File_Name: Enter the name of Packed or Zipped file if packing anything.

Source_Path: The Path of Data which needs to be processed.

Destination_Path: The Path on which you want the final output.

34

4.Then Run the code to start your given task. Finally Enjoy!😎

⛔NOTE: Enter Source Path and Destination Path between "" to avoid any Path Error.

⚠️Error Messages

MCT Tool is a multi purpose tool, it comes with various useful features, but you may tackle some Error codes which are their to prevent wrong configurations or progress loss! Below are common Error codes which you may tackle or tackled in Tool:

1.Google Drive Error: Google Account is not connected!

This Error code is self explanatory and caused when you try to use MCT Tool without connecting your Google Account. The main aim of MCT Tool is to save/edit/export important configurations from/to your Google Drive, this is because Colab provides temporary runtimes which got reset due to inactivity.

2.Restore point not exist in your Google Drive, use additional feature of MCT Tool and it will be automatically created.

As usual, this error code is also self explanatory and caused only when you try to fetch your previously saved MCT Tool configurations from My Drive. MCT Tool creates and saves all your important input or simply values of specific variables in "MCT-Config" folder located in your My Drive and that's why it checks before fetching any configurations or simply values that if the "MCT-Config" folder exist or not. Basically this check is their to save your time and make MCT Tool work faster because it will take time to check the existance of 4-5 files at place of checking the existance of only folder and only importing configurations only if they exist.

3.MEGA Credentials not found or not valid!

This Error happens when you ask program to download MEGA Links using MEGA Account, but you not provided the MEGA Account credentials or wrong credentials.

4.Path Error: One or more required path is missing! Check if you connected Google Account or not?

When you don't provide required paths like Source path or Destination path then this problem happens. This can also be happen if Google Account is not connected but this condition is already no more exist after v4.0.2 of MCT Tool.

5.Destination field is empty, hence downloaded files will be moved to downloads folder.

If you leave Destination field empty while downloading files from MEGA Links then it will trigger this error. In that case your MEGA Link will be downloaded in "downloads" folder of runtime allotted to you by Google Colab. Basically I was tired of using sys.exit() so I ignored this error and tried to warn user at place of interuptting whole work.🤣

6.Source field can't be empty.

Uff...🤣 Read Error again! and reason that why I added this error is written in error 5 (Destination Error).

7.Input Error: Invalid input parameter! Please enter 0 or 1 as input.

MCT Tool requires user's favor before doing particular task and hence it follows a simple format where '0' (Zero) means No and '1'(One) means Yes and entering something else trigger this error.

🚅Modes of Transfer

As this project use Rclone as a base! and Rclone support multiple options to help you move or clean your Data, in this Project those options are known as Modes. Here is the list of the Modes that this Project supports:

Copy: Copy files from source to dest, skipping identical files.

Copyto: This can be used to upload single files to other than their current name.

CopyURL: Download a URL's content and copy it to the destination without saving it in temporary storage.

Move: Moves the contents of the source directory to the destination directory.

Moveto: If source:path is a file or directory then it moves it to a file or directory named dest:path.

Check: Checks the files in the source and destination match. It compares sizes and hashes (MD5 or SHA1) and logs a report of files that don't match. It doesn't alter the source or destination.

Sync: Sync the source to the destination, changing the destination only. Doesn't transfer files that are identical on source and destination, testing by size and modification time or MD5SUM. Destination is updated to match source, including deleting files if necessary (except duplicate objects, see below).

Bisync: Bisync provides a bidirectional cloud sync solution in rclone. It retains the Path1 and Path2 filesystem listings from the prior run. On each successive run it will:

Cat: Rclone cat sends any files to standard output.

CleanUp: Clean up the remote if possible. Empty the trash or delete old file versions. Not supported by all remotes.

ls: List all the objects in the path with size and path.

lsd: List all directories/containers/buckets in the path.

lsl: List all the objects in the path with size, modification time and path.

test makefile: Make files with random contents and of any size.

md5sum: Produce an md5sum file for all the objects in the path.

sha1sum: Produce a sha1sum file for all the objects in the path.

size: Return the total size and number of objects in path.

genautocomplete: Output shell completion scripts for rclone.

gendocs: Output markdown docs for rclone to the directory supplied.

obscure: Obscure password for use in the rclone.conf.

tree: List the contents of the remote in a tree like fashion.

Following Modes can easily cause Data Loss:

Delete: Remove the files in path.

Purge: Will delete whole Directory and leaving nothing behind.

rmdir: Completely Delete the given path.

rmdirs: Remove any empty directories under the path.

🎯Points to be Remembered

1.Always choose No n for Auto-Config because we are not going to use Rclone on our Local Device/Machine otherwise login will not work!

2.Don't forget to type ! before executing any custom Rclone Command.

3.For Safety! Rclone will always save progress in your My Drive! Not in any of your Shared Drive.

4.Use the Dry Run option to check both Source and Destination path without copying anything.

5.You can either choose the path of Cloud Account folder or any other folder inside it.

🗃️Supported Command List

Show List

🤔Why use this?

1.Simple UI: Rclone is really a very powerful command line tool! but some users are really not comfortable with Command line and hence it become difficult to use this type of useful tools, so this Project provides a very interactive User Interface (UI) to operate it with less inputs.

2.Rclone creates "rclone.conf" file, which is the most important thing to make rclone useful. But when you change your device, then you have to create a new rclone.conf file. But this Project saves your rclone.conf file to Google Drive and you can use the same file repeatedly, what you just need is your Google Account and nothing.

3.Heavy flexibility: You can use this project anytime & anywhere because it is totally cloud based, that means if you are moving files in your PC and want to check its progress in your phone then you just have to login on Google Collab in your Phone.

4.Variety of features: It comes with various amazing Add-ons which will make your task or work simpler. It combines the power of Rclone with other engines which make a combination that can do your work by taking less efforts and giving more output.

🔐 Safe or Not? ✅

1.Don't Worry! No data will be shared with anyone, if you use the Original code.🔒

2.This code do not share even a single piece of data to any third party source and not create any log of that!🔑

3.Do not trust any other copy of this Code.📚

⛑Contact Us!

Join our Update Channel at Telegram: Join Now!

Directly Contact the Developer using Telegram @HelpAutomatted_Bot

❤️Credits & Thanks

Dr.Caduceus: For making this Project and Guide.

Rclone: The Backbone of this Powerful Project.

multi-cloud-transfer-tool's People

Contributors

thecaduceus avatar

Watchers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.