Wednesday 11 Jan 2023

Rclone

There are many cloud storage providers out there: AWS S3, Backblaze, OneDrive, pCloud, Mega, etc. I like to use a few of these to create cloud redundancy for essential data. It would be a pain to log into each app, or download software for every cloud provider to upload data.

The solution I use is rclone - an open source command-line application to upload/download data. I am a heavy user of rclone: it's literally the only tool I use to upload to the cloud. My use case for cloud storage is back-up only, my primary data are created and edited locally.

After a little reading, it's easy to use. Because it's a command-line application you can write scripts and set-up scheduled tasks to run those scripts. Do not trust yourself to run back ups manually - automate the process to ensure they back-up tasks are completed. Read more about my back-up strategy.

I am not going to rehash how to install, set-up etc. This is clearly detailed in the rclone manual, a place I like to visit often.

Encryption

A small note, "the cloud" is somebody else's computer. You can not trust that they encrypt or protect your data; what if they are hacked. To mitigate this, I encrypt all my data prior to uploading. Rclone allows me to do this seamlessly.

To enable encryption, first create a remote host; by now, you should have created a remote host using this guide. I append "clear" so I know not to upload this, for example "cloud_clear". Then you need to add an encryption configuration on top of the remote.

rclone config

Select n (for new remote)
Choose a name, for example "cloud"
From the list select 12

Then type the name of your clear remote, for example "cloud_clear:"
Choose filename encryption. Depending on the remote, I like to see file names so I select 3 (I like the .bin extension for me to easily see it's an rclone encryption; one remote is using legacy tools).

Next you can decide to encrypt the name of folders (directories). Some storage providers have file length restrictions, so if you run into this, edit the remote (same settings) and use simple obfuscate option. This is not secure, but better than plain readable. I had to use simple obfuscation for OneDrive.
You can now create your own SALT and password. You can hit "g" for rclone to generate its own, or add your own. Make sure they are strong, and choose 1024 bits for additional strength. Passwords should be greater than 12 characters, ideally randomly generated by a password manager, such as:

fWo&7$D75d56z9qNU@DLEa#h

Test your new encrypted remote:

rclone lsd cloud://

How I use rclone

Image copright

I use rclone to back-up my data (to the cloud). It's fast, secure and cross-platform. On Windows, I copy the rclone data to C:\Users\Me\ so I can easily run from command line as: rclone config.

This is the command I use to copy data. I do not sync data, I only copy to my cloud providers:

rclone copy C:\Users\Me\Documents\ cloud://documents/

The above command will copy all files and directories from my documents folder to the remote called cloud. This will be encrypted by my local machine and sent to the cloud.

I use rclone in the background via a script that is run on a schedule. I just use a BAT script and Windows Task Scheduler to run the script.

However, to help troubleshoot I like the script to create a log file. To do this, I add:

--log-file=C:\Users\Me\logs\cloud.log --log-level INFO

Log level "INFO" will only log added and deleted files, but if you are having problems with your back-up, or testing, change "INFO" to "DEBUG".

Conclusion

I love rclone - it's one of my favourite command-line applications. I use it often and is an integral part of my back-up strategy to "off-site" data. There is community available for support and rclone continues to be maintained.

Read more about crypt.
Read about rclone mount

Backlinks: Computing:Backup Journal:Index Journal:2023:01 Journal:2023:02:01 Journal:2023:02:06