Software for Syncing files between storages (Linux)

Syncthing? git-annex?

rsync, SyncThing, or Nextcloud. I use all three, often on the same machine at the same time, for various purposes.

rsync is great for scheduled syncs (but intentionally not TOO frequent – where blowing away one copy would almost instantly blow away the other copy could be a concern.) and outstanding for one-off syncs. With the right command line options it can even create a rough approximation of snapshots with copy-on-write rolling incremental backups. I believe this is the original documentation for that particular trick although it has spawned many genuine backup tools which operate very similarly using rsync under the hood, like rsnapshot.

SyncThing is great for large, intensive datasets that you want to keep perfectly synchronized across multiple computers at the same time. One example is my saved games, so without relying on Steam “cloud saves” or anything, I can just save my game and hop from gaming on one computer onto my laptop and pick up right where I left off.

Nextcloud is what I use for keeping separate copies of all my documents, photos from my phone, etc. It’s easier than SyncThing to setup and manage in my opinion, it’s simple and reliable and reasonably user-friendly in most use-cases, and seems to be more compatible with a lot of software due to its similar design to something like Dropbox. It isn’t ideal for handling huge numbers of files or very large folders in my personal opinion, but it does work if you want to try it.

Syncthing is the closest to what you want.

I do something like that with rsync. The syntax is a bit tricky to figure out at first, but the idea is that you take a known-good copy of your data and run rsync to copy it to the removable drive. After the first sync, you can run the command to check for any changes and push the changes from your main data to the copy on the drive.

You’ll want to research the command and the options, but you can theoretically run something like: rsync -avP --delete /mnt/Data_Drive/Data/ /run/media/user/Backup/Data/ where a is archive mode which helps to do things like preserve file attributes, v is verbose so you can see more details when you run the command, and P gives a progress line and keeps any partial files on the receiving end if the transfer is interrupted for some reason. --delete removes any files on the receiving end that have been deleted from your main data folder since the last time you’ve run rsync. The mnt folder path can be replaced with wherever your main data backup lives and the /run folder is a hypothetical destination folder on a removable drive.

An additional trick is to run the command with -n at the end of it before you run it without -n at the end. -n performs a ‘dry run’ where it will output what the changes are going to be when you actually run the sync operation. This can help you spot issues like mistyping the file path in your command. If you’re only syncing one or two changes and the dry run indicates that it is going to basically re-transfer your entire data folder, then you probably got something wrong.

Another fun thing about rsync is that if you’ve got ssh server set up on your home computers, you can run rsync to sync files from one computer to another one on your network.