• 2 posts
  • 15 comments
Joined 3 years ago
Cake day: December 12th, 2023

For a while now I’ve been using rsync to manage my file transfers and backups. I find it to be quite a powerful and flexible tool but sometimes it’s just a bit too awkward to use in a terminal.

I created this script to simplify the use of rsync to be a similar experience to using interactive cp (cp -i) but with a preview generated by rsync --archive --partial --progress --verbose --human-readable --dry-run. This gives me a chance to properly review any file changes before I decide to commit any changes.

For example, if I want sync the contents of ~/directory_a to ~/directory_b, I can do the following:

$ srsync ./directory_a/ ./directory_b/
Rsync File Transfer: DRY RUN
Source:		./directory_a/
Destination:	./directory_b/

sending incremental file list
./
file_1

sent 86 bytes  received 22 bytes  216.00 bytes/sec
total size is 0  speedup is 0.00 (DRY RUN)

Continue? [Yes/No]: y
Continuing...

Rsync File Transfer: TRANSFER
Source:		./directory_a/
Destination:	./directory_b/

sending incremental file list
./
file_1
              0 100%    0.00kB/s    0:00:00 (xfr#1, to-chk=0/2)

sent 122 bytes  received 38 bytes  320.00 bytes/sec
total size is 0  speedup is 0.00

I added a couple of other options to make syncing a bit easier. ./srsync --delete adds the option to delete files at the destination path and ./srsync --reverse reverses the paths without manually changing the paths in the command line. Continuing from the previous snippet:

$ rm ./directory_b/file_1
$ touch ./directory_b/file_0
$ srsync --delete --reverse ./directory_a/ ./directory_b/
Rsync File Transfer: DRY RUN, REVERSE, DELETE
Source:		./directory_b/
Destination:	./directory_a/

sending incremental file list
deleting file_1
./
file_0

sent 85 bytes  received 35 bytes  240.00 bytes/sec
total size is 0  speedup is 0.00 (DRY RUN)

Continue? [Yes/No]: y
Continuing...

Rsync File Transfer: TRANSFER, REVERSE, DELETE
Source:		./directory_b/
Destination:	./directory_a/

sending incremental file list
deleting file_1
./
file_0
              0 100%    0.00kB/s    0:00:00 (xfr#1, to-chk=0/2)

sent 125 bytes  received 55 bytes  360.00 bytes/sec
total size is 0  speedup is 0.00

For ease of use in scripts, I’ve also added options to replace the default rsync options, add to the current default options or include an SSH command for network transfers.

Here is an example script that I used for testing but also for making quick backups of a directory I’m currently working in:

#!/bin/sh

srsync \
 ${@} \
 --addon='
  --exclude=.backup/
  ' \
 './' \
 './.backup/'

And one more example script which I use to sync a remote directory to my Android phone that uses Termux:

#!/bin/sh

srsync \
 ${@} \
 --rsh='ssh -p 8022 -i ${HOME}/.ssh/id_ed25519' \
 --options='
  --partial --progress --verbose --human-readable
  --archive --chown=1000:1000
  --mkpath
  --compress
  ' \
 '[email protected]:/storage/emulated/0/sync/' \
 '/media/archive/phone/sync/'

I can easily pass the --reverse or --delete option flags to these scripts and sync in either direction with ease.

I also wrote this script to be as POSIX portable as possible. It’s only dependency is rsync and whatever dependencies rsync requires. Hopefully this means it works on any POSIX compliant shell but I’ve only tested it with busybux-ash, dash and bash so I can’t make any guarantees about other shells.

This current script has been an ongoing evolution of previous scripts I’ve made around the use of rsync, but I think this current version is everything I wanted back nearly 2 years ago when I first started working with linux and rsync.

I use it quite a lot and maybe it can be of use to someone else. If anyone has the time to review the code and offer any feedback, that would be appreciated. I am always interested in learning new POSIX scripting tricks and am open to any feedback.

  • Further down this comment chain I explain why I feel AI is causing so many issues.

    The blog post linked but the user lepire does an even better job explaining how AI destroys online communities.

    Personal convenience should not come at such a deep and destructive cost.

    I’ve been on the internet since 1998 and I’ve watched slowly as big money and bigger tech companies slowly destroy everything that made the internet inclusive and weird. I’m sure people who have had access to the Internet for longer than myself would have even more to say about the current state of the internet.

  • It’s completely overwhelming how many new projects there are that are full of AI/LLM code and documentation. AI has also crept into existing projects, including those that deal with security or encryption. It’s even in Linux now. This is all incredibly troubling from an outside perspective.

    It’s dangerous to live on top of so many layers of abstraction. If we don’t understand our tools and machinery, they will not be useful to us when they eventually break. And all things eventually break. Just because it exists and is available, does not mean it is good, acceptable or safe to use.

    It’s very saddening that the ethical issues about the use of AI is constantly being pushed to the side of every conversation. It’s frustrating to deal with people who are Pro-AI that are consistently rude and condescending. It’s frustrating for Pro-AI people to consistently ignore a person’s question to aggressively push more AI products on other people. This behaviour has killed any sense or feeling of community.

    This technology didn’t always exist. This technology doesn’t need to have such complete control over our future. It is completely acceptable for people to seek AI free alternatives and not be treat like they are troublesome idiots.

    All the rage and insecurities in the world won’t erase people’s desire to feel community and express their humanity. It’s a shame that the few who are angry and insecure are also incredibly fucking loud.

  • I’ve lost all motivation to continue any sort of self-hosting. AI/LLM has crept into every corner of the internet. It’s killed the community vibe I once grew up with. There’s a certain lifelessness with these projects and the internet as a whole that’s hard to describe but it’s just not fun anymore.

    I’m in the progress of changing my hobby to be more offline. Writing my own code for my own purpose and entertainment. I’ll still host my code on Codeberg. Codeberg, this Lemmy account and that one bash focused discord server will be my entire online presence going forward. Whatever happens, happens and I won’t feel a loss about losing access to any of those things.

    Capitalism and the forces that promote or enable capitalism just exploits everything until there is no life, fun or joy left…

  • I don’t expect perfection out of anyone. I would not make friends if I held them to such high standards or expectations. Everyone learns, adapts and grows at their own pace and I would do better to support them along the way through their journey

    Myself and the people around me are doing our best in this situation we are in with the tools that are currently available to us

    I don’t believe trying to reduce the complex human experience to 100% right or 100% wrong will lead to any useful discussion when there’s so much in the middle we could be discussing

  • This is great, thank you for taking the time for this write up :) The provided scripts are a huge help to me

    So far my only question I have is about the directories you use. I was wondering if you could provide the directories you use or even just an example so I could better understand the file tree. I’m very particular with my files and have a whole system dedicated to maintaining neat and organized files

    I agree about not using /etc for server related stuff. I keep all my server/container related stuff in /srv so it’s easier for me to manage

  • Impossible? There are people who still write code by hand. There are people who oppose AI, some more actively (or destructively) than others.

    Harder to avoid seems like a more reasonable take.

    However I am coping, by actively seeking, talking about and supporting alternatives with the hope of spreading that knowledge to those who would like to avoid the use of AI or enable those who support AI.

    Human creativity has a much longer and far more interesting history when compared to AI or machine learning. AI hasn’t always existed and does not need to have complete influence over our future.

Recently it’s come to my attention that Caddy has an AI sponsor so I have been looking at moving away from Caddy.

I’m currently looking for another reverse proxy to use in place of Caddy. For TLS I am looking into using CertBot and it appears there’s a module (https://github.com/desec-io/certbot-dns-desec) I can use that works for https://desec.io/ to handle my certs.

I have two questions, the first is about CertBot. Since Caddy is handling my certs automatically, how often would I want to renew my certs? Desec.io has this command to obtain a cert:

certbot certonly \
     --authenticator dns-desec \
     --dns-desec-credentials /etc/letsencrypt/secrets/$DOMAIN.ini \
     -d "$DOMAIN" \
     -d "*.$DOMAIN"

Would I be required to run the same command periodically to renew my cert?

My second question is a bit more open ended. I am looking to hear any suggestions or experiences about different reverse proxies that are preferably free of AI. There is a list here with some suggested alternatives: https://codeberg.org/ethical-foss/open-slopware#web-servers