awful.systems now has a board for slop-free jobs. post your favorites! trade 'em like pokemon!

This is the awful.systems community for people who are seeking or looking to hire for jobs or contract work in technology where you don’t have to touch the slop or otherwise engage with LLMs or other forms of generative AI.

To prevent any confusion, here’s our definition of slop-free job: a job where you don’t have to produce or engage with any form of LLM or generative AI output as an explicit or implicit part of your duties. This includes even uses of generative AI or LLMs that you personally think result in high-quality output; all generative AI and LLM output is slop. This is non-negotiable.

It is loosely acceptable to post a job here where executives or coworkers produce slop under their own power, but this must be made explicit in the job posting submission. Please be honest about what candidates can expect, if you have that information.

Jobs posted here may or may not be with a company you work for, but you are responsible for exercising a reasonable amount of caution regarding the accuracy of job postings you submit. There will be consequences for intentional misrepresentation.

While jobs posted here should be technology-oriented, the jobs themselves do not necessarily need to be in traditionally technical roles. Jobs in design, technical writing, product management, and similar roles are welcome.

To submit a job posting here, gather the following information:

• Job type (full-time, part time, contract work, temporary)
• Job duration if temporary or contract work
• Skills required for candidates (I’d prefer your own words for this if you have first-hand knowledge)
• Location (call out remote jobs specifically)
• Timezone, language, and residency or citizenship requirements
• Pay range and benefits if you have this information
• A link to a public job or contract posting if available
• A disclosure of any exposure to slop the candidate may encounter in the job
• If the posting is time-limited, specify when we should delete the post

Send the above and any related information to the email address blessley at working dot <the original domain this community is from> with [slop-free] somewhere in the subject line. We won’t get the post at all if you screw up either the email address or the tag.

Note that by default, we won’t share any information about who submitted the job posting, and we encourage candidates to take steps to prevent their account on fedi from being linked with their identity. If you would like credit for a job submission or applicants should list you as the person who referred them, please mention that in the job posting email.

3 days ago on the 12th I pushed a new version of the awful.systems infrastructure flake and our production ruggedized Lemmy fork to try and fix the problem where AI scrapers would come in and push over the instance, resulting in a bunch of 503s and general slowness. I’m happy to announce we’re now running with iocaine, which seems to have fixed things nicely – though as always, let me know if anything seems off and ping me on Mastodon if the instance goes down. in addition to iocaine, I’ve added a suite of live metrics available to instance admins and moderators. if I haven’t sent you credentials for this yet and you think you should have them, shoot me a message. I apologize that these changes took me so long to get working, but I’m very happy they seem to be effective. I’ve also got some notes on running iocaine in production I can share with any interested parties, and I’ll be reaching out to iocaine’s developer with some suggestions around making it a little bit easier to work with, since at this point it’s a crucial part of running an independent web site.

the treat

is still in the oven

the instance is going down for a bit at 8 AM GMT on November 12th to finally put us behind iocaine, which should hopefully reduce or eliminate the amount of 503 errors we’re seeing (since all of them are scrapers pushing over the instance) and expose some metrics for admins trying to keep track of how our instance is performing

I’m also working on a bit of a treat for our regulars who’ve been putting up with the 503s for a while. that will probably be announced a bit after the downtime completes.

Want to wade into the sandy surf of the abyss? Have a sneer percolating in your system but not enough time/energy to make a whole post about it? Go forth and be mid: Welcome to the Stubsack, your first port of call for learning fresh Awful you’ll near-instantly regret.

Any awful.systems sub may be subsneered in this subthread, techtakes or no.

If your sneer seems higher quality than you thought, feel free to cut’n’paste it into its own post — there’s no quota for posting and the bar really isn’t that high.

The post Xitter web has spawned soo many “esoteric” right wing freaks, but there’s no appropriate sneer-space for them. I’m talking redscare-ish, reality challenged “culture critics” who write about everything but understand nothing. I’m talking about reply-guys who make the same 6 tweets about the same 3 subjects. They’re inescapable at this point, yet I don’t see them mocked (as much as they should be)

Like, there was one dude a while back who insisted that women couldn’t be surgeons because they didn’t believe in the moon or in stars? I think each and every one of these guys is uniquely fucked up and if I can’t escape them, I would love to sneer at them.

(Credit and/or blame to David Gerard for starting this.)

we’re finally on Lemmy 0.19.12! check out the changes here:

• our infrastructure deployment flake
• the frontend
• the backend

here’s a quick summary of what changed:

• our instance fork is now in line with the stable upstream version of Lemmy, 0.19.12. big shoutout to @[email protected] for their significant labor in documenting the upstream changes we could expect from 0.19.3 to the new version, in figuring out what the stable version of Lemmy even is (the 0.20.0 and 1.0.0 series of releases don’t even talk to their own frontends), and to froztbyte, @[email protected], and @[email protected] for moral support during the upgrade process.
• all our instance features merged fine into the new version (this, shockingly, wasn’t the hard part).
• our Lemmy Nix module configuration has been moved out of the infrastructure repo, flakeified, and brought in line with the current state of the Lemmy NixOS module. in the process, I fixed two major bugs in the Lemmy NixOS module around secret handling and federation. I will not be upstreaming these changes because the Nix people like murderbots and fascists more than they like having contributors.
• we’re now running on the latest stable NixOS, 25.05.
• I’ve removed the infrastructure code for the now-unused staging instance; now we just have prod and dev.
• we’ve migrated to PostgreSQL 16, the version currently in use by the Lemmy Docker container.

as always, post here or in the testing thread if anything seems extremely broken

the good news: we’re now on the newest stable lemmy!

the bad news: federation feels a little off to me? sometimes this is a federation queue thing that resolves itself, sometimes it’s an indication of a problem.

things to test if you want to help out:

• see if you can see your posts on other lemmy and mastodon instances
• post here from other instances
• see if you can load communities, threads, and comments in non-local communities (this is a big one)
• see if you can load our communities from other instances and see up-to-date threads and comments
• make sure your own profile settings are as they should be
• if you aren’t getting email notifications and should be, let me know

I’ll push all my changes and post a full changelog once we know 0.19.12’s running stable!

our version of lemmy is old enough that clients like mlem are starting to break due to API drift, so I’m finally upgrading us to the latest stable version of lemmy. this will involve a bit of downtime and potentially a number of breakages; keep an eye out for anything that doesn’t look right after the upgrade and let us know!

after some extended downtime, I rolled out the following changes to our instance:

• pict-rs was migrated to version 0.4 then 0.5. this should hopefully fix an issue where pict-rs kept leaking TCP sockets and exhausting its resources, leading to our image uploads and downloads becoming non-functional. let me know if you run into any issues along those lines!
• NixOS was updated to 24.11.
• the instance’s storage was expanded by 100GB. this increased the monthly bill for our instance by €1.78 per month. to keep the bill low, I disabled an automated backup feature that became unnecessary when we started doing Restic backups.

I have one more thing I want to implement before our big Lemmy upgrade; I expect I should be able to fit it in tomorrow. I’ll update this thread with details when I start on it.

since we’ve been experiencing a few image cache breakages, I’m scheduling some maintenance for January 24th at 8AM GMT to upgrade our pict-rs version, increase the total amount of storage available to our production instance, and do a handful of other maintenance tasks. this won’t include a lemmy upgrade, but I plan to do one soon after this maintenance round. I anticipate the maintenance should take around 2-4 hours, but will post updates on the instance downtime page and Mastodon if anything changes.

this is somewhat of a bigger update, and it’s the product of a few things that have been in progress for a while:

email

email should be working again as of a couple months ago. good news: our old provider was, ahem, mildly inflating our usage to get us off their free plan, so this part of our infrastructure is going to cost a lot less than anticipated.

backups

we now have a restic-based system for distributed backups, thanks to a solid recommendation from @[email protected]. this will make us a lot more resilient to the possibility of having our host evaporate out from under us, and make other disaster scenarios much less lethal.

writefreely

I used some of the spare capacity on our staging instance to spin up a new WriteFreely instance where we can post long-form articles and other stuff that’s more suitable for a blog. post your gibberish at gibberish.awful.systems! contact me if you’d like an invite link; WriteFreely instances are particularly vulnerable to being turned into platforms for spam and nothing else, so we’re keeping this small-scale for instance regulars for now.

alongside all the ordinary WriteFreely stuff (partial federation, a ton of jank), our instance has a special feature: if you have an account, you can make a PR on this repository and once it’s merged, gibberish will automatically pull its frontend files from that repo and redeploy WriteFreely. currently this is only for the frontend, but there’s a lot you can do with that – check out the templates, pages, less, and static directories on the repo to see what gets pulled. check it out if you see some jank you want to fix! (also it’s the only way to get WriteFreely to host images as part of a post, no I’m not kidding)

what’s next?

next up, I plan to turn off Hetzner’s backups for awful.systems and use that budget to expand the node’s storage by 100GB, which should increase the monthly bill by around 2.50 euros. I want to go this route to expand our instance’s storage instead of using an object store like S3 or B2 because using block storage makes us more resilient to Hetzner or Backblaze evaporating or ending our service, and because it’s relatively easy to undo this decision if it proves not to scale, but very hard to go from using object storage back to generic block storage.

after that, it’ll be about time to carefully upgrade to the current version of Lemmy, and to get our fork (Philthy) in a better state for contributions.

as always, see our infrastructure deployment flake for more documentation and details on how all of the above works.

we’ve exceeded the usage tier for our email sending API today (and they kindly didn’t email me to tell me that was the case until we were 300% over), so email notifications might be a bit spotty/non-working for a little bit. I’m working on figuring out what we should migrate to — I’m leaning towards AWS SES as by far the cheapest option, though I’m no Amazon fan and I’m open to other options as long as they’ve got an option to send with SMTP

big update, awful.systems is now a federated lemmy instance. let me know if anything looks broken! here’s what to expect:

• to pull up an awful.systems community on another instance, just paste that community’s URL into the other instance’s search bar
• federation with other lemmy instances should work, and probably kbin too? there’s no way I can find to pull in pre-federation posts on remote instances though, so send your friends here to read the backlogs
• we can’t federate with most of mastodon right now because lemmy doesn’t implement authorized_fetch, which is a best practice setting for mastodon instances. if your instance doesn’t use it, try entering something like @sneerclub@awful.systems into your mastodon search; lemmy communities are represented to mastodon as users
• this is pretty much an experimental thing so if we have to turn it off, I’ll send out another post
• reply to this post with ideas for moderation tools and instances you’d like to see blocked (and a reason why) and we’ll take action on anything that sounds like a good idea

federation was made possible by

• lemmy’s devs skipping their release process and not telling anyone 0.18.2 was released on friday? so we’re on 0.18.2 now
• updating all of the deployment cluster’s flake inputs just in case
• @[email protected] shouting yolo