cross-posted from: https://slrpnk.net/post/25347642
You must log in or register to comment.
- Pratai@piefed.caEnglish6 days
The moment people get off their dumb asses and realize that the entire AI infrastructure is designed around maximizing user interaction, should be the same moment people realize that it’s all bullshit and is not anywhere close to a source of useful information.
14 daysI haven’t used Google as a primary search engine in a decade or more. Nothing I do depends on Gemini (I don’t even use Apple Intelligence that’s a Gemini white label).
Personally, I think what Gemini is destroying is a market-based financial model that many came to associate with the Internet, information access and lifestyle.
Most of the websites I visit pre-date Google and some have gone as far as blocking all bots via robots.txt. They survive not via predatory ad networks but by donations and volunteers.
It’s why I like Lemmy; it follows the same model.
- floofloof@lemmy.caEnglish14 days
I don’t think today’s bots take any notice of robots.txt. It’s a relic from a more civilized time.
- Clearwater@lemmy.worldEnglish13 days
I run a few personal sites with some being available to the wider internet, and I have found that they generally do respect it. All the bots which advertise themselves as OpenAI, Anthropic, or Google do obey. However, I do occasionally see a stealth bot appear which uses a normal browser’s user agent and those just do whatever they want.
- Ecco the dolphin@lemmy.mlEnglish13 days
Meta absolutely does not respect robots.txt. they are scraping what I have at around 200 hits / minute.
I also notice there is some data center somewhere (or multiple) proxying all its requests through residential proxies so I can’t tell who is scraping. They don’t scrape like Meta though. Far slower. These are the stealthy bois using spoofed user agents you mention. Can’t tell who they are, though. How do they get so many residential IPs?
I have no organic traffic (I am literally just running a crawler tarpit. My page has nothing) so its really obvious that Im watching AI scrappers. My tarpit generates random links that all resolve to the same place so its super obvious its not human. Its also super obvious when two distinct IPs follow the same random word salad link milliseconds apart.
- jsproc@lemmy.worldEnglish13 days
It is a business model. People install free apps on their phone. These apps generate their income by acting as residential proxy for scrapers. They form a botnet of millions of ips, actual phones, without the owners knowing it.
- Clearwater@lemmy.worldEnglish13 days
I’ll have to double check my stats later. Haven’t looked at it for a while.
I certainly don’t recall them ignoring robots.txt, but I also don’t remember seeing Meta in my dash at all, so it’s entirely possible they, for wherever reason, never found my site.
They don’t—but they DO use search index data to figure out where to crawl.
- givesomefucks@lemmy.worldEnglish13 days
Nah, Gemini is really bad.
There’s a push to use AI where I work and we have our own Gemini, openAI, and Grok models and they literally check to make people are at least “trying” to use it…
Gemini will spin “thinking” for maybe 30 seconds, then something like “checking google results” and then just spitting out the same “ai overview” you’d get from searching the same thing.
All Gemini can do is Google something and summarize the first few results, but at that point a human could just pick a trustworthy site themselves instead of going with what has the best search optimization all mashed up.
With that one (and likely a lot of others) the big search engines becoming shit from AI slop and search optimization is going to also make AI even shittier, then the top links the next AI will be shittier.
We’ll get AI rewrites of AI rewrites, repeating itself exponentially faster as more money is sunk into it and hallucinations just keep repeating.
Like, this is the natural results of monopolies. Corporations get so large they can’t help but ruin the parts that work by integrating parts no one wants to try and make them profitable.
Because one corporation can’t function and be as large as Google. It’s just too much to keep organized while constantly growing profits. It’s like how the square cube law limits the size of buildings and animals.
The larger an animal gets, the weaker it gets pound for pound, until it would reach a point where it isn’t even strong enough to breathe…
That’s where these giant tech companies are headed.
- 13 days
This is exactly the kind of circular cascade failure that we are going to see. Once the AI engines all start training on their own data and sourcing each others slop it just gets worse and worse.
- givesomefucks@lemmy.worldEnglish13 days
If it wasn’t already an issue, AI companies wouldn’t be destroying old books to guarantee they weren’t training on AI slop…
The problem is if AI replaces writers, there will never be anything new to train AI on.
Humans haven’t hit a wall on innovation, were just no longer in the massive boom from early internet.
Historically the only way to cause those booms is to connect humans together to facilitate the exchange of ideas and collaboration.
We’re headed so fast to the opposite I don’t know if tech bros are too overconfident to understand they were a product of their time and not the other way around, or if they’re smart and selfish enough that they’re intentionally trying to freeze human innovation so that there won’t be another boom so no other group could clean up and replace them.
- LittleBorat3@lemmy.worldEnglish13 days
I am back to early web 2.0 in many cases , forums and such. Simply because they needlessly fucked something that worked perfectly before 2015 or so.
- billwashere@lemmy.worldEnglish13 days
I think it’s cute you think the AI sites are paying attention to robots.txt. 🤣
I don’t; I think googlebot is paying attention to it, and AI sites get a LOT of their initial visibility and site ranking from the Google index.
Case in point: the sites I spend time on that set robots.txt to deny all bots before the LLM wave started still aren’t getting hammered with AI bot requests, while the ones that didn’t are.
I can see it on my own websites as well; I’ve got one that’s in Google’s index, and it gets a constant low volume traffic from AI bots. The ones that aren’t listed? They just get traffic from exploit scanners.
- newbeni@lemmy.worldEnglish13 days
So, I disabled everything AI the I can think of, use DDG as a search engine, ad block wherever, blah blah blah, my search results SUCK. Is there a better way to fix it?
And I use Linux at home…no blows crap
- brb@sh.itjust.worksEnglish13 days
Kagi is pretty decent but the sad truth is that something like duck.ai works better as a search engine than normal search engines
- Hudell@lemmy.dbzer0.comEnglish13 days
Suck in what way? And what kind of stuff do you usually search?
I used to think search was kind of an universal thing but after looking at some people’s examples of what is a good or bad result I noticed that is definitely not the case.
For me a good search result is a list of sites that contain the words I typed on the search bar, sorted by how closely the words match and then perhaps subsorted by some website ranking system. For that, DDG has been quite decent.
Not OP, but I find the DDG results are SEO sites at the top for pages.
My search: how to fit a dog collar
Top 50 results from DDG:
fitting a dog collar is difficult and many people wonder how to fit a dog collar. This article will show you how to fit a dog collar.
A dog collar is difficult to fit. Perhaps you’ve wondered how to fit a dog collar.
To be fair, SEO ruined the internet and Internet searches well before AI and it’s not DDG’s fault. But I would love it if blatant SEO was filtered out of my results.
- Hudell@lemmy.dbzer0.comEnglish13 days
True, I think I probably just skip over those results and don’t even register them anymore. They definitely happen a lot but it’s not something I even remembered when I thought about search results.
Part of it might be how Google trained everyone to use natural language queries in their searches.
Instead of “how to fit a dog collar” does
“pet advice” +”dog collar” “guidance for new dog owners”
give you better results?
The idea is to search for a cluster of terms that are likely to be on the page you want to find, but aren’t hyper-focused like SEO pages.
- chunes@lemmy.worldEnglish13 days
Very curious whether you’re old enough to remember how google used to be in say, 2007.
- Hudell@lemmy.dbzer0.comEnglish13 days
I am and I do, but I didn’t mean to say DDG is just as good. It’s just been good enough for me.
- non_burglar@lemmy.worldEnglish13 days
robots.txt has been summarily ignored for nearly 20 years, my friend. Sorry to bear the news.
- buddascrayon@lemmy.worldEnglish13 days
Just a quick correction. Robots.txt doesn’t block anything, it simply politely asks the search engine to not look here or use what is contained within. A lot of the search engines ignore this and catalogue the contents of the website anyway. And pretty much all of the AI database downloaders ignore it with impunity.
- Telorand@reddthat.comEnglish14 days
It’s worse than that. Not only are people going with the AI Overview as gospel, the search results themselves have nothing to do with the normal SEO that’s been the standard for the last 20+ years. People are picking the AI Summary, because the search itself is fundamentally broken.
Gone are the days that website content + traffic determine the order or relevance. Now websites have to include specifically formatted fluff that the AI summarizers like in order to show as relevant results. Just ask Louis Rossman.
- Hudell@lemmy.dbzer0.comEnglish13 days
The problem with search is not that it’s broken. The problem is that an effective search system like google had on the early days is only truly effective for “power users” who try to understand how to use search effectively.
The majority of people have been using Google as if it was an AI long before it turned into one. People search for questions expecting to see answers, so search engines tried to adapt to that. And then when you’re trying to guess what the user wants to search for instead of just searching literally for what they asked, your search engine will never be 100% effective for anyone.
Like, on the old days for each person who could get great search results from Google you had 9 people who failed to use Google at all. That was one happy user and 9 very unhappy ones. Nowadays they have all 10 of them getting mediocre to ok-ish results. They are never getting anyone as satisfied as that one person used to be, but in general the average satisfaction has gone up.
- 13 days
Yeah, I do read the AI result often, but that’s mostly because the search has turned into dog shit. Gemini is retarded, to boot. I always get better answers and lists from GPT.
- fodor@lemmy.zipEnglish13 days
Well no, the title is wrong. Google itself destroyed Google Search. And the Internet does not depend on Google Search. And most of us don’t use Google’s AI.
Of course there are some real damages, but the title itself is bait click. It’s just false at face value. Do better next time.
- zerofk@lemmy.zipEnglish13 days
Most people on the fediverse might not (although I have my doubts about that), but even among my fellow developers the majority use Google Chrome and Google search. Looking at my circle outside tech, it’s pretty much everyone.
- MyVeryRealName@lemmy.worldEnglish13 days
Most people use Google’s AI. It has 2.5 billion monthly active users. 📈💸
- ElChinchilla@lemmy.worldEnglish13 days
I tried to use it yesterday to summarize a parts list a customer sent over, like combine same items, etc. It fucked it up so badly. That was my first and last true attempt to use it
- 13 days
Sadly, there is an astounding number of
stupidignorant, gullible people who have not been trained in media literacy.In fact, the problem is so epidemic we should have more organizations that create and proliferate media literacy content.
Such as If a news story sounds dubious, or if you are particularly invested in a story being true, search for other coverage from other news agencies, and (if the story isn’t verbatim) check the facts. The details that are consistent between separate coverage are more reliable than the rest.
- robador51@lemmy.mlEnglish13 days
The article mentions but then bypasses how the ai summaries link out to aggregation websites which seem to be showing AI summaries themselves, which in turn link out to other sources than the original. The problem is therefore not only google’s ai summaries, but also the ease that ai enables these other sites to publish with the speed they do (fully automatic probably), and they’re probably optimized to be featured in ai summaries.
That’s not a fight they’re going to win by writing higher quality content, and it’s only going to get worse. I’m not sure what remediation there is.
It really sucks.
- MyVeryRealName@lemmy.worldEnglish13 days
“That’s not a fight they’re going to win by writing higher quality content” - Why not?
- robador51@lemmy.mlEnglish13 days
I think because of the instant gratification a summary gives. It’s a bit like how spoilers work, no need to watch that movie anymore of you know all the plot twists. No matter how beautiful the camera work is 🤷♂️
- blackwall@lemmy.worldEnglish12 days
I (unfortunately) work in an industry where companies used to live or die by the amount of website traffics they received. 5 years ago the goal was to write the best content for your users, abiding by the EEAT principle - Experience, Expertise, Authoritativeness, Trustworthiness. Basically good advice given by proven experts. Since the rise of LLMs it’s like we don’t even consider human users anymore. All content is geared towards having your brand show up in AI search. The content itself is all AI generated drivel, but no one cares about that as long as ChatGPT mentions your name when asked about a related topic.
Edit: to answer your question more directly, websites exist for users. Your website only gets users if it appears in search. It used to be that higher quality content was the way to do that, but now that search is largely driven by gen AI, the quality of the content is less important than the specific structure and “freshness” (i.e. how recently it was published) that AI platforms prefer.
- blackwall@lemmy.worldEnglish12 days
100% agree and can confirm (from working in an industry overrun by gen AI) that the vast majority of content is now AI generated with the sole purpose of being cited in AI search, and that quantity and newness are both factors in being cited.
The remediation techniques I’ve found are to refer to “trusted” sources of information like Wikipedia or to ask a question here on Lemmy. Both are unfortunately still prone to AI generated content, but better than what typically comes up in a Google or DDG search.
- 13 days
Today I searched for a CD ISO filename on Google. Just the first part of the ISO name. I knew it was an ISO for Win 7, just not which one. I wanted to find a site that had the ISOs in an archival listing so I knew which version it was.
Google gave me zero results except AI telling me what I already knew. It’s a filename.
Zero.
Google is garbage. Google search is dead.
- Buddahriffic@lemmy.worldEnglish12 days
When I was watching Naruto Shippuden on Netflix, I eventually caught up to the episodes NF had but obviously it wasn’t the end of the series. I quickly found alternative places to stream the rest, and from there would occasionally find new animes to watch. Sites would last for a while and then get taken down, then the usual response on my end would be to just do another search and pick up where I left off.
The last time a site went down for me, I wasn’t able to find another one quickly. I’m sure they still exist, but my assumption is that the big search engines are using LLMs to filter results based on what their owners deem acceptable, at least when they can say it wasn’t legal anyways.
I’ve also been surprised at some of the “zero results” searches. Can’t think of any examples offhand, just that some were surprising.
Though I use DDG instead of Google, so YMMV and it might just be Microsoft doing this.
- CultLeader4Hire@lemmy.worldEnglish12 days
Google just, talks to me now? I’m not searching technical stuff I have an old habit of just plugging random subjects or ideas into a search bar and seeing what the internet thinks not for any reason other than I find it interesting, I’ve been doing this since the Ask Jeeves days it’s just a thing I do at this point. But now Google wants to talk to me about those ideas or topics instead of showing me what the internet at large is saying/thinking about them. It’s definitely no longer a tool for finding new or unique ways of seeing or thinking about things.
- oakward@feddit.orgEnglish11 days
Google won’t say what the internet is saying, it will say what the highest bidder is saying
- chilicheeselies@lemmy.worldEnglish12 days
Search has been bad for a long time. We are back to the alta Vista days. Aggressive SEO ruined the internet.
- mechoman444@lemmy.worldEnglish13 days
I’ve noticed an interesting phenomenon: ChatGPT’s search function is substantially better than Google’s AI search. And Google’s AI search is, frankly, so broken that it can barely be considered functional in some situations.
I think part of the problem is that Google is asking its AI to serve two fundamentally conflicting purposes. It doesn’t just want an LLM that does LLM things. It wants an LLM that also functions as a marketing engine. And that makes sense from Google’s perspective because Google is, at its core, an advertising company that happens to operate some enormously successful technology platforms. Advertising is how Google makes the overwhelming majority of its money, so naturally it wants its AI to become part of that ecosystem.
The problem is that optimizing an AI for advertising inevitably compromises its usefulness as an information system. The AI’s job should be to determine what information is most relevant to the user’s question and present it as accurately and efficiently as possible. An advertising platform has an entirely different objective: determine what can be sold, who will pay for access to the user, and how to maximize engagement and revenue. Those objectives can overlap, but they are not the same objective.
And this gets even more interesting when you consider what increasingly capable AI systems are supposed to become. The fundamental value of an LLM, AGI, or whatever comes after AGI is that it can act as an intermediary between a human being and an enormous body of information, reasoning about what the human actually needs rather than simply presenting whatever happens to be profitable to show them.
You can certainly put advertisements around an AI. You can even have an AI identify products or services that are relevant to a user. But the moment the AI itself becomes financially incentivized to favor one answer, product, company, or piece of information over another, you have fundamentally compromised the thing that makes the AI useful in the first place.
A calculator doesn’t work better because the person who made it gets paid when you enter a particular number. A search engine shouldn’t give you a different answer because someone paid it to prefer that answer. And an intelligent system certainly shouldn’t be quietly optimizing its reasoning around whoever is willing to pay for access to your attention.
The irony is that the more capable AI becomes, the less compatible that model should be with traditional advertising. Advertising depends on controlling what people see. Intelligence depends on being able to determine what is actually relevant. Those are fundamentally different incentives.
As usual, the limiting factor may not be the technology itself. It may simply be our inability to resist turning every useful invention into another machine for extracting money from one another.
- 13 days
Why does the author assume I’m dumber than them. I basically never use AI. It’s stupid and pulls shit from the Internet and just assumes it’s correct. I, on the other hand, have media literacy and can figure out which info is useful and likely accurate myself. It’s like asking a chronic liar a question. It’s a waste of time when we have DDG and a few brain cells.
- galaxy_nova@lemmy.worldEnglish13 days
The amount of times I’ve gone to a top ranked website by duckduckgo and read the first paragraph only to go “oh this is just more slop” is ridiculous. I have to append reddit to every search to get anything useful and even then it could be bots.
- heartSagan5@lemmy.zipEnglish13 days
i don’t know why Google would be okay with their AI destroying those Promotional links.
