Block LLM in a website

Here, too, there is a debate. Why should we block content scraping by artificial intelligence models and not by traditional search engines?

Let me elaborate. If your website is informational, it’s true that you wouldn’t want an LLM (Large Language Model) displaying your content, just as it’s not ideal for your content to appear in full in a rich snippet, preventing users from visiting your website.

However, if you’re selling products or services, wouldn’t it be somewhat “foolish” to hide available information from these LLMs? After all, no one can describe your products or services better than your own website. If these LLMs have access to this information, they could present positive aspects of your products or services in response to certain queries, with information sourced directly from your website.

After all, although SEOs tend to align with Google, our goal shouldn’t be simply to appear on Google or to defend it at all costs. Instead, we should adapt to whatever the user uses to search for information, as it may ultimately lead them to our website. And today, that often leads to LLMs.

On the other hand, I know that not all LLMs follow robots.txt directives. So, if you want to block them, I <a href="/en/article/blocking-access-to-user-agents/">would do it from the serve</a>r, as giving a disallow directive to their User-Agent might not have the desired effect.