How “4XX not found” errors affect SEO

Even at the risk of writing an article that could be considered a soft 404 for using the magic words (404), I believe this is useful information for the reader and I want to contribute to the SEO community.

Since even the pages that do not exist on your website require a certain level of optimization, the truth is that a page returning a 404—no matter how much you customize it or make it readable—is officially a page that does not exist.

It’s part of the set of 4xx response codes, which are considered “client errors” (we’ll clarify what that means in a moment). Except for code 429, Google treats all of them the same way. (In another post I’ll talk about 429, how it can save you from DOS and DDOS attacks, and why Google considers it a 5XX.)

All 4xx errors, except 429, are treated the same way: Googlebot tells the indexing systems that the content does not exist. — Google Developers

So yes, according to Google, a 410 (pages that used to exist but no longer do) is treated the same as a 404. Sorry if you fell for the myth that it caused faster deindexation; officially, according to documentation and my own tests, it works the same from a technical standpoint. However, not everything is technical; it may still make sense from a UX perspective to differentiate them, as it could be better for the user.

404s from a technical perspective

404s must exist from a technical standpoint, because your website cannot contain all the infinite URL possibilities and therefore all the non-existent pages. If they don’t exist, you must indicate that they shouldn’t exist.

So yes, in case you’re wondering, 404s are our friends, and it’s not a matter of redirecting every 404 we come across. It’s also not good practice to redirect all 404s so that none exist.

"If they are supposed to be 404s, then there’s nothing to do. 404s for pages that don’t exist are fine. It’s technically correct to have them return 404. These being flagged don’t mean you’re doing something wrong, if you’re doing the 404s on purpose." —John Mu

Indexing

A 404 will never be indexed if the response code is correct. We must ensure that the response code of a 404 is indeed a 404, which we can verify with DEVTOOLS or Chrome extensions such as Redirect Path or Perseo.

It’s completely irrelevant whether that custom 404 contains index metas or canonicals, because these are signals that no search engine will take into account once it sees the response code. But we shouldn’t get too relaxed, because customizing it is important—and we’ll see why.

Linking

Whenever possible, we should avoid any type of linking to a 404 page, and we can redirect them when there are external links pointing to our page. This can be checked quickly with Ahrefs, or if your budget is tighter, with Bing Webmaster.

Forcing 404s

It’s possible that we find pages that “exist” for reasons we don’t fully control—pages generated by the CMS, or even a SEO virus, or blocks of pages inherited from older projects.

We can make the server override them and return a 404, even if those pages technically “exist” because they’re generated automatically.

404s from a Usability Perspective

It turns out we also need to help the user as much as possible, and to do that we must provide response codes that not only inform the user of what happened, but also help them understand how to proceed once it happens. Not everyone has the quick reflex of going back, and if they came from another URL, the user has already left our website. In other words, it must be a functional page.

Although we have to keep in mind an interesting point. Besides usability, they also help with ranking.

Who said a non-indexable page can’t help with our positioning?

Well, it can help, and so much that Google recommends having a well-built and customized 404 to improve user experience. They even say it in their official guidelines, section 9.2.

That guide is for Google’s human evaluators, who must assess the UX of a page. This section explains how a well-implemented custom 404 can earn a good score.

What parameters do evaluators consider when assessing 404s?

Well-designed and useful error message

• The page must clearly explain the issue to the user and confirm that the URL does not exist or has changed.

Satisfactory main content (MC)

• The 404 page must offer more than a simple error notice.
• It may include original and useful elements such as images, graphics, a personalized message, or a creative explanation.

Options to navigate the site

• It should provide relevant links that help the user find the information they were looking for.
• A well-implemented internal search bar is a positive factor.

Effort and creativity in the design

• A 404 created with effort and originality improves user experience.
• It can include illustrations, humor, or interactive elements that make the page memorable.

Positive reputation of the website

• If the page belongs to a site with excellent reputation and authority, it is more likely to receive a high score.

Satisfactory user experience

• The page should convey a high level of professionalism and care for user experience.
• If the 404 page is fun, useful, and well designed, it is considered high quality.

404 on my social media

Discrepancies

MJ Cachón conducted a test to check the effectiveness of deindexation. In her tests on one website (it would be interesting to see a larger and more up-to-date sample), her conclusions were that a 410 did deindex faster than a 404.