Fixing 404 File Not Found frustrations (SEM 101)

You’ve seen it. So have I. Nearly every person who has actively browsed the Web for more than 15 minutes has seen it. I’m talking about the dreaded 404 File Not Found error. When it occurs, users simply abandon their search on that site and go elsewhere. That’s a potential lost sale, subscription, or download opportunity (aka conversion) for the affected site! It has been estimated that up to 10% of traffic to large websites on the Web is looking for pages that don’t exist, so this is a big problem.

Back in the early days of the Web, when you entered an incorrect URL, you got a nearly blank, stark white screen containing nothing but the simple words, “404 File not found.” Yeah, thanks. That’s really helpful. But back in the day, when people actually wrote webpages in Notepad, that behavior was de rigueur. That was the way things were. Back then. But that was then. This is now. We can do better.

Unusable URLs

There are a great many ways for URLs to be rendered unusable. They can be mistyped or misspelled by the user, the page can be moved, renamed, or deleted by the webmaster, and the URL can be incorrectly written by external webmasters who create the outbound link from their sites to your, which is the most frustrating situation for the webmaster of the intended destination.

So when a potential customer of yours, interested in learning more about what you do or what it is your site has to offer, uses an erroneous URL for a page within your website today, what do they get? Do they get the Web’s equivalent of the blue screen of death, a useless page that stops them dead in their tracks and forces them to move on to a competitor’s site? Or do they get a page with helpful guidance that keeps them in your site, offering to assist them with finding the information they seek?

As webmaster, you control how you rename and remove pages from your site (see information on using redirects to point to moved and renamed pages in an earlier SEM 101 article). But you can’t control the content of your inbound links, nor can you nudge that user who can’t spell or remember your page-naming scheme (yet another good reminder to make the file names of your pages logical and easy to remember). So instead of cursing the darkness of silly users over whom you have no control, instead light a metaphorical candle by creating a custom 404 error page.

Custom 404 error messages

Creating a custom 404 error page is not that hard to do, but so many webmasters overlook doing this small but crucial step. Your custom 404 error page will appear in place of a generic 404 error message when the URL to your site is broken. By anticipating what users will likely want to know when they come to your site, you can proactively give them enough information in the custom error message to keep them in your site, and then provide them with easy means to find the information they want.

OK, so this is a great idea, but how do you do it? Well, I’ve got you covered there. What you do depends on which web server platform you are using to host your site. Let’s take a look at what you need to do to establish this safety net for your users. After all, adding this one little feature to your site might make the difference between a bounce and a conversion!

Dynamic or static?

Before we discuss anything else, you first must decide how you want to approach this situation. You can create a single, static custom 404 error page for your entire site. That will be pretty simple to implement and is likely a good fit for smaller sites, but larger sites might feel constrained by a single, static page. There are technologies available that allow you to create dynamic error pages based on script, and that might suit a larger site’s needs better, but the caveat exists that if the host scripting engine suffers a failure, then you’ll have no custom 404 message at all. Given that this is not a developer blog, I’ll just point all webmasters interested in creating dynamic 404 error pages out to Bing searches for information on both Internet Information Services (IIS) and Apache platforms.

If you’re interested in learning more about basic, static custom 404 pages, follow on. After all, this is SEM 101, right?

Create the custom page content

You need to decide what information and features you want to include in your custom 404 page. I suggest the following:

  1. Use the page template for your website to maintain consistency with your site’s look and feel on this new page.
  2. Include your site’s navigation scheme in this page. If you have created an HTML sitemap page or a dedicated site search box, include access to those features as well.
  3. In the page’s text, first acknowledge that the URL for the page the user was intending to see does not exist. Then offer a quick description of your site’s subject-matter theme and list the products/services/opportunities it offers. Follow that with a suggestion that the reader use the on-page site navigation (menus, sitemap, search box, etc.) to look for the information they are interested in.

Don’t add huge, hard-to-read fonts, auto-play music, videos, or animations, or include anything else that may be off-putting to the first-time reader. Leave off the advertisements as well – this will distract the user from the critical mission at hand – to get the user back onto a real page within your site. Remember, the whole point of a custom 404 error page is to prevent a bounce (aka a single page visit session in which the visitor abandons the site without visiting any other pages). Keep the page clean and easy to read.

Save this custom page file to the root directory of your site (for this discussion, I’ll call the new file 404.htm). Now let’s cover how to employ it on the various web server platforms.

Apache

Apache users can configure a special text file found in the root directory of their site to implement a custom 404 error page. The file, named .htaccess (the dot precedes the file name and contains no typical file name extension), can be edited in Notepad to include the following line (using our sample error page file):

ErrorDocument 404 /404.htm

Of course, you must name and store the custom 404 error page file identified in the location as specified or even it will return a 404 File Not Found error! Use Bing search results to find more information on what to do in Apache for custom 404 messages.

IIS

If you are using IIS, the implementation of a custom 404 page is simple. Here’s how:

  1. First, open IIS and select the website you want to customize.

In IIS versions 5 and 6:

  1. Right-click the website and select Properties.
  2. Click the Custom Errors tab.
  3. Double-click the listing for status code 404 to edit that setting.
  4. In the Edit Custom Error Properties dialog box, set the Message type drop down list to URL.
  5. In the URL text box, per our example earlier, type /404.htm.
  6. Click OK to save your work.

In IIS 7 and higher:

  1. In the right pane, double-click Error Pages.
  2. Double-click the listing for status code 404 to edit that setting.
  3. In the Response Action group, select the option Insert content few static file into the error response.
  4. Click Set.
  5. In the Root directory path text box, either type the physical path of the root directory (up to the portion of the path that branches into a specific locale directory) of the custom error page file or click Browse and navigate to the error file root directory and then click OK.
  6. In the Relative file path text box, type the relative path of the localized error file, which if our example earlier was for US English only, could be written as EN-US404.htm.
  7. If you are implementing localized versions of a custom 404 error page, ensure the Try to return the error file in the client language check box is selected, and then click OK twice to save your work.

IIS 7.0 and higher users can alternatively edit their web.config file to include a snippet of code to accomplish the same task. Using the sample file 404.htm referenced earlier, here is an example code snippet:

<?xml version=”1.0″ encoding=”UTF-8″?>
<configuration>
  <system.webServer>
    <httpErrors errorMode=”DetailedLocalOnly”>
      <remove statusCode=”404″ />
      <error statusCode=”404″ path=”404.htm” responseMode=”File” />
    </httpErrors>
  </system.webServer>
</configuration>

Check out Bing search results for more information on creating custom 404 error pages in IIS.

Bing Web Page Error Toolkit

Aside from creating your own home-grown, dynamic custom 404 page solution for IIS, there is another option for IIS users that kicks it up a notch. The Bing Developer Center team has augmented the IIS experience in creating even more useful customized 404 error pages. Straight from the Bing Developer Center blog, check out the post from this past summer, Customize your 404 error pages with the Bing API Web Page Error Toolkit. The post reveals that the toolkit, built on the Bing API, replaces the default IIS 404 error page with a dynamically created Bing search page containing customized search results derived from keywords extracted from either the source Uniform Resource Identifier (URI) or the HTTP request. This creates a search list of relevant, alternative pages on your site, helping the user more easily and quickly find the information they originally wanted without abandoning their visit to your domain.

The Bing Web Page Error Toolkit is available as a free download. If you’re using IIS, give it a try. Your users will show their gratitude with more conversions and fewer bounces! Note that you will need Microsoft Visual Studio to use this tool.

Test, test, test

Once you have implemented your custom 404 page, test it out. In a browser, type your domain name followed by a short, random string of characters you know does not match any existing file or directory name. If everything was implemented correctly, you should get the custom 404 page you created in response to that bad URL. From that point forward, your potential customers will also see that page, and if the new error page’s content is well done, they will more likely stay on your site. And that’s the goal after all, right?

If you have any questions, comments, or suggestions, feel free to post them in our SEM forum. Later…

— Rick DeJarnette, Bing Webmaster Center

* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *

UPDATE: Cool news! The Bing Developer Center team has announced today (11/18/2009) the release of the Web Page Error Toolkit for PHP users. For more information, check out the Dev Center blog post titled, Finding the page not found is now even easier.

Rick