We are taking a quick time-out on our series on large site optimization to provide feedback on a question we are getting asked frequently in the Webmaster Center forum.
Site owners want to control which content gets indexed. We have worked hard to ensure that we follow and respect the Robots Exclusion Protocol as directed in your robots.txt file. But occasionally there are pages that get indexed because they were not blocked or for some other technical reason and you want to have the URLs or site removed from the Live Search index.
If there is content that you want removed, the best thing to do is to specify this exclusion in either your robots.txt file, your HTTP header, or add a meta tag exclusion within the <head> tag at the top of your HTML page for example:
<meta name="robots" content="noindex"/>
Once the page is recrawled, the page will no longer be linked to in the SERP. However, if time is an issue or the page is still alive in the cache, we have a process that you can use to have URLs removed more quickly.
If you need URLs removed from our index, please take the following steps:
1. Ensure the pages that need to be removed are unavailable to the crawler. To remove a URL at the next crawl ensure that the page displays a 404 file not found error.
2. Launch the Live Search Support form. Go to the form and begin filling it out.
3. Identify from where in Live Search you want the URL removed. To quickly remove a URL, select Content Removal Request from the form’s drop-down list. Select one of these resulting options for removal:
- Remove my content. If you want the URL removed from the SERP, select this option. This is a permanent removal. Should you want this URL indexed again in the future, you will need to fill out a Content Inclusion Request from the same support form
- Cache removal. If you just want the cached page removed, use this option. Note that this will not remove the URL from our index.
4. Complete the rest of the form. Submit the URL or URLs to be removed, the query used to find the URL, complete the rest of the form, and then click Submit.
Once we have received your request, we will process the request to remove the URL within 48 hours of the request being accepted. In the future, we may provide an automated tool for these requests, but for now, this is the most effective way to quickly remove a URL from the Live Search index. If you have additional questions on controlling the content that we index, feel free to ask in our forums.
Jeremiah Andrick – Program Manager Live Search Webmaster Center