"Access is denied. Check that the Default Content Access Account has access to this content, or add a crawl rule to crawl this content"

Recently, in my testing environment, I encountered the following error in the crawl log: "Access is denied. Check that the Default Content Access Account has access to this content, or add a crawl rule to crawl this content". After verifying the search content service account had all the necessary permissions to access the data for indexing, I was stumped. Then I found the following article issued by Microsoft:

http://support.microsoft.com/default.aspx/kb/896861

According to the article, Windows Server 2003 SP1 includes a loopback check security feature that is designed to help prevent reflection attacks on your server. Therefore, authentication fails if the FQDN or custom host header that you use does not match the local server name.

Since my testing environment is running Windows Server 2003 SP2 and uses different host headers for different web applications in IIS 6.0, I experienced this symptom. I applied the Method 1 workaround offered in the article and was then successful in crawling content.

Note: You only receive this error message if you try to browse the Web site directly on the server. If you browse the Web site from a client computer, the Web site works as expected.