Error in crawl log: “The SharePoint item being crawled returned an error when attempting to download the item”

Another error in the crawl log:

SNAGHTML3dde524

Using Fiddler I figured out that the pages report an “incompatible browser” error if the SharePoint crawler opens that pages. It seems that the gantt view only renders content for an up-to-date Internet Explorer – identified by the user agent.

The solution is to change the user agent used by the SharePoint crawler component.

Microsoft’s “procmon” showed me this registry key:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Office Server\14.0\Search\Global\Gathering Manager\UserAgent

I set this to the IE9 user agent string found here: http://blogs.msdn.com/b/ie/archive/2010/03/23/introducing-ie9-s-user-agent-string.aspx

It’s:

Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)

After the next crawl the errors related to this problem where gone.

2 thoughts on “Error in crawl log: “The SharePoint item being crawled returned an error when attempting to download the item””

  1. Thanks, Ingo. Your solution worked for me.
    I had to reset the SharePoint Search Service before the next crawl in order to make the problem go away.
    Regards, Matthias

  2. Hello Ingo,
    just wanted to add, that this solution broke the crawls for applications with mixed authentication. I had to rewind to the original useragent. In an application that only uses NTLM authentication everything was fine.
    Regards, Urs

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.