Error in crawl log: "The SharePoint item being crawled returned an error when attempting to download the item"

April 20, 2012 · Ingo Karstein

SharePoint Search Config Walkthrough SharePoint 2010

Another error in the crawl log:

Using Fiddler I figured out that the pages report an “incompatible browser” error if the SharePoint crawler opens that pages. It seems that the gantt view only renders content for an up-to-date Internet Explorer - identified by the user agent.

The solution is to change the user agent used by the SharePoint crawler component.

Microsoft’s “procmon” showed me this registry key:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Office Server\14.0\Search\Global\Gathering Manager\UserAgent

I set this to the IE9 user agent string found here: http://blogs.msdn.com/b/ie/archive/2010/03/23/introducing-ie9-s-user-agent-string.aspx

It’s:

Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)

After the next crawl the errors related to this problem where gone.

2 comments

Matthias Feb 27, 2013

Thanks, Ingo. Your solution worked for me. I had to reset the SharePoint Search Service before the next crawl in order to make the problem go away. Regards, Matthias

Urs Jan 18, 2021

Hello Ingo, just wanted to add, that this solution broke the crawls for applications with mixed authentication. I had to rewind to the original useragent. In an application that only uses NTLM authentication everything was fine. Regards, Urs

2 comments

Leave a comment