Strange Error on Web Frontend Server of MOSS 2007 farm: stsadm operation “enumallwebs” with “includewebparts” options results in lots of missing WebParts

During a migration project from MOSS 2007 to SharePoint 2010 we got a strange error that took some hours to resolve.

For an automated migration tasks I tried to create a PowerShell script to enumerate all WebParts on all ASPX pages using SPLimitedWebPartManager on a WFE of the MOSS 2007 farm. But I only got “ErrorWebPart” for every WebPart on the Web Application!

Then I used “stsadm” to enumerate all WebParts:

stsadm –o enumallwebs –databasename sharepoint_content_local_sharepoint_local –includewebparts > allwebparts.txt

This is the result:

<Databases>
  <Database SiteCount="2" Name="sharepoint_content_local_sharepoint_local" DataSource="server">
    <Site Id="12345678-716d-44a3-a973-fffffffffff" OwnerLogin="domainspadmin" InSiteMap="True">
      <Webs Count="5">
        <Web Id="12345678-17d9-4870-8d94-fffffffffff" Url="/sites/test1" LanguageId="1031" TemplateName="STS#0" TemplateId="1">
          <WebParts>
            <WebPart Id="00f5bad9-f117-4eca-8d77-194bb598d7c6" Count="1" Status="Missing" />
            <WebPart Id="e398c8f4-1750-b857-cca9-2542cadaee8f" Count="1" Status="Missing" />
            <WebPart Id="d23f666e-372e-8f7d-e9f4-341c7fab5653" Count="1" Status="Missing" />
            <WebPart Id="9f7a9f30-76a0-2d43-0d84-402ba6c3c125" Count="1" Status="Missing" />
            <WebPart Id="ce9aa113-48cf-ddee-0c03-597445e5b7ab" Count="1" Status="Missing" />
            <WebPart Id="e60f6c95-e86c-4717-2c0d-6d8563c9caf7" Count="1" Status="Missing" />
            <WebPart Id="293e8d0e-486f-e21e-40e3-75bfb77202de" Count="103" Status="Missing" />
            <WebPart Id="6bbb6e7d-fe0d-5d79-6c79-76a90de47e1c" Count="1" Status="Missing" />
            <WebPart Id="9f030319-fa14-b625-4892-89f6f9f9d58b" Count="1" Status="Missing" />
            <WebPart Id="b9a7f972-708a-cd77-4ffd-a235dfed5c38" Count="1" Status="Missing" />
            <WebPart Id="2242cce6-491a-657a-c8ee-b10a2a993eda" Count="182" Status="Missing" />
            <WebPart Id="db128878-9a93-4768-2256-cc2c390ffb57" Count="1" Status="Missing" />
          </WebParts>
        </Web>
        <Web Id="12345678-3bda-4109-aacd-fffffffffff" Url="/sites/test1/subweb1" LanguageId="1031" TemplateName="STS#1" TemplateId="1">
          <WebParts>
            <WebPart Id="ce9aa113-48cf-ddee-0c03-597445e5b7ab" Count="1" Status="Missing" />
            <WebPart Id="293e8d0e-486f-e21e-40e3-75bfb77202de" Count="9" Status="Missing" />
            <WebPart Id="2242cce6-491a-657a-c8ee-b10a2a993eda" Count="7" Status="Missing" />
          </WebParts>
        </Web>
        <Web Id="12345678-b479-4157-91e5-fffffffffff" Url="/sites/test1/subweb2" LanguageId="1031" TemplateName="STS#1" TemplateId="1">
          <WebParts>
            <WebPart Id="ce9aa113-48cf-ddee-0c03-597445e5b7ab" Count="1" Status="Missing" />
            <WebPart Id="293e8d0e-486f-e21e-40e3-75bfb77202de" Count="6" Status="Missing" />
            <WebPart Id="2242cce6-491a-657a-c8ee-b10a2a993eda" Count="5" Status="Missing" />
          </WebParts>
        </Web>
        <Web Id="12345678-25f3-450f-b44d-fffffffffff" Url="/sites/test1/subweb3" LanguageId="1031" TemplateName="STS#0" TemplateId="1">
          <WebParts>
            <WebPart Id="ce9aa113-48cf-ddee-0c03-597445e5b7ab" Count="1" Status="Missing" />
            <WebPart Id="293e8d0e-486f-e21e-40e3-75bfb77202de" Count="21" Status="Missing" />
            <WebPart Id="2242cce6-491a-657a-c8ee-b10a2a993eda" Count="22" Status="Missing" />
          </WebParts>
        </Web>
        <Web Id="12345678-b341-4370-b0c6-fffffffffff" Url="/sites/test1/subweb4" LanguageId="1031" TemplateName="STS#0" TemplateId="1">
          <WebParts>
            <WebPart Id="ce9aa113-48cf-ddee-0c03-597445e5b7ab" Count="1" Status="Missing" />
            <WebPart Id="e60f6c95-e86c-4717-2c0d-6d8563c9caf7" Count="1" Status="Missing" />
            <WebPart Id="293e8d0e-486f-e21e-40e3-75bfb77202de" Count="23" Status="Missing" />
            <WebPart Id="2242cce6-491a-657a-c8ee-b10a2a993eda" Count="27" Status="Missing" />
          </WebParts>
        </Web>
      </Webs>
    </Site>
    <Site Id="12345678-730c-46fd-a114-fffffffffff" OwnerLogin="domainspadmin" InSiteMap="True">
      <Webs Count="2">
        <Web Id="12345678-7cd6-447d-8107-fffffffffff" Url="/sites/test2" LanguageId="1031" TemplateName="STS#0" TemplateId="1">
          <WebParts>
            <WebPart Id="d55b3b6b-6281-707b-73d0-0c49581475ad" Count="1" Status="Missing" />
            <WebPart Id="aadc2962-fc63-03e2-f119-13e7096bab20" Count="1" Status="Missing" />
            <WebPart Id="2f1510c7-75d5-921f-b120-2ce98fe3afe3" Count="1" Status="Missing" />
            <WebPart Id="f5c3ff60-e752-3a90-84f8-3677f8384e2d" Count="2" Status="Missing" />
            <WebPart Id="f2c50a02-9894-4ace-bb3f-4146a24cd940" Count="2" Status="Missing" />
            <WebPart Id="ce9aa113-48cf-ddee-0c03-597445e5b7ab" Count="2" Status="Missing" />
            <WebPart Id="e60f6c95-e86c-4717-2c0d-6d8563c9caf7" Count="2" Status="Missing" />
            <WebPart Id="293e8d0e-486f-e21e-40e3-75bfb77202de" Count="83" Status="Missing" />
            <WebPart Id="9f030319-fa14-b625-4892-89f6f9f9d58b" Count="1" Status="Missing" />
            <WebPart Id="c9b34b5d-bf06-dc91-d23e-94ecad31cd0a" Count="2" Status="Missing" />
            <WebPart Id="2242cce6-491a-657a-c8ee-b10a2a993eda" Count="83" Status="Missing" />
            <WebPart Id="74bd016c-baa0-14a8-d5d8-b75dc7e6f429" Count="1" Status="Missing" />
            <WebPart Id="94e9c166-264a-f84b-2377-bccefb8b3771" Count="1" Status="Missing" />
            <WebPart Id="fb35a198-aea0-3c26-e40c-df473fe9b07b" Count="2" Status="Missing" />
            <WebPart Id="669602d9-e116-ccb8-eea3-e37ad589b14b" Count="1" Status="Missing" />
            <WebPart Id="f5897322-ddd4-c990-d012-f9d4fe2180ad" Count="2" Status="Missing" />
          </WebParts>
        </Web>
        <Web Id="12345678-80e5-425d-b0d9-fffffffffff" Url="/sites/test4" LanguageId="1031" TemplateName="STS#0" TemplateId="1">
          <WebParts>
            <WebPart Id="ce9aa113-48cf-ddee-0c03-597445e5b7ab" Count="1" Status="Missing" />
            <WebPart Id="e60f6c95-e86c-4717-2c0d-6d8563c9caf7" Count="1" Status="Missing" />
            <WebPart Id="293e8d0e-486f-e21e-40e3-75bfb77202de" Count="81" Status="Missing" />
            <WebPart Id="2242cce6-491a-657a-c8ee-b10a2a993eda" Count="105" Status="Missing" />
          </WebParts>
        </Web>
      </Webs>
    </Site>
  </Database>
</Databases>

(This file is modified: I’ve changed Web and Site IDs and I’ve removed lots of webs and sites.)

EVERY WEBPART IS MISSING!! Also all default MOSS 2007 WebParts with default SharePoint DLLs!

BUT… On the second Web Frontend Server all WebParts are Status=”Installed” !!! Strange…

In the events viewer we found:

clip_image002

image

It’s a SharePoint Feature DLL that is missing!

This error is not present on the second WFE!

So we installed the missing DLL in the GAC.

And… now it works! – All WebParts have status “Installed” in the stsadm output!

Summary

The feature deployment on the WFE was not successfull. There must be an error but MOSS 2007 doesn’t tell us about it. All solutions seems to be deployed in the Solution Management of the farm…