Enterprise Solutions Part 1: Content Type Syndication

This is the first blog in my little series on SharePoint 2010 enterprise solutions. The series has the following topics which I keep running into while working on various SharePoint projects:

  1. Content Type Syndication (this post)
  2. Document Sets (coming soon)
  3. External Content Types (coming soon)
  4. Taxonomies (coming soon)
  5. Metadata Navigation (coming soon)
  6. Web Service Delegation (coming soon)

Content type syndication is one of the great new features in SharePoint 2010. It solves the problem of distributing content types across site collection boundaries and keeps your custom content types in sync where ever these are used. It consists of a content type hub site where the syndicated content types are published. Other site collection can subscribe to the content type hub site. SharePoint timer jobs (scheduled to run every 15 minutes per default) do the actual publishing of the content types to the subscriber site collections. Alternatively, when a new site collection is being created, the published content types are instantly available on it.

I will cover the following aspects of content type syndication in this blog post:

  1. Limitations
  2. Setting it up
  3. Auto-publishing content types
  4. Updating content types

1. Limitations

Lookup fields (single and multi-value) are not supported by content type syndication. This makes good sense because lookup fields work within site collection scope. By design, a lookup field cannot reference a list in another site collection. Content type syndication is designed to work across site collection boundaries and therefore it would make no sense to support lookup columns. If you need lookup functionality and want to use content type syndication you must switch to use taxonomies (Managed Metadata, see Part 4 of this series).

You might run into an issue when syndicating custom document set content types. When creating a document set based on your custom content  type you get the error message: “The content type ‘…’ is read-only.”. The error is triggered by the ProvisionCTEventReceivers method in the SharePoint DocumentSet class when the code is adding the ItemAdded and ItemUpdated event receiver. Published content types are by design read-only in the subscriber site collections. Because SharePoint tries to add the two mentioned event receivers to the read-only content type you get the error. The solution is to add the event receivers to your custom content type. For details on how to do this see Part 2 post of this series.

2. Setting it up

The  following steps are necessary to set content type syndication up:

  1. Create site collection for the hub
  2. Configure Managed Metadata service application to point to hub site
  3. Create a subscriber site collection

2.1 Create Hub Site Collection

Create a site collection that is used for nothing else but the content type hub. You do not want anything else in this site collection: No content pages, no custom lists or libraries. This is to make sure that it is always possible to update the content types in the hub site collection easily. When there are no lists or lists items provisioned in a site collection based on a custom content type it can simply be updated by re-activating the feature that deployed it. There can only be one content type hub per Managed Metadata service application. So it is up to you where you want to put it. You can create an entire web application where you place the hub in the root site collection of it, a host header site collection or a hub site collection under a managed path of one of the existing web applications such as http://…/sites/cthub. The disadvantage of creating an entire web application is that it allocates server resources and there it is not recommended to have more than ten web applications per farm. In the example here I chose to create a separate web application for the content type hub.

Because I do not want any content on the hub site collection I simply create a blank site as shown below.

Post214-1

Next, configure the hub site collection:

  • Activate the built-in site collection scoped feature ‘Content Type Syndication Hub’ (Feature ID: 9a447926-5937-44cb-857a-d3829301c73b)
    Powershell: Enable-SPFeature -Identity 9a447926-5937-44cb-857a-d3829301c73b -Url http://cthub
  • Activate the built-in, hidden site collection scoped feature ‘TaxonomyFieldAdded’ (Feature ID: 73EF14B1-13A9-416b-A9B5-ECECA2B0604C)
    Powershell: Enable-SPFeature -Identity 73EF14B1-13A9-416b-A9B5-ECECA2B0604C -Url http://cthub
  • Activate your custom content type feature (or create the content types manually if you want)
  • Publish the syndicated content types

The TaxonomyFieldAdded feature is included in most of the SharePoint out-of-the-box site definitions but not in the blank site that I created. When this feature is not activated you will be missing the ‘Content type publishing’ link under Site Settings / Site Collection Administration.

Post214-2

The ‘Content type publishing’ dialog is shown below. It is though most useful on the subscriber site collections as it shows you which content types currently are syndicated. This is the number one trouble-shooting tool. Also it shows you the URL of the content type hub. So you can easily check whether the configuration is ok. In our case this simply points to http://cthub.

Post214-3

The final step is to create a content type and publish it. Go to the content type via Site Settings / Content Types, select the content type you want to publish and than click on the ‘Manage publishing for this content type’ link. If the content type is not yet published the radio button ‘Publish’ is selected. Click ok to actually publish it. For an already published content type you can choose between unpublish and republish.

Post214-4

2.2 Point to Hub

To make use of the content type hub we created in the previous step it must be configured. The Managed Metadata service application has the setting for it. You can configure exactly one content type hub per Managed Metadata service application. The configuration can be done via Central Administration or PowerShell. To set it up, do the following:

  1. Go to Central Administration / Application Management / Manage Service Applications
  2. Select the Managed Metadata Service and click on the Properties button in the ribbon
  3. Scroll to the bottom of the modal pop-up and enter the URL for the content type hub site as well as check the ‘Report syndication import errors…’ check-box

Post214-6

As you can see on the screen shot above there is no way to change the URL through the UI once it has been configured. That is a bit impractical but you can fall-back to PowerShell to set it:

Set-SPMetadataServiceApplication -Identity "Managed Metadata Service" -HubURI http://cthub -ErrorAction:SilentlyContinue -Confirm:$False

 

2.3 Create a Subscriber Site Collection

Create a new site collection in a different web application and the syndicated content type published earlier is immediately available. You can check this via Site Settings / Content type publishing in the subscriber site collection. All subscribed content types are listed there. In our case it is only the ‘Bike’ content type I created in step 2.1.

Post214-5

When working with feature deployed content types you also have to activate the content type feature (containing the syndicated content types) in the subscriber site collections. This is safe to do and you can verify that the content types really are syndicated via the Site Settings / Content type publishing function as well as checking that the content types are marked as ‘Read-only’.

3. Auto-publishing Content Types

I would like my content types to be published automatically. In an enterprise solution with many content types it would be tedious to have to click three times for each content type to publish it. So I created a feature with a feature receiver to do it. The code is quite generic as it reads the content types to publish from the feature properties which enables you to reuse the helper class in other projects.

    /// <summary>
    /// Helper class for SharePoint content type hub functionality.
    /// </summary>
    public class ContentTypeSyndicationManager
    {
        /// <summary>
        /// Publishes the content types via content type hub based on the content type names specified in a site collection scoped feature properties.
        /// </summary>
        /// <param name="properties">The properties.</param>
        public void PublishContentTypesFromFeatureProperties(SPFeatureReceiverProperties properties)
        {
            var site = properties.Feature.Parent as SPSite;

            if (ContentTypePublisher.IsContentTypeSharingEnabled(site))
            {
                var publisher = new ContentTypePublisher(site);

                var contentTypeNames = properties.Feature.Properties
                    .Cast<SPFeatureProperty>()
                    .Where(p => p.Name.StartsWith("ContentTypeName", StringComparison.OrdinalIgnoreCase))
                    .Select(p => p.Value);

                foreach (var contentTypeName in contentTypeNames)
                {
                    var contentType = site.RootWeb.AvailableContentTypes
                        .Cast<SPContentType>()
                        .FirstOrDefault(ct => ct.Name.Equals(contentTypeName, StringComparison.OrdinalIgnoreCase));

                    if (contentType == null)
                    {
                        continue;
                    }

                    if (!publisher.IsPublished(contentType))
                    {
                        publisher.Publish(contentType);
                    }
                }
            }
        }
    }

 

Here is the feature receiver of the publish content types feature:

    public class PublishContentTypesEventReceiver : SPFeatureReceiver
    {
        /// <summary>
        /// Occurs after a Feature is activated.
        /// Publishes the content types specified via feature properties where the key starts with 'ContentTypeName'.
        /// Note: The feature properties must have unique key names.
        /// </summary>
        /// <param name="properties">An <see cref="T:Microsoft.SharePoint.SPFeatureReceiverProperties"/> object that represents the properties of the event.</param>
        public override void FeatureActivated(SPFeatureReceiverProperties properties)
        {
            var manager = new ContentTypeSyndicationManager();
            manager.PublishContentTypesFromFeatureProperties(properties);
        }
    }

 

Here are the feature properties of the publish content types feature:

<Feature Title="Publish Content Types" Scope="Site">
  <Properties>
    <!-- The below content types will be published in the content type hub. -->
    <Property Key="ContentTypeName1" Value="Bike" />
  </Properties>
</Feature>

 

4. Updating Content Types

Once the setup is completed you can begin to make use of the content type syndication and update some content types. To get updates published to subscriber site collections the following steps are necessary:

  1. Update the content type:
    1. Either directly on the hub site via SharePoint UI (if the content type was manually created) or
    2. Change in XML, deploy the WSP and re-activate the content type feature on the hub site
  2. Verify that the content types are updated correctly on the hub site collection (only necessary for feature deployed content types)
  3. Wait for the syndication SharePoint timer jobs to execute (every 15 minutes) or start these manually via Central Administration
  4. Verify that the content types were updated as expected one of the subscribe site collection

The screen-shot below shows the content type syndication SharePoint timer jobs from my environment from Central Administration / Monitoring / Review Job Definitions. There is one for the hub and one timer job per subscriber web application. To start the timer jobs manually you can simply click on each of them and click the ‘Run Now’ button. Than you can check the timer job history page on Central Administration to see when the jobs are done.

Post214-7

Timer Jobs and SharePoint Deployment

While deploying a SharePoint solution to my development environment with PowerShell I got this surprising error:

Failed to create receiver object from assembly "****", class "****.EventReceiver" for feature "****" (ID: ****).: System.ArgumentNullException: Value cannot be null

When trying to redeploy the solution I got this error (which is expected):

A feature with ID **** has already been installed in this farm.  Use the force attribute to explicitly re-install the feature.

The last error occurs because some features were installed before the original deployment error occurred. An efficient way to uninstall these features is to retract the solution (even though it is not deployed):

stsadm -o retractsolution –name ****.wsp -url http://**** -immediate

If this does not work you can use the -local parameter instead of –immediate and run the command on all servers in the farm. But typically it works fine on a single server farm.

Redeploying the solution gives now this error message:

Failed to load receiver assembly "****" for feature "****" (ID: ****).: System.IO.FileNotFoundException: Could not load file or assembly ‘****’ or one of its dependencies. The system cannot find the file specified.

I found this very confusing to begin with because the actual error message changes. After some digging into the problem I determined that there is nothing wrong with the assembly, its references or the feature event receiver class – which is the original error message. The reason for the failure is that the updated assembly that contains a new feature event receiver is not being loaded by the SharePoint Timer Job. Then I tried the following:

  1. Stopping and starting the timer job (net start sptimerv4 and net stop sptimerv4)
  2. Clearing the SharePoint Configuration Cache
  3. Rebooting the server (ok, getting desperate)
  4. Disabling all SharePoint timer jobs that are in the same assembly that causes the failure
  5. Deleting all SharePoint timer jobs that are in the same assembly that causes the failure with PowerShell (Get-SPTimerJob | ? { $_.Name.StartsWith("****") } | % { $_.Delete() }

It turns out that in my situation the following sequence solves the deployment problem:

  1. Retract the solution package (as described above)
  2. Delete all timer jobs that are in the same assembly that causes the failure
  3. Clear the SharePoint Configuration Cache
  4. Deploy the solution package

What I learned from this is that you should either place SharePoint timer jobs in a separate assembly that will never contain feature event receivers or update the assembly version of the assemblies containing SharePoint timer jobs. The latter might cause some other issues in case the solution uses hard-coded references to that assembly such as in user controls (ASCX), HTTP handlers (ASHX) or custom web.config sections. This can be avoided in most situations with the new token “$SharePoint.Project.AssemblyFullName$” in SharePoint 2010 to reference the current assembly.

Customize XSLT List View Web Part

The new list view web part in SharePoint 2010 is great because it allows you to customize it easily using XSLT stylesheets instead of CAML.

Although it is possible to add XSLT list view web parts via mark-up directly (either in the page directly or in elements file) you should not do so. The XSLT list view web part has no ‘Export’ option and the reason for this is that it should not be import in anyway. If you still want to add a web part via mark-up note these issues:

  • The value of the title attribute must be different from the display name of the list. Otherwise the web part will show ‘Untitled’ as title.
  • To reference the list use the ListUrl attribute. It is relative to the current web.
  • To set the Url for the title use the TitleUrl attribute. It is relative to the current location of the page.
  • To remove the check-boxes for selection (left column) put the TabularView=”FALSE” attribute on the View element.
  • When updating to a newer version of SharePoint (e.g. installation of service pack or CU update) the earlier created views might break.

One thing that is good to know is this: List view web parts added to pages via mark-up will always result into an error when editing the web part in page through the SharePoint UI or SharePoint Designer. In case it is required to be able to further customize the pre-added web parts at a later stage the list view web parts must be added by code or by using a View element in the module (and not mark-up). The only work-around is to remove the ‘failing’ web part from the page and add a new one.

The proper way to add XSLT list view web part is via the View element in a module:

<Module Name="Module1" Url="$Resources:cmscore,List_Pages_UrlName;">
  <File Url="default.aspx" NavBarHome="True" >
    <View List="Lists/Contacts" BaseViewID="1" WebPartOrder="1" WebPartZoneID="Header"/>
  </File>
</Module>

It seems essential to specify the BaseViewID even though the MSDN documentation specifies it as optional. Without a BaseViewID attribute I get a ‘Cannot complete this action’ error. The title of the created XSLT list view web part is always the list name. If you want to change that or any other property of the XSLT list view web part that is not exposed by the View element you can do this through the SharePoint object model. This is not ideal because it requires custom code.

Also note that there are (at least) to View elements in the MSDN documentation. The View Element (Site) and the View Element (List). The first one is used in modules and the latter on in list schema definitions. The View Element (List) contains a lot of child elements for backward compatibility, which are ignored with the XSLT list view web part in SharePoint 2010. These element are described here. An ok general introduction to the XSLT based rendering mechanism can be found here.

Feature Stapling Revisted

Feature stapling in SharePoint enables developers to execute a set of features each time a SharePoint site based on a specific site definition is created by a user or administrator. It is a very good approach to either customize SharePoints OOTB site definitions with custom features or to separate custom functionality from custom site definitions.

I always assumed that the scope of a feature stapler (in other words: the feature that staples a set of features on a site) can be either Farm og WebApplication and that it does not matter whether you choose one or the other. Using WebApplication scope can makes a lot of sense if the solution is running on a farm that hosts multiple applications. For example if you want to apply some branding to all team sites for one application you probably do not want to apply the very same application specific branding to team sites in the other application. This is what happen if using Farm scope and what does not happen with WebApplication scope.

Unfortunately it turns out that SharePoint behaves differently when using Farm or WebApplication scope for the feature stapler. When using Farm scope it seems that features are activated in this sequence:

  1. Features from site definition
  2. Features without activation dependencies from stapler
  3. Features with activation dependencies from stapler (in correct sequence)

When using WebApplication scope there does not seem to be any specific order. In my example I had the SharePoint publishing features in the site definition and some content types, list definitions and pages in the feature stapler. This resulted in that the feature with custom pages was activated before the publishing features leading to a NotAuthorizedException, because the Pages document library was not yet created. The exact same setup (site definition, feature stapler, features) works perfectly when the feature stapler is set to Farm scope.

As mentioned earlier this is a bit unpractical in situations with multiple applications on the same farm. I am not sure whether this is intended SharePoint behavior or simply a bug. One work-around is to create Farm scoped feature staplers and only staple features on custom site definitions. In case OOTB site definitions are customized these features must only be added to the stapler if they check for the current web application. Another work-around is to use WebApplication scope for the stapler feature and to create one feature that activates all other features in the correct sequence. This approach requires though that there are no dependant features in the site definition and it requires some custom code for the feature activator.

A completly different scenario could be that the customer or the company hosting the application requires all Web and Site scoped features to be hidden. Some times this is requested to protect the application from site administrators that could disable certain crutial functionality by mistake. Hidden features are a bit special because they must not have feature dependencies. In a scenario with dependant features and a hidden feature requirement you must create a feature activator feature that can activate a number of features in a given sequence.

Computed Field XML Definition

Today I wanted to create a XML based field definition for a computed field for a SharePoint 2010 solution. The requirement is to show the difference of two date/time fields in the column. The field should be used in an XML based content type that inherits from the document.

That should be easy to do. The field XML schema has all the necessary options. But as soon as I declared a field as type ‘Computed’ it was no longer deployed. Checking the site columns gallery via site settings on the site collections showed that the field was not available and therefore not part of the custom content type. Even the simplest field definition did not work:

<Field ID="{GUID}" Name="ComputedColumn" Type="Computed" />

As a next step I changed the type to calculated and that got the field deployed, appeared in the site columns gallery and was part of the content type as expected:

<Field ID="{GUID}" Name="CalculatedColumn" Type="Calculated">

  <Formula>=0</Formula>

</Field>

Then I added my custom content type with the CalculatedColumn to the Documents library of a collaboration site. The new site column appeared in the fields list on the document library settings page. But when uploading a document to that library I got the error message that the file I was uploading does not exist: <nativehr>0×81020030</nativehr><nativestack></nativestack>The URL ‘Documents/SomeDoc.docx’ is invalid. It may refer to a nonexistent file or folder, or refer to a valid file or folder that is not in the current web.

Removing the CalculatedColumn from the document library proved that it caused the error. So I added the calculated field via SharePoint UI instead and looked at the generated schema XML in SharePoint Manager. After a bit of trial and error I add a couple of attributes that made everything work:

<Field ID="{GUID}" Name="CalculatedColumn" Type="Calculated" ResultType="Text" ReadOnly="TRUE" Required="FALSE">

  <Formula>=INT((SomeDateColumn-Modified)*86400)&gt;=0</Formula>

  <FieldRefs>

    <FieldRef Name="SomeDateColumn" />

    <FieldRef Name="Modified" />

  </FieldRefs>

</Field>

What is the moral of the story? I do not know. Maybe you can get the Computed column to work properly with the right attributes.

Fixing the Unfixable: How to Fit a Bottom Bracket without Threads

I have this fantastic MBK steel frame racing bike which was totally refurbished a couple of years ago. During the process of replacing old parts the bottom bracket got upgraded to a Shimano 105 model with stainless steel caps. I thought that this was going to last forever. It also did but what did not last was the right side of the bottom bracket threading in the frame. The threads were a bit run down when the bottom bracket was replaced and so the new tough model could work its’ way through the threads ending up totally loose on the right side.

Seeking professional help at the bike shop I learned that I rather should buy a new frame. But I was not ready to throw this great frame away. It still has a couple of ten thousands of kilometers in it. I went to the attic at found a good old adjustable alloy Dura Ace BB-7700 bottom bracket where the threading is bit broader than on the new models. This worked for a couple of weeks until the right side worked itself loose again. Than a trip to the hardware store. I bought the strongest two component adhesive they had and two short M6 screws. Back at home I drilled two small holes in the frame and bottom bracket cap, threaded the holes in the frame and glued the right cap of the BB-7700 into the frame! The two M6 screws should keep the right cap in its place. I expected this to hold a couple of weeks, too. But this time it lasts. I have been riding with this work-around for a year now, about 8,000 km. The BB-7700 is still available in online shops. With the right cap glued in it is still possible to replace the left cap, bearings and axle.

Migrate Microsoft Project Central 2000

I am currently working on a migrating and older version of Microsoft Project server, Project Central 2000. The goal is to extract all the task with their assigned resources from the database. The data model seemed a bit confusing for me at first glance. I found this article really helpful although it does not list all the relevant tables in my case.

I want to get this information from the Project Central database:

  • Project name
  • Task name
  • Completed Percentage
  • Resource name
  • Start date
  • Finish date
  • Is milestone

Joining these tables where proj_id is part of the join condition does the trick:

  • msp_projects
  • msp_tasks
  • msp_assignments
  • msp_resources

So, to obtain the task and resource name the following SQL can be used:

SELECT t.task_name, r.res_name
FROM msp_tasks t
INNER JOIN msp_assignments a on a.task_uid = t.task_uid and a.proj_id = t.proj_id
INNER JOIN msp_resources r on r.res_uid = a.res_uid and r.proj_id = a.proj_id

Make SharePoint fly, Part 1

How to speed up Sharepoint 2007 and SharePoint 2010 list data access

 

In this article I want to share my experiences working with real-life lists in SharePoint with regards to data access performance. To access list data in SharePoint can be rather slow but it can actually be pretty fast. In order to code efficient SharePoint data access code you must bear a couple of rules in mind:

  1. Keep the list size below 10.000 items
  2. Use the fastest data access object model methods
    1. PortalSiteMapProvider.GetCachedListItemsByQuery(PortalWebSiteMapNode, string, SPQuery, SPWeb)
    2. SPList.GetItems(SPQuery)
  3. Always use CAML queries to get data. The only exceptions are:
    1. SPList.GetItemById can be used in SharePoint 2007, but never use SPListItemCollection.GetItemById
    2. SPList.GetItemByIdSelectedFields should be preferred over List.GetItemById in SharePoint 2010
    3. LinqToSharePoint with SharePoint 2010 can be used but you must always review the generated CAML
  4. Index the site columns you query on
  5. Verify in production

In the following I will discuss the above rules more detailed and give a couple of references. Everything written here is based on my personal experience and I cannot guarantee that the information provided is flawless. So please share your comments and experiences here and help me to improve the article. On contrary these rules of thumb have helped some seriously troubled SharePoint applications. In one case the use of SPList.GetItems(SPQuery), which is good, still lead to frequent server break-downs. The database server was getting to much load. Replacing it with PortalSiteMapProvider.GetCachedListItemsByQuery(PortalWebSiteMapNode, string, SPQuery, SPWeb) in the hot spots of the calling code took a lot of pressure from the database server. In another example we could improve the average page rendering from 8s to 2s by using the PortalSiteMapProvider.GetCachedListItemsByQuery(PortalWebSiteMapNode, string, SPQuery, SPWeb) and replacing SPListItemCollection.GetItemById with SPList.GetItemById.

But how can you determine the weak spots in your application? Like in the first example with the troubled database server you should take a look at a SQL Server trace. Calls to a stored procedure called spexecutequery (I am not 100% sure on the name) indicate the execution of CAML queries from the SharePoint application. CAML is translated into a so-called ad-hoc SQL query. To determine what lists and queries are generating the load on the database server you can only tell if you have the actual SQL included in the trace. Normally this is not an option in a production system. Get a copy of the production content database on your test environment, create a load test and enable the tracing on the database server in the test environment. Another option to find weak spots is to populate lists with production or production like data and run a ASP.NET profiler such as Red Gates ANTS Profiler on certain pages on your development machine.

A note on load testing: When testing on a contained environment you might notice that regular list data access via SPList.GetItems at some point becomes very fast. This will happen if you query the same data multiple time so that SQL Server will cache the query results. Use the following T-SQL statement to clean all SQL Server buffers:

DBCC DROPCLEANBUFFERS;

A simple and free tool for load testing is Load UI from eviware. It allows you to request pages at variable rates. It has a drag&drop user interface in the same way as good old Lab View. If you need more complex load patterns, do custom stuff, click on things in the browser a good starting point is Visual Studio for Testers or Ultimate edition with its web test/load test capabilities.

1. List Size

The list size is the key issue with list data performance. If you have a list with ten items you get great performance no matter how you access the data in the list. On the contrary a list with ten million items cannot be used at all in terms of normally expected application performance no matter how you access the data. What you really need to understand is how the performance scales in between these two extremes. That depends a lot on how you access the data. The best starting point is to ready the Microsoft white paper on big lists.

When designing a SharePoint application you should, as a rule of thumb, only create lists with maximum 10.000 items. When you need more items either create folders or multiple lists. Folders and lists perform in the same way. Using multiple folders within a list complicates querying a little bit, because you have to set the folder on the SPQuery object. You should avoid to use recursive CAML queries because it will be slower and you might match items from another folder that you are not interested in.

2. Data Access Methods

The lesson learned from the big list white paper is that you should either use PortalSiteMapProvider.GetCachedListItemsByQuery(PortalWebSiteMapNode, string, SPQuery, SPWeb) or SPList.GetItems(SPQuery). Nothing else. In some rare case it might be acceptable to use List.Items but only if you are 100 % sure that the list will have few items. Working with SharePoint 2010 you can also use Linq to SharePoint with absolute care (see the following paragraph for details).

The PortalSiteMapProvider is a great way to access list data via the SharePoint object cache. So when using this data access method make sure to configure the object cache to something useful. As with every cache this method incurs an overhead for adding items to the cache and checking the caches validity. The most technical information I could find on this method is from the SharePoint team blog. In a situation with many reads compared to writes you will get a great performance boost from this method. Otherwise it will slow down your application. One limitation with this data access method is that you cannot access list item attachments or document library documents through it. It provides only all the site columns of the list or document library. The PortalSiteMapProvider uses security trimming and updates changed items instantly in the cache.

In situation where you cannot use the PortalSiteMapProvider you should opt for SPList.GetItems(SPQuery) or the Linq to SharePoint solution.

3. CAML Only

CAML queries are the only efficient way to query list data in SharePoint. The alternative is to retrieve all list items from the content database and than filter these in code. It is obvious that the alternative is slow.

So, PortalSiteMapProvider.GetCachedListItemsByQuery, SPList.GetItems and Linq to SharePoint use CAML queries to query the content database. That is why you must be extremely careful when using Linq to SharePoint. It hides the CAML generation from you by abstraction. Underneath it creates a CAML query in the Linq to SharePoint provider which you must inspect because not all Linq capabilities are supported by CAML. For example when you use IEnumerable<T>.Contains in a Linq to SharePoint query it will not generate CAML or clauses but retrieve all list items and than filter these with Linq to Objects in memory afterwards (two-stage query). And that is exactly what you do not want to happen. Check-out this MSDN article for more details about two-stage queries. To see the auto-generated CAML attach a TextWriter (e.g. StringWriter) to DataContext.Log, see here.

An interesting issue exists with the GetItemById method of the SharePoint object model. There is a big performance difference between using SPList.GetItemById and SPListItemCollection.GetItemById. I have no idea why but the Microsoft best practices do not recommend to use SPListItemCollection.GetItemById and my own experience backs this up. See this article for a reference. So it is safe to use SPList.GetItemById when know the ID of the list item and want to retrieve it. In SharePoint 2010 you should consider and prefer to use the SPList.GetItemByIdSelectedFields method as it only loads data for the specified fields and hence performs better.

4. Indexing

SharePoint site columns can be indexed. When you have a list with a couple of thousands list items a you always query on a site column called Name you should index that site column. See this article how to create indexes in SharePoint. It is the same principle as creating indexes in databases. Some sort of hash map is maintained for your index in memory so that items can be matched much faster than by a retrieving and filtering approach.

5. Verify

One thing is to code for performance the other thing is verify that everything behaves as expected. I would suggest two things to make sure that your optimizations are effective:

  1. Check the object cache.
    Because the PortalSiteMapProvider.GetCachedListItemsByQuery uses the object cache you want to make sure that it works efficiently. This is not a trivial task. You can find more information here. The bottom line is that you should monitor the ‘SharePoint Publishing Cache’ performance counter on the production server. The number of ‘Total number of cache compactions’ should be 0 to 1 within an hour.
  2. Use SPMonitoredScope (SharePoint 2010 only) for data access methods.
    Wrapping the data access method with a using(new SPMonitoredScope(“Scope name”)) { … } adds an entry to the developer dashboard. This is rather targeted at your development and environment than production. But it gives you the chance to evaluate the data access performance all the time easily in the browser.

 

Well, I hope this article helps you to writer faster data access code and makes your SharePoint applications fly. Please drop a comment with your experience, advice or challenges.

 

The next part will be about caching in SharePoint applications. I always disliked the way I used to do control output caching so far. I am working on a more general approach that uses the ASP.NET (partial) page output cache and the ASP.NET cache with a CacheDependency or HttpCachePolicy.AddValidationCallback.

Setup Squeezebox Duet and Synology NAS

My setup consists of a Logitech Squeezebox Duet and a Synology DS-109+ NAS. The Synology NAS runs the Squeezebox Server software so that I do not need a running PC to play music. I struggled a little bit with reconfiguring the Squeezebox so I thought it might be useful to create a post on that. Here are the steps to setup the Squeezebox, get the NAS ready and prepare the music files.

  1. Download cover art for your music.
    Cover art are pictures of album front covers that are stored in the folder together with the music files of the album. A good free tool to help you with downloading and saving the cover art images is the Album Art Downloader on sourceforge. Still, you will need some patience because you have to pick an image for each album you have in your collection.
  2. Download and install the latest firmware version for the NAS from Synology’s website. Choose the correct NAS version in dropdown box in the top of the download page. To install Squeezebox Server you need at least DSM2.2-0958. I am currently running with DSM3.0-1337.
    Locate the DSM Update/Firmware function on the web interface of the Synology disk station and follow the instructions.
  3. Download and install the Squeezebox Server latest version from Synology’s website. Again make sure to choose the correct NAS version.
    From the dropdown in the upper left corner (DSM3.0) on the Synology web interface choose Package Management and follow the instructions. When the installation is completed you have to start the service and note the HTTP address of it. Via this address you can configure the Squeezebox Server.
  4. Power-up the Squeezebox base station. Make sure that the center button on the base station is flashing red.
  5. Configure the Squeezebox controller
    If it is the first time you are using the controller simply follow the wizard on the controller. If it has been used with another Squeezebox account or on another network earlier you have to reset the controller: Go to Settings / Advanced / Restore Factory Settings. Than follow the wizard as the controller restarts. If you want to use the controller for music playback: Go to Settings / Advanced / Beta Features / Audio Playback and activate Enable Playback. You can than use either the built-in loudspeaker or connect headphones via the 3.5 mm jack stick to the controller. To make this feature work properly you have to restart the controller.
  6. Configure the Squeezebox Server.
    Enter the address such as http://192.168.178.100:9002 from step 3 in a browser and follow the instructions to select the music as well as playlist folder on the NAS drive. To fine-tune the Squeezebox Server click on the Settings link in the lower right corner. On the Basic Settings tab you want to run a rescan so that all music on the NAS folder you pointed to is added to the Squeezebox music library. Check the other tabs for more options in case you want to change something. Than go to the Plugins tab and follow the Settings link for the Rescan Music Library plug-in. Set the plug-in to Enabled and set the time of day when you want to run a rescan of your music library. If you do not activate this then newly added music will not be accessible through the Squeezebox. With the plug-in enabled it will update the Squeezebox music library on a daily basis hence new music will show up on the Squeezebox once the next rescan has run. 

Now that I have been running SqueezeCenter (aka Squeezebox Server) for a while a came across the following problems and solutions:

  • Problem: Without doing anything special Squeezebox Server 7.5.3 simply stops running and cannot be started again. When being started again it stops after about one or two seconds. The log only contains one entry that the Squeezebox Server is starting up.
    Solution: Logon to the NAS with a Putty terminal as root, go to /var/packages/Squeezecenter/target/Logs and change the access rights to the server.log file with the command ‘chmod 646 server.log’
  • Problem: After upgrading to a newer version of Squeezebox Server (e.g. from 7.5.0 to 7.5.3) it cannot start. It starts up, runs for about five seconds and then shuts down again. The log file says ‘Couldn’t connect to database’.
    Solution: Logon to the NAS with a Putty terminal and go to /var/packages/Squeezecenter/Cache and remove its’ contents. In case you should delete the whole Cache folder you have to recreate it and give appropriate access rights with chmod 646. Than restart the NAS.

Migrating Projects to TFS 2010

I am so disappointed with TFS 2010. Ok, the new functions are really cool but NO tooling for migrating existing projects. Microsoft – what were you thinking? I suppose that it would be a little task for a TFS engineer to create a migration tool. Now we have to fiddle with it.

Here is the deal: You created a project on TFS 2008 with let’s say the Agile 4.2 template, then you migrate your server to TFS 2010 and you will notice that the migrated project still use the Agile 4.2 process template and SharePoint site, more or less. There is no automated way to migrate to the Agile 5.0 process template. You will find a couple of blogs describing how to upgrade certain parts for the TFS 2010 beta version. Which is good for inspiration but does not fully work on our TFS 2010 RTM. The best source I could find so far is the Visual Studio 2010 upgrade project on codeplex.

I will update this post along with going through the painful process of manually upgrade my Agile 4.2 project. I just wish that I had waited a little longer and created the project from scratch with the new Agile 5.0 process template on TFS 2010.

Here are my upgrade steps:

  1. Download the Agile 5.0 process template
    Right-click the root node in team explorer/process template/choose Agile 5.0 and select download (you must be TFS project collection administrator to do so)
  2. Manually upload the six work item templates (*.wit) files to my team project with TFS powertools, you can also use witadmin.exe for that purpose
    You might have noticed that some of the work item template have the same name as the old ones such as Bug or Task. These have undergone a major face lift and it would be a shame not to upgrade those. Especially support for link management to parent and child elements as well as an estimate field (Task template) are very nice. But be aware that you will loose all information that was stored in custom fields you added to the old work item template. In that case you should export the current values to an Excel spreadsheet so that you can copy it back into TFS after you updated the work item template.
  3. Iteration planning workbook
    I simply followed the description from the codeplex document and it fully work. The iteration planning workbook is a great help to create user stories and/or spread these over iterations.
  4. Product planning workbook
    The description from the codeplex document worked partially. The most interesting feature of this workbook where you can see the distribution of products does not work for me. I get a meaningless error when choosing the tab.
  5. Test management
    If you want to create test suites and test cases you must use the new Test Management tool that comes with TFS 2010. For the tool to work you must upload the new categories and link types to TFS using the witadmin tool. You can find a detailed walk-through in blog post that is referred to in the codeplex upgrade project mentioned above.