Avoid UNC file share to prevent File Change Notification issues

Recently we were experienced many unexpected application pool recycling with File Change Notifications (FCN) on our newly created Sandbox Sitecore CM instances. Every few hours (ranging from 20-40 hours) only one of two CM instances was getting restarted automatically with below error.

Change Notification for critical directories. File Change Notification Error in App_LocalResources HostingEnvironment initiated shutdown CONFIG change HostingEnvironment caused shutdown

What we tried

After spending a good amount of time on it, we hypothetical thought for few workarounds to do like

• Stop Anti-virus on server
• Use Process Monitor to check a File System for Web-root and Temporary Internet Files, etc. folders to know what's actually causing this
• Check there's no access given on the web-root to any unauthorized user. 

But all looked good for us, even not getting much help from Google as well. So final option we had to debug the crash dumps.

One more thought came in mind that recycling is happening only on one instance, which is using Sublayouts from other Sitecore instance using UNC file share, and that other instance never got recycled! We know that DFS was the best option here, but we were not able to digest that UNC share can really cause this issue. At the same time we found a nice post where someone already faced the same issue and they fixed it by stopping UNC path sharing for sublayouts. And yes, it worked for us too.
(Note: Recycling is not happening because of number of re-compilations - "numRecompilesBeforeAppRestart")

- http://www.dnnsoftware.com/forums/threadid/318762/scope/posts/file-change-notification-issues-web-farm-over-unc-share
- http://blogs.msdn.com/b/tess/archive/2006/08/02/686373.aspx

What we Learned and Investigated:

- We learnt one more reason that can recycle the application pool
- Avoid using UNC sharing for ASPX, ASCX, RESX, App_Code files, etc. compilable files and Use DFS for them, what Sitecore recommends in Sitecore Scalability Guide.

Our Learning on Publishing Sitecore Sublayouts

After we started using Web Deploy for publishing sublayouts on our multisite environment, we were getting random caching issues on our CD servers. We have multiple publishing target databases and load balanced multiple CD servers. So, as I mentioned in my previous blog, when a sublayout is published, it is published with help of Web Deploy to one server and then the sublayout is replicated to all other servers using DFS.

What kind of issues we found?

Sometimes we found that the published sublayout gets reflected on few servers, on few servers we still get older content. But surprisingly, the published sublayout was replicated on all servers and still we were getting different output from different servers. So, when this happens, we used to clear caches on servers where such issues occurred assuming this might be because of Sitecore caching issues, later on we found workaround to publishing those sublayouts again after 1 or 2 minutes.

Now you will understand how critical it would be to publish sublayouts and getting them reflected on live servers quickly to make go-lives, re-brandings, news releases or press releases successful in one go.

Note: We found this only for those Cacheable sublayouts.

Our learnings to fix such issues

1. Web Deploy publishing should be synchronous.

We analyzed below sequence happened rarely.

1. First we started publishing, so as per my previous blog, Web Deploy will start deploying sublayout in async mode (By default Web Deploy is configured on Sitecore is asynchronous). So, sublayout item publishing and sublayout physical file deploying are done in parallel.
2. So, chances that item gets published before sublayout file is copied.
3. Now, item is published, so CD servers will invoke "publish:end:remote" event and clear HTML (sublayout) cache.
4. Now, before the new published sublayout file gets deployed to CD server, end-user requested a page which uses the same sublayout. So, the HTML cache will be generated again for older Sublayout.
5. Now, Web Deploy sent a new Sublayout. (So, we have HTML cache of older sublayout)

Learning: Sublayout publishing should always be synchronous, so publishing will get on hold till the sublayout is not deployed to the CD server, which can be configured in WebDeploy.config as below.

<event name="publish:begin">
 <handler type="SitecoreTactics.Publishing.BeginWebDeploy, SitecoreTactics" method="PublishSublayouts">
  <synchronous>true</synchronous>
  <tasks hint="list:AddTask">
   ......
  </tasks>
 </handler>
</event>

2. Target Database should be the first publishing target.

We have 3 publishing targets say, web, web-2, web-3. Means, any publishing will be done in this sequence itself. Earlier we found the target database inside the WebDeploy.config was set as last publishing target, means web-3. We might have done this in past to make sure the item is published before sublayout getting deployed.

But, as per our recent experiences and findings, we should keep it as first publishing target. So, in our case, it should be web. This will help us when there are multiple servers in DFS, so when DFS is taking some more time in replicating to other servers. So,  for reducing such chances of delayed Sublayout deploy, we should keep web as the publishing target in WebDeploy.config as below.

<tasks hint="list:AddTask">  
 <default type="Sitecore.Publishing.WebDeploy.Task">  
   <!-- It should be the first Publishing target database -->  
   <targetDatabase>web</targetDatabase> 
 </default>
</tasks>

3. We can make few seconds delay in clearing HTML Cache.

If we have done above cases, there are no chances that the CD server where Web Deploy is sending Sublayout file will get any issues. But just consider a worst case where DFS is taking more time to replicate sublayouts to other servers. So, chances that HTML Cache will get cleared before the sublayout replication is done.

To avoid such cases, we can add a delay of few seconds, say 2 or 3 seconds before clearing HTML Cache (Only when the sublayouts are getting published)

Finally, we have hassle free one-time Sublayout publishing working without any caching issues very well! I'm sure this will be helpful to others who are facing same kind of issues.

Sitecore Publish Selected Sublayouts using WebDeploy

On our Multisite Sitecore instance, we have thousands of sublayouts. So, content authors or developers should be able to modify and publish selected sublayouts. Means, only selected sublayout should be deployed to CD servers along with the items.

We achieved this using Web Deploy, that can be configurable as guided is Sitecore Scalability Guide.

What approach we chose to conditionally sync Sublayouts?

1. On CM environment, we have all sublayouts stored in a folder SiteSublayouts, now on publishing if sync this folder with CD server's relevant folder, then all sublayouts will get synced instead of just publishing the selected one. So, we applied an idea create another directory PublishedSublayouts on same level.
2. So, on publishing a sublayout, it will be first copied from SiteSublayouts to PublishedSublayouts folder and then invoke WebDeploy. So,this will sync all sublayouts from PublishedSublayouts (Actually published or publishable sublayouts) to live server's SiteSublayouts folder.

Note: Here, we created SiteSublayouts and PublishedSublayouts folders outside the Webroot to ease of use.

How we implemented this approach?

1. Configured WebDeploy settings in Sitecore. Enable App_Config\Include\Webdeploy.config file and do changes as below.

   <?xml version="1.0" encoding="utf-8" ?>
   <configuration>
     <sitecore>
       <events>
         <event name="publish:begin">
           <handler type="SitecoreTactics.SublayoutPublish, SitecoreTactics" method="SublayoutPublish">
             <tasks hint="list:AddTask">
               <default type="Sitecore.Publishing.WebDeploy.Task">
                 <!-- Publishing to the target database will trigger this deployment task. -->
                 <!-- You should prefer to write here first publishing target (if have multiple target DBs) -->
                 <targetDatabase>web</targetDatabase>
   
                 <!-- Target server is where we want to send sublayouts. If omitted, operation is performed on the local server. -->
                 <targetServer>x.x.x.x</targetServer>
   
                 <!-- userName and password are optional. If omitted, local user identity or credentials saved in Windows Vault will be used to connect to the server. -->
                 <userName>Administrator</userName>
                 <password>Password</password>
   
                 <!-- localRoot is optional. If omitted, the website root is used. -->
                 <localRoot>E:\CMS\Sitecore\PublishedSublayouts</localRoot>
   
                 <!-- remoteRoot is physical path where sublayouts are stored on remote server -->
                 <remoteRoot>E:\CMS\Sitecore\SiteSublayouts</remoteRoot>
                 
                 <!-- Paths, relative to the localRoot, which will be deployed to the remote location. -->
                 <items hint="list:AddPath">
                   <media>SiteSublayouts/</media>
                 </items>
                 
               </default>
             </tasks>
           </handler>
         </event>
       </events>
     </sitecore>
   </configuration>
   

   Here, we synced the PublishedSublayouts folder of CM server with relevant SiteSublayouts folder of CD server using Web Deploy. And we customized Sitecore's default WebDeploy handler for copying sublayouts mentioned in step 2. 
2. Create a class as below by inheriting with Sitecore.Publishing.WebDeploy.PublishHandler as below. Here, when any sublayout is started publishing, on begin:publish event we defined in step 1, we copy the sublayout from SiteSublayouts folder to PublishedSublayouts folder and invoke WebDeploy as below code.

3. namespace SitecoreTactics
   {
    public class SublayoutPublish : Sitecore.Publishing.WebDeploy.PublishHandler
    {
     string SourceFolder = "E:\CMS\Sitecore\SiteSublayouts";
     string DeployFolder = "E:\CMS\Sitecore\PublishedSublayouts";
   
     protected void DeploySublayout(object Sender, EventArgs args)
      {
      Item RootItem = ((Sitecore.Publishing.Publisher)(((Sitecore.Events.SitecoreEventArgs)(args)).Parameters[0])).Options.RootItem;
        if (RootItem.Paths.Path.ToLower().IndexOf("/sitecore/layout/sublayouts/") >= 0)
      {
       string sublayoutSourceFolder = SourceFolder + <Relative Path of the sublayout>;
       string sublayoutDeployFolder = DeployFolder + <Relative Path of the sublayout>;
   
       // Copy publishing sublayout to Deployable folder   
       File.Copy(sublayoutSourceFolder, sublayoutDeployFolder);
   
       // Invoke WebDeploy to sync published sublayouts
       base.OnPublish(Sender, args);
      }
     }
    }
   }

4. We have multiple CM servers, so we replicated all these published sublayouts with help of DFS across all servers.

It's done! We can also use the same approach for publishing file based media items.

Very soon I will post few leanings we had after implementing sublayouts publishing!