Showing posts with label scalability. Show all posts
Showing posts with label scalability. Show all posts

Avoid UNC file share to prevent File Change Notification issues

Recently we were experienced many unexpected application pool recycling with File Change Notifications (FCN) on our newly created Sandbox Sitecore CM instances. Every few hours (ranging from 20-40 hours) only one of two CM instances was getting restarted automatically with below error.

Change Notification for critical directories. File Change Notification Error in App_LocalResources HostingEnvironment initiated shutdown CONFIG change HostingEnvironment caused shutdown

What we tried

After spending a good amount of time on it, we hypothetical thought for few workarounds to do like
  • Stop Anti-virus on server
  • Use Process Monitor to check a File System for Web-root and Temporary Internet Files, etc. folders to know what's actually causing this
  • Check there's no access given on the web-root to any unauthorized user. 
But all looked good for us, even not getting much help from Google as well. So final option we had to debug the crash dumps.

One more thought came in mind that recycling is happening only on one instance, which is using Sublayouts from other Sitecore instance using UNC file share, and that other instance never got recycled! We know that DFS was the best option here, but we were not able to digest that UNC share can really cause this issue. At the same time we found a nice post where someone already faced the same issue and they fixed it by stopping UNC path sharing for sublayouts. And yes, it worked for us too.
(Note: Recycling is not happening because of number of re-compilations - "numRecompilesBeforeAppRestart")

- http://www.dnnsoftware.com/forums/threadid/318762/scope/posts/file-change-notification-issues-web-farm-over-unc-share
- http://blogs.msdn.com/b/tess/archive/2006/08/02/686373.aspx

What we Learned and Investigated:

- We learnt one more reason that can recycle the application pool
- Avoid using UNC sharing for ASPX, ASCX, RESX, App_Code files, etc. compilable files and Use DFS for them, what Sitecore recommends in Sitecore Scalability Guide.

Our Learning on Publishing Sitecore Sublayouts

After we started using Web Deploy for publishing sublayouts on our multisite environment, we were getting random caching issues on our CD servers. We have multiple publishing target databases and load balanced multiple CD servers. So, as I mentioned in my previous blog, when a sublayout is published, it is published with help of Web Deploy to one server and then the sublayout is replicated to all other servers using DFS.

What kind of issues we found?

Sometimes we found that the published sublayout gets reflected on few servers, on few servers we still get older content. But surprisingly, the published sublayout was replicated on all servers and still we were getting different output from different servers. So, when this happens, we used to clear caches on servers where such issues occurred assuming this might be because of Sitecore caching issues, later on we found workaround to publishing those sublayouts again after 1 or 2 minutes.

Now you will understand how critical it would be to publish sublayouts and getting them reflected on live servers quickly to make go-lives, re-brandings, news releases or press releases successful in one go.

Note: We found this only for those Cacheable sublayouts.

Our learnings to fix such issues

1. Web Deploy publishing should be synchronous.

We analyzed below sequence happened rarely.
  1. First we started publishing, so as per my previous blog, Web Deploy will start deploying sublayout in async mode (By default Web Deploy is configured on Sitecore is asynchronous). So, sublayout item publishing and sublayout physical file deploying are done in parallel.
  2. So, chances that item gets published before sublayout file is copied.
  3. Now, item is published, so CD servers will invoke "publish:end:remote" event and clear HTML (sublayout) cache.
  4. Now, before the new published sublayout file gets deployed to CD server, end-user requested a page which uses the same sublayout. So, the HTML cache will be generated again for older Sublayout.
  5. Now, Web Deploy sent a new Sublayout. (So, we have HTML cache of older sublayout)

Learning: Sublayout publishing should always be synchronous, so publishing will get on hold till the sublayout is not deployed to the CD server, which can be configured in WebDeploy.config as below.
<event name="publish:begin">
 <handler type="SitecoreTactics.Publishing.BeginWebDeploy, SitecoreTactics" method="PublishSublayouts">
  <synchronous>true</synchronous>
  <tasks hint="list:AddTask">
   ......
  </tasks>
 </handler>
</event>

2. Target Database should be the first publishing target.

We have 3 publishing targets say, web, web-2, web-3. Means, any publishing will be done in this sequence itself. Earlier we found the target database inside the WebDeploy.config was set as last publishing target, means web-3. We might have done this in past to make sure the item is published before sublayout getting deployed.

But, as per our recent experiences and findings, we should keep it as first publishing target. So, in our case, it should be web. This will help us when there are multiple servers in DFS, so when DFS is taking some more time in replicating to other servers. So,  for reducing such chances of delayed Sublayout deploy, we should keep web as the publishing target in WebDeploy.config as below.
<tasks hint="list:AddTask">  
 <default type="Sitecore.Publishing.WebDeploy.Task">  
   <!-- It should be the first Publishing target database -->  
   <targetDatabase>web</targetDatabase> 
 </default>
</tasks>

3. We can make few seconds delay in clearing HTML Cache.

If we have done above cases, there are no chances that the CD server where Web Deploy is sending Sublayout file will get any issues. But just consider a worst case where DFS is taking more time to replicate sublayouts to other servers. So, chances that HTML Cache will get cleared before the sublayout replication is done.

To avoid such cases, we can add a delay of few seconds, say 2 or 3 seconds before clearing HTML Cache (Only when the sublayouts are getting published)

Finally, we have hassle free one-time Sublayout publishing working without any caching issues very well! I'm sure this will be helpful to others who are facing same kind of issues.

Sitecore Publish Selected Sublayouts using WebDeploy

On our Multisite Sitecore instance, we have thousands of sublayouts. So, content authors or developers should be able to modify and publish selected sublayouts. Means, only selected sublayout should be deployed to CD servers along with the items.

We achieved this using Web Deploy, that can be configurable as guided is Sitecore Scalability Guide.

What approach we chose to conditionally sync Sublayouts?

  1. On CM environment, we have all sublayouts stored in a folder SiteSublayouts, now on publishing if sync this folder with CD server's relevant folder, then all sublayouts will get synced instead of just publishing the selected one. So, we applied an idea create another directory PublishedSublayouts on same level.
  2. So, on publishing a sublayout, it will be first copied from SiteSublayouts to PublishedSublayouts folder and then invoke WebDeploy. So,this will sync all sublayouts from PublishedSublayouts (Actually published or publishable sublayouts) to live server's SiteSublayouts folder.
Note: Here, we created SiteSublayouts and PublishedSublayouts folders outside the Webroot to ease of use.

How we implemented this approach?

  1. Configured WebDeploy settings in Sitecore. Enable App_Config\Include\Webdeploy.config file and do changes as below.
    <?xml version="1.0" encoding="utf-8" ?>
    <configuration>
      <sitecore>
        <events>
          <event name="publish:begin">
            <handler type="SitecoreTactics.SublayoutPublish, SitecoreTactics" method="SublayoutPublish">
              <tasks hint="list:AddTask">
                <default type="Sitecore.Publishing.WebDeploy.Task">
                  <!-- Publishing to the target database will trigger this deployment task. -->
                  <!-- You should prefer to write here first publishing target (if have multiple target DBs) -->
                  <targetDatabase>web</targetDatabase>
    
                  <!-- Target server is where we want to send sublayouts. If omitted, operation is performed on the local server. -->
                  <targetServer>x.x.x.x</targetServer>
    
                  <!-- userName and password are optional. If omitted, local user identity or credentials saved in Windows Vault will be used to connect to the server. -->
                  <userName>Administrator</userName>
                  <password>Password</password>
    
                  <!-- localRoot is optional. If omitted, the website root is used. -->
                  <localRoot>E:\CMS\Sitecore\PublishedSublayouts</localRoot>
    
                  <!-- remoteRoot is physical path where sublayouts are stored on remote server -->
                  <remoteRoot>E:\CMS\Sitecore\SiteSublayouts</remoteRoot>
                  
                  <!-- Paths, relative to the localRoot, which will be deployed to the remote location. -->
                  <items hint="list:AddPath">
                    <media>SiteSublayouts/</media>
                  </items>
                  
                </default>
              </tasks>
            </handler>
          </event>
        </events>
      </sitecore>
    </configuration>
    
    Here, we synced the PublishedSublayouts folder of CM server with relevant SiteSublayouts folder of CD server using Web Deploy. And we customized Sitecore's default WebDeploy handler for copying sublayouts mentioned in step 2. 
  2. Create a class as below by inheriting with Sitecore.Publishing.WebDeploy.PublishHandler as below. Here, when any sublayout is started publishing, on begin:publish event we defined in step 1, we copy the sublayout from SiteSublayouts folder to PublishedSublayouts folder and invoke WebDeploy as below code.
  3. namespace SitecoreTactics
    {
     public class SublayoutPublish : Sitecore.Publishing.WebDeploy.PublishHandler
     {
      string SourceFolder = "E:\CMS\Sitecore\SiteSublayouts";
      string DeployFolder = "E:\CMS\Sitecore\PublishedSublayouts";
    
      protected void DeploySublayout(object Sender, EventArgs args)
       {
       Item RootItem = ((Sitecore.Publishing.Publisher)(((Sitecore.Events.SitecoreEventArgs)(args)).Parameters[0])).Options.RootItem;
         if (RootItem.Paths.Path.ToLower().IndexOf("/sitecore/layout/sublayouts/") >= 0)
       {
        string sublayoutSourceFolder = SourceFolder + <Relative Path of the sublayout>;
        string sublayoutDeployFolder = DeployFolder + <Relative Path of the sublayout>;
    
        // Copy publishing sublayout to Deployable folder   
        File.Copy(sublayoutSourceFolder, sublayoutDeployFolder);
    
        // Invoke WebDeploy to sync published sublayouts
        base.OnPublish(Sender, args);
       }
      }
     }
    }
  4. We have multiple CM servers, so we replicated all these published sublayouts with help of DFS across all servers.
It's done! We can also use the same approach for publishing file based media items.

Very soon I will post few leanings we had after implementing sublayouts publishing!

Solved Sitecore PDF Loading Issue

Are getting problem in viewing PDF files in your newly upgraded Sitecore environment? We came across the same bug while upgrading Sitecore to version 7.2 that in some browsers while opening PDF files, they get stuck. Surprisingly after that, pressing Ctrl+F5 was allowing to view PDF file. See below screenshot when the downloading stuck:

Sitecore PDF viewing stuck randomly

Workaround from Sitecore

In Sitecore 7.2, they gave workaround to solve the issue by forcing download of PDFs by doing below setting:
        <mediaType name="PDF file" extensions="pdf">
          <mimeType>application/pdf</mimeType>
          <forceDownload>true</forceDownload>
        </mediaType>
But, this is not the solution for our problem. We can't say our clients to download and view the PDF, it should be viewed in browser itself. We applied many tricks, and finally we solved it.

So, what was the issue?

After reverse engineering MediaRequestHandler, we came to know that Sitecore is downloading PDF content partially in range using HTTP 206 Status Code, which might be creating problems in different browser. In Sitecore versions earlier than before 6.5, Sitecore was downloading PDF or any other media in one go, which never failed.

So, one thing was sure that this bug was just because of Range Retrieval mechanism provided by Sitecore or the plug-in our browser does not support it. See below screenshot which shows how Sitecore downloads PDF content in range.

 Sitecore downloads PDF content partially in range

Final Solution

We just tried disabling the Range Retrieval Mechanism as below and found the bug resolved.
    <setting name="Media.EnableRangeRetrievalRequest" value="false" />
Many thanks to our colleague Sandeep Gupta who helped in the investigation.

Enjoy accurate media download now!!

Other Sitecore 7.2 Bugs Solved!



Sitecore Bug - Media Attach Detach Getting Slower

While upgrading to Sitecore 7.2, we faced another big bug regarding slowness. When we attach or detach media file or edit image file online, we found heavy load and slowness on DB server which subsides subsequently, but the browser becomes unresponsive during this time. And slowness is observed across CM platform. (We have not checked but this issue is found from Sitecore 6.5 onwards)

We have New Relic configured on CM server, which shows slow transactions. The stacktrace in all cases has same portion mentioned below:
Sitecore.Data.DataProviders.Sql.SqlDataApi.CreateReader(:0)
Sitecore.Data.DataProviders.Sql.SqlDataProvider.Exists(:0)
Sitecore.Data.DataProviders.Sql.SqlDataProvider.CheckIfBlobShouldBeDeleted(:0)
Sitecore.Data.DataProviders.Sql.SqlDataProvider.RemoveOldBlobs(:0)
Sitecore.Data.DataProviders.Sql.SqlDataProvider.SaveItem(:0)
Alongwith above stacktrace, we also found following as slowest SQL operation which is implemented in CheckIfBlobShouldBeDeleted method, which tells whether blob is used or not:
SELECT tmp.[Id] FROM 
   (SELECT sf.[Id] FROM [SharedFields] sf 
      WHERE sf.[Value] LIKE @blobId 
    UNION 
    SELECT uf.[Id] FROM [UnversionedFields] uf 
      WHERE uf.[Value] LIKE @blobId 
    UNION 
    SELECT vf.[Id] FROM [VersionedFields] vf 
      WHERE vf.[Value] LIKE @blobId 
    UNION 
    SELECT af.[FieldId] [Id] FROM [ArchivedFields] af 
      WHERE af.[Value] LIKE @blobId 
    ) tmp
Credit goes to Muktesh Mehta for drilling down the issue and Sergey Kravchenko from Sitecore Support who gave us solution for it.

We raised this to Sitecore with all above information, Sitecore provided a new config file and assembly for overriding SqlServerDataProvider, which solved our issue. You can get the assembly and config from Sitecore Support using Ticket# 412563.

Hope, this will help you if you are facing the same issue!

Media Performance Related Posts:
- IRequiresSessionState slowing down Sitecore Media Requests
- Improve Sitecore Media Performance using Reverse Proxy
- Upload Sitecore media items in selected languages

Other Sitecore 7.2 Bugs Solved!



IRequiresSessionState slowing down Sitecore Media Requests


For last few weeks we faced a performance issue on our Sitecore production servers. Page response time was substantially increased by 50% and getting slowness in full page loading. While investigation we found that this was just because of our customized MediaRequestHandler.

Our requirement was to display some Disclaimer Page when a user request for some Sitecore media files (not for any image), after accepting disclaimer we can continue to download requested media. All other media items should be served directly.

We used session for managing this disclaimer. For using session in the MediaRequestHandler, we implemented iRequiresSessionState interface in it. See below code snippet
    class CustomRequestHandler : Sitecore.Resources.Media.MediaRequestHandler, IRequiresSessionState
    {
        protected override bool DoProcessRequest(HttpContext context)
        {
            if (context.Request.Url.ToString().ToLower().Contains("/~/media/files/"))
            {
                // Check session, if session does not available, then send it to disclaimer
                // Once user accepts Disclaimer, generate a session with user details.
            }
            else
            {
                // Proceed all image requests
                base.DoProcessRequest(...);
            }
        }
    }

So, what the reason for the slower page rendering? Yes, that's IRequiresSessionState which was a halts in performance.

Can IRequiresSessionState really make my page slower?

Below image shows how ASP.NET Session state works:


Now we can definitely say YES, it can slow down. State access requires that the aspx pages or handler, run sequentially. Each page/handler may read from and write to the session state. To handle this without creating any errors in the session state, .NET will only run one at a time using locking mechanism, so no handlers will run when the aspx page itself is running.

Tactics:
  • Can we customize handler not to use session state?
    - No, this will disable session so we cannot achieve disclaimer functionality
  • We can decide the request needs session or not and use session dynamically.
    - Is it possible? Yes, it is, using HttpModules

Solved by Dynamically deciding Session State Behavior programmatically

Changing ASP.NET session state behavior dynamically is possible only using HttpModules or Global.asax. It is not possible in HttpHandlers or other code behind. This solution is possible only in .NET 4.0 and later. Yes, we can define it at Page Directives but that's static.

As per our requirement, we have to show Dislaimer for selected Sitecore Media Files only (Not for any Media Images), we thought to create a HttpModule, which will check whether the request is for media files or images. If it is a file, then enable session state and if it is an image, disable session state.

Step-1:

We removed IRequiresSessionState from our CustomMediaRequestHandler, so it will not use Session State directly.

Step-2:

Created HttpModule, which will decide whether to use Session State or not.
    public class CustomMediaRequestSessionModule : IHttpModule
    {
        public void Init(HttpApplication application)
        {
            application.BeginRequest += (new EventHandler(this.Application_BeginRequest));
        }

        private void Application_BeginRequest(Object source, EventArgs e)
        {
            HttpApplication application = (HttpApplication)source;
            var CurrentContext = HttpContext.Current;

            if (CurrentContext.Request.Url.ToString().ToLower().Contains("/~/media/files/"))
            {
                // Files need Disclaimer functionality, so session needed
                CurrentContext.SetSessionStateBehavior(System.Web.SessionState.SessionStateBehavior.Required);
            }
            else if (CurrentContext.Request.Url.ToString().ToLower().Contains("/~/media/images/"))
            {
                // Images do not need Disclaimer, so no session needed
                CurrentContext.SetSessionStateBehavior(System.Web.SessionState.SessionStateBehavior.Disabled);
            }
        }
        
        public void Dispose()
        {
        }
    }

Step-3:

Now, in web.config, register this module under system.webServer/modules section:
 <add type="SitecoreTactics.Resources.Media.CustomMediaRequestSessionModule, SitecoreTactics" name="CustomMediaRequestSessionModule" /> 

And now we checked Page Response time, now feeling heavenly. All problem gone, Page Response Time now dropped down to earlier stage.

Good reads for improving Sitecore Media Item Performance

Improve Sitecore Media Performance using Reverse Proxy

- Are you facing slowness in serving media library items?
- Are you getting increased response time due to media requests?
- Is your Sitecore instance serves media slower even if you have applied output (response) cache?

Then this post will surely help you, which describe how we can improve Sitecore Media Library performance by implementing Reverse Proxy Server.

How Reverse Proxy will help to improve media performance?

The Reverse Proxy will play a role being a proxy between the client and Sitecore web server. Reverse Proxy provides caching mechanism, which caches all media items. So, once any user has requested any media file from server, will be get cached on Reverse Proxy itself. So, from second time onwards, Reverse Proxy will not get media from Sitecore Web Server but will serve it from its own cache.

We can use URL Rewrite Module and Application Request Routing (ARR) to implement a Reverse Proxy Server.


Step - 1 : Install ARR and Url Rewrite on IIS?

  1. Setup IIS 7.0+ on your Server which will work as Proxy.
  2. Install URL Rewrite module. You can download it from here.
  3. Install ARR module. You can download it from here.

Step - 2 : Configure URL Rewrite module:

  1. Create a new Website in IIS or use Default website, and click on Website, then click on URL Rewrite option under IIS section
  2. Edit inbound rules as below:





    Above configuration shows that if requested host is www.patelyogesh.in, then this rule will be applied for its all requests (*).
  3. Configure Rewrite URL for above configurations as below. This will make sure that all requests coming from http://www.patelyogesh.in/ will be rewrited to http://rp.patelyogesh.in/*.



    When we apply above configurations, it will generate a web.config file under the website directory, which will look like below:

    <system.webServer>
        <rewrite>
          <rules>
             <rule name="Sitecore-Production" stopProcessing="true">
                 <match url="(.*)" />
                 <action type="Rewrite" url="http://rp.patelyogesh.in/{R:1}" />
                 <conditions>
                     <add input="{HTTP_HOST}" pattern="www.patelyogesh.in" />
                 </conditions>
             </rule>
          </rules>
        </rewrite>
      </system.webServer>
    

    We can also configure multiple domain's URL rewrites in the same way.

Step - 3 : Configure ARR (Application Request Routing) Cache

  1. Select the Server Node, now select Application Request Routing Cache option.

  2. Add Drive where the caching will be stored by ARR Module. The below image shows how we can configure ARR module and how it will look.
  3. Make sure the Identity user of the Application Pool should have read/write access of the drives configured here. So, ARR will store all cache files here only.

  4. Enable Proxy As per below image, click on Proxy Settings on the right side bar and enable Proxy.


Finally, Reverse Proxy setup finished, took just 15 minutes only!!

Now, Request to http://www.patelyogesh.in/. This will serve you content from http://rp.patelyogesh.in itself by traversing through Reverse Proxy. It's really easy and simple, isn't it?

How to confirm Reverse Proxy working fine?

We can provide our Custom Response Headers using Proxy Settings from ARR Cache option as shown in above image. So, if Reverse Proxy is working well, we will get those headers in response.

How to confirm ARR Caching working fine?

In File Explorer, open the Drive folder we configured in ARR Cache settings. We will get all files cached by ARR.

Cache for the URL: http://www.patelyogesh.in/~/media/Images/yogi.png (Rewrited URL: http://rp.patelyogesh.in/~/media/Images/yogi.png) will be stored at : <website root>\rp.patelyogesh.in\~\media\Images\yogi.png.full.

How to delete ARR cache programmatically?

In ARR Cache settings, you will find a button Delete Specific Cached Objects which can clear specific URL cache. It also supports wild cards for clearing cache.

We can create a web service on the ARR website, which will get a URL as input and will clear cache accordingly using below code. Now, on each media item publish from Sitecore, we will clear the ARR cache. We can decice How and when to make a call to the ARR web service to cache clear according to our architecture.

Source code to clear ARR cache programmatically:
[WebMethod]
public static ClearCache(string urlToCacheClear)
{
      var url = urlToCacheClear == "ALL" ? string.Empty : urlToCacheClear;

      var m = new ServerManager();
      var x = m.GetApplicationHostConfiguration().GetSection("system.webServer/diskCache");
      var method = x.Methods["FlushUrl"].CreateInstance();
      method.Input.SetAttributeValue("url", url);
      method.Execute();
}

Enjoy Reverse Proxy! Enjoy improved Sitecore Media Library performance!!

Good reads on Reverse Proxy:
- http://www.agarwalnishant.com/2013/04/improve-sitecore-media-library.html

- http://www.iis.net/learn/extensions/url-rewrite-module/reverse-proxy-with-url-rewrite-v2-and-application-request-routing

Sitecore Parallel Publishing Implementation

Many readers of my blogs asked me the practical way to implementing parallel publishing using multiple publish queue on Sitecore. Earlier I have posted a blog on it explaining theoretical approach for Parallel Publishing in Sitecore using custom EventQueue. It is really difficult to explain that approach on a blog. So here, I am going to explain easiest and quickest implementation of multiple publish queue practically.

Why Parallel Publishing from different instances?

In our multisite environment, hundreds of users work at a time, add/update thousands of items and publish them frequently. Daily 15,000+ items get published. Sometimes this number crosses 50,000 per day. So, mostly we see long publishing queue and users have to wait for minutes to get their items published. To make publishing more scalable, we thought to have multiple publishing instances in our CM environment and we dedicated them for different types of users based on business need.

What we wanted to achieve:

We wanted to have three different Publishing Instances for our users.
Publish Instance - 1(PI-1) Dedicated to Client Content Authors.
Publish Instance - 2(PI-2) Dedicated to Internal Content Authors.
Publish Instance - 3(CM-1) Dedicated to all other users. This is the Content Management Server, where users use Page Editor or Content Editor.

For this, we created three different roles named Publish Instance One, Publish Instance Two and Content Management Instance. So, depending on role assigned, we will allocate Publishing Instance to the user.

How we hooked Publishing Process

Below image shows how we hooked and modified startPublishing event to choose Publish Instance depending on user publishes.

Parallel Publishing Architecture with Multiple Sitecore Instances

Let's see practical implementation of Parallel Publish in Sitecore?

You can see How Parallel Publishing works on Sitecore in below video.
In this video, below is the allocation done for users:

User Role Allocated Publishing Instance
sitecore\admin Publish Instance One Machine-PI-1
sitecore\yogesh Publish Instance Two Machine-PI-2
All other users Content Management Instance Machine-CM

So, we have three different Publishing Queues available.



Now see how we can achieve this in simple steps below:
  1. Setup three different CM instances (in a cluster) with same master, web and core databases.
  2. Parallel publishing requires 1+ Sitecore Instances So indirectly we can say parallel publishing is possible on n number of CM instances.

  3. Enable EventQueue on each instance
  4. On all the Sitecore Instances, it is must to enable EventQueue from web.config or App_config\Include\ScalabilitySettings.config file.
         <setting name="EnableEventQueues" value="true" />
       
  5. Configure allocation of Roles & Publishing Instances
  6. On all the Sitecore Instances, create a config file: App_config\Include\ParallelPublish.config file. This file should remain same on each instance.
    <configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
      <sitecore>
        <parallelpublishRoles>
          <!-- Roles needed for Parallel Publising -->
          <role name="sitecore\Publish Instance One" publishinstance="Machine-Sitecore-PI-1" />
          <role name="sitecore\Publish Instance Two" publishinstance="Machine-Sitecore-PI-2" />
          <role name="sitecore\Content Management Instance" publishinstance="Machine-Sitecore-CM" />
        </parallelpublishRoles>
      </sitecore>
    </configuration>
       
  7. Override publish:startPublishing Event with your custom code as below
  8. namespace SitecoreTactics.Publishing
    {
        public class RemotePublishingEventHandler : Sitecore.Publishing.RemotePublishingEventHandler
        {
            public override void OnStartPublishing(object sender, EventArgs args)
            {
                RemoteEventArgs<Sitecore.Publishing.StartPublishingRemoteEvent> args2 = args as RemoteEventArgs<Sitecore.Publishing.StartPublishingRemoteEvent>;
                if (args2 == null)
                {
                    throw new InvalidOperationException("Unexpected event args: " + args.GetType().FullName);
                }
    
                Sitecore.Globalization.Language lang = Sitecore.Globalization.Language.Parse(args2.Event.ClientLanguage);
                StartPublishingRemoteEvent myArgs = new StartPublishingRemoteEvent(args2.Event.Options, args2.Event.StatusHandle, args2.Event.UserName, lang);
    
                // Findout Dedicated Publishing Instance for current user
                string allocatedPublishInstance;
                using (new SecurityDisabler())
                {
                    User user = User.FromName(args2.Event.UserName, true);
    
                    allocatedPublishInstance = GetPublishInstanceForUser(user);
                }
    
                // Allocate Publishing Instance to current publishing
                myArgs.SetPublishInstance(allocatedPublishInstance);
    
                // Allocated Publishing Instance will pick up the publishing
                if (this.IsPublishingServer(myArgs))
                {
                    Sitecore.Publishing.DistributedPublishingManager.StartPublishing(myArgs);
                }
            }
    
            protected string GetPublishInstanceForUser(User user)
            {
                // Find role and allocated publishing instance
                foreach (XmlNode node in Factory.GetConfigNodes("parallelpublishRoles/role"))
                {
                    string role = XmlUtil.GetAttribute("name", node);
                    string publishInstance = XmlUtil.GetAttribute("publishinstance", node);
                    if (user.IsInRole(role))
                        return publishInstance;
                }
    
                // Return default Publish Instance
                return Settings.Publishing.PublishingInstance;
            }
        }
    
    
        public class StartPublishingRemoteEvent : Sitecore.Publishing.StartPublishingRemoteEvent
        {
            public StartPublishingRemoteEvent(Sitecore.Publishing.DistributedPublishOptions[] options, Sitecore.Handle statusHandle, string userName, Sitecore.Globalization.Language clientLanguage)
                : base(options, statusHandle, userName, clientLanguage)
            {
            }
    
            // Methods
            [Obsolete("Use StartPublishingRemoteEvent(DistributedPublishOptions[] options, Handle statusHandle, string userName, Language clientLanguage) instead.")]
            public StartPublishingRemoteEvent(Sitecore.Publishing.DistributedPublishOptions[] options, Sitecore.Handle statusHandle, string userName) :
                base(options, statusHandle, userName, Sitecore.Data.Managers.LanguageManager.DefaultLanguage)
            {
            }
            
            public void SetPublishInstance(string piName)
            {
                base.PublishingServer = piName;
            }
        }
    }
    
  9. Update your custom publish:startPublishing event in web.config with below changes
  10. <event name="publish:startPublishing">
            <!-- handler type="Sitecore.Publishing.RemotePublishingEventHandler, Sitecore.Kernel" method="OnStartPublishing" / -->
    
          <handler type="SitecoreTactics.Publishing.RemotePublishingEventHandler, SitecoreTactics.Publishing" method="OnStartPublishing" />
            
          </event>
    
       
Now try in your environment, parallel publishing is surely working for you too. If it's not working, check
  • EventQueue is disabled, enable it
  • All CM instance should have exact time. If two instances have time difference of 5 minutes, then there the instance running late will get updates of other instance after 5 minutes, so sync will never be done between them.
  • Give proper Roles Name and Publishing Instance Name in the ParallelPublish.config file.
  • Check all your Sitecore instances are pointing to same databases - master, web, core.
  • Configure the publish:startPublishing event and ParallelPublish.config on each Sitecore Instance.
  • ParallelPublish.config file should have same contents on all the instances.
  • Instance which needs to do publish should be up and running.


Now enjoy parallel publishing without waiting time, without queue stuck!

Invoke Sitecore Scheduling Agent Programmatically

Sitecore provides facility to run scheduled jobs at specified regular interval like DatabaseAgent, UrlAgent, etc. We created our own agent to make scheduled publishing. So, this agent is executed every one hour and starts pending publishing, which can take several minutes say 20-30 minutes.

This can be achieved by creating a scheduling agent in web.config as below:
    <agent name="ScheduledPublish" type="SitecoreTactics.Tasks.StartPublish" method="StartPublish" interval="01:00:00" />
Here, every 1 hour, Sitecore invokes the agent named ScheduledPublish and executes StartPublish method of SitecoreTactics.Tasks.PublishManager class.

Requirement to invoke Sitecore Agent

Now, we had a requirement to invoke our custom agent in between through a Web Page when user wants to do instant publish instead of waiting for next turn of its execution. Should we call our code from the web page itself or invoke the Agent directly programmatically? There was no clear idea which can be a better approach? Below are two approaches we worked on.

Wrong approach we chose earlier to execute code directly

Earlier, we called the PublishManager's StartPublish method directly from the web page, when user clicks on a button as below.
protected void btnStartPublish_Click(object sender, EventArgs e)
{
    SitecoreTactics.Tasks.PublishManager publisher = new SitecoreTactics.Tasks.PublishManager();
    publisher.StartPublish();
}
The limitation in the approach was, the code was executed by the thread of web page or a web request, which needed to be executed more than 30 minutes. But, after the Server's Script timeout (default 5 minutes), the execution was stopped every time and could not complete full job of 30+ minutes. But if same code was executed as a Sitecore Agent, it finished all execution, because Sitecore Agents are executed by separate ManagedThreadPools, which never has TimeOut until they finish job or Application ends.

Correct approach to invoke Sitecore Scheduling Agent

We searched a lot on Google about invoking Sitecore Agent. Later on, Reflector came to our help. Using it we found the code from where Sitecore invokes its agents and we implemented the same code as below:
protected void btnStartPublish_Click(object sender, EventArgs e)
{
 foreach (XmlNode node in Factory.GetConfigNodes("scheduling/agent"))
 {
  object obj = Factory.CreateObject(node, true);
  string name = XmlUtil.GetAttribute("name", node);
 
  if (name == "ScheduledPublish") // Scheduling Agent Name
  {
   string method = XmlUtil.GetAttribute("method", node);
 
   JobOptions options = new JobOptions(name, "", "", obj, method);
   options.AtomicExecution = true; // Allow multiple instances of the agent running simultaneously
   JobManager.Start(options);
 
   break;
  }
 }
}
Now, we are not worried for executing agents's code. Agent's process needs to be executed for one hour, two hours, ten hours? No worries, we can now call Sitecore Jobs programmatically!!

Show PDF Thumbnail as Icon in Content Editor

Sitecore shows PDF icon as a thumbnail, so it becomes very difficult to find out a PDF file from a big list of uploaded files. Just imagine, life would be so easy when Sitecore provides PDF thumbnails as the icons just like images!!

It is quite possible and easy to show PDF thumbnails in different dimensions just by overriding the MediaRequestHandler of Sitecore. See my earlier post, PDF Thumbnail Handler blog. You can also find PDF Thumbnail Handler on Sitecore MarketPlace.

Use of PDF Thumbnails Handler

Once the concept of PDF Thumbnail Handler is understood, we can achieve this easily. Do following:
  1. Install PDF Thumbnail Handler to your Sitecore and make it up and running.
  2. Update PDF item's Icon field. Replace ~/media to ~/mediathumb
  3. Now, check Sitecore Content Editor will show PDF thumbnails as icons.
By default PDF icons are available as below image:

Sitecore shows PDF icon as thumbnail
The Icon has value: ~/media/36C02213E38441D9BA1AA82DB86A80E0.ashx?h=16&thn=1&w=16, which will load icon of PDF which is defined in the sitecore itself.

As per PDF Thumbnail Creation Handler, by using ~/mediathumb handler by updating its value to: ~/mediathumb/36C02213E38441D9BA1AA82DB86A80E0.ashx?h=16&thn=1&w=16. See below image which shows how PDF thumbnail is shown as icon.


We can show PDF thumbnail as icon like this



 Let's make PDF thumbnails working in Content Editor

Our requirement is to show thumbnails like below image:

Show PDF thumbnails by overriding MediaProvider


Override MediaProvider of Sitecore, for that you need to do changes in web.config file.
   <!-- override Sitecore MediaProvider -->
   <mediaProvider type="SitecoreTactics.MediaProvider, SitecoreTactics"/>

Below is the code required in MediaProvider class. In the GetMediaUrl function, when the request of any PDF file is there, then replace existing ~/media/ handler with ~/mediathumb/.
namespace SitecoreTactics
{
    public class MediaProvider: Sitecore.Resources.Media.MediaProvider
    {
        public override string GetMediaUrl(MediaItem item, MediaUrlOptions options)
        {
            string mediaUrl;
            mediaUrl = base.GetMediaUrl(item, options);

            // When item is PDF and Thumbnail is requested
            if (item.Extension == "pdf" && options.Thumbnail)
                mediaUrl = mediaUrl.Replace(Config.MediaLinkPrefix, "~/mediathumb/");

            return mediaUrl;
        }
    }
}



Wow, let's enjoy easier life with PDF thumbnails in Content Editor!!

Related Posts:
- PDF Thumbnail Handler
- Sitecore HTTP Custom Handler

PDF Thumbnail Creation Handler in Sitecore

I recently published a Sitecore Marketplace module PDF Thumbnail Creater Handler. Basically it allows to generate thumbnail on-the-fly (dynamically) for the uploaded PDF in sitecore by passing width and/or height. This will allow user to request thumbnail for any height or width and the thumbnail will be stored as a media cache in Sitecore.

Requirement

Suppose user uploaded PDF in Sitecore.
  • User want to generate thumbnails of the fist page of uploaded PDF. 
  • User can choose height and/or width of thumbnails without any configurations.
  • If user replaces a new file instead of that PDF, it should serve thumbnail of newly uploaded PDF. 
  • Similarly the thumbnail URL should work if user moves/copies/deletes the PDF item.
  • Finally, the conversion process should be scalable and quick enough so it does not affect performance and the thumbnails should be cached as media cache too

Below are the outputs of same PDF but different size thumbnails: 

PDF Thumb Original
http://sitecore/~/mediathumb/Files/MyPDF.pdf
PDF Thumb Width=300
http://sitecore/~/mediathumb/Files/MyPDF.pdf?w=300
PDF Thumb Width=150
http://sitecore/~/mediathumb/Files/MyPDF.pdf?w=150

Theoretical Concept

I just thought to achieve this overriding Sitecore MediaRequestHandler. Sitecore allows to generate different sized thumbnails of uploaded image files, See how. Why should not I use the same concept to generate thumbnails from PDF? Only concern I looked was to convert PDF to JPG only, but was not that much easy. 

So, what I wanted to achieve:
 
PDF / Thumbnail Details PDF / Thumbnail Path/URL
PDF Sitecore Path /sitecore/media library/Files/MyPDF
PDF URL http://sitecore/~/media/Files/MyPDF.pdf
PDF Thumbnail URL http://sitecore/~/mediathumb/Files/MyPDF.pdf
or
http://sitecore/~/mediathumb/Files/MyPDF.jpg
PDF Thumbnail URL
Width = 100 px
http://sitecore/~/mediathumb/Files/MyPDF.pdf?w=100
PDF Thumbnail URL
Height = 200 px
http://sitecore/~/mediathumb/Files/MyPDF.pdf?h=200
PDF Thumbnail URL
Width = 100 px
Height = 200 px
http://sitecore/~/mediathumb/Files/MyPDF.pdf?w=100&h=200

How this achieved

PDF to JPG conversion can be done using GhostScript (With GPL License, which is free), which is very efficient and gives flexibility with many other options.

You can read my older post regarding Sitecore Custom HTTP Handler, I have described there in detail.

I created own Sitecore Custom Handlers (SitecoreTactics.ThumbnailManager.PDFThumbnailRequestHandler) to generate thumbnails of media(PDF) items. See below config changes this requires:
    
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
  <sitecore>

    <!-- Define Custom Handler -->
    <customHandlers>
      <handler trigger="~/mediathumb/" handler="sitecore_media_thumb.ashx"  />
    </customHandlers>

    <!-- Define Media Prefix -->
    <mediaLibrary>
      <mediaPrefixes>
        <prefix value="~/mediathumb" />
      </mediaPrefixes>
    </mediaLibrary>
  </sitecore>


  <!-- Define Web Handler -->
  <system.webServer>
 <handlers>
     <add verb="*" path="sitecore_media_thumb.ashx" type="SitecoreTactics.ThumbnailManager.PDFThumbnailRequestHandler, SitecoreTactics.ThumbnailManager" name="SitecoreTactics.PDFThumbnailRequestHandler"/>
 </handlers>
  </system.webServer>
</configuration>    
Handler's source code to process thumbnail and use media cache as below.
namespace SitecoreTactics.ThumbnailManager
{
    public class PDFThumbnailRequestHandler : Sitecore.Resources.Media.MediaRequestHandler
    {
        protected override bool DoProcessRequest(HttpContext context)
        {
            Assert.ArgumentNotNull(context, "context");
            MediaRequest request = MediaManager.ParseMediaRequest(context.Request);

            if (request == null)
                return false;

            Sitecore.Resources.Media.Media media = null;
            try
            {
                media = MediaManager.GetMedia(request.MediaUri);
                if (media != null)
                    return this.DoProcessRequest(context, request, media);
            }
            catch (Exception ex)
            {
                Log.Error("PDF Thumbnail Generator error - URL:" + context.Request.Url.ToString() + ". Exception:" + ex.ToString(), this);
            }

            if (media == null)
            {
                context.Response.Write("404 - File not found");
                context.Response.End();
            }
            else
            {
                string itemNotFoundUrl = (Context.Site.LoginPage != string.Empty) ? Context.Site.LoginPage : Settings.NoAccessUrl;

                if (Settings.RequestErrors.UseServerSideRedirect)
                    HttpContext.Current.Server.Transfer(itemNotFoundUrl);
                else
                    HttpContext.Current.Response.Redirect(itemNotFoundUrl);
            }
            return true;
        }

        protected override bool DoProcessRequest(HttpContext context, MediaRequest request, Sitecore.Resources.Media.Media media)
        {
            Assert.ArgumentNotNull(context, "context");
            Assert.ArgumentNotNull(request, "request");
            Assert.ArgumentNotNull(media, "media");

            if (this.Modified(context, media, request.Options) == Sitecore.Tristate.False)
            {
                Event.RaiseEvent("media:request", new object[] { request });
                this.SendMediaHeaders(media, context);
                context.Response.StatusCode = 0x130;
                return true;
            }

            // Gets media stream for the requested media item thumbnail
            MediaStream stream = ProcessThumbnail(request, media);
            if (stream == null)
            {
                return false;
            }
            Event.RaiseEvent("media:request", new object[] { request });
            this.SendMediaHeaders(media, context);
            this.SendStreamHeaders(stream, context);
            using (stream)
            {
                context.Response.AddHeader("Content-Length", stream.Stream.Length.ToString());
                WebUtil.TransmitStream(stream.Stream, context.Response, Settings.Media.StreamBufferSize);
            }
            return true;
        }

        private MediaStream ProcessThumbnail(MediaRequest request, Sitecore.Resources.Media.Media media)
        {
            MediaStream mStream = null;
            
            ParseQueryString(request);

            mStream = MediaManager.Cache.GetStream(media, request.Options);

            if (mStream == null)
            {
                string tempPath = Settings.TempFolderPath + "/PDF-Thumbnails/";

                tempPath = MainUtil.MapPath(tempPath);

                if (!Directory.Exists(tempPath))
                    Directory.CreateDirectory(tempPath);

                // Prepare filenames
                string pdfFile = tempPath + media.MediaData.MediaId + ".pdf";
                string jpgFile = tempPath + media.MediaData.MediaId + ".jpg";

                string resizedJpgFile = tempPath + media.MediaData.MediaId + "_" + request.Options.Width.ToString() + "_" + request.Options.Height.ToString();

                if (!File.Exists(jpgFile))
                {
                    // Save BLOB media file to disk
                    MediaConverter.ConvertMediaItemToFile(media.MediaData.MediaItem, pdfFile);

                    // Convert PDF to Jpeg - First Pager
                    MediaConverter.ConvertPDFtoJPG(pdfFile, 1, jpgFile);

                }

                // Resize Image
                MediaConverter.ReSizeJPG(jpgFile, resizedJpgFile, request.Options.Width, request.Options.Height, true);

                // Convert resized image to stream
                MediaStream resizedStream = new MediaStream(File.Open(resizedJpgFile, FileMode.Open, FileAccess.Read, FileShare.Read), "jpg", media.MediaData.MediaItem);

                // Add the requested thumbnail to Media Cache
                MediaStream outStream = null;
                MediaManager.Cache.AddStream(media, request.Options, resizedStream, out outStream);

                if (outStream != null)
                {
                    // If Media cache is enabled
                    return outStream;
                }

            }

            // If Media cache is disabled
            return mStream;
        }

        public void ParseQueryString(MediaRequest mediaRequest)
        {
            HttpRequest httpRequest = mediaRequest.InnerRequest;

            Assert.ArgumentNotNull((object)httpRequest, "httpRequest");
            string str1 = httpRequest.QueryString["as"];
            if (!string.IsNullOrEmpty(str1))
                mediaRequest.Options.AllowStretch = MainUtil.GetBool(str1, false);
            string color = httpRequest.QueryString["bc"];
            if (!string.IsNullOrEmpty(color))
                mediaRequest.Options.BackgroundColor = MainUtil.StringToColor(color);

            string str2 = httpRequest.QueryString["dmc"];

            mediaRequest.Options.Height = MainUtil.GetInt(httpRequest.QueryString["h"], 0);
            string str3 = httpRequest.QueryString["iar"];
            if (!string.IsNullOrEmpty(str3))
                mediaRequest.Options.IgnoreAspectRatio = MainUtil.GetBool(str3, false);

            mediaRequest.Options.MaxHeight = MainUtil.GetInt(httpRequest.QueryString["mh"], 0);
            mediaRequest.Options.MaxWidth = MainUtil.GetInt(httpRequest.QueryString["mw"], 0);
            mediaRequest.Options.Scale = MainUtil.GetFloat(httpRequest.QueryString["sc"], 0.0f);
            string str4 = httpRequest.QueryString["thn"];
            if (!string.IsNullOrEmpty(str4))
                mediaRequest.Options.Thumbnail = MainUtil.GetBool(str4, false);

            mediaRequest.Options.Width = MainUtil.GetInt(httpRequest.QueryString["w"], 0);
        }
    }
}

You can get full source code (of older version) of this module from Sitecore Marketplace.
Update:
Module available on Sitecore Marketplace contains older code, having a bug on media cache that when someone overwrite media files (using detach/attach), it was serving older thumbnail. This bug has been fixed in above code, and will be available on marketplace very soon. Meanwhile, you can download the source code (excluding DLLs) from https://drive.google.com/file/d/0B1otw7vE3rGTQmU1U1l2TTJHQTA/view?usp=sharing:

Benefits of this approach

  1. Dynamic conversion of PDF to Thumbnail when requested
  2. Allows to convert different size thumbnails
  3. Repeated thumbnails will be served from media cache.
  4. Conversion is fast using GhostScript and media cache adds more power.


Related Posts:
- Show PDF Thumbnail Icons in Content Editor
- Sitecore HTTP Custom Handler

Render Sitecore Content Item with name of language

One issue reported in Sitecore SDN forum regarding rendering an content item under Home which has name of a language. I mean, there is an item /sitecore/content/Home/uk. Now, Sitecore consider /uk/ as an language, so it will render the Home item with uk (Ukranian) language (even if uk language is not added to Sitecore Languages). Its expected behavior is to render /Home/uk item in default language itself.

See what's the issue

We have an item named uk under Home.




Now, see what happens when we access http://patelyogesh.in/uk

Sitecore content item with language name

Why this happens

StripLanguage processor in preprocessRequest pipeline checks the FilePath and extracts expected language name from it. If the FilePath contains a valid language, it will assign it to Context.

So, in our URL: http://patelyogesh.in/uk, it will get FilePath as uk. So, assumes the uk item as uk language and set it to Context Language and Home is set to Context item.

How to solve this

One solution came in my mind to write a custom processor in httpRequestBegin, which will over-write Context Language and Context Item both, using it it is possible to solve. But, as John West suggested in forum, this might be possible using StripLanguage. So, thought to implement this and yes, it solves and is easy too. We can solve this issue overriding StripLanguage processor in preprocessRequest.
<!--<processor type="Sitecore.Pipelines.PreprocessRequest.StripLanguage, Sitecore.Kernel"/>-->
<processor type="SitecoreTactics.StripLanguage, SitecoreTactics">
  <ignoreLanguages hint=”list”>
    <language>uk</language>
    <language>en-gb</language>
    …
  </ignoreLanguages>
</processor>
To override the StripLanguage, we need to modify only one function named ExtractLanguage, which extracts the language. Now, as per our requirement, as need to ignore uk and en-gb languages. So, we will not consider language when it is uk or en-gb.

Below code should be implemented for this, but it does not contain code to read above ignoreLanguages setting. Here I have hard-coded these languages for better understanding:
public class StripLanguage : PreprocessRequestProcessor
{

    // Other methods
    ......

    private static Language ExtractLanguage(HttpRequest request)
    {
        Language language;
        Assert.ArgumentNotNull(request, "request");
        string str = WebUtil.ExtractLanguageName(request.FilePath);

        // Our code starts here
        // If the found language is uk, then set it to Empty, 
        // so sitecore will consider this as no language
        if (str == "uk" || str == "en-gb")
            str = string.Empty;
        // Our code ends here

        if (string.IsNullOrEmpty(str))
        {
            return null;
        }
        if (!Language.TryParse(str, out language))
        {
            return null;
        }
        return language;
    }

    ....
    // Other methods
}
Now, see below screen, referring /Home/uk page now refers to uk page itself with en language instead of Home page with uk language.



Finally, StripLanguage worked, we might need some extra code when implementing Multisite environment!!

Some Sitecore Publishing Findings

Well, I do not want to waste your time in explaining how Sitecore publishing works, there are many blogs explaining What is publishing, publishing modes, ways to publish content, versions, target databases, etc. So, I would like to share some hidden secrets about Sitecore Publishing Mechanism, which I learn while exploring it.


Below are the questions mostly Sitecore developers always eager to know:

Why my publish fails or when publish:fail event is called?

While publishing any runtime error occurs like SQL Connectivity loss/Timeout, any SQL Exception, etc. the publish fails. The items already published, cannot be rollback and the items pending to publish needs to be published again. Means, Sitecore publishing does not maintain Transactions.

Check the publishing code from Sitecore.Publishing.Publisher.Publish. The code also contains comments which can explain how publishing works.
public virtual void Publish()
{
    object obj3;
    Monitor.Enter(obj3 = this.GetPublishLock()); // Locks the publishing. That's why publishing is a sequential process.
    try
    {
        using (new SecurityDisabler())
        {
            this.AssertState();
            this.NotifyBegin(); // Raises publish:begin event
            this.PerformPublish(); // Sends the current publish job to start publish
            this.NotifyEnd(); // Raises publish:end event
            this.UpdateLastPublish(); // Updates last publish date in source database
        }
    }
    catch (Exception exception)
    {
        // This function raises publish:fail event.
        this.NotifyFailure(exception);
        throw;
    }
    finally
    {
        Monitor.Exit(obj3);
    }
}

Why my publish expires or when publish:expire event is called?

You might get below configuration setting in web.config file.
     <setting name="Publishing.TimeBeforeStatusExpires" value="02:00:00"/>
If your publishing is taking more than time specified in the setting, the publishing job expires. Here, if your publishing job is taking more than 2 hours, it will get expired. Increasing its value to 5 or 10 hours can solve your problem of expiration if you have to publish thousands of items or heavy media items in one go.

Why my items not getting published?

There can be many reasons behind it:
  • The user publishing the item has not having its rights. Giving rights to the user can do publish.
  • When default/anonymous user's access rights are removed for the item or its parent. Giving rights to the user can do publish.
  • If still you cant find out issue, enable event timing level to high. Now, check logs on publish instance while doing publishing, this will log all events in details and help to identify the cause of problem in publish.
       <events timingLevel="high">
    
  • If you have set a publish instance, check Publish Instance is running or not. It should be up & running.
  • If it is not above case, and still publish not happening, then you need to enable eventqueue on CM as well as PI.
  • If first publish is going on after restart, it can be slow too. It needs to generate more cache and that's why it taking more time. Once cache generated, it will start publishing normally.

What are the reasons for slower publishing?


  • Publishing is a slower process by default as per its process. It requires a lot of processing time, so it consumes lots of CPU resources on Publish Instance. It needs constant updates on web database, so while publishing many insert, update or delete queries are executed. Many caches are cleared.
  • You are publishing many as well as heavy items (items with more size like media files)
  • Have you recently updates Access Viewer to give rights to any Role?
  • Have you checked your target database server is performing well? Checked its IOPS(Input/Output Operations Per Second)?
  • Many publishing or other jobs are in queue. Check below setting in your web.config on your PI.
    <setting name="MaxWorkerThreads" value="20" />
    
    This config determines how many worker threads can be running simultaneously. If your publishing or any other jobs have occupied these (in the example 20) threads, next queued job/publishing has to wait till it gets a free thread. Also, in this situation, your publishing might get stuck.

    You can increase its value as per need. Also remember, greater value can allow more jobs to execute, so may slow down the whole instance.

    You can use Publish Queue Viewer to know how many jobs are running on your instance on:
    - Sitecore Publish Queue Viewer - 1
    - Sitecore Publish Queue Viewer - 2

What can be the optimized publishing approach?

To get best performance with publishing, you might need to take care below things:
  • Proper Cache Tuning on PI as per need
  • Prevent frequent and long publishes
  • Allow Publish Basket facility, means adding media and other items' references
  • Allow scheduling publishing
I'll be posting regarding this very soon!!

How can I setup and use Sitecore Publish Instance?

Refer my earlier post regarding Publishing Scalability or Setting Publish Instance. This post describes how we can set separate Publish Instance and how it works.

Can I use multiple publish instance to support parallel publish?

Many have asked me the question, is it really possible to create Multiple Instances which can do publish parallel? The answer is YES. Although, Sitecore does not recommended this approach as per its architecture. But, Sitecore architecture is so scalable, we can still achieve it.

Refer my earlier post regarding Multiple Publish Instance or Parallel Publishing

Must read posts for Sitecore Publish

- Intelligent Publish in Sitecore - The most optimized approach
- Setup Publish Instance
- Sitecore Parallel Publishing using Multiple Publish Instances

Hostname Resolution without editing Host File

Have you ever thought to map hostname without changing host file? If your Sitecore or any other website is working with Load Balanced environment with multiple servers, you surely have thought to point your request to a particular server (IP Address). When you need to check your functionality working/not working on any of these servers, you are surely going to make host entry for individual servers one by one and test the site.

Uffff... it's so boring and lengthy task and chances of human mistakes. Let's see how we can automate this.

Traditional way for Host Name Resolution Using a Hosts File

Find Hosts file in the %SystemRoot%\system32\drivers\etc directory. Read more about Host Name Resolution. Following is an example of the contents of the Hosts file:
#
# Table of IP addresses and host names
#

127.0.0.1 localhost
192.168.220.111 www.patelyogesh.in # Server-1
#192.168.220.112 www.patelyogesh.in # Server-2

How we can bypass hosts file for host name resolution

Traditional way to get response using WebRequest is requesting to http://www.patelyogesh.in but this will get content from any of the servers which Load Balancer choose for our request.

We can get content from particular server by requesting to: http://<IP Address>/<Page> and passing host/domain name as request.Host as below:
      
var request = (HttpWebRequest)WebRequest.Create("http://<IP Address>/<Page>");
request.Host = HostName;
Let us see how whole source code / GetResult function will be to achieve this:
      string strIPAddress = "192.168.220.111";
      string strHostName = "www.patelyogesh.in";

      string result = GetResult(strIPAddress, strHostName);

      public string GetResult(string strIPAddress, string strHostName)
      {
            var request = (HttpWebRequest)WebRequest.Create("http://" + strIPAddress);
            request.Host = strHostName;

            request.Credentials = CredentialCache.DefaultCredentials;
            HttpWebResponse response = (HttpWebResponse)request.GetResponse();

            // Get the stream containing content returned by the server.
            Stream dataStream = response.GetResponseStream();

            // Open the stream using a StreamReader for easy access.
            StreamReader reader = new StreamReader(dataStream);

            // Read the content. 
            string responseFromServer = reader.ReadToEnd();

            // Cleanup the streams and the response.
            reader.Close();
            dataStream.Close();
            response.Close();
            return responseFromServer;
      }
Again, www.patelyogesh.in is mapping to 192.168.220.111. Now suppose we want to request to http://www.patelyogesh.in/AboutUs.html page, we can pass parameters as:

var request = (HttpWebRequest)WebRequest.Create("http://192.168.220.111/AboutUs.html");
request.Host = "www.patelyogesh.in";
Thus we can access all the pages using C# .NET 4.5 bypassing host file using above source code.

Hope you like this idea! Cheers!!

Sitecore bypass login without password

This blog explains how to bypass login for any Sitecore user or how to implement Single Sign-On on Sitecore to automate user login without knowing password.

Practical Use

Suppose, you have a non-Sitecore application, which has different users and all its users also access Sitecore by login into it. Now, switching between these two sites is really repetitive and time-consuming task, need to remember passwords for two websites, etc. If we provide SSO (Single Sign-On) here so that the user logs in only once and access both Sitecore and non-Sitecore websites with single login.

How to achieve

Create a web page in Sitecore website suppose sso.aspx, that will do login in Sitecore with passed userame via querystring. After login, we have to redirect user to welcome.aspx.

So, from the non-sitecore application, we will call url: http://mysitecore.com/sso.aspx?username=user_sitecore. On load of the page, we will login using user_sitecore and redirect user to welcome.aspx.

But make sure you encrypt username before passing in querystring. You can also pass a unique token number (A random generated number/string) along with username and encrypt it to make it more secured.

Login without password is possible using Sitecore.Security.Authentication.AuthenticationManager.Login(username) function. See below source code:

public partial class SSO : System.Web.UI.Page
{
    protected void Page_Load(object sender, EventArgs e)
    {
         string strUserName = Request.QueryString["username"];
         if (Sitecore.Security.Authentication.AuthenticationManager.Login(strUserName))
         {
               Response.Redirect("welcome.aspx", true);
         }
    }
}

Hurrey.... we did login without giving user password.

Sitecore Publish Instance

Are you not getting any direction to improve your Content Management server performance due to publishing? Are you facing slowness while doing publishing or having publishing queue stuck-up issues? Separate Publish Instance is one of the solution.

How to setup Sitecore Publish Instance?

  1. Sitecore PI is nothing but a clone instance of Sitecore CM environment.
  2. On CM server, open \Website\App_Config\Include\Scalability.config file. Update Publish Instance Name as below.
            <setting name="Publishing.PublishingInstance">
               <patch:attribute name="value">PI-Sitecore</patch:attribute>
            </setting>
    
    If the value is empty Sitecore will use the instance name by combining the machine name and the IIS Site name. So for the IIS Site Sitecore on the server PI the instance name would become PI-Sitecore.
    Just to remember, do not configure above setting on Publish Instance.
    Note: Sitecore allows to add only one publish instance name in this setting.
  3. Enable EventQueue on both CM and PublishInstance servers. To enable EventQueues, from web.config, find EnableEventQueues setting. Set its value to true. This setting can also be set from \App_Config\Include\ScalabilitySettings.config, which will be given more precedence over web.config settings.
          <setting name="EnableEventQueues">
            <patch:attribute name="value">true</patch:attribute>
          </setting>
    
  4. Disable any scheduled tasks or agents running if not required.

How Publishing works on Publish Instance?

Below image shows the flow of publishing from CM to PI server.
Sitecore Publish Instance Architecture


Suppose, Sitecore CM Application has Instance named Staging-Sitecore (In image: CM1), and Publish Instance is named PI-Sitecore (In image: CM-PI-1). Now, as per above settings, Sitecore CM1 knows that it's publishing should be done by Publish Instance: PI-Sitecore.

Now, when a user sets a publish job, Sitecore adds its entry in EventQueue table with a flag/column of PublishInstance. It simply adds PI-Name in that column's value. So, every publish job is assigned its Publish Instance Name. Now, as per EventQueue architecture every Sitecore Instance checks for EventQueue every few seconds to fetch which events/jobs it has to perform. Here, when CM1 server checks EventQueue, it will not find any publish job, where PI will find records with its name. So, it will fetch those publish details and start publishing those publish jobs.

Now, as per architecture, all other instances will start updating cache of those published and related items, to get updated.

Can we utilize Publish Instance for other tasks?

- PI is not only PI, which can still be used as CM. So, we can have two Content Management servers. So, we can make less used CM as PI.

- If we are using it only for Publishing, then we can reduce load of CM server by moving all backend tasks or jobs(except Sitecore default) to PI.

How many instances Sitecore supports?

Logically, unlimited. We can have multiple Sitecore Instances working as CM with/without load-balanced environment with one PI.

As per Sitecore architecture, only one PI is recommended. But after reviewing publishing code and my experiements done locally, I concluded that we can have two PIs too. But, that will work with minor architecture change and only one limitation that an item can not be published on both PIs simultaneously.

Must read posts for Sitecore Publish

- Sitecore Publishing Facts
- Intelligent Publish in Sitecore - The most optimized approach
- Sitecore Parallel Publishing using Multiple Publish Instances

Sitecore on Cloud - Windows Azure and Amazon EC2

We investigated a lot of time which cloud service is better for our Sitecore architecture. It is always good to know about reliability, efficiency, scalability and agility for the cloud service we are going to use.

Here are some points I concluded while working on both of Microsoft Azure and Amazon EC2. Azure is a IaaS, while EC2 is PaaS, so surely both are having some pros and cons. Below are few points I collected while working on it.


Points Windows Azure Amazon EC2
Computing Category Infrastructure as Service Platform as Service
Guaranteed Network Availability

99.999%

99.9%
SQL Database
Database Capacity

Allows maximum 150GB size of SQL Azure database. We can then use a cloud VM for using SQL.

No limitation on size of database, as need a separate VM.

Database Backup/Restore

Many limitations in backup/restore on SQL Azure a more mature environment for backup/restore than Azure
Load Balancers
Sticky Session Configuration Does not support Sticky sessions, so session based functionalities will not work Still, they are paid. - Free and allows sticky session configurations.
Multi-region load-balancing Can not support multi-region servers. Can not support multi-region servers, but we can use 3rd party services to handle multi-region servers
Deployment
Quick deployment Sitecore Azure works superbly if you have valid Sitecore and Sitecore Azure versions, which makes automated deployment so easy and quick. Very easy to create multiple sitecore instances using it. Deployment is not automated as they provides a remote console, still automated deployment is possible using WebDeploy.
Troubleshooting It is very difficult to troubleshoot deployment problems faced using Sitecore Azure. Manual deployment does not create any issues
Backup and Monitoring Has some basic monitoring system AWS
Support Paid Support Free Support

If you want to try and experiment cloud services, go to Windows Azure or Amazon EC2. Both of them provides free cloud services for limited time period.

Upload Sitecore media items in selected languages

One day, one idea came in my mind to create media items in specific languages only, which is very useful thought. By default, Sitecore media item gets created in many languages, where we need actually one or two out of them. Want to know why? Then read why sitecore media item gets created in multiple languages.

NOTE: This issues was recreated in Sitecore 7.2 and older versions. Sitecore 8 and later creates media items only in context language.

Specific languages for Unversioned Media Item

How it's benefitial?
In Unversioned media item, item gets created in all languages exist under /sitecore/system/languages/. Creating its version in selected languages is benefitial:
  1. Less insert queries to DB tables, less records fetched while fetching media item details.
  2. DataCache and ItemCache will be created in selected languages only instead of all languages.
  3. Reduced Database size, where millions of media are stored.
  4. So, overall performance improvement while uploading media item and while using/fetching media item.
How to achieve it?
  1. Create your custom class inherited from the Sitecore.Resources.Media.MediaCreator one, say SitecoreTactics.Resources.Media.MediaCreator. In this class you should override CreateItem method to retrieve there the set of the languages from the web.config setting. This set should be used as the languageArray in the foreach cycle. Thus the unversioned mediaitem with certain language versions will be created.

    See how it looks:
    namespace SitecoreTactics.Resources.Media
    {
     using System.Reflection;
     public class MediaCreator : Sitecore.Resources.Media.MediaCreator
     {
      protected override Item CreateItem(string itemPath, string filePath, MediaCreatorOptions options)
      {
       // Override the CreateItem function method and write custom code to create selected language versions
      }
      }
    }
    
    
  2. Create your custom class inherited from Sitecore.Resources.Media.MediaProvider one. In the constructor of this class you should assign an instance of our custom MediaCreator class (from Step#1) to the MediaProvider "creator" private field using the Reflection. So, the MediaProvider will use our own code of MediaCreator.

    How the code looks:
     namespace SitecoreTactics.Resources.Media
     {
      using System.Reflection;
      public class MediaProvider : Sitecore.Resources.Media.MediaProvider
      {
       public MediaProvider() : base()
       {
        typeof(Sitecore.Resources.Media.MediaProvider).GetField("creator", BindingFlags.NonPublic | BindingFlags.Instance).SetValue(this, new SitecoreTactics.Resources.Media.MediaCreator());
       }
      }
     }
    
    
  3. Change in config, register our custom mediaprovider class as below:
    <mediaLibrary>
        ...
        <mediaProvider type="SitecoreTactics.Resources.Media.CustomMediaProvider,SitecoreTactics.Resources" />
        ...
    </mediaLibrary>
    
    

Wow, it worked well!

Specific languages for Versioned Media Item

Versioned Media Item does not need an automated way to upload media in selected languages. The reason is, this kind of media does not share Media File. So, it is used only when we need to upload different files for different languages of same media item. If still someone need to achieve it, here is the way:
  1. Add a settings in the web.config file containing set of the pipe separated needed languages. For example:
    <setting name="SetOfLanguages" value="en|ru-RU|en-GB" />
    
  2. Add an extra checkbox "Versionable For Certain Languages" on the Media Folder Upload Form using this file: \Website\sitecore\shell\Applications\Media\MediaFolder\MediaFolder.xml
  3. Create a custom class inherited from Sitecore.Pipelines.Upload.UploadArgs and add here an extra "SpecialVersioned" property. This property will indicate to Sitecore.Pipelines.Upload.Save processor to execute the custom actions.
  4. Create a custom class based on Sitecore.Shell.Applications.FlashUpload.Advanced.UploadTarget one and define it in \Website\sitecore\shell\Applications\FlashUpload\Advanced\UploadTarget.aspx file instead of the default one.
  5. In this class you should assign a value to your custom UploadArgs.SpecialVersioned property similar to the UploadArgs.Versioned one.
  6. Create your custom class based on the Sitecore.Pipelines.Upload.Save one and register it in the web.config <uiUpload> pipeline instead of the default one.
  7. An example of its rewritten Process method is here: Expand Code
It's working great, isn't it?
And yes, many thanks to Yuriy Zakharov from Sitecore Support to provide such a nice solution!!