Sitecore Parallel Publishing Implementation

Many readers of my blogs asked me the practical way to implementing parallel publishing using multiple publish queue on Sitecore. Earlier I have posted a blog on it explaining theoretical approach for Parallel Publishing in Sitecore using custom EventQueue. It is really difficult to explain that approach on a blog. So here, I am going to explain easiest and quickest implementation of multiple publish queue practically.

Why Parallel Publishing from different instances?

In our multisite environment, hundreds of users work at a time, add/update thousands of items and publish them frequently. Daily 15,000+ items get published. Sometimes this number crosses 50,000 per day. So, mostly we see long publishing queue and users have to wait for minutes to get their items published. To make publishing more scalable, we thought to have multiple publishing instances in our CM environment and we dedicated them for different types of users based on business need.

What we wanted to achieve:

We wanted to have three different Publishing Instances for our users.
Publish Instance - 1(PI-1) Dedicated to Client Content Authors.
Publish Instance - 2(PI-2) Dedicated to Internal Content Authors.
Publish Instance - 3(CM-1) Dedicated to all other users. This is the Content Management Server, where users use Page Editor or Content Editor.

For this, we created three different roles named Publish Instance One, Publish Instance Two and Content Management Instance. So, depending on role assigned, we will allocate Publishing Instance to the user.

How we hooked Publishing Process

Below image shows how we hooked and modified startPublishing event to choose Publish Instance depending on user publishes.

Parallel Publishing Architecture with Multiple Sitecore Instances

Let's see practical implementation of Parallel Publish in Sitecore?

You can see How Parallel Publishing works on Sitecore in below video.
In this video, below is the allocation done for users:

User Role Allocated Publishing Instance
sitecore\admin Publish Instance One Machine-PI-1
sitecore\yogesh Publish Instance Two Machine-PI-2
All other users Content Management Instance Machine-CM

So, we have three different Publishing Queues available.



Now see how we can achieve this in simple steps below:
  1. Setup three different CM instances (in a cluster) with same master, web and core databases.
  2. Parallel publishing requires 1+ Sitecore Instances So indirectly we can say parallel publishing is possible on n number of CM instances.

  3. Enable EventQueue on each instance
  4. On all the Sitecore Instances, it is must to enable EventQueue from web.config or App_config\Include\ScalabilitySettings.config file.
         <setting name="EnableEventQueues" value="true" />
       
  5. Configure allocation of Roles & Publishing Instances
  6. On all the Sitecore Instances, create a config file: App_config\Include\ParallelPublish.config file. This file should remain same on each instance.
    <configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
      <sitecore>
        <parallelpublishRoles>
          <!-- Roles needed for Parallel Publising -->
          <role name="sitecore\Publish Instance One" publishinstance="Machine-Sitecore-PI-1" />
          <role name="sitecore\Publish Instance Two" publishinstance="Machine-Sitecore-PI-2" />
          <role name="sitecore\Content Management Instance" publishinstance="Machine-Sitecore-CM" />
        </parallelpublishRoles>
      </sitecore>
    </configuration>
       
  7. Override publish:startPublishing Event with your custom code as below
  8. namespace SitecoreTactics.Publishing
    {
        public class RemotePublishingEventHandler : Sitecore.Publishing.RemotePublishingEventHandler
        {
            public override void OnStartPublishing(object sender, EventArgs args)
            {
                RemoteEventArgs<Sitecore.Publishing.StartPublishingRemoteEvent> args2 = args as RemoteEventArgs<Sitecore.Publishing.StartPublishingRemoteEvent>;
                if (args2 == null)
                {
                    throw new InvalidOperationException("Unexpected event args: " + args.GetType().FullName);
                }
    
                Sitecore.Globalization.Language lang = Sitecore.Globalization.Language.Parse(args2.Event.ClientLanguage);
                StartPublishingRemoteEvent myArgs = new StartPublishingRemoteEvent(args2.Event.Options, args2.Event.StatusHandle, args2.Event.UserName, lang);
    
                // Findout Dedicated Publishing Instance for current user
                string allocatedPublishInstance;
                using (new SecurityDisabler())
                {
                    User user = User.FromName(args2.Event.UserName, true);
    
                    allocatedPublishInstance = GetPublishInstanceForUser(user);
                }
    
                // Allocate Publishing Instance to current publishing
                myArgs.SetPublishInstance(allocatedPublishInstance);
    
                // Allocated Publishing Instance will pick up the publishing
                if (this.IsPublishingServer(myArgs))
                {
                    Sitecore.Publishing.DistributedPublishingManager.StartPublishing(myArgs);
                }
            }
    
            protected string GetPublishInstanceForUser(User user)
            {
                // Find role and allocated publishing instance
                foreach (XmlNode node in Factory.GetConfigNodes("parallelpublishRoles/role"))
                {
                    string role = XmlUtil.GetAttribute("name", node);
                    string publishInstance = XmlUtil.GetAttribute("publishinstance", node);
                    if (user.IsInRole(role))
                        return publishInstance;
                }
    
                // Return default Publish Instance
                return Settings.Publishing.PublishingInstance;
            }
        }
    
    
        public class StartPublishingRemoteEvent : Sitecore.Publishing.StartPublishingRemoteEvent
        {
            public StartPublishingRemoteEvent(Sitecore.Publishing.DistributedPublishOptions[] options, Sitecore.Handle statusHandle, string userName, Sitecore.Globalization.Language clientLanguage)
                : base(options, statusHandle, userName, clientLanguage)
            {
            }
    
            // Methods
            [Obsolete("Use StartPublishingRemoteEvent(DistributedPublishOptions[] options, Handle statusHandle, string userName, Language clientLanguage) instead.")]
            public StartPublishingRemoteEvent(Sitecore.Publishing.DistributedPublishOptions[] options, Sitecore.Handle statusHandle, string userName) :
                base(options, statusHandle, userName, Sitecore.Data.Managers.LanguageManager.DefaultLanguage)
            {
            }
            
            public void SetPublishInstance(string piName)
            {
                base.PublishingServer = piName;
            }
        }
    }
    
  9. Update your custom publish:startPublishing event in web.config with below changes
  10. <event name="publish:startPublishing">
            <!-- handler type="Sitecore.Publishing.RemotePublishingEventHandler, Sitecore.Kernel" method="OnStartPublishing" / -->
    
          <handler type="SitecoreTactics.Publishing.RemotePublishingEventHandler, SitecoreTactics.Publishing" method="OnStartPublishing" />
            
          </event>
    
       
Now try in your environment, parallel publishing is surely working for you too. If it's not working, check
  • EventQueue is disabled, enable it
  • All CM instance should have exact time. If two instances have time difference of 5 minutes, then there the instance running late will get updates of other instance after 5 minutes, so sync will never be done between them.
  • Give proper Roles Name and Publishing Instance Name in the ParallelPublish.config file.
  • Check all your Sitecore instances are pointing to same databases - master, web, core.
  • Configure the publish:startPublishing event and ParallelPublish.config on each Sitecore Instance.
  • ParallelPublish.config file should have same contents on all the instances.
  • Instance which needs to do publish should be up and running.


Now enjoy parallel publishing without waiting time, without queue stuck!

Comments

  1. Hi Yogesh, where can I find "publish:startPublishing Event "? what file is it?

    ReplyDelete
    Replies
    1. Hi Borj, you can find this event in web.config. Refer above mentioned step 5. Your profile does not show your email address.

      You can drop an email to me at contact@patelyogesh.in, I will surely try to help you!!

      Delete
  2. Wanted to convey to my blog readers, this parallel publishing is totally different than what Sitecore has provided in Sitecore 7.2 which released later this March. Sitecore does publishing parallel using multi-threading, while the approach mentioned here does parallel publishing on different Publish Instances.

    ReplyDelete

Post a Comment