Sitecore Multiple Publish Queue (Parallel Publishing)

Are you facing a long long queue stuck up with Sitecore publishing and clients are complaining a lot? Then surely you might have think about having multiple publish queue.

See my other post regarding Practical Implementation of Sitecore Parallel or simultaneous Publishing using multiple Publish Queue.

Benefits of Parallel Publishing:

1. More than one user can do publishing simultaneously, so no more "Publishing Queued".
2. Less load on CM environment, where actually users are editing or previewing pages as publishing load is now shared with Publish Instances.
3. Less chances of Publish Queue stuck-up as Publish Jobs threads are now divided between multiple instances.
4. If one publish instance is stuck-up, then we can move publishing traffic to another instance.

What is expected?

The expectation is to do parallel publishing on two or more Sitecore Instances. Say, our Content Authors' publish should go to PI-1 and clients' publish should go to PI-2. Or, Content Editor publish should go to PI-1 and Page Editor publish should go to PI-2.

This idea is quite possible and easier too. Let's see how can we use multiple publish queue. For that, you might need to understand How Sitecore Publish Instance works.

What is the Challenge?

As we see from the above link, Sitecore allows to add only one PublishInstance name, as per below setting so publishing will be done from that server itself.

        <setting name="Publishing.PublishingInstance">
           <patch:attribute name="value">CM-PI-1</patch:attribute>
        </setting>

So, the problem is well described in below image. See the red balloon points.
Ohhhh.... how to come out from this challenges, as Sitecore does not support this? Well, we have an alternate to overcome this issue.

Solution to Setup Multiple Publish Instances

As we seen, when Sitecore CM sets publish, it is recorded in EventQueue and then the related PI will pickup that publish and start publishing. Can't we do the same process by bypassing EventQueue? I mean we can prepare our own Event Queue.

So, that's really possible now. Also, don't worry about caching updates, means items updated/published on CM or PI-1 or PI-2 will get reflected on other servers immediately using Event Queue. So, our main concern should be publishing only.

There can be two different approaches to achieve this:

1. Creating own table same as EventQueue
2. By Invoking Web Service to start Publish

1. Creating own table same as EventQueue

• We can create own table, which can have fields like below:
  
  

  Fields in MyPublishQueue
  ID
  ItemId
  SubItems
  Languages
  PublishType
  PublishInstance

  
  So, when user sets publish from CM, we will not start publish. We will just make entries in above table, with appropriate Publish Instance Name.
• Now, both PI will have a our new Sitecore scheduler running. This scheduler should be triggered every few time interval as per our need, same as EventQueue checks. Write down some code to find any publishes for their own instance, and starts publish there itself. Refer Alex Shyba's blog to Create Sitecore Scheduler.

Below image shows how to achieve this approach. Here, as what we expect, Content Editor publish should go to PI-1 and Page Editor publish should go to PI-2, which is maintained by our Custom EventQueue table.

This approach is very useful when we use for Scheduled Publish.

2. By Invoking Web Service to Start Publish

We can create a web service on both PIs, with input parameters as below:

• ItemId
• SubItems
• Languages
• PublishType (Smart/Republish)

This web service will do publish according to the passed items. So, when user sets publish from CM for PI-1 or PI-2, web service of respective PI should be invoked. So, publishing will be started by the webservice itself.
Below image shows how to achieve this approach.

This approach is very useful when we are not using Scheduled Publish.

Finally, you might be knowing how to start publishing using Sitecore. If No? See the code below:

public void PublishItem(Sitecore.Data.Items.Item item, User user)
{
 using(UserSwitcher(user))
 {
  // The publishOptions determine the source and target database,
  // the publish mode and language, and the publish date
  Sitecore.Publishing.PublishOptions publishOptions =
  new Sitecore.Publishing.PublishOptions(item.Database,
              Database.GetDatabase("web"),
              Sitecore.Publishing.PublishMode.SingleItem,
              item.Language,
              System.DateTime.Now);  // Create a publisher with the publishoptions
  Sitecore.Publishing.Publisher publisher = new Sitecore.Publishing.Publisher(publishOptions);

  // Choose where to publish from
  publisher.Options.RootItem = item;

  // Publish children as well?
  publisher.Options.Deep = true;

  // Do the publish!
  publisher.Publish();
 }
}

Note: Using multiple Publish Instance can be done with 3 Sitecore instances CM, PI-1 & PI-2. We can also do parallel publish with total two instances too CM & PI-1.

Sitecore does not recommend to have Multiple Publishing Instances. But, using above approaches, we implemented Parallel Publishing in 2012 with Sitecore 6.4.1, published millions of items with parallel publish, and still counting.

No need to say but having multiple Sitecore instances requires license accordingly.

No words to say about great Sitecore Architecture!!

Must read posts for Sitecore Publish

- Sitecore Publishing Facts
- Intelligent Publish in Sitecore - The most optimized approach
- Sitecore Setup Separate Publish Instance