excel distribution network

Distributed Excel Processing in Azure

In Azure, Big Data by Stuart, Product & DataLeave a Comment


Note: This article has been updated with more information. It was originally published August 12, 2015


 

A business problem

In various sectors there are businesses that have pushed Microsoft Excel to its limit and who have critical business processes running in an Excel compute ‘grid’ in conjunction with using products like SharePoint Excel Services or “home grown” PC based grids.

Excel is an amazing tool but it’s fundamentally a desktop application and not a background processing engine and using it as such is not without its issues.

With this in mind, we have defined a scalable Microsoft Azure based architecture to process Excel workbooks without the need for Excel or SharePoint. Our Azure Calculation Engine (ACE) enables organisations to focus on running their operations, and leave the Excel processing in Azure to computing.

How we achieved Excel processing in Azure

1. The first task, was to find a way to calculate Excel workbooks from code and without using Excel itself.

2. Secondly we had to define the Azure architecture using all Azure PaaS and SaaS services that would allow for consistent scaling when load is heavy.

We decided to use the Azure service bus queue and Azure worker roles with the Queue-based load levelling pattern:



excel processing image

We deployed a 3rd party component to handle the Excel processing, which across various performance and compatibility tests thus far has been very strong.

3. Our application adds a number of tasks to a queue, the auto scaling workers pull those tasks (task = calculating an Excel workbook) from the queue for processing, then they process and write the output to the relevant storage location.

Excel processing in reality

From a technical perspective, taking something that is theoretical to vast data quantities may seem like a flag on the moon moment.

In reality, we are delighted to say that all parts of the equation have come together to form a fantastic tool for those pushing Excel to its best. See how Telereal Trillium made the jump and the impacts it has had.



excel processor


Written by Stuart Anderson | Principal Consultant, RedPixie | See his LinkedIn Profile | Follow him on Twitter


Leave a Comment