How will we meet the BIG data processing challenges of the future?
There is no doubt the solving the BIG data processing challenges is not going to be easy, I will be honest we have found ourselves going back to the drawing board several times already in the last 2 years. After some considerable degree of trial and error recently I believe we have come up with an architecture that will meet these challenges.
Ok, so we are building a BIG data platform in the cloud and what is so different about what we are doing then ? Well, while I cannot offer specifics details of our latest architecture I can offer some basic insights into how it works. We have come up with a design that is based around independent processes which communicate in a common way using a central task based mechanism. The infrastructure dynamically scales processing depending on what tasks are waiting to be executed.
In this way logical processing scales to meet demand and new types of processing logic can be introduced independently of existing ones. Because all process engines use a common, centralised method of communication each individual process does not need to know how or what the others are doing. This is fundamentally different to simply adding more hardware resources; instead the architecture increases resources for specific processing tasks as and when required. The application still scales horizontally in the traditional sense, if more processing power is needed then extra server nodes can be added dynamically.