BIG data processing challenges

How will we meet the BIG data processing challenges of the future?

There is no doubt the solving the BIG data processing challenges is not going to be easy, I will be honest we have found ourselves going back to the drawing board several times already in the last 2 years. After some considerable degree of trial and error recently I believe we have come up with an architecture that will meet these challenges.

Ok, so we are building a BIG data platform in the cloud and what is so different about what we are doing then ? Well, while I cannot offer specifics details of our latest architecture I can offer some basic insights into how it works. We have come up with a design that is based around independent processes which communicate in a common way using a central task based mechanism. The infrastructure dynamically scales processing depending on what tasks are waiting to be executed.

In this way logical processing scales to meet demand and new types of processing logic can be introduced independently of existing ones. Because all process engines use a common, centralised method of communication each individual process does not need to know how or what the others are doing. This is fundamentally different to simply adding more hardware resources; instead the architecture increases resources for specific processing tasks as and when required. The application still scales horizontally in the traditional sense, if more processing power is needed then extra server nodes can be added dynamically.


Intelligent cloud management comes built in to NetNow

Intelligent cloud management

NetNow will have intelligent cloud management services built in so our customers don’t need to worry about managing their NetNow cloud service.

Multiple hosting providers

We host in multiple providers to spread the risk of any outage in a single hosting provider impacting the service we provide to our customers. Our application has been specifically designed and built from the ground up to run across multiple data-centres and multiple providers ensuring maximum resilience.

Auto Scaling

NetNow has sophisticated in-built monitoring inside the application. Monitoring agents watch the internal message queues constantly looking for bottlenecks in processing and automatically scale up and down appropriate worker nodes on demand. This means if you get a flood of messages or a surge of user connections the application detects this and starts new worker nodes to deal with the increased load.

Auto Recovery

NetNow has a distributed architecture and each node in the architecture is independant of the others. If any single nodes fails others are waiting to pick up its workload and replacement nodes are added and removed dynamically. By spreading workers nodes out across multiple data centres and hosting providers our application is fault tolerant and your data is replicated in real-time.

Monitoring & Performance data

NetNow will automatically produce monitoring, management and performance data which can be sent directly to your own systems. This enables our customers to externally monitor the service we provide to them and this is achieved automatically and without additional costs. Monitoring data can be sent out via SNMP, RSS or WEB Services.


Big Data, BI & monitoring meets, cue Real Time Intelligence

While our developers are very busy right now preparing for our General Availability (GA) release, I wanted to share some insights regarding our vision for the NetNow cloud platform.

Back in the development labs work is at fever pitch marrying a massively scalable cloud data platform with an easy to use and feature rich cloud application that will realise our vision of Real Time Intelligence. Dealing with data volume and velocity is one thing but we need to monitor that data in real time and provide easy ways for our App to unlock intelligence from all the noise.

NetNow has a vision of creating a real time big data platform in the cloud capable of sophisticated event management, data enrichment, automation, notification, impact management, service modelling, change detection, change verification & predictive analysis. NetNow wants to provide a cloud platform not only for IT technicians but for non-technical users who don’t want to code solutions to deliver this type of functionality.

McKinsey & Company says “There will be a shortage of talent necessary for organizations to take advantage of big data.”

NetNow wants to solve this problem by providing an easy to use, modern and social cloud platform with powerful tools to make leveraging intelligence from big data a reality. NetNow intends to bring a sophisticated suite of data visualisation, management and monitoring features together in the cloud creating a new Real Time Intelligence layer based on top of our users existing data.

McKinsey & Company says  “Access to data is critical—companies will increasingly need to integrate information from multiple data sources, often from third parties, and the incentives have to be in place to enable this.

NetNow intends to have the most extensive library of data integrations possible, our integration team is already working on over 20 different pre-built App integration modules for the NetNow platform. These new Apps includes integrations for legacy tools from HP, Microsoft, CA, IBM, BMC & Oracle as well as cloud based solutions like Amazon, Rackspace, ServiceNow & others. We will also provide utility integrations for generic functions like TCP/IP, SMNP, Databases & Web Services. We will continue to build out our capabilities in this area moving forward and will also actively encourage the developer community to join us and help build integration plugins to other 3rd party data sources and applications.

Mark Beyer from Gartner recently said

“IT organizations should introduce a metadata management program, Beyer suggested. They should also plan on extending their existing data repository environments to the cloud and build at least one pilot analysis that combines text, social, and operational data.”


Reference Sources

McKinsey & Company

Big Data: the next frontier for innovation, competition, and productivity


Beyond Big Data : Gartner on “Extreme” Information Management


NetNow General Availability (GA) of platform planned for Q2 2013

General Availability (GA) date announced for the NetNow platform.

After two years of prototyping and a further eighteen months of intensive development on the first release of the NetNow platform our development teams deserve a big round of applause! We expect to be releasing new iterations of the NetNow platform on a quarterly basis and at this time we are planning for the first GA release in Q2 2013.

Beta testing has been teaching us a lot and we have made massive strides in improving the platform thanks to our beta participants. We have made huge gains in the area of performance from the early Alpha release with throughput of the platform now running at hundreds of messages a second on a small single sandbox server.

We have introduced a JMS based message broker into the architecture to improve scalability and modularity of the platform, especially useful features in here for our plugin developers of the future!

We have been playing with different caching architectures and database types to support the different requirements of the application. We expect to run the platform on multiple databases including NoSQL and RDBMS flavours depending on the needs of the App. In our testing we certainly confirmed the need for a stateless modular application to enable horizontal scalability and NoSQL seems a must for supporting enormous data volumes. It is also true to say there is also a requirement to have traditional RDBMS functionality in places. We overcome this by splitting the application into a modular architecture and connecting everything with the message broker.

Security, resilience, scalability and usability are all as important as each other so it’s no good having a super-fast platform that is horrible and complex to configure. Or how about a very secure and friendly platform that won’t scale up. Our development teams have been working to make sure we deliver a new kind of platform that delivers in all these areas, one which we are certainly proud of.

I hope our growing community of users become proud of NetNow too and enjoy using what we are sure will be a great solution development platform for the cloud.


What is Real Time Intelligence ?

Real-time Intelligence (RTI) is the process of delivering information about your data as it happens. The speed of today’s processing systems has moved classical data warehousing into the realm of real-time. In this context, real-time means a range from milliseconds to a few seconds after the event has occurred. While traditional business intelligence presents historical data for manual analysis, real-time intelligence compares current data with historical patterns to detect problems or opportunities automatically.

  • Business transactions can be fed to the NetNow service as they occur to maintain the current state of the enterprise.
  • Real-time intelligence not only supports the ability to derive information and knowledge from past enterprise activity, but it also provides real-time tactical support.
  • Drive enterprise actions and react immediately to events as they occur.
  • Automated analysis capability enables corrective actions to be initiated and/or business rules to be adjusted to optimize business processes.

NetNow creates a “single source of truth” from all your data sources and gives you the power to easily create new ways to see your data, just the way you need to. Knowledge is Power and NetNow brings intel to your fingertips.
Download the info-graphic


Real Time Intelligence – General Availability of the Service

Real Time Intelligence Update

With around 20 sites currently beta testing the product we have been learning lots and thank all our beta users for their invaluable feedback. We are currently working on making sure we have the right hosting infrastructure in place ready for moving the NetNow service to general availability, currently we are planning for an October 2012 announcement.

So what can you expect one we go GA? Well the NetNow service will be accepting users directly, all you need to do is make sure you have Signed up to the site and as soon as you login to the NetNow service your Free Trial will automatically start. You can undertake your trial without obligation and there won’t be any sales people pestering you! We are here to help and waiting to answer any questions you may have during your trial. Our consultants will be available to give you any assistance you may require should you ask for it.

We have many exciting announcements to make in the coming weeks and months including more details on how to make the most of your trial along with many fantastic offers for our users! Stay tuned in to NetNow.

NetNow Corporate BLOG

Want to move to cloud based services?

How to integrate your existing applications and data and move to cloud based services

NetNow Apps
There are many benefits to cloud computing and businesses want to synchronize data exchange between legacy investments and new applications and processes. Solutions are needed that automate the integration of these data and systems to the cloud. Wouldn’t it be ideal to easily layer a cloud solution on-top of your existing systems and data? Do you want to be able to deliver new systems, processes and interfaces in your projects rapidly using modern cloud based solutions. Do you want to be able to develop and test new systems or solutions rapidly without the need to procure and deploy new hardware and software?

With NetNow you can

  • Use one tool to connect to and from all existing applications and data to instantly integrate without the need for programming.
  • Share data and processes by using integrations that support different environments and platforms
  • Allow legacy systems to easily connect and consume NetNow Web services. This effectively integrates legacy applications with NetNow including those on and off site.
  • Future proof developing business processes by reducing the influence of underlying technology.
  • Retain control over proprietary business systems by keeping them in-house and integrating them with NetNow.


What are the benefits of Cloud based software solutions?

  • Reduced Cost – Cloud technology is paid incrementally, saving organizations money.
  • Increased Storage – Organizations can store more data than on private computer systems.
  • Highly Automated – No longer do IT personnel need to worry about keeping software up to date.
  • Flexibility – Cloud computing offers much more flexibility than past computing methods.
  • More Mobility – Employees can access information wherever they are, rather than having to remain at their desks.
  • Allows IT to Shift Focus – No longer having to worry about constant server updates and other computing issues, organizations will be free to concentrate on innovation.
  • Monitor projects more effectively – Stay within budget and ahead of completion cycle times.
  • Less personnel training is needed – It takes fewer people to do more work on a cloud, with a minimal learning curve on hardware and software issues.



NetNow Apps – data integration software

How to integrate your data with NetNow?

NetNow Apps
We are just about to release the first collection of our new NetNow Apps for data integration. Our promise here at NetNow is to make managing your data a pleasure not a chore.

A key part of this process is making it easy for you to hook up to your data to the NetNow service. Our data integration software is designed to make this process as simply as possible.

To integrate your data you just need to download or activate the NetNow App for the platform, application or data source you want to integrate, configure it’s connection settings and away you go!

About data integration

Depending on the source of the data the integration App may be able to connect to it directly using a web service API for example. Other data sources or applications may require a component to be downloaded as placed near the data source. In these cases we use a highly secure connection back to the NetNow service and all data is encrypted at the source end as it would be better to first encrypt the content before storing into the Cloud (and automatically decrypt it upon retrieval).

In addition, NetNow data integration Apps will provide useful secondary functions such as compression and support for splitting/merging files that may be larger than a given threshold. Strong security algorithms and providers are be used, and this tool supports a variety of encryption key types.


We are about to release three new NetNow data integration Apps and 50+ to follow. v1.0 – Initial release.

  • nnApp for files
  • nnApp for mySQL
  • nnApp for Amazon CloudWatch


The cryptographic operations performed by these tools are enabled by OpenSSL. By default, OpenSSL is used as it enables the greatest level of portability. Configuration via the NetNow service is provided using a simple, easy to use interface.


Risk free trial offers

Because we are now able to offer a Free trial and low subscription based pricing, many more businesses or even individuals can now use the NetNow service. You can now evaluate our product without making investments in hardware and software. We can provide a risk free / investment free approach to the evaluation process for those interested in the NetNow service. For those prospects who like the product we can then offer either a cost effective subscription service with NetNow or alternatively they may engage with Interlink Software for on premise solutions.


Develop applications in the cloud

New enterprise projects can often get off to a slow start while project managers have to wait for testing and development resources to be put in place. SaaS offers a great way for the enterprises to bypass this headache. Many managers have projects they run that need solutions to be developed / piloted quickly and don’t have time to wait for harware to be pocured or systems to be built. SaaS is the perfect place to do this type of work and develop or test out your ideas and prototypes without the cost of designing/purchasing/deploying internal systems to facilitate the work. Assuming your company does not want to run the production application in a cloud environment the solutions built in the SaaS environment can be later moved in house to secure production ready systems.  This ability to cheaply leverage a SaaS environment when deploying new solutions in development and testing environments can present huge project cost saving opportunities to enterprise businesses.