CEO

NetNow General Availability (GA) date.

NetNow General Availability (GA) date

In order to take advantage of the very latest web technology and agile development methods to keep netnow.co ahead of the field we have decided to further delay our GA date. We are rebuilding much of the core server architecture at this time to deliver dramatic performance improvements inside the platform. We want NetNow to be a truly new generation of software that delivers outstanding results with ease, NetNow will manage all your data effortlessly. We expect to be releasing new iteration of the NetNow platform on a quarterly basis and at this time we are planning for the first GA release in Q4 2015.

About NetNow

NetNow is a Software-as-a-Service (SaaS) or on-demand cloud-based service, meaning there is no software installation or maintenance required and new features are automatically added without the need for expensive and lengthy upgrades. Data is accessed live, meaning there is no need to upload data into the cloud.

Connect to any data source

NetNow creates a single version of the truth through connection to multiple and diverse (on-premise and online) data sources from a single, easy-to-use, cloud-based interface.

CEO

Cloud Data

Managing your cloud data

As we speak, the entire NetNow development team is laser focussed on the final push towards our GA release. Now we’re going to give you a little sneak peek at what we’ve been working on in recent months. Here are some of the new features we have been building into the platform.

configData Manager

The data manager provides an easy an intuitive way to establish a connection to your data sources. A step-by step wizard guides you through connecting to different types of data including format such as csv, data tables, URL sources and many more. Logic can be applied to cleanse, transform, merge or enrich the incoming data.

 monitorFilter Builder

The filter builder is a modern free text search based tool that enables users to easily search their data and find what they need. Searching works just like Google with support for advanced queries containing conditional logic. Filters can be saved, combined and re-used to build complex search criteria.

 emMessaging

Your data is managed on a messaging architecture so you can subscribe to data topics and have updates and notifications pushed to your dashboards in real time.

  

slmDisplay Wizards

Organising and displaying your data is made easy using our Display Wizards. These wizards walk you through the process of configuring our extensive library of widgets to display your data.

bsmOur Developers

Our developers use work in using agile development techniques and work with a variety of modern development languages and technologies including Apache Camel, MongoDB, ActiveMQ, Grails, Java, and OSGi.

About NetNow

NetNow is a Software-as-a-Service (SaaS) or on-demand cloud-based service, meaning there is no software installation or maintenance required and new features are automatically added without the need for expensive and lengthy upgrades. Data is accessed live, meaning there is no need to upload data into the cloud.

Connect to any data source

NetNow creates a single version of the truth through connection to multiple and diverse (on-premise and online) data sources from a single, easy-to-use, cloud-based interface.

Xen Virtual Machine Conversion HVM/PV

VMware to Xen Virtualised Image Conversion

We recently converted a RHEL 6.0 virtual image from VMware to Xen (Xen Cloud Platform 1.6) by using the XenServer Conversion Manager. After installing the XenTools on to the image XenCenter still showed them as “not installed”. We registered the VM with the RHN (via rhn_register) and for RH 6.1 updates and upgraded the kernel from vmlinuz-2.6.32-71.40.1.el6.x86_64 to 2.6.32-131.38.1.el6.x86_64 via the following command:

yum update kernel.x86_64

After this and a reboot the tools the Virtualization state appeared as “Optimized (version 6.1 installed)” and we could set the memory ballooning minimum and maximum in the XenCenter “memory” tab.

Hardware Virtualised Machine By Default

We later found out that it was running as a Hardware Virtualised Machine (HVM) instead of Paravirtualised (PV), which is less efficient. Apparently the actual performance difference varies depending on what hardware it’s running on but we wanted to aim for PV. From what I’ve read on various sites and posts I believe that HVM was used because we had converted the image instead of creating a new one from a Xen template.

To validate this from the XenServer console we got the uuid of our image via a “xe vm-list” and checked the HVM-boot-policy to see if it was set to “BIOS order”, e.g:

xe vm-param-get uuid=<VM uuid> param-name=HVM-boot-policy

If it is set to “BIOS order” (or other non-empty string?) then HVM is enabled.

Converting a Xen Virtual Machine from HVM to PV

We couldn’t find any formal documentation about converting VMs to PV but we came across this posting which pointed us in the right direction.

In order to convert the image to use PV:

1. Go in to your Linux VM and save the output from the “mount” command and /boot/grub/menu.1st contents.

2. Take a snapshot of your image in XenCenter to roll back to just in case!

3. From the XenServer console get the VM ‘uuid’ using the command:

xe vm-list

NB: You can list all VM params with: xe vm-param-list uuid=<VM uuid>

4. Set PV mode by running:

xe vm-param-set uuid=<VM uuid> HVM-boot-policy=""

5. Set the PV boot loader:

xe vm-param-set uuid=<VM uuid> PV-bootloader=pygrub

6. Set the kernel and ramdisk params from the grub menu.1st contents saved earlier:

xe vm-param-set uuid=<VM uuid> PV-bootloader-args="--kernel <kernel location> --ramdisk <initrd location>"

e.g.

xe vm-param-set uuid=c66eb11e-75a3-91a5-2f90-9aa3c03ec61b PV-bootloader-args="--kernel /vmlinuz-2.6.32-131.38.1.el6.x86_64 --ramdisk /initramfs-2.6.32-131.38.1.el6.x86_64.img"

7. List the disks in your VM:

xe vm-disk-list uuid=<VM uuid>

8. Get the VDI disk uuid (e.g. for Disk 0 VDI) and then:

xe vdi-param-get uuid=<VDI uuid> param-name=vbd-uuids

9. Get the vbd-uuids value and then set bootable=true:

xe vbd-param-set uuid=<vbd-uuids> bootable=true

10. Set the root device to point to that from the output of the ‘mount’ command from earlier:

xe vm-param-set uuid=<VM uuid> PV-args="root=<root device>"

e.g.

xe vm-param-set uuid=<VM uuid> PV-args="root=/dev/mapper/vg_vmrhel6-lv_root"

11. Reboot/Start the VM.

12. Unfortunately we could not get the framebuffer (/dev/fb0) device to work and so X wouldn’t start, but this wasn’t really an issue in our environment since we were only concerned with console access.

A few days later we came across a handy script in the following link which performs all of the above commands, which is probably much less error prone. Disclaimer – We have not tested this.

Reverting Back to HVM

To revert back to HVM either drop back to your previous Xen snapshot or run the following from the Xen server console:

xe vm-param-set uuid=<vm uuid> HVM-boot-policy="BIOS order"

By the way, Xen snapshots are great! The ability to branch and revert without losing changes is really handy.

– Steve –

CEO

NetNow General Availability (GA) date delayed.

NetNow General Availability (GA) date delayed

We are very sorry to announce that our GA date has been delayed. Due to performance problems with our database, development plans have been pushed back by 3 months. We have now changed our back end database and resolved the issue. We expect to be releasing new iteration of the NetNow platform on a quarterly basis and at this time we are planning for the first GA release in Q2 2013.

About NetNow

NetNow is a Software-as-a-Service (SaaS) or on-demand cloud-based service, meaning there is no software installation or maintenance required and new features are automatically added without the need for expensive and lengthy upgrades. Data is accessed live, meaning there is no need to upload data into the cloud.

Connect to any data source

NetNow creates a single version of the truth through connection to multiple and diverse (on-premise and online) data sources from a single, easy-to-use, cloud-based interface.

Steve White

SVN Keywords and PropSet PowerShell Script

How we are using SVN Keywords and PropSet PowerShell Script

Some of our source code remains in an SVN repository and uses keyword substitution. By using SVN Keywords and PropSet PowerShell Script together we can provide up to date version and revision information. Keywords are generally defined in comments or variables in the format “$TagName: TagValue $:” in any source code. See the following link for more information.

A C source file may use keywords to insert version header information which can be pulled out by a version parameter (-v) passed to it, e.g:

 

const char *SVNVERSION={“@(#) $Revision: 1234 $:”};
const char *CODEAUTHOR={“@(#) $Author: steve.white $:”};
const char *DATEMOD={“@(#) $Date: 2013-01-01 12:00:00 +0100 (Tue, 01 Jan 2013) $:”};

 

Quite often, running a UNIX “strings” on the C generated binary will return the above information in its output which is useful for identifying binaries from an unknown source. So far I’ve noticed this works well with binaries produced by gcc on Linux, Solaris, Windows and AIX.

Eventually our code will all be migrated to a distributed version control system (DVCS) such as Git or Mercurial. However, keywords are not recommended so we will have to have a re-think as to how this is done.

Source Code with Missing Keywords

We recently discovered that TortoiseSVN on some of our developer machines had SVN keywords disabled. This meant that newly added source files hadn’t had them added and as a result keywords were not being updated upon a commit. Instead of running an SVN command manually against each source file I wrote a quick PowerShell script to do the legwork for us:

 

# Save as: svnPropSetSrc.ps1
# Change to source root and run: PowerShell .\path\to\svnPropSetSrc.ps1
#
# Recursively set SVN properties for common source files

$srcFiles=gci . -recurse -name -include @(“*.cpp”,”*.c”,”*.h”,”*.js”,”*.java”,”*.cs”)
Foreach ($Item in $srcFiles)
{
Write-Host $Item
svn propset “svn:eol-style” native $Item
svn propset “svn:keywords” “Author Date Id HeadURL Revision” $Item
}

 

This uses “gci” which is an alias for the Get-ChildItem PowerShell Cmdlet and outputs the file it’s modifying to standard output. It’s probably worth running this and piping output to a file so you can track what’s changed and see if there have been any errors reported.
 

CEO

BIG data processing challenges

How will we meet the BIG data processing challenges of the future?

There is no doubt the solving the BIG data processing challenges is not going to be easy, I will be honest we have found ourselves going back to the drawing board several times already in the last 2 years. After some considerable degree of trial and error recently I believe we have come up with an architecture that will meet these challenges.

Ok, so we are building a BIG data platform in the cloud and what is so different about what we are doing then ? Well, while I cannot offer specifics details of our latest architecture I can offer some basic insights into how it works. We have come up with a design that is based around independent processes which communicate in a common way using a central task based mechanism. The infrastructure dynamically scales processing depending on what tasks are waiting to be executed.

In this way logical processing scales to meet demand and new types of processing logic can be introduced independently of existing ones. Because all process engines use a common, centralised method of communication each individual process does not need to know how or what the others are doing. This is fundamentally different to simply adding more hardware resources; instead the architecture increases resources for specific processing tasks as and when required. The application still scales horizontally in the traditional sense, if more processing power is needed then extra server nodes can be added dynamically.

CEO

Intelligent cloud management comes built in to NetNow

Intelligent cloud management

NetNow will have intelligent cloud management services built in so our customers don’t need to worry about managing their NetNow cloud service.

Multiple hosting providers

We host in multiple providers to spread the risk of any outage in a single hosting provider impacting the service we provide to our customers. Our application has been specifically designed and built from the ground up to run across multiple data-centres and multiple providers ensuring maximum resilience.

Auto Scaling

NetNow has sophisticated in-built monitoring inside the application. Monitoring agents watch the internal message queues constantly looking for bottlenecks in processing and automatically scale up and down appropriate worker nodes on demand. This means if you get a flood of messages or a surge of user connections the application detects this and starts new worker nodes to deal with the increased load.

Auto Recovery

NetNow has a distributed architecture and each node in the architecture is independant of the others. If any single nodes fails others are waiting to pick up its workload and replacement nodes are added and removed dynamically. By spreading workers nodes out across multiple data centres and hosting providers our application is fault tolerant and your data is replicated in real-time.

Monitoring & Performance data

NetNow will automatically produce monitoring, management and performance data which can be sent directly to your own systems. This enables our customers to externally monitor the service we provide to them and this is achieved automatically and without additional costs. Monitoring data can be sent out via SNMP, RSS or WEB Services.

CEO

Big Data, BI & monitoring meets, cue Real Time Intelligence

While our developers are very busy right now preparing for our General Availability (GA) release, I wanted to share some insights regarding our vision for the NetNow cloud platform.

Back in the development labs work is at fever pitch marrying a massively scalable cloud data platform with an easy to use and feature rich cloud application that will realise our vision of Real Time Intelligence. Dealing with data volume and velocity is one thing but we need to monitor that data in real time and provide easy ways for our App to unlock intelligence from all the noise.

NetNow has a vision of creating a real time big data platform in the cloud capable of sophisticated event management, data enrichment, automation, notification, impact management, service modelling, change detection, change verification & predictive analysis. NetNow wants to provide a cloud platform not only for IT technicians but for non-technical users who don’t want to code solutions to deliver this type of functionality.

McKinsey & Company says “There will be a shortage of talent necessary for organizations to take advantage of big data.”

NetNow wants to solve this problem by providing an easy to use, modern and social cloud platform with powerful tools to make leveraging intelligence from big data a reality. NetNow intends to bring a sophisticated suite of data visualisation, management and monitoring features together in the cloud creating a new Real Time Intelligence layer based on top of our users existing data.

McKinsey & Company says  “Access to data is critical—companies will increasingly need to integrate information from multiple data sources, often from third parties, and the incentives have to be in place to enable this.

NetNow intends to have the most extensive library of data integrations possible, our integration team is already working on over 20 different pre-built App integration modules for the NetNow platform. These new Apps includes integrations for legacy tools from HP, Microsoft, CA, IBM, BMC & Oracle as well as cloud based solutions like Amazon, Rackspace, ServiceNow & others. We will also provide utility integrations for generic functions like TCP/IP, SMNP, Databases & Web Services. We will continue to build out our capabilities in this area moving forward and will also actively encourage the developer community to join us and help build integration plugins to other 3rd party data sources and applications.

Mark Beyer from Gartner recently said

“IT organizations should introduce a metadata management program, Beyer suggested. They should also plan on extending their existing data repository environments to the cloud and build at least one pilot analysis that combines text, social, and operational data.”

 

Reference Sources

McKinsey & Company

Big Data: the next frontier for innovation, competition, and productivity

Gartner

Beyond Big Data : Gartner on “Extreme” Information Management

Steve White

Samba 3.6.7 undefined symbol libintl_gettext, compile problem on solaris 10,

About the Samba Solaris 10 Problem – undefined symbol libintl_gettext

Recently I had to install the latest version of Samba on one of our Solaris 10 development boxes. Initially I tried to use the packages linked on http://www.sunfreeware.com but it appears that these had been compiled for a higher patch level of Solaris 10. Samba installed without issue and so did the dependencies listed on there, but none of the binaries would launch (nmbd, smbd, smbclient, etc…). Since our it’s critical this particular box stays up I didn’t want to patch the OS and have any downtime and so I downloaded the source from http://www.samba.org.

Compiling From Source on Solaris 10

After unpacking the tarball I changed to the ‘source3′ directory and tried to compile with the following:

./configure --without-ldap --without-ads --without-sys-quotas --without-manpages --with-libsmbclient --without-winbind --without-python --enable-shared --disable-cups
make

I disabled certain built in features during the configure to reduce the number of dependencies required and as a result the extra libraries which I may have had to install. However, no matter what I changed I kept coming across an undefined reference for libintl:

Undefined first referenced
 symbol in file
 libintl_gettext utils/net.o
 libintl_textdomain utils/net.o
 libintl_bindtextdomain utils/net.o

I’ve compiled this many times before without fault on various Linux distributions so I was surprised to have come across this problem. Now I know libintl.so was installed in /lib/ and I knew it worked fine as other applications which depended on it were fully functioning.

After searching through various google results I decided to edit the Samba Makefile. The first change I made was an edit to ./source3/Makefile in which I appended -lintl to the first instance of the LIBS= variable. To my surprise after running the make again everything compiled fine and a make install put it all in the right place. I wish every linking issue was that easy to fix.

Regards,

Steve