Teaching the Big Boys to Share

Jake Anthony – Sales Engineering Manager, LogPoint

Big Data and Big Data Analytics (BDA), have been on the tip of everyone’s tongue for what seems like forever. Initially something that was a nebulas idea, ringfenced for only the biggest and most forward thinking of companies (I am looking at you Tech Giants and Telecoms companies), last year saw an explosion of articles across the web telling businesses that if they weren’t using Big Data they were being left behind. Although I should point out that none of those articles seemed to point towards exactly what the rest of us mere mortals should be using Big Data for…

To their credit some of the larger consultancy firms have started providing seminars or presentations steering people in the vague direction, just sign away your life to get access (definitive answers come at a price as we all know). That is probably because, unlike a lot of technology that is forced down people’s throats nowadays, there is no way to easily tell people exactly what they should be doing to get benefits from BDA. The benefit becomes crucially clear when considered in the context of the individual business needs and landscape.

For example, in financial services, an industry where I spend a lot of my time, BDA is absolute key to their success.  Not only from a compliance and regulatory perspective but the fact that most us are now banking online, rather than popping down to our local high street branch, means the analysis of data to remain competitive is essential.  The question is, are they deploying it correctly.

Below are a few potential applications of BDA. These only provide a source of inspiration, unless your business fits neatly in the same sector/circumstances as those referenced;

  • Credit card company and business intelligence – Hindsight reporting and trailing indicators can only take a business so far, so this company started looking for indicators that could predict loyalty and developed sophisticated models to analyse historical transactions. Using over 115 variables against the data collected, accurate forecasts potential churn were created. The company believes it can now identify 24% of accounts that will close within the next four months.
  • Fast food and video – This company points cameras on its drive-through lanes to determine what to display on its digital menu board. When the lines are longer, the menu features products that can be served up quickly; when the lines are shorter, the menu features higher-margin items.
  • Supermarket and performance efficiency – This supermarket chain collected refrigerator-related data points coming off its units and fed them into a dedicated data warehouse. Those data points were analyzed to keep better tabs on performance, gauge when the machines might need to be serviced and do more proactive maintenance to cut down on energy costs.

So, no, this blog won’t be the magic bullet that explains how and why you should use BDA, it will instead outline the other side of BDA. Specifically, what the current options for implementing BDA within a business are, and to hopefully peak your interest (at least I am honest!) in how LogPoint’s approach can fill that crucial gap.

What are the options

Let’s start with a big claim… the current issue with utilizing BDA in the mid-market is the fact that the two realistic options provide unique challenges that are only solvable at the large enterprise.

The first potential option is to invest in the development of a bespoke architecture based on the Hadoop open source platform. This promises limitless flexibility and the ability to deliver the exact needs, nothing more and nothing less. This sounds amazing in theory, however, as everyone knows, too much of a good thing can be bad.

By providing limitless flexibility, these open-source architectures inherently introduce a level of complexity into the deployments. This can only be solved through the introduction of specialist Data Scientist resources/teams that can quickly spiral in terms of size and cost. These specialists are not only expensive, they are also incredibly in demand, the shortage of expertise in the IT sector is a well-known phenomenon that I am sure I don’t need to go into here…

Alternatively, if the idea of toying with your Big Data platform at a system level doesn’t appeal to your business, then there are a multitude of off the shelf products that could fit the bill. These tools are almost exclusively based on the Hadoop architecture, but claim to do all the heavy lifting so that you don’t have to. Brilliant! In theory. In practice, these off the shelf products generally fall into one of two categories which have their own inherent challenges.

Firstly, those products which are locally hosted and build to customer specifications. Able to offer significant value to business of all sizes and solve challenges of various kinds they come with one rather important caveat. I could, in theory, buy a Bugatti Chiron to solve my commuting problem, except I don’t have a cool £2,400,000 sat around to do so!

Secondly, the business could embrace Cloud. Although I am sure your business already has embraced Cloud, from the sounds coming out of the industry every man and his dog has a cloud environment that is eating up whole swathes of compute within the major cloud providers gigantic black boxes. However, if like a lot of companies, you are currently still in the process of dipping your toe in the proverbial shallow end of cloud adoption, then suddenly sending all manner of data points to the cloud without considering all the potential security implications is naïve at best, and downright negligent at worst. Now I think about it that’s probably the beginnings of an article unto itself….

Anyway, back on track, the question businesses must ask themselves is this, how do you attack a problem that you are being expected to solve without the inherent advantages that come from having a limitless bank balance?

In a world where the amount of data being produced is increasingly exponentially, companies that were once “too small” to get benefit from BDA, are now producing as much data as the large enterprise was when they first dipped a toe into the world of Big Data. Reducing all this data into actionable intelligence will make you more valuable to your customers and allow for continued business growth.

If you fancy learning how LogPoint’s NoSQL-based SIEM platform can help deliver analytics on a Big Data scale without the drawbacks outlined above why not get in touch?

Jake Anthony