A look Into Requirements for Real-Time Analytics
What is needed on the hardware side to upgrade big data analytics to meet real-time
performance requirements?
Real-time analytics can be defined as enabling instant or near-instant access and use of analytical data. It could be suggested that real-time analytics involves the data being used within one minute of it being entered into the system.
As multiple users and applications perform data access in a big data capacity across a scaled infrastructure, the amount of memory lookups increase exponentially – much more than in typical enterprise computing.
You are likely to employ a hypervisor, such as VMWare® vSphere in your network. If your jobs are distributed equally across the various job types (CPU bound, Disk I/O bound, or Network I/O bound), your cluster has balanced workload pattern. A virtualized Hadoop cluster can take advantage of VMware’s high availability and fault tolerant capabilities as a built-in protection for critical components. |
 |
Big data analytic tasks, such as sorting, require very little compute power but rely more on the I/O bound capacity of the cluster (for example if you have lot of cold data). For this type of workload, investing in more disks per box is recommended. The master nodes will factor in immensely, so IT should plan for reasonable investment for these systems, such as Dual Xeon E5 servers.
Also on the hardware side, real-time analytics needs to employ a new memory model to make it all happen. Due to its need for speed, it’s all about the RAM. So what is commonly being adopted is the integration of a software layer to access RAM and storage volume equally along the infrastructure. In this technology, the software provisioning of RAM resources is able to equally assign an SSD volume to act as RAM. Therefore, DDR modules on the board and storage are combined to define a virtual resource to take on the properties of RAM, giving the ability to run databases completely in memory. The In-Memory Database solutions business is expected to explode in the next three years.
As business needs drive enterprise IT departments with big data projects, it is often the IT department’s onus to inform executive management of the need to assess current resources for upcoming real-time analytics needs as well. It’s all around the corner.
