DICE Monitoring Platform

Big Data is certainly a big hype nowadays and there are a tremendous number of frameworks available that enable companies to develop Big Data applications. The development of data-intensive applications, like development of any other software application, involves testing, validation and fine-tuning processes to ensure the performance and reliability the end-users expect. Throughout these processes the execution of the application needs to be constantly monitored in order to extract execution trends and spot the anomalies. And this is only the beginning. Once in production, monitoring of the application, together with its underlying infrastructure, is a must. But Big Data applications generate Big Monitoring Data, and not only this: the data is generated in different formats, is available either in log files, or via APIs.

There are monitoring systems on the market to help you with that. Plenty of them, some open source, some commerical, others using a freemium model. But they rather tend to be focused on specific areas. Nagios, Ganglia could be used to easily monitor your infrastructure. Others, such as Apache Chukwa or Sematext, could be used to monitor Apache Hadoop, Apache Storm, or Apache Spark. However, all these tools need to be deployed in your infrastructure, and you will certainly need to scale them to cope with scaling of your infrastructure. Or else, let external services transfer data out of your infrastructure, in case of SaaS platforms. Hmmm… Big Data seems to mean Big Problems.

DICE monitoring platform (shortly, DMon) tries to make your life easier when it comes to collecting, searching, analyzing and visualizing, in real time, your data-intensive application. Firstly, due to the leveraging on the Elastic‘s open source stack – Elasticsearch for indexing and searching the data, Logstash for pre-processing incoming data and Kibana for real time visualization – DMon platform is fully distributed, highly available and horizontally scalable. All these core components of the platform have been wrapped in microservices accessible using HTTP RESTful APIs for an easy control.

Secondly, DMon is able to monitor your infrastructure, thanks to collectd plugins. Additionally, it collects data from multiple Big Data frameworks, such as Apache HDFS, YARN, Spark, or Storm (for now). With DMon you have one platform for monitoring both your infrastructure and your Big Data frameworks.

Next, it streamlines the control and configuration of its core components. With DMon controller service, you have a unique HTTP RESTful API you can use both to control the core components of the platform (change configuration parameters, start/stop) and administer the monitored cluster, making it possible to add, update, remove monitored nodes or start/stop services on them via GET/POST/PUT/DELETE calls. We will also provide a Web user interface wrapping DMon controller API to have all administration jobs at your fingertips by clicking of a button.

Visualization of collected data is fully customizable and can be structured in multiple dashboards based on your needs, or tailored to specific roles in your organization, such as administrator, quality assurance engineer or software architect.

The deployment of the platform is integrated with Chef configuration maganagement system and we also provide a Vagrant script for a single node installation, which you will find useful for your development environment. If you are using the full DICE toolchain, it is going to be even simpler for you because the DICE deployment tool will take care of the platform deployment and deploying the agents on monitored nodes.

You can either use DMon as a stand-alone platform to monitor your infrastructure, or as a raw-data provider for high-level simulation and optimisation tools available in DICE toolchain. For more details about DMon please visit its Github page

Daniel Pop (IEAT)