Next Generation Infrastructure Monitoring

Customer is into developing digital and physical information management software solutions. Organizations across the globe trust the company to store and protect their information and assets in digital format. The company offers information governance solutions, to address issues such as distribution, collaboration, management, and governance for physical and digital information; RFID data protection services; and Web Portal that enables users to access information governance solutions. The company also provides document and information management solutions.

Background:

Customer is a progressive company in terms of software development. Their challenges were mostly focused around the monitoring of high volume of data, tools and methodologies to introduce automation, the application caters to 100,000+ users globally.

Customer approached CT to analyze their current scenario for monitoring analyzing their business critical applications. However, the challenge is to be able to decipher if the traffic is unusually high and detect the anomaly in the way the data is generated as to when, at a certain time of the day in their customer portal or if the average response time is high for the application or if any unusual processes running on the host (App server, Load balancer).

To resolve these issues CT’s started off by using next gen monitoring of time series anomaly detection with Elastic X-Pack Machine Learning.

The Elastic Stack enables to reliably and securely take data from any source in any format and search, analyze, and visualize it in real time. Elasticsearch is a real-time, distributed storage, search, and analytics engine. It can be used for many purposes, but one context where it excels is indexing streams of semi-structured data, such as logs or decoded network packets.

CT used this to

• Automated analysis of time-series data
• Create accurate baselines of normal behavior in the data
• Identify anomalous patterns
• Create baselines of normal behaviors in data and identify anomalies
• Unsupervised machine learning algorithms
• Detect, Score and Link with statistically significant influencers in the data
• Anomalies related to temporal deviations in counts, frequencies
• Statistical rarity
• Unusual behaviors for a member of a population

The following diagrams depicts anomalies in the number of request received by the load balancer in the anomaly explorer.

Advantages

With this next gen monitoring approach, there is no necessity to define algorithms, create/manage rules and keep tweaking the threshold/baseline values. The anomaly detection model is based on unsupervised machine learning algorithm which learns and improves with ingested data over a period of time.