API series - eG Innovations: APIs as IT monitoring tools
This piece is written by Rachel Berry in her capacity as product consultant at eG Innovations, a company known for its cloud-based application performance and IT-Infrastructure monitoring solutions.
The importance of automation and integration is becoming well understood. With the increasing adoption of DevOps culture and Infrastructure-as-Code (IaC) initiatives, such integrations are now a must.
Monitoring tools are rarely being used in isolation and with the increasing complexity of today’s IT infrastructures and applications, it is no longer viable to have just one tool to address all the use cases and requirements of IT personnel. In this article, Berry explores the different types of APIs that monitoring tools must support, both consume and provide themselves for other products to leverage.
Berry writes as follows…
Let us think about monitoring tools as ‘consumers’ of APIs.
Monitoring platforms rely on many APIs supported by software and hardware vendors. For the modern app/infrastructure world, APIs have become almost as standard as SNMP was for the networking world 10 years ago (probably still is!). Everything from virtualisation platforms to container technology to storage devices to cloud technologies and applications provides and supports APIs for monitoring and metric collection.
Even more sophisticated network equipment like load balancers and security devices support APIs because there is greater flexibility to provide metrics and details with an API than when using SNMP. APIs provide details about configuration, usage, performance and health of different applications and infrastructure tiers.
Vendors of IT technologies must take care to craft good APIs and metrics to ensure that monitoring tools provide the level of detail that is necessary to quickly identify if their technology is not the cause of an IT service problem. Otherwise, costly support interactions may result, costing the vendor hours and money.
Given the diversity of metrics and capabilities of different applications and infrastructure elements, there is no standard API that monitoring tools can use. So, monitoring tools have been customised to use different APIs for monitoring different application and infrastructure tiers.
Figure 1: Different APIs and mechanisms used for metric collection by the eG Enterprise monitoring tool.
APIs are also increasingly used by monitoring tools to initiate remedial actions automatically or through manual intervention – e.g., to restart a VM, reboot a crashed server or reconfigure an application.
APIs beyond metrics for monitoring
While it is fairly obvious that IT monitoring products are likely to leverage APIs to collect metrics about usage, performance and health of applications and infrastructure that they monitor and manage, it is less obvious that enterprise-grade monitoring products now must provide APIs themselves so they can be integrated into other technologies and workflows used by an organisation. Here are different uses for APIs in a monitoring tool…
Automating deployment of the monitoring tool
IT systems now are often dynamic and often cloud-based. Systems are designed to auto-scale up and down, especially if using architectures and technologies such as containers, microservices and Kubernetes orchestration. Declarative infrastructure and Infrastructure-as-Code are now mainstream enterprise strategies and monitoring tools need APIs to allow users to deploy monitoring alongside the infrastructure.
Common activities performed via APIs include adding a server to be monitored, deploying the appropriate agents and then configuring them to enable data collection. Monitoring tool APIs to enable deployment are needed if you are considering automation strategies including technologies such as Terraform. See: Infrastructure-as-Code series: Practical monitoring in an IaC universe – CW Developer Network (computerweekly.com) for more details on IaC monitoring workflows. With the right APIs and automation, deployment of monitoring for entire datacentres can be done in minutes, rather than days or months.
Integration with ITSM for auto-discovery
Many organisations maintain configuration management databases (CMDBs) in ITSM tools. Integration with the CMDB API can enable a monitoring tool use discovered inventory of servers, devices, applications and so on for monitoring. This capability is especially important if the monitoring tool does not have in-built auto-discovery features.
Automatic alerting
When a monitoring tool detects an alert, it has to communicated to IT personnel. Many organisations have centralised alerting tools that serve as a central repository of IT staff, their contact details, preferred mode of communication, shift timings, etc. Monitoring tools can use APIs to communicate alert information to these alerting tools and letting them handle multi-modal communications to users.
Integration with ITSM for automatic incident management
ITSM tools like ServiceNow, HP ServiceDesk, etc. are used by organisations for generating and tracking tickets when incidents happen. All of these tools have APIs. Using these APIs, the monitoring tool can inform the ITSM tool about a new alert (by creating an incident), update priorities of existing incidents or even automatically close an incident when a problem is resolved.
Integration with identity management solutions for single sign-on
Many organisations have implemented single sign-on mechanisms. While some use standard protocols like SAML, others may have proprietary APIs for user identification and authorisation. To support single sign-on, monitoring tools must integrate with the corresponding APIs of the identity management systems.
Integration with analytics tools
Monitoring tools collect and consolidate data from numerous application and infrastructure components. While these tools have built-in dashboards and reports, there may be scenarios where the data from the monitoring tool may need to be made available to other tools. For example, a manager-of-managers solution may be in use in an organisation and it may be required to provide a consolidated dashboard including data from multiple tools.
Another use case is off-line analysis of the metrics. Some IT experts may prefer to export the monitoring data into enterprise visualisation tools such as Microsoft PowerBI and Tableau to align with their goals. The monitoring tool must include APIs that can be queried by analytics tools for historical data extraction. On the other hand, external dashboarding tools may need queries on real-time metrics.
Integration with subscriber-management tools and online marketplaces
In a service provider context, users may need to be automatically created and provisioned in the monitoring tool when they sign up or renew a service. APIs must be supported by the monitoring tool to support tight integration with subscriber management systems. In a similar vein, many monitoring tools are deployed as SaaS and integrated with online marketplaces. In such cases too, when a user signs up through the marketplace APIs ensure their details automatically updated in the monitoring tool. Subscription extension, account closures, etc. also have to be supported by these monitoring tool APIs.
APIs are monitoring evaluators
As you have seen, APIs have become increasingly important to integrate monitoring tools with existing customer deployments. Monitoring tools that do not have API support and cannot be integrated into customer workflows often end up being shelfware – i.e., used for a short while and discarded thereafter.
It is wise to evaluate carefully what APIs a monitoring tool supports, how extensible they are and whether they can be integrated with your existing toolsets.
At the same time, when you are evaluating APIs for IT management and monitoring tools you really need to look at and ask about the security and auditing around them. These are invasive products that not only access information at the core of your IT systems but also that be leveraged to extract data and even perform privileged actions such as shutting down servers.
Whilst you can monitor APIs for security, you really want to be choosing those that are secure by design.
Our own API support experience
In an attempt to further validate some (if not all) of these populations and ruminations, please allow me to tell you about our experiences at eG Enterprise. The technology itself exists as enterprise-grade monitoring and reporting platform that consumes metrics, events and logs from a large number of applications and IT infrastructure technologies to provide end-to-end observability, alerting and anomaly detection.
The data and signals ingested are processed and correlated within the platform’s AIOps engine and raises alerts when performance or availability issues occur alongside automated root-cause diagnosis information. Basically, the product answers the questions – “Why is my application slow?” and “Why can’t I access my application?”. Beyond this monitoring tools usually provide out-of-the-box and custom visualisation and analytic tools such as dashboards, reports and topology diagrams.
For a number of years, we have included a richly featured REST API, enabling DevOps and administrators to develop programs and code to automate their monitoring deployments and data retrieval and integrate with other enterprise products. Its REST API enables tasks such as:
- Ability to automate admin activities (e.g., auto-provision monitoring when a VM (Virtual Machine) is spun up).
- Extract and analyse performance metrics automatically.
- Integration with other management portals to provide a seamless user interface.
- Integration and consolidation with asset/configuration tracking systems.
Concluding thoughts
In summary, whilst a monitoring platform is usually regarded as a heavy user of APIs, increasingly you need to understand what APIs they offer to make a wise choice and extract the maximum value from such a product securely. To learn more about eG Enterprise’s approach to APIs, please see: APIs for IT Monitoring Solutions | eG Innovations.