How to Simplify Systems Integration in an Enterprise Environment

Getting heterogeneous systems to talk to each other can be a make or break exercise, especially in hybrid cloud infrastructures, critical to supporting digital transformation processes.

We are seeing more and more of the current generations of IT professionals involved in this transformation, who have a growing expectation of a frictionless enterprise IT experience.
A seamless integration between IT systems can be a critical ingredient to reshaping an infrastructure after the experience offered by modern consumer-grade public cloud services.

My goal in this blog post is to describe how new tools and practices can improve your ability to realize, scale and maintain the integration of infrastructure components over time.

System integration is a well-known practice, but, with the sole purpose of describing how and where the aforementioned tools can be used, I’ll frame it as a two-way process, each way characterised by a different approach:

  • Decision-Oriented: to “know something”, and use this information to make decisions
  • Action-Oriented: to “do something”, as a reaction to an event or a made decision

Depending on your business needs and the nature of systems involved, you may be able to use just one or both of these approaches, combined together in a two-way integration process.

In this context we’ll see how:

Cloud Management Platforms to support Decision-Oriented Integrations

Decision-Oriented Integrations are aimed at collecting data about your IT environment and making decisions about it.

While you can use ad-hoc connectors or rely on APIs to collect the data, you still need a place where that information can be analyzed, correlated, compared against a set of policies, and possibly used to finally generate recommendations.

When your IT environment is a hybrid cloud infrastructure, an ideal place to perform these tasks is a cloud management platform (CMP) like Red Hat CloudForms.

The reasons why you may want to collect metrics from infrastructure systems can be various. As an example, you may want to correlate them to understand whether an application is performing within contracted service levels and then decide to deprovision the resources or reallocate them to meet your business needs. This could be the case for a set of customer-facing services that your company is offering. You may also want to use the collected information in a different way, to decide whether a customer should be offered a discount or suggested a different kind of service.

To better explain this concept, I will use the smart home appliances as a metaphor. Imagine that your IT environment is a centralised smart home system. Now imagine that the smart home system is able to use GPS data from your smartphone to understand when you leave the apartment and trigger an “away” mode (e.g., turning off the smart lights, adjusting the smart thermostat, etc.). To achieve this result, this centralised system must be able to integrate all the different home appliances to collect their data, correlate it, decide whether or not you left your apartment and what elements must be adjusted because of that.

Some energy providers already collect and analyze energy usage data coming from smart meters or smart thermostats, to tailor heating and cooling settings to specific buildings or suggest better energy plans to their customers.

Similarly, a CMP can correlate the current utilization of your systems (e.g. physical clusters or Infrastructure-as-a-Service (IaaS) regions) with the utilization trend of your workloads to decide the optimal placement. Furthermore, it can collect usage data from all the tiers of a service, to determine whether to scale up or down one or more components.

To provide this unified governance, a CMP uses native integration points or APIs to ingest information, alerts and metrics from systems such physical clusters, IaaS engines, virtual machines, etc.

Then, the CMP can correlate and evaluate them against technical, business and compliance policies.

CloudForms can collect and analyze capacity and utilization data (e.g. CPU usage and states, disk & network I/O, memory usage, running VMs and hosts, etc) from many different engines in your virtual infrastructure. The smart placement capability can use this data to understand the optimal placement for new workloads. The same information can be used to determine if a service is reaching its capacity and, according to conditions pre-defined by the organization, scale all the needed components.

IT automation to simplify Action-Oriented Integrations

Action-Oriented Integrations are designed to enable systems to act on an event or a decision.

In this scenario, IT automation could reduce the complexity by acting as an integration broker. The more flexible and powerful the tool, the easier it is to scale and diversify the integration.

I will use the behind the scenes of a self-service portal as an example of integrations aimed at triggering actions.

The goal of a self-service portal is to empower users to deploy a set of predefined and sometimes preconfigured assets (from VMs and containers all the way up to business services), to support the business.

The challenge is to compose these assets by tapping into many different kind of resource pools provided by bare metal-server environments, network or storage arrays, virtualized and containerized environments, public and on-premises IaaS cloud environments, etc.

Let’s go back to our smart home analogy.

Imagine that your resource pool is made of all the smart lights, smart thermostats and the few sensors and smart appliances you have in your smart home environment. Ideally, you should be able to issue a command through an interface (e.g., a smartphone app or a voice-recognition device) and trigger a predefined and preconfigured scenario (e.g. “chill out evening”, which means “dimming the lights and turn on the TV”).

IT automation tools, like Red Hat Ansible automation, offer many advantages which can make them critical to managing a large-scale IT environment. One of them is to provide a single integration layer to deal with, removing the need to enable and maintain many different integration points.

In the smart home analogy, consumer-oriented automation, provided by companies like IFTTT and Zapier, similarly offers the ability to integrate and govern a wide range of systems through a single, common language.

If two systems are not designed to work together (e.g. your self-service portal does not support your network devices), you must write ad-hoc code to integrate them. This means that you have to figure out how your endpoints communicate and code your way through the self-service portal. Repeat this for any other unsupported system in your IT environment and maintain your integration code throughout the many software and hardware updates that your endpoints will have before their decommissioning.

Back to our analogy now. How do you tie together your smart lights, which support Apple Home Kit, and your smart tv, which only supports Amazon Alexa? Once again, an intermediary like IFTTT can solve that complexity and incompatibility, allowing users to combine devices and services in workflows unique to their specific needs. Furthermore, as user needs evolve (e.g., they buy a different smart TV, a new device to monitor plant watering is introduced in the smart home environment), the integration layer enables them to adapt and scale existing workflows, extend control to new components and adapt to replacement smart devices.

Red Hat Ansible automation, used as the integration layer, offers an easy to use, human-readable language, to abstract a set of actions throughout the IT stack. To further reduce the effort to extend and maintain such a large integration surface, Ansible adopts a modular strategy where both the community and certified partners continuously contribute to the support of new technologies.

First collecting data with CloudForms to make a decision, and then acting on that decision with Ansible as a broker to execute commands, is a good example of Decision-Oriented and Action-Oriented integrations combined together in a single process.

Open source to reduce the long-term risk of integration

So far, I discussed how tools like IT automation engines and CMPs can contribute to mitigate the integration complexity in an IT environment that grows in scale and diversity.

However, maintaining multiple integrations can bring a completely different set of challenges.

What if you are the first to attempt a certain integration?

What’s the advantage of contributing to an open source community in this scenario?

Anyone who has worked in IT for a large organization has likely written some custom integration code at least once, probably more than once. Both open and closed solution starts from the same point: the initial cost of writing code. No matter if this is the cost of a system integrator or your own developers.

Open source opens a unique opportunity, though: by submitting this code to the project maintainers, your integration effort is evaluated in terms of stability, scalability and readability, and then, potentially, enters the downstream. The business advantage in that, is the possibility to reduce the recurring cost of maintaining the integration you made.

The factors that determine if this is going to happen are the same that define the success of a project: how critical is the technology for the community and how broad and active that community is.

If we look at Decision-Oriented integrations, one of the biggest challenges you may have is to harmonize all the different data sets.

Different systems collect data with different logics and mostly for internal consumption, you have to extrapolate what you want, and process it through a lowest common denominator to make it usable with other data sets.

To make things even harder, the data formats you have to reconcile will mutate over time.

Open Banking Standard is an example of open source practices used to mitigate this particular integration challenge, promoting community-driven API standardization.

If we look at Action-Oriented integrations, one of the main challenges is to consistently support all the integration points over time. This means being able to evolve at the same pace of both the technology you are integrating and the business requirements served by these integrations.

Using Red Hat Ansible as the integration layer, you have access to more than 1,000 modules to integrate with the most disparate of systems, from infrastructure components, such as networking, up to application servers and databases, and more than 12,000 roles* to support an enormous range of use cases, from provisioning and configuration to remediation.

This broad range of components (that reduce the initial complexity of integration) is the result of a large and active community which dramatically increases the chances of having a return from your contribution.

In summary, you integrate IT systems for two main reasons:

  • to collect data about your IT environment and make decisions (Decision-Oriented Integrations)
  • to act on an event or a decision (Action-Oriented Integrations)

and, to mitigate the complexity of your integration effort, your strategy should consider:

  • leveraging tools like Cloud Management Platforms and IT Automation engines as integration layers to manage more easily an IT environment as it grows in scale and diversity
  • adopting open source to potentially reduce the costs of maintaining the existing integration points over time

Massimo Ferrari
Management Strategy Director

*Ansible Roles are a standardized file structure, to organize and group together a series tasks and the related metadata, variables, etc.. Grouping content by roles also ease reutilization and sharing with other users.