Share

In the previous post (part 1) we have covered how visualizing microservices is very useful when monitoring an environment with Microservices.  This post will look at a number of practical ways of understanding the relationships within and between services in the environment, and how to implement them with ITculate.

In order to map the relationships, I would like to first break the entire “map” into a number of components and relationship types so that we can see how to cover each one of them. There are two types of components:

  1. Physical objects: Instances and objects like ELBs, ASGs, RDS etc. Usually the AWS management console will show these objects.
  2. Logical objects: these are objects represent a concept and usually are a higher level than a single physical object. These can be “a service” or “a product”. Typically no management console will display logical objects, but are more important from an architecture standpoint. These are the concepts that we use when thinking about the architecture of the system.

Similarly we can look at different relationship types and break them to simple relationships and other:

  1. Simple relationships: these are relationships that are a clear result of a configuration. For example – the ELB configuration determines all the instances that will work with that ELB. Usually, these relationships are easily detectable through APIs (like boto for aws-ec2)
  2. Other relationships: those are relationships that harder to find. For example, a relationship between two logical objects may be something that only a user can identify.

Building the map

Building a map of an environment can be challenging. Microservices, by definition may be highly distributed and very dynamic. We will look at how to do that step by step

Physical objects and simple local relationships

The most simple place to start is using well known APIs. For example, for an environment that is running in AWS, using an interface like boto enables getting the physical objects and AWS services (instances, ebs, elbs, and services like rds, kinesis etc.). However, only some of the relationships can be determined at that time. These are things that are specific to AWS ec2 configuration. The most common example if relationships between instances and their ELB or ASG. A more advanced example would be step functions internal relationship. With ITculate, this is a part of the initial value that our customers can get, ten minutes after installation.

Local relationships discovered

Flow logs – automatically finding the communication patterns

ITculate automatically detect the relationships and any changes to them, using VPC flow-logs. This is a key capability that we have not seen in other products. This functionality ensures that the customer’s network is automatically kept up to date without any effort on the customer side. For more details about these capabilities, check out the post Using VPC Flow Logs to visualize your network.

Using tags to discover logical objects and relationships to physical objects

In most medium size environments, the team is using tags extensively. We found that in many cases, there is a financial motivation for maintaining these tags. After a short discussion we can usually determine which tag holds the name of the service, and in many cases, the product (a higher level of abstraction above a service).

Using the service name and product name can quickly achieve the following thing:

  1. Identify logical objects of services and products, such as microservices and their names.
  2. Create physical to logical relationships between resources such as instances ELBs and others to the logical objects.
  3. Create logical to logical relationship between services and resources.

Discovered services that are not yet connected

Note that if the tags as set correctly, this phase can be achieved very quickly. It is something that we use with almost every customer because it is successfully leveraging previous work that was done (placing the tags) in the environment.

Last step – service to service relationship and other connections

At that point, we have all the services (and maybe the products) and their internal structure. What is left is the last final touches – setting the relationships between the services and other internal connections. For example a instance may be using a certain RDS or a certain queue. As soon as possible after an instance starts, one should report what it is going to use. We recommend that to report everything “locally”. That means that the team that operates service A will report about what service A is using, team B report what service B is using and so on. This is important because in a distributed scenario, it prevents the need for more inter team coordination. Every team reports its information and ITculate takes care of combining the data into a single large picture. Doing that is very simple using the ITcualte API. Here is an example:

Note how the services are defined on lines 21 and 25, and connected on line 29.

Are there any other ways of creating relationships?

Yes. At a later stage we recommend customers to consider using the provisioning scripts in order to update relationships. To that end ITculate provides simple integration with the major provisioning script languages. This is possibly the best way to declare what an instance is is going to use, as soon as it is starting. Using the provisioning scripts ensure that ITculate represents the environment in an updated as tightly as possible. Furthermore, this can serve as an alternative to using the tags, and has additional possible functionality like reporting logical to logical relationship, defining logical clusters and many other things.

Connected services

Conclusion

Using ITculate there is a very simple path to build the relationships within and between services. Furthermore it is possible to build the relationships in an automated fashion. This is the base for successful visualization of a microservices environment. But at least as important: automation this is the only way to have live visualization that is up to date to the minute.

 

About ITculate

ITculate.io provides a monitoring solution for DevOps environments. ITculate’s solution captures not only raw and custom metrics but also the architecture of the customer’s environment. ITculate provides a more intuitive way of data exploration, as a result of the overlay of metrics on top of architecture components. We believe that this dramatically improves the user experience of monitoring and troubleshooting over existing products in the market. Please check us out at ITculate.io to learn more!

Share

One Comment

  1. How to visualize microservices environment as a part of microservice monitoring

    […] the next part we will review how to map the relationships within and inside microservices with the ITculate […]

Comments are closed.