Managing Computers with Automation

High availability apps in the IBM Cloud

Dima Rekesh and I wrote an HA-in-the-cloud article that has recently appeared on IBM's DeveloperWorks.  Article abstract is below

The new features of the IBM® Cloud enable application developers and architects to eliminate single points of failures in applications. This article provides a detailed guide on those features. It includes a discussion of the approach the IBM Cloud takes (added support for virtual IP addresses); how to prepare your cloud instances to take advantage of this feature; how to set up a highly available website; and how to test that site.

Read more...

A network discovery proposal for highly scaleable monitoring with O(1) overhead

In my previous article on scalable monitoring, I described an interesting and highly scalable low level infrastructure for monitoring servers and services.  Although that proposed architecture would work without network topology information, it worked best with a certain minimal amount of network topology information - which machines were connected to which switches, and what subnets these various servers are connected to.

This article proposes a high-level design for collecting and managing this network topology information.

Basic information Outline

The previous article described a three-level ring structure with each ring representing a different level in the network hierarchy.  Although other topologies are possible, and some may even be especially interesting, this article will expand on the three-level ring topology.  These three levels were (from top to bottom):

• Collections of subnets
• Collections of switches providing a given subnet
• Collections of hosts attached to switches

Towards this end, there are a few types of data items that need to be collected.  These include:

• A (hostid, switchid) tuple.  Depending on the protocol (CDP, LLDP) a given switch might have different types of switchids.  One might be the fully qualified domain name of the switch, and another might identify itself by the base MAC address of the switch.
• A set of one or more (switchid, subnet-description) pairs collected per-host.  For IPv4, the subnet description can be a standard CDL address/mask format.  I don't know what the right terminology is for a corresponding IPv6 physical network segment.

This information is sufficient for creating a basic network topology map from which one can create the 3-level ring structure as described in my previous blog post.

Some Considerations

For both LLDP and CDP, the client cannot solicit an announcement - you just have to wait for it.  The default interval for CDP announcements is one per minute.  This creates some complications in handling newly booted hosts since there is no way to tell when, if ever, an LLDP or CDP packet will arrive.  In addition, since one can move switch ports around easily, the affiliation of a given host with a given switch can change dynamically.

This implies that a general design for managing network topology must be prepared for a host to have no switch identification, for it to be supplied later, and for it to change.

Minion Design Outline

As you will recall from my previous article, I use the term minion to refer to a server which sends and receives heartbeats and reports failures.  For a minion, the job is relatively simple.  Here are a few elements of what it has to do beyond the tasks identified in the previous article:

• On initialization, send a startup packet containing network segment information for the host.
• Forward LLDP or CDP packets to the overlords when it first hears one and when something significant changes in the packets it hears.  Significant is meant to include changing of the identity of the switch, what port the host is connected to, and perhaps a small number of other conditions.  Minions only have to understand a small number of fields in CDP or LLDP packets

Overlord Topology Design Outline

Overlord is a term I used in my first article on this topic to refer to a host that is performing a management function.  There may be many overlord machines performing a variety of functions, and they may be in a cluster or other method of enhancing reliability.  For the purposes of this article, I will describe a single node implementation of the topology management function.

For managing topology, an overlord has to perform the following functions:

• Receive announcements from minions announcing their startup.  When this happens, the overlord should initially add it to its subnet ring as described below.
• Receive update CDP/LLDP packets from hosts and depending on how things have changed, one or more of the following actions can be taken:
  • Move it from the subnet ring into the ring for its switch.
  • Move it from one intra-switch ring to another
  • Create new higher level ring(s) for its switch and/or subnet.
  • Assign it a position in a higher-level ring.
  • Take away its position in a higher-level ring.
  • Give its position in higher level ring(s) to other nodes.
  • Remove subnets and/or switches from the active configuration.
  • Update the active configuration database with the forwarded CDP or LLDP packet.
• Receive "apparent death of host" notices from minions announcing that another minion has apparently died.  It will need to take one or more of the following actions when this happens.
  • Put a message in syslog for the death of this host
  • Send a detailed message to a higher-level management entity.
  • Close the intra-switch ring this host was a member of (if any) by sending the dead machine's neighbors directives telling them to send heartbeats to each other instead of the dead machine.
  • If the ring had two nodes before, move the surving node to its subnet ring.
  • Assign its role in higher level rings to another eligible host.
  • Remove the host information from the active configuration.
  • Remove a subnet or switch from the active configuration.

A few special cases need to be accounted for:

• A two-node ring - each only needs to send the other a single packet per interval - not two.
• A configuration with only one minion.

Although there are a number of cases to be taken care of, the details resemble the well-known algorithms for inserting and deleting from doubly-linked circular lists.  Of course, this case is more complex because things might fail in the middle of these various rearrangements.  What comes to mind are finite state machines to handle timeout and other failure conditions appropriately.

No doubt there are things I've overlooked, but this should be enough to demonstrate feasiblity and approximate level of difficulty.

Really Big Clusters: A Scalable membership proposal

This blog entry is a bit different than previous entries - I'm proposing some enhanced capabilities to go with the LRM and friends from the Linux-HA project.  I will update this entry on an ongoing basis to match my current thinking about this  proposal.

This post outlines a proposed server liveness ("membership") design which is intended to scale up from a dozen or so servers to tens of thousands of servers to be managed as an entity.

Scalability depends on a lot of factors - processor overhead, network bandwidth, and network load.  A highly scalable system will take all of these factors into account.  From the perspective of the server software author (for example, me), one of the easiest to overlook is network load.  Network load depends on a number of factors - number of packets, size of the packets, how many switches or routers it has to go through, and how many endpoints will receive the packet.  To best accomplish this task, it is desirable that the majority of "normal" traffic be network topology aware.  To scale up to very large collections of computers, it is also necessary that as much as possible be monitored as locally as possible.  In addition, since switching gear is not optimized for multicast packets, and multicast packets consume significant resources when compared to unicast packets, it is desirable to avoid using multicast packets during normal operation.

The Basic Concept - network aware liveness

Although the LRM in Linux-HA is not network-enabled, it tries to minimize monitoring overhead by distributing the task of monitoring resources to the machine providing the resource, and only reporting failures upstream.

To extend this idea of local monitoring into system liveness monitoring, one might imagine a standard 48-port network switch with 48 servers attached to it.  If one were to choose a server to act as proxy for monitoring the servers on that switch, then the other 47 nodes on the switch would send that unicast heartbeats to that node.  That node in turn would report on failure-to-receive-heartbeat events from the other 47 nodes on the switch.

Although this is simple to understand, it isn't ideal either - the work of detecting failures within a switch is  concentrated in a single node - and what happens when a switch fails.  So, let's try a different idea...

What if instead of having a hierarchy, the nodes within a switch arranged in a sort of doubly linked circular list of nodes in a switch - where each node sent heartbeats to the node "ahead" of or "behind" it on the list - making a ring-like structure.    This would spread the work for detecting failures across to more machines. For the 48-port switch, then each would send a heartbeat to the node with the next lower port number, and to the node with the next higher port number.  The first port would send heartbeats to port 2 and port 48, and port 48 would send heartbeats to ports 47 and 1. Since each node is monitored by two other nodes, monitoring has no single point of failure.   This design  is different from normal communication rings because there is no token being passed around the whole ring - just heartbeats going to a single node ahead and a single node behind.  So, conventional thoughts about latency in rings don't apply here.

If one were to implement this in the context of a cloud-like infrastructure, or a monitoring environment it is likely to be desirable to also run the LRM (or something like it) on these same machines so one can perform more detailed monitoring and/or effect changes to the managed servers.

Some Tentative Terminology

In order to facilitate discussion, I'll use the following terms (which are, of course, subject to change):

• Overlord - the policy-aware management entity at the top of the food chain.  The Overlord layer might be a cloud management layer, it might be a System management package like IBM director or Nagios, or a high-performance scientific cluster management entity, or some other entity interested in managing large numbers of computers.  The term overlord is used to refer to a management capability - not necessarily implemented in a single computer.
• Minion - a monitoring node which is listening to heartbeats from other nodes and reporting failure-to-receive-heartbeat events to an Overlord.  The minions at the lowest level in the system are only part-time minions - with the rest of their resources being spent on performing other tasks.  Dedicated minion servers should be capable of monitoring thousands (or tens of thousands) of other minions.  It is anticipated that the minion function would be performed by some software which also acts as a network proxy for an LRM client.  Minions send heartbeats, listen for heartbeats, report problems up the line, and act as LRM proxies.

Datagram Protocols

I anticipate the need for two different datagram transmission protocols, UDP (unreliable datagram protocol) and RDS (reliable datagram service).  UDP would be used for heartbeats, and RDS for communication between the Overlords and the LRM proxies (regardless of role).  It is expected that digital signatures are sufficient for heartbeats, but encryption may be desirable for some Overlord communication.

For this application, it is anticipated that the reliable datagram communication would not be high bandwidth, or likely need congestion control.  Other options in place of RDS would be an application protocol over UDP, or something like Qpid (AMQP).  I suspect any of these would work nicely.

LRM Proxy Design Goals

Here are a few design goals which are good to keep in mind during detailed design and implementation:

• Continue the policy-free approach that the LRM implements.  Although this implementation is intended to be topology aware, the LRM proxy code should rely on directions from is Overlord to tell it where and how often to send heartbeats, which heartbeats to expect (and how often), and other configuration information.  The Overlord is be the entity that "understands" the overall heartbeat topology.  Nevertheless, it is possible that the LRM proxy may gather some low-level network configuration for its Overlord.
• Assume a hostile network environment, and design communications accordingly.
• Minimize resource consumption - CPU, memory, network bandwidth should be held to a minimum.  Although this component is inherently lightweight, it will exist on every server in the collection - so resource waste is multiplied by thousands or tens of thousands of instances.
• Minimize latency.  Even if there is a hierarchy of watchers watching for failures doesn't mean that reporting problems should go back through the hierarchy.
• Minimize membership workload on non-dedicated minions.
• Try to design the API so that it is flexible enough and stable enough that version mismatches are unlikely to cause problems.
• Keep it simple.  The LRM proxy is a simple concept, the detailed design and implementation should be simple as well.

LRM Proxy Design Outline

The LRM proxy would implement the following capabilities:

• Send multicast (or broadcast) packets on boot-up to announce our presence
• Receiving requests from authenticated Overlords.  Requests currently anticpated include:
  • Requests to forward an encapsulated request to the LRM
  • Requests to send heartbeats to a certain address at a certain rate
  • Requests to stop sending heartbeats to a certain address
  • Requests to expect heartbeats from a certain address at a certain rate
  • Requests to stop expecting heartbeats from a certain address
  • Requests to add an address of an Overlord
  • Requests to delete an address of an Overlord
  • Requests to register a new Overlord authentication key
  • Requests to drop an Overlord authentication key
  • Requests to change the outgoing LRM proxy authentication key
  • Requests to stop sending boot-up multicasts
• Sending failure-to-receive-heartbeat messages to Overlords
• Sending LRM return result messages back to the registered Overlords

As implied above, at startup time, each LRM proxy would send out a multicast or broadcast packet announcing it was now up and ready to accept requests.  It will continue to send them until it receives an authenticated message from an overlord telling it to stop.  From then on the normal proxy API would be employed.

Some Thoughts about Heartbeating Topologies

This design could also be extended to create multiple levels of bidirectional rings between all the switches on a subnet, and between all the subnets.   The point is not that this is an optimal topology, but that the design of the LRM proxy component is topology-independent - it should support any unicast-based topology.

Below is an illustration of what a multi-level ring topology might look like.

For this smallish network, there are 54 servers.  12 of them send and receive heartbeats from 4 peers.  The remaining 42 have two peers - averaging 2.4 connections per server.  In a larger network, this number would be closer to 2.

If one wanted a higher degree of redundancy, one could imagine two shifted rings - with each node talking to node n-2, n-1, n+1 and n+2.  This would increase the number of packets sent and receive per interval to 4.  This could be used at all levels, or at selectively at higher levels.

There are no doubt other interesting topologies one could imagine as well.

 Advantages to this LRM proxy design

Here are a few advantages to this design:

• The LRM proxy design is simple and straightforward.
• Once a few releases have gone out, it is likely that the LRM proxy code and API will be stable for a long time.  This is a significant advantage for a component which is deployed on tens of thousands of computers.  It also simplifies upgrades, since it would be a very rare occurrence that an LRM proxy update would have to be performed in order to update the Overlord software.
• The LRM code will not have to change in order to fine tune the network resource usage.  Since where to send packets, and who to send them to and how often to send them are commands that come from its Overlord, the LRM proxy neither knows nor cares what the topology is.
• Although this was designed with network topology-awareness as a goal, it is not necessary that an Overlord implement topology-awareness, either initially, correctly, or at all.  The LRM proxy concept will function correctly without topology awareness, it will simply consume more network resources than necessary.

Potential problems with this design

• Providing switch topology information to the Overlord(s) is a potential difficulty.  There are some protocols implemented by some switches which provide information similar to what's required.  Further investigation is required to determine how difficult discovery and auto-configuration of network topology will be. The Link Layer Discovery Protocol (LLDP) may be of help here.
  
• The scope of the problem being addressed is very large.  The addition of layers of monitoring will likely introduce difficulties in debugging problems. Infrastructure for determining the origin of anomolous behavior should be provided.
• The flexibility in setting up monitoring topologies adds complexity to the system (even though it keeps it out of the LRM proxy).
• There is the possibility that the Overlord will receive conflicting reports regarding the liveness of a particular node.  This potential lack of consensus and timing issues will add complexity to the corner cases in the Overlord.

Open Issues

• There are many possible topologies for handling the upper layer Minion workload distribution.  It is not yet clear what the set of good choices are, and what the tradeoffs are between those different topologies.
• It is possible that it will prove desirable for this same infrastructure to collect statistical information for the Overlords.  Conversations with experts in high-performance computing and examination of tools like Ganglia will likely prove helpful in better understanding this problem.  What I have in mind is some amount of periodic piggybacking of data on the heartbeats, and then aggregation by nodes in minion roles, and periodic uplinks of this data to the overlords.  The goal would be to reduce the number of uplinks packets while having minimal impact on the timeliness of the data.  Not sure if this is a good idea or not.
• What about IPv6?

Concluding Thoughts

This proposal only provides liveness information and distributed control for a large collection of managed servers.  In many respects, this is perhaps the easiest component to scale up.  It is a long way from a complete cloud infrastructure, clustering software, or enterprise-scope system management package.  Scaling the Overlord components above this layer to the same standard of size is a very interesting and challenging task.  I suspect that this design is sufficiently scalable that it is likely that other components in the system architecture are likely to be the limiting factors in system scaling.

An Update

I have now posted a follow-on article outlining some ideas for the "overlord" portion of the infrastructure.