Managing Computers with Automation

Rules to automatically monitor services using OCF resource agents

In my previous post on automatic service monitoring rules, I explained the general principles behind the Assimilation project's rules for automatically monitoring with LSB-style init scripts based on what we've discovered.  In this post, I'll extend that further and explain how to monitor using Open Cluster Framework (OCF) resource agents - which are far more powerful, but a bit more interesting to configure.

As you will recall from the previous post, LSB resource agents rules were written using JSON with comments.  The rules for recognizing and configuring OCF resource agents are very similar - and are also described in JSON.  Faithful readers may recall that I wrote a blog post on how OCF monitoring works - which may be of interest to review.

With an LSB resource agent, all you had to do was recognized that it controlled this particular service - there is nothing to configure.  Either it matches, and you can issue a status operation, or it doesn't.  With OCF resource agents, they support configuring multiple instances of resources on the same machines, through a series of name/value parameters in the environment.  Some parameters have defaults, but some have to be configured in order to properly monitor an instance.  The challenge for the Assimilation code is to figure out these parameters from the discovery information and plug them into the right places.

As part of the Assimilation project, we've written an neo4j OCF resource agent for monitoring and controlling Neo4j which we'll discuss here.  The LSB init script merely looks to see if the neo4j process is running.  The OCF resource agent goes further - it issues a query to Neo4j and makes sure it gets a sane result returned.  For good measure, it implements multiple levels of monitoring - one with a simpler query and one with a more complex query.  This allows us to detect when services hang or give erratic results - which is far more interesting than merely looking to see if the process is in the process table.

There are a few interesting parameters for this resource agent which we can derive from our discovery information.  These are:

• ipport - the IP:port address that neo4j is listening on for REST queries
• neo4j_home - the root directory of the neo4j installation
• neo4j - the name of this Neo4j instance

For the purposes of monitoring, only the first parameter is of interest.  The other two parameters are necessary in order to know how to properly start and stop the service (which is part of the OCF resource agent specification).  So, let's see how we tell the Assimilation code how to find and fill in these parameters.

{
    "class":    "ocf",
    "type":     "neo4j",
    "provider": "assimilation",
    "classconfig": [
#       OCF parameter   expression-to-evaluate      regular expression
        [null,          "@basename()",               "java$"],
        [null,          "argv[-1]",                 "org\\.neo4j\\.server\\.Bootstrapper$"],
        ["ipport",      "@serviceipport()",          "..."],
        ["neo4j_home",  "@argequals(-Dneo4j.home)", "/"],
        ["neo4j",       "@basename(@argequals(-Dneo4j.home))","."]
	]
}

The last two columns are the same as in the LSB example in my previous blog post, but the first column is new.  It gives the name of the parameter that corresponds to the expression in the second column.  If this expression and regular expression are only used for recognizing the service and doesn't have a corresponding parameter, then we use the JSON tag null to indicate that.  Our previous matching patterns are the same as they were, and don't have a corresponding parameter to go with it.  To fill in the ipport parameter in our example, we call the serviceport() function which examines the IP address/port combinations that neo4j is listening on, and returns a single IP/port that it is listening on.  This function prefers to return ipv4 addresses, and lower numbered ports.  If it is listening on an ANY address it returns the appropriate loopback address (127.0.0.1 or ::1).

This data that serviceipport() returns is derived from data discovered by our tcpdiscovery discovery agent.  The tcpdiscovery agent currently uses netstat to gather this information.  Because of this we know exactly how to monitor this particular instance of neo4j.  The other two parameters use the argequals() function.  This function searches the command line of this service (collected by the tcpdiscovery agent) for a "-Dneo4j.home=something" argument and returns the something part.  Because you're one of my brilliant readers, you doubtless have no figured out that neo4j is invoked with that argument as a parameter - enabling us extract it from the command line.

As time goes on and we start recognizing more and more services, we will no doubt add more functions to simplify writing these rules.  When the Assimilation code is extended to use other monitoring APIs, these same principles will enable us to configure those monitoring methods using nearly identical formats to either the LSB or OCF formats discussed in these two blog posts.

What's your reaction to this?  Do these kinds of rules seem straightforward?  Incomprehensible?  Useful?  A waste of time?

 

Related articles

Neo4j: The 'Thinking in Graphs' Learning Curve

Neo4j 2.0 A Dream Graph Database

How Open Cluster Framework monitoring works

Another thing that I've taken for granted is that people understand how HA systems like Linux-HA or Pacemaker monitor their resources.  This is important to me in the Assimilation Monitoring Project (AMP) because we use the same techniques.  So this post concentrates on how typical resource agents following the Open Cluster Framework (OCF) API (which we like a lot) work.  Since these OCF resource agents were written for high-availability systems, they tend to be both both flexible and robust.  The OCF API is one of those things that's worked out very well - much better than expected.  Both Pacemaker and AMP implement other APIs as well, but for both this is their preferred APIs.

Years ago, I created and headed a team of people from several companies and several different operating systems to create a standard API for managing high-availability resources.  Resource is a vague and general concept - it's the name we used for the things HA systems manage.  It might be a web server, it might be a piece of hardware, it might be an IP address, a filesystem mount, or even something as abstract as a set of firewall rules.  The key thing is you have to assign a meaning to starting it, to stopping it, and you have to be able to tell if it is started (running) or not (stopped).  If you can meet these criteria, then you can make an OCF resource that models it.  The part we'll explore in this article is "can we tell if it's running properly or not"?.  Like the concept of resource, the idea of whether something is properly running is a very flexible idea, and can mean almost whatever you want it to mean.  The only real constraints are that if the start operation succeeds, then the monitor should normally succeed right after the start.  Conversely, if you have issued a stop, then it should monitor as "not running" until a start is issued.

The key to monitoring anything is to monitor the right thing - whatever that might be - and to not make it too trivial.  For example, looking to see if a daemon is running is usually too trivial for meaningful monitoring.  It's better than nothing, but not a lot... Instead you want to monitor something that is highly correlated to the correct functioning of the resource in question.

For example, in the case of a web server, you want to be able to give it a URL to retrieve, and get the right result.  You want to give it a "normal" web page - whose success is well-correlated to the success of the web server and the application behind it.  But you don't want to have to continually update the monitoring script every time you update the web server, so typically such a resource would return the right successful return code (200), and matches a regular expression.  For the OCF Apache resource, the default regular expression is very simple - it looks for an "</html>" at the end of the received stream.  It's very simple, but if the web server is broken, normally it won't look like that.  Of course, you can easily override it with some other value.  This kind of flexibility is common in OCF resource agents.

Similar things apply to other resource agents.  For example, in the MySQL, DB2 and Oracle resource agents, they perform a simple query - that is cheap and should always succeed.  In both cases, the query is against the authorized users, and the result is counted.  The result should be >= 1.  If it's not, then it failed.  Although it's simple, it's a real query that exercises most layers of the database system, and whose result is easily verified.

In general, you want resource agents that are cheap to run and you can run frequently.  However, there are occasionally needs for more in-depth monitoring actions - things you would like to do, but are too expensive to check frequently.

There is a Filesystem resource agent that mounts a filesystem.  The normal level check is something simple like checking to see if it's still mounted, and if a df command on it succeeds.  This is cheap and can be run frequently.  But what if that filesystem has been corrupted - by hardware failures or software bugs?  A more extensive check that might be worth running once a month would be to snapshot the filesystem and peform a full fsck on the snapshot.  The OCF resource agent API supports these multiple levels of monitoring through the OCF_CHECK_LEVEL parameter.  The invoker of a monitoring action is permitted to say how gnarly the test is supposed to be - and if the resource agent supports that level, it will act accordingly.  If it doesn't support that level, it will assume the next lowest level that it does support.

So, and OCF resource agent specification is flexible in two dimensions - it can take parameters to tell it what the resource is, and details of things like regular expressions, and also can be told how expensive a check it is expected to perform.

No matter what you want to check, it is likely that you can write an OCF resource agent to check it for you - and you can even select which check to perform at run time through the use of the OCF_CHECK_LEVEL parameter.  They can be as lightweight as you want, or as "meaningful" as you want (where you get to define what meaningful means to you).

To top it all off, OCF resource agents also provide metadata describing all the paramters that they take and what they mean - potentially in multiple languages.

One other note about implementations of the OCF standard - all the ones I know about monitor resources on the machines where the resources are.  For example, they execute the monitor action on a machine which is running the database.  They then use other techniques to watch this watcher - as is common in reliable systems.

Since AMP implements the OCF resource agent specification (among others), this also means that it will eventually be able to use this API to start and stop resources as well as monitor them.  Extensions to the API also support migrating virtual machines - and there are resource agents written for most common types of virtual machines.

Related articles

How to implement "no news is good news" monitoring reliably

The network is reliable

Really Big Clusters: A Scalable membership proposal

This blog entry is a bit different than previous entries - I'm proposing some enhanced capabilities to go with the LRM and friends from the Linux-HA project.  I will update this entry on an ongoing basis to match my current thinking about this  proposal.

This post outlines a proposed server liveness ("membership") design which is intended to scale up from a dozen or so servers to tens of thousands of servers to be managed as an entity.

Scalability depends on a lot of factors - processor overhead, network bandwidth, and network load.  A highly scalable system will take all of these factors into account.  From the perspective of the server software author (for example, me), one of the easiest to overlook is network load.  Network load depends on a number of factors - number of packets, size of the packets, how many switches or routers it has to go through, and how many endpoints will receive the packet.  To best accomplish this task, it is desirable that the majority of "normal" traffic be network topology aware.  To scale up to very large collections of computers, it is also necessary that as much as possible be monitored as locally as possible.  In addition, since switching gear is not optimized for multicast packets, and multicast packets consume significant resources when compared to unicast packets, it is desirable to avoid using multicast packets during normal operation.

The Basic Concept - network aware liveness

Although the LRM in Linux-HA is not network-enabled, it tries to minimize monitoring overhead by distributing the task of monitoring resources to the machine providing the resource, and only reporting failures upstream.

To extend this idea of local monitoring into system liveness monitoring, one might imagine a standard 48-port network switch with 48 servers attached to it.  If one were to choose a server to act as proxy for monitoring the servers on that switch, then the other 47 nodes on the switch would send that unicast heartbeats to that node.  That node in turn would report on failure-to-receive-heartbeat events from the other 47 nodes on the switch.

Although this is simple to understand, it isn't ideal either - the work of detecting failures within a switch is  concentrated in a single node - and what happens when a switch fails.  So, let's try a different idea...

What if instead of having a hierarchy, the nodes within a switch arranged in a sort of doubly linked circular list of nodes in a switch - where each node sent heartbeats to the node "ahead" of or "behind" it on the list - making a ring-like structure.    This would spread the work for detecting failures across to more machines. For the 48-port switch, then each would send a heartbeat to the node with the next lower port number, and to the node with the next higher port number.  The first port would send heartbeats to port 2 and port 48, and port 48 would send heartbeats to ports 47 and 1. Since each node is monitored by two other nodes, monitoring has no single point of failure.   This design  is different from normal communication rings because there is no token being passed around the whole ring - just heartbeats going to a single node ahead and a single node behind.  So, conventional thoughts about latency in rings don't apply here.

If one were to implement this in the context of a cloud-like infrastructure, or a monitoring environment it is likely to be desirable to also run the LRM (or something like it) on these same machines so one can perform more detailed monitoring and/or effect changes to the managed servers.

Some Tentative Terminology

In order to facilitate discussion, I'll use the following terms (which are, of course, subject to change):

• Overlord - the policy-aware management entity at the top of the food chain.  The Overlord layer might be a cloud management layer, it might be a System management package like IBM director or Nagios, or a high-performance scientific cluster management entity, or some other entity interested in managing large numbers of computers.  The term overlord is used to refer to a management capability - not necessarily implemented in a single computer.
• Minion - a monitoring node which is listening to heartbeats from other nodes and reporting failure-to-receive-heartbeat events to an Overlord.  The minions at the lowest level in the system are only part-time minions - with the rest of their resources being spent on performing other tasks.  Dedicated minion servers should be capable of monitoring thousands (or tens of thousands) of other minions.  It is anticipated that the minion function would be performed by some software which also acts as a network proxy for an LRM client.  Minions send heartbeats, listen for heartbeats, report problems up the line, and act as LRM proxies.

Datagram Protocols

I anticipate the need for two different datagram transmission protocols, UDP (unreliable datagram protocol) and RDS (reliable datagram service).  UDP would be used for heartbeats, and RDS for communication between the Overlords and the LRM proxies (regardless of role).  It is expected that digital signatures are sufficient for heartbeats, but encryption may be desirable for some Overlord communication.

For this application, it is anticipated that the reliable datagram communication would not be high bandwidth, or likely need congestion control.  Other options in place of RDS would be an application protocol over UDP, or something like Qpid (AMQP).  I suspect any of these would work nicely.

LRM Proxy Design Goals

Here are a few design goals which are good to keep in mind during detailed design and implementation:

• Continue the policy-free approach that the LRM implements.  Although this implementation is intended to be topology aware, the LRM proxy code should rely on directions from is Overlord to tell it where and how often to send heartbeats, which heartbeats to expect (and how often), and other configuration information.  The Overlord is be the entity that "understands" the overall heartbeat topology.  Nevertheless, it is possible that the LRM proxy may gather some low-level network configuration for its Overlord.
• Assume a hostile network environment, and design communications accordingly.
• Minimize resource consumption - CPU, memory, network bandwidth should be held to a minimum.  Although this component is inherently lightweight, it will exist on every server in the collection - so resource waste is multiplied by thousands or tens of thousands of instances.
• Minimize latency.  Even if there is a hierarchy of watchers watching for failures doesn't mean that reporting problems should go back through the hierarchy.
• Minimize membership workload on non-dedicated minions.
• Try to design the API so that it is flexible enough and stable enough that version mismatches are unlikely to cause problems.
• Keep it simple.  The LRM proxy is a simple concept, the detailed design and implementation should be simple as well.

LRM Proxy Design Outline

The LRM proxy would implement the following capabilities:

• Send multicast (or broadcast) packets on boot-up to announce our presence
• Receiving requests from authenticated Overlords.  Requests currently anticpated include:
  • Requests to forward an encapsulated request to the LRM
  • Requests to send heartbeats to a certain address at a certain rate
  • Requests to stop sending heartbeats to a certain address
  • Requests to expect heartbeats from a certain address at a certain rate
  • Requests to stop expecting heartbeats from a certain address
  • Requests to add an address of an Overlord
  • Requests to delete an address of an Overlord
  • Requests to register a new Overlord authentication key
  • Requests to drop an Overlord authentication key
  • Requests to change the outgoing LRM proxy authentication key
  • Requests to stop sending boot-up multicasts
• Sending failure-to-receive-heartbeat messages to Overlords
• Sending LRM return result messages back to the registered Overlords

As implied above, at startup time, each LRM proxy would send out a multicast or broadcast packet announcing it was now up and ready to accept requests.  It will continue to send them until it receives an authenticated message from an overlord telling it to stop.  From then on the normal proxy API would be employed.

Some Thoughts about Heartbeating Topologies

This design could also be extended to create multiple levels of bidirectional rings between all the switches on a subnet, and between all the subnets.   The point is not that this is an optimal topology, but that the design of the LRM proxy component is topology-independent - it should support any unicast-based topology.

Below is an illustration of what a multi-level ring topology might look like.

For this smallish network, there are 54 servers.  12 of them send and receive heartbeats from 4 peers.  The remaining 42 have two peers - averaging 2.4 connections per server.  In a larger network, this number would be closer to 2.

If one wanted a higher degree of redundancy, one could imagine two shifted rings - with each node talking to node n-2, n-1, n+1 and n+2.  This would increase the number of packets sent and receive per interval to 4.  This could be used at all levels, or at selectively at higher levels.

There are no doubt other interesting topologies one could imagine as well.

 Advantages to this LRM proxy design

Here are a few advantages to this design:

• The LRM proxy design is simple and straightforward.
• Once a few releases have gone out, it is likely that the LRM proxy code and API will be stable for a long time.  This is a significant advantage for a component which is deployed on tens of thousands of computers.  It also simplifies upgrades, since it would be a very rare occurrence that an LRM proxy update would have to be performed in order to update the Overlord software.
• The LRM code will not have to change in order to fine tune the network resource usage.  Since where to send packets, and who to send them to and how often to send them are commands that come from its Overlord, the LRM proxy neither knows nor cares what the topology is.
• Although this was designed with network topology-awareness as a goal, it is not necessary that an Overlord implement topology-awareness, either initially, correctly, or at all.  The LRM proxy concept will function correctly without topology awareness, it will simply consume more network resources than necessary.

Potential problems with this design

• Providing switch topology information to the Overlord(s) is a potential difficulty.  There are some protocols implemented by some switches which provide information similar to what's required.  Further investigation is required to determine how difficult discovery and auto-configuration of network topology will be. The Link Layer Discovery Protocol (LLDP) may be of help here.
  
• The scope of the problem being addressed is very large.  The addition of layers of monitoring will likely introduce difficulties in debugging problems. Infrastructure for determining the origin of anomolous behavior should be provided.
• The flexibility in setting up monitoring topologies adds complexity to the system (even though it keeps it out of the LRM proxy).
• There is the possibility that the Overlord will receive conflicting reports regarding the liveness of a particular node.  This potential lack of consensus and timing issues will add complexity to the corner cases in the Overlord.

Open Issues

• There are many possible topologies for handling the upper layer Minion workload distribution.  It is not yet clear what the set of good choices are, and what the tradeoffs are between those different topologies.
• It is possible that it will prove desirable for this same infrastructure to collect statistical information for the Overlords.  Conversations with experts in high-performance computing and examination of tools like Ganglia will likely prove helpful in better understanding this problem.  What I have in mind is some amount of periodic piggybacking of data on the heartbeats, and then aggregation by nodes in minion roles, and periodic uplinks of this data to the overlords.  The goal would be to reduce the number of uplinks packets while having minimal impact on the timeliness of the data.  Not sure if this is a good idea or not.
• What about IPv6?

Concluding Thoughts

This proposal only provides liveness information and distributed control for a large collection of managed servers.  In many respects, this is perhaps the easiest component to scale up.  It is a long way from a complete cloud infrastructure, clustering software, or enterprise-scope system management package.  Scaling the Overlord components above this layer to the same standard of size is a very interesting and challenging task.  I suspect that this design is sufficiently scalable that it is likely that other components in the system architecture are likely to be the limiting factors in system scaling.

An Update

I have now posted a follow-on article outlining some ideas for the "overlord" portion of the infrastructure.

A Complete Cluster Stack for Linux

Recently, I've had some folks ask me offline what exactly would a “complete” Linux cluster stack look like.  That's a good question, and this posting is intended to address that question.

So let's start with – what kind of cluster?  For the purposes of this posting, I'm primarily talking about a full-function high-availability enterprise-style cluster, not primarily a load balancing cluster, and not a high-performance scientific (Beowulf-style) cluster.

A few caveats before proceeding – much of what I'll reference below will be relative to the Linux-HA[1] framework, but the concepts are easily translated to any other clustering framework one might have in mind.

It's also worth noting that not every application, nor every configuration needs every component.   Adding unnecessary components adds complexity, and complexity is the enemy of reliability.

Many of the components (cluster filesystem, DLM) are primarily needed by cluster-aware applications.  Note that at this time (early 2008) very few applications are cluster-aware.

The Full Cluster Stack Exposed!

Below is a picture of the full cluster stack – which I'll describe in more detail later.  For the most part, the components higher up in the picture build on the components lower down in the picture.  To simplify the drawing, I didn't add all the who-uses-whom lines that one might want to make a detailed study of this subject.

 

Cluster Comm - Intracluster  communications

The most basic component any cluster needs is intracluster communications. There are a variety of different possibilities, but guaranteed packet delivery is a requirement.  Linux-HA has its own custom comm layer for doing this.  It's not perfect, but it works.  At one time in the past, we provided support for the AIS cluster APIs, and if you use OpenAIS today, then you can still have a reasonable cluster using Linux-HA and providing compatible support for the AIS protocols.  As will become clear, it's not a perfect configuration, but it's a reasonable one.  (Of course, like everything, it can always be improved even more)

Very large clusters (hundreds to tens of thousands of nodes) will likely require a different communication protocol, since most guaranteed delivery multicast protocols don't scale that high. 

Nevertheless, in an ideal world, all cluster components and cluster-aware applications would sit on top of the same set of communications protocols.

Membership – who's in the cluster?

Looking to the right of the cluster comm box on our architecture chart, you'll see the membership box.  The next basic function that a cluster has to provide is membership services.  Membership closely related to communication – since a simplistic view of membership is just who we can communicate with.  It is highly desirable that everyone in the cluster be able to communicate with everyone else. It's the job of the membership layer to provide this information to the cluster.

When your communication fails in weird ways, it's the job of the membership layer to present a view of the cluster that makes sense – in spite of the weird kinds of failures that might be going on.

If we eventually wind up with multiple kinds of communications methods, then we'll also have multiple ways of becoming a member.

Linux-HA (with or without OpenAIS) supports the AIS membership APIs.

Like I mentioned for communication, in an ideal world,  all cluster components would use exactly the same membership information.  However, it is important to note that the membership one uses must be computed using the communication method being used by the application.  So, unless every cluster-aware application uses both the common communication method and the common membership, it risks getting its membership out of sync with respect to its communication and other components using other communication methods.  In many cases, this can't be avoided.  Methods for coping with such discrepencies are discussed in more detail at the end of this post.

Fencing

Fencing is the ability to “disable” nodes not currently in our membership without their cooperation  Many of you will remember having discussed this in some detail in an earlier post[2]. As I explained in more detail there, fencing  is vital to ensure safe cluster operation

Our current implementation is STONITH[3]-based - STONITH == Shoot The Other Node In The Head

Quorum

Quorum is a topic we've talked about extensively in a couple of earlier posts [2] [4]. Quorum encapsulates the ability to determine whether the cluster can continue to operate safely or not

Quorum is tied closely to both fencing and membership.  In practice, as we discussed before, it is often highly desirable to implement multiple types of quorum.  Linux-HA currently provides multiple implementations and can provide more through plugins.  Like membership and communication, it is desirable for all cluster components to use the same quorum mechanism.  All the interesting and legal ways that quorum can interact with fencing and membership and the communication layer are too detailed for this posting.

Cluster Filesystems

Cluster filesystems allow multiple machines to sanely mount the same filesystem at the same time.  This is a great boon to parallel applications.  Cluster filesystems typically don't use the network or another server involved when doing bulk I/O. Each node mounting the filesystem is normally expected to have access to the data.  This typically requires a SAN.

Typically, this is done to improve performance, but convenience and manageability are common secondary goals  Cluster filesystems are related to, but distinct from, network filesystems like NFS and CIFS. On Linux, there are several cluster filesystems available including

• Red Hat's GFS

• Oracle's OCFS2

• IBM's GPFS

• etc.

Normally, when they're being used for performance reasons, cluster-aware applications are required.  You can't typically just run 'n' copies of your favorite cluster-ignorant application and have it work.  The filesystem won't scramble the data, but your application typically will.  It goes without saying that high-performance cluster filesystems run in the kernel – unlike all the other items we've talked about before.  Because of the high-performance, it's common for a cluster filesystem to have its own communication and membership code – not using the typically userland communications and membership code.  Since membership isn't high bandwidth or really low latency information, it is possible to feed membership from a user-space membership layer into the kernel.  Of course, then the membership and the communications layer are out of sync.  It is arguable whether this is an improvement or not.

Cluster filesystems typically need cluster lock managers – described in the next section.

DLM - Distributed Lock Manager

A DLM (Distributed Lock Manager) provides locking services across the cluster, and it's an interesting piece of code to implement them – particularly the error recovery. 

To some degree, DLMs are analogous to System V semaphores but – cluster-aware.  In addition, they provide much more sophisticated API and semantics.  Although DLM APIs are fairly well understood, there is no formal standard, so switching from one to another can be annoying.  Red Hat has a reasonable kernel-based DLM which they use with GFS.  DLMs commonly have their own separate communications and membership code.  The comments about getting membership from user-space and having them be potentially different from cluster filesytems also apply here.

Cluster Volume Managers

You might think that you really don't need a cluster-aware volume manager.  Sometimes you might be right.  More often, if you thought that, you'd be wrong.   A cluster volume manager is just like a regular volume manager – only cluster aware.  This is to keep different nodes from getting inconsistent views of the layout of a set of disks or volumes.   The current cluster-aware volume managers are EVMS and CLVM.  Only CLVM is expected to survive into the long term.

The big challenges for cluster volume managers are high-performance mirroring and snapshots.  These operations are potentially very difficult to implement right and fast.  Cluster-aware volume managers often have both kernel and user-space components.  The membership inconsistency issues here are similar to those for cluster filesystems and the DLM.

CRM - Cluster Resource Manager

Every HA cluster has something like a CRM, but they may divide up these functions differently.  Our CRM is a policy-based decision maker for what should run where – handling failed services and failed cluster nodes.

The CRM is similar to UNIX/Linux startup init scripts – it starts everything up – but across a cluster following some policies, and managing failures.

The Linux-HA CRM is arguably the best cluster resource manager around today – at least in terms of flexibility and power.  It has usability issues, and can be extended, but those are solvable.

The Linux-HA CRM function is largely divided between the PE and TE – which are described below.

PE - Policy Engine

The Policy Engine is a key component of the CRM and does two distinct things.

• It determines what should run where (cluster layout)

• It creates a graph of actions of how to get from the current state of affairs to the new desired state

This graph of actions is then given to the TE (described below).

The system would have more flexibility if he PE were split into two parts for these two functions, and supported plugins for the cluster layout function. 

It currently isn't aware of resource cost, nor of absolute resource limits and load balancing considerations, which complicate optimal placement.   Those would be good things to add to it in the future.  Having plugins for doing resource placement would also be a highly useful and desirable thing.

TE - Transition Engine

Receives a graph of actions to perform from the policy engine, then uses the LRM proxy to communicate with the LRMs to carry out the actions

Its main jobs are action sequencing, error detection and reporting

CIB - Cluster Information Base

The CIB manages information on cluster configuration and current status.  The cluster configuration includes the configuration and policies as defined by the system administrator.

Its key difficulty is to keep a consistent copy replicated across the cluster, resolving potential version differences.

All the data it manages is XML, and the CIB has a minimal knowledge of the structure of this XML.

LRM - Local Resource Manager

In the Linux-HA architecture, a local resource manager runs on every machine and carries out the tasks given to it. Everything that gets done gets carried out by the LRM.  Examples are:

• start this resource

• stop this resource

• monitor this resource

• migrate this resource

• etc.

The LRM provides interface matching to the various kinds of resources through Resource Agents.  The Linux-HA LRM supports several classes of Resource Agents.

The LRM is not at all cluster-aware.  It can support an arbitrary number of clients, one of which is the LRM communications proxy (below).

LRM Communications Proxy

The LRM proxy communicates between the CRM and the LRMs on all the various machines.  This function is currently built into the CRM.  This architectural decision was based on expedience more than anything else.

To support larger clusters this needs to be separated out, made more scalable, and more flexible.  This would allow a large number of LRMs to be supported by a small number of LRM proxies.   In large systems, this would probably use the ClusterIP capability to provide load distribution (leveling) across multiple LRM proxies.

Init - Initialization and recovery

This code does really three things:

• Sequences the startup of the cluster components

• Recovers from component failures (restart or reboot)

• Sequences the shutdown of all the various cluster components

This is currently provided by Linux-HA and bundled with the Linux-HA communications code.  This likely needs to be separated out to a separate proxy function (process) in the future.

Infrastructure

The Linux-HA infrastructure libraries (“clplumbing”) does a wide variety of things.  A few samples include:

• Inter-Process Communication

• Process management

• Event management and scheduling

• Many other miscellaneous functions

Surprisingly, these libraries amount to about 20K lines of code.

Quorum Daemon

The quorum daemon is an unusual daemon, because it's the only daemon we have that's intended to run outside the cluster proper.  It is instrumental in solving certain knotty quorum problems – especially for:

• 2-node clusters (very common)

• Split-site (disaster recovery) clusters

This was discussed extensively in previous postings [4] and [5].

Management Daemon

Provides Complete Authenticated Configuration and Status API.  This includes both information contained in the CIB, and also information about the communications configuration and so on.

The management daemon is used by the:

• GUI

• SNMP agent

• CIM agent

Clients are authenticated using PAM, and all communications is via SSL, so its clients can safely be outside cluster, or even outside a firewall.  This daemon should provide different levels of authorization depending on the authenticated user, and should log its actions in a format suitable for Sarbanes-Oxley (SOX) auditing purposes.

GUI

Provides a Graphical User Interface providing configuration and status information.  It also supports creating and configuring the cluster.  Note that at the present time, there are a number of useful cluster configurations it cannot create.

CIM and SNMP agents

The CIM and SNMP agents provide CIM and SNMP management interfaces for systems management tools.  The CIM interface supports status updates and configuration changes, whereas the SNMP interfaces only report status.

Disadvantages of this architecture

For a variety of reasons, kernel space doesn't have access to user-space cluster communications or membership.

As a result, both the DLM and most cluster filesytems implements their own membership and communications.

This is in contradiction to the “ideal world” statements earlier.  This can result in some odd cases where one communication method is working in a particular case, but another method is not.  This results in differences in membership – which can have bad effects.

Why this might not be quite as bad as it seems

One reason why one might not worry about this as much as one might, is because it's a problem which one can't make go away.  A cluster system will always have to interface with software packages which do their own communication, and compute their own membership for a variety of usually good reasons.  As a result, this is a problem which we can't make go away.  Instead we have to deal with it effectively.  There are basically two cases to consider:

1. The “Main” membership thinks that node X should not be in the cluster, whereas the “Other” membership thinks it should be.

2. The “Other” membership thinks that node X should not be in the cluster, whereas the “Main” membership thinks it should be.

Let's take these two cases one at a time:

Case 1:

If the main membership thinks node X is not in the cluster, then it will simply not start any resources on node X.  This takes care of the problem.

Case 2:

If the “Other” membership discovers that a particular node should be dropped from its view of membership, and it can inform the CRM not to start its resources on that machine, then the local view of this membership from the perspective of the resources it deals with is effectively made to exclude these Other-errant nodes.  In the Linux-HA CRM this is easily done having the Other-resources write node attributes to cause those nodes to be excluded, and the rules would then be written to exclude those nodes from consideration for running Other-related resources.

Although Case 2 isn't pretty, it works, and no amount of wishing and hoping is likely to ever make this kind of problem go away in the general case - particularly when one involves proprietary applications  So, even if there is some membership discrepancy, it can is always possible to manage it appropriately assuming you can get a tiny bit of cooperation from the application.

References

[1] http://linux-ha.org/
[2] http://techthoughts.typepad.com/managing_computers/2007/10/split-brain-quo.html
[3] http://linux-ha.org/STONITH
[4] http://techthoughts.typepad.com/managing_computers/2007/10/more-about-quor.html
[5] http://techthoughts.typepad.com/managing_computers/2007/11/quorum-server-i.html

Availability, MTBF, MTTR and other bedtime tales

If we let A represent availability, then the simplest formula for availability is:
    A = Uptime/(Uptime + Downtime)

Of course, it's more interesting when you start looking at the things that influence uptime and downtime.  The most common measures that can be used in this way are MTBF and MTTR.

    MTBF is  Mean Time Between Failures
    MTTR is Mean Time To Repair   

 A = MTBF / (MTBF+MTTR)

One interesting observation you can make when reading this formula is that if you could instantly repair everything (MTTR = 0), then it wouldn't matter what the MTBF is - Availability would be 100% (1) all the time.

That's exactly what HA clustering tries to do.  It tries to make the MTTR as close to zero as it can by automatically (autonomically) switching in redundant components for failed components as fast as it can.   Depending on the application architecture and how fast failure can be detected and repaired, a given failure might not be observable by at all by a client of the service.  If it's not observable by the client, then in some sense it didn't happen at all.  This idea of viewing things from the client's perspective is an important one in a practical sense, and I'll talk about that some more later on.

It's important to realize that any given data center, or cluster provides many services, and not all of them are related to each other.  Failure of one component in the system may not cause failure of the system.  Indeed, good HA design eliminates single points of failure by introducing redundancy.  If you're going to try and calculate MTBF in a real-life (meaning complex) environment with redundancy and interrelated services, it's going to be very complicated to do.

    MTBFx is  Mean Time Between Failures for entity x
    MTTRx is Mean Time To Repair for entity x
    Ax is the Availability of entity x   

Ax = MTBFx / (MTBFx+MTTRx)

In practice, these measures (MTBFx and MTTRx) are hard to come by for nontrivial real systems - in fact, they're so tied in to application reliability and architecture, hardware architecture, deployment strategy, operational skill and training, and a whole host of other factors, that you can actually compute them only very very rarely.  So, why did I spend your time talking about it?  That's simple - although you probably won't compute them, you can learn some important things from these formulas, and you can see how mistakes you make in viewing these formulas might lead you to some wrong conclusions.

Let's get right into one example of a wrong conclusion you might draw from incorrectly applying these formulas.

Let's say we have a service which runs on a single machine, which you put onto a cluster composed of two computers with a certain individual MTBF (Mi) and you can fail over to the other computer ("repair") a computer in a certain repair time (Ri).  With two computers, they'll fail twice as often as a single computer, so the system MTBF becomes Mi/2.  If you compute the availability of the cluster, it then becomes:

    A = Mi/2 / (Mi/2+Ri)

Using this (incorrect) analysis for a 1000 node cluster performing the same service, the system MTBF becomes Mi/1000.

    A = Mi/1000 / (Mi/1000+Ri)

If you take the number of nodes in the cluster to the limit (approaching infinity), the Availability approaches zero.

    A = 0/(0+Ri) = 0/Ri = 0

This makes it appear that adding cluster nodes decreases availability.  Is this really true?  Of course not!  The mistake here is thinking that the service needed all those  cluster nodes to make it go.  If your service was a complicated interlocking scientific computation that would stop if any cluster node failed, then this model might be correct.  But if the other nodes were providing redundancy or unrelated services, then they would have no effect on MTBF of the service in question.  Of course, as they break, you'd have to repair them, which would mean replacing systems more and more often, which would be both annoying and expensive, but it wouldn't cause the service availability to go down.

To properly apply these formulas, even intuitively, you need to make sure you understand what your service is, how you define a failure, how the service components relate to each other, and what happens when one of them fails.  Here are a few rules of thumb for thinking about availability

• Complexity is the enemy of reliability (MTTR).  This can take many forms
  
  • Complex software fails more often than simple software
  • Complex hardware fails more often than simple hardware
  • Software dependencies usually mean that if any component fails, the whole service fails
  • Configuration complexity lowers the chances of the configuration being correct
  • Complexity drastically increases the possibility of human error
    • What is complex software? - Software whose model of the universe doesn't match that of the staff who manage it.
• Redundancy is the friend of availability - it allows for quick autonomic recovery - significantly improving MTTR.  Replication is another word for redundancy.
• Good failure detection is vital - HA and other autonomic software can only recover from failures it detects.  Undetected failures have human-speed MTTR or worse, not autonomic-speed MTTR.  They can be worse than human-speed MTTR because the humans are surprised that it wasn't automatically recovered and they respond more slowly than normal.  In addition, the added complexity of correcting an autonomic service and trying to keep their fingers out of the gears may slow down their thought processes.
• Non-essential components don't count - failure of inactive or non-essential components doesn't affect service availability.  These inactive components can be hardware (spare machines), or software (like administrative interfaces), or hardware only being used to run non-essential software.  More generally, for the purpose of calculating the availability of service X, non-essential components include anything not running service X or services essential to X.

The real world is much more complex than any simple rules of thumb like these, but these are certainly worth taking into account.