Managing Computers with Automation

Really Big Clusters: A Scalable membership proposal

This blog entry is a bit different than previous entries - I'm proposing some enhanced capabilities to go with the LRM and friends from the Linux-HA project.  I will update this entry on an ongoing basis to match my current thinking about this  proposal.

This post outlines a proposed server liveness ("membership") design which is intended to scale up from a dozen or so servers to tens of thousands of servers to be managed as an entity.

Scalability depends on a lot of factors - processor overhead, network bandwidth, and network load.  A highly scalable system will take all of these factors into account.  From the perspective of the server software author (for example, me), one of the easiest to overlook is network load.  Network load depends on a number of factors - number of packets, size of the packets, how many switches or routers it has to go through, and how many endpoints will receive the packet.  To best accomplish this task, it is desirable that the majority of "normal" traffic be network topology aware.  To scale up to very large collections of computers, it is also necessary that as much as possible be monitored as locally as possible.  In addition, since switching gear is not optimized for multicast packets, and multicast packets consume significant resources when compared to unicast packets, it is desirable to avoid using multicast packets during normal operation.

The Basic Concept - network aware liveness

Although the LRM in Linux-HA is not network-enabled, it tries to minimize monitoring overhead by distributing the task of monitoring resources to the machine providing the resource, and only reporting failures upstream.

To extend this idea of local monitoring into system liveness monitoring, one might imagine a standard 48-port network switch with 48 servers attached to it.  If one were to choose a server to act as proxy for monitoring the servers on that switch, then the other 47 nodes on the switch would send that unicast heartbeats to that node.  That node in turn would report on failure-to-receive-heartbeat events from the other 47 nodes on the switch.

Although this is simple to understand, it isn't ideal either - the work of detecting failures within a switch is  concentrated in a single node - and what happens when a switch fails.  So, let's try a different idea...

What if instead of having a hierarchy, the nodes within a switch arranged in a sort of doubly linked circular list of nodes in a switch - where each node sent heartbeats to the node "ahead" of or "behind" it on the list - making a ring-like structure.    This would spread the work for detecting failures across to more machines. For the 48-port switch, then each would send a heartbeat to the node with the next lower port number, and to the node with the next higher port number.  The first port would send heartbeats to port 2 and port 48, and port 48 would send heartbeats to ports 47 and 1. Since each node is monitored by two other nodes, monitoring has no single point of failure.   This design  is different from normal communication rings because there is no token being passed around the whole ring - just heartbeats going to a single node ahead and a single node behind.  So, conventional thoughts about latency in rings don't apply here.

If one were to implement this in the context of a cloud-like infrastructure, or a monitoring environment it is likely to be desirable to also run the LRM (or something like it) on these same machines so one can perform more detailed monitoring and/or effect changes to the managed servers.

Some Tentative Terminology

In order to facilitate discussion, I'll use the following terms (which are, of course, subject to change):

• Overlord - the policy-aware management entity at the top of the food chain.  The Overlord layer might be a cloud management layer, it might be a System management package like IBM director or Nagios, or a high-performance scientific cluster management entity, or some other entity interested in managing large numbers of computers.  The term overlord is used to refer to a management capability - not necessarily implemented in a single computer.
• Minion - a monitoring node which is listening to heartbeats from other nodes and reporting failure-to-receive-heartbeat events to an Overlord.  The minions at the lowest level in the system are only part-time minions - with the rest of their resources being spent on performing other tasks.  Dedicated minion servers should be capable of monitoring thousands (or tens of thousands) of other minions.  It is anticipated that the minion function would be performed by some software which also acts as a network proxy for an LRM client.  Minions send heartbeats, listen for heartbeats, report problems up the line, and act as LRM proxies.

Datagram Protocols

I anticipate the need for two different datagram transmission protocols, UDP (unreliable datagram protocol) and RDS (reliable datagram service).  UDP would be used for heartbeats, and RDS for communication between the Overlords and the LRM proxies (regardless of role).  It is expected that digital signatures are sufficient for heartbeats, but encryption may be desirable for some Overlord communication.

For this application, it is anticipated that the reliable datagram communication would not be high bandwidth, or likely need congestion control.  Other options in place of RDS would be an application protocol over UDP, or something like Qpid (AMQP).  I suspect any of these would work nicely.

LRM Proxy Design Goals

Here are a few design goals which are good to keep in mind during detailed design and implementation:

• Continue the policy-free approach that the LRM implements.  Although this implementation is intended to be topology aware, the LRM proxy code should rely on directions from is Overlord to tell it where and how often to send heartbeats, which heartbeats to expect (and how often), and other configuration information.  The Overlord is be the entity that "understands" the overall heartbeat topology.  Nevertheless, it is possible that the LRM proxy may gather some low-level network configuration for its Overlord.
• Assume a hostile network environment, and design communications accordingly.
• Minimize resource consumption - CPU, memory, network bandwidth should be held to a minimum.  Although this component is inherently lightweight, it will exist on every server in the collection - so resource waste is multiplied by thousands or tens of thousands of instances.
• Minimize latency.  Even if there is a hierarchy of watchers watching for failures doesn't mean that reporting problems should go back through the hierarchy.
• Minimize membership workload on non-dedicated minions.
• Try to design the API so that it is flexible enough and stable enough that version mismatches are unlikely to cause problems.
• Keep it simple.  The LRM proxy is a simple concept, the detailed design and implementation should be simple as well.

LRM Proxy Design Outline

The LRM proxy would implement the following capabilities:

• Send multicast (or broadcast) packets on boot-up to announce our presence
• Receiving requests from authenticated Overlords.  Requests currently anticpated include:
  • Requests to forward an encapsulated request to the LRM
  • Requests to send heartbeats to a certain address at a certain rate
  • Requests to stop sending heartbeats to a certain address
  • Requests to expect heartbeats from a certain address at a certain rate
  • Requests to stop expecting heartbeats from a certain address
  • Requests to add an address of an Overlord
  • Requests to delete an address of an Overlord
  • Requests to register a new Overlord authentication key
  • Requests to drop an Overlord authentication key
  • Requests to change the outgoing LRM proxy authentication key
  • Requests to stop sending boot-up multicasts
• Sending failure-to-receive-heartbeat messages to Overlords
• Sending LRM return result messages back to the registered Overlords

As implied above, at startup time, each LRM proxy would send out a multicast or broadcast packet announcing it was now up and ready to accept requests.  It will continue to send them until it receives an authenticated message from an overlord telling it to stop.  From then on the normal proxy API would be employed.

Some Thoughts about Heartbeating Topologies

This design could also be extended to create multiple levels of bidirectional rings between all the switches on a subnet, and between all the subnets.   The point is not that this is an optimal topology, but that the design of the LRM proxy component is topology-independent - it should support any unicast-based topology.

Below is an illustration of what a multi-level ring topology might look like.

For this smallish network, there are 54 servers.  12 of them send and receive heartbeats from 4 peers.  The remaining 42 have two peers - averaging 2.4 connections per server.  In a larger network, this number would be closer to 2.

If one wanted a higher degree of redundancy, one could imagine two shifted rings - with each node talking to node n-2, n-1, n+1 and n+2.  This would increase the number of packets sent and receive per interval to 4.  This could be used at all levels, or at selectively at higher levels.

There are no doubt other interesting topologies one could imagine as well.

 Advantages to this LRM proxy design

Here are a few advantages to this design:

• The LRM proxy design is simple and straightforward.
• Once a few releases have gone out, it is likely that the LRM proxy code and API will be stable for a long time.  This is a significant advantage for a component which is deployed on tens of thousands of computers.  It also simplifies upgrades, since it would be a very rare occurrence that an LRM proxy update would have to be performed in order to update the Overlord software.
• The LRM code will not have to change in order to fine tune the network resource usage.  Since where to send packets, and who to send them to and how often to send them are commands that come from its Overlord, the LRM proxy neither knows nor cares what the topology is.
• Although this was designed with network topology-awareness as a goal, it is not necessary that an Overlord implement topology-awareness, either initially, correctly, or at all.  The LRM proxy concept will function correctly without topology awareness, it will simply consume more network resources than necessary.

Potential problems with this design

• Providing switch topology information to the Overlord(s) is a potential difficulty.  There are some protocols implemented by some switches which provide information similar to what's required.  Further investigation is required to determine how difficult discovery and auto-configuration of network topology will be. The Link Layer Discovery Protocol (LLDP) may be of help here.
  
• The scope of the problem being addressed is very large.  The addition of layers of monitoring will likely introduce difficulties in debugging problems. Infrastructure for determining the origin of anomolous behavior should be provided.
• The flexibility in setting up monitoring topologies adds complexity to the system (even though it keeps it out of the LRM proxy).
• There is the possibility that the Overlord will receive conflicting reports regarding the liveness of a particular node.  This potential lack of consensus and timing issues will add complexity to the corner cases in the Overlord.

Open Issues

• There are many possible topologies for handling the upper layer Minion workload distribution.  It is not yet clear what the set of good choices are, and what the tradeoffs are between those different topologies.
• It is possible that it will prove desirable for this same infrastructure to collect statistical information for the Overlords.  Conversations with experts in high-performance computing and examination of tools like Ganglia will likely prove helpful in better understanding this problem.  What I have in mind is some amount of periodic piggybacking of data on the heartbeats, and then aggregation by nodes in minion roles, and periodic uplinks of this data to the overlords.  The goal would be to reduce the number of uplinks packets while having minimal impact on the timeliness of the data.  Not sure if this is a good idea or not.
• What about IPv6?

Concluding Thoughts

This proposal only provides liveness information and distributed control for a large collection of managed servers.  In many respects, this is perhaps the easiest component to scale up.  It is a long way from a complete cloud infrastructure, clustering software, or enterprise-scope system management package.  Scaling the Overlord components above this layer to the same standard of size is a very interesting and challenging task.  I suspect that this design is sufficiently scalable that it is likely that other components in the system architecture are likely to be the limiting factors in system scaling.

An Update

I have now posted a follow-on article outlining some ideas for the "overlord" portion of the infrastructure.

A Complete Cluster Stack for Linux

Recently, I've had some folks ask me offline what exactly would a “complete” Linux cluster stack look like.  That's a good question, and this posting is intended to address that question.

So let's start with – what kind of cluster?  For the purposes of this posting, I'm primarily talking about a full-function high-availability enterprise-style cluster, not primarily a load balancing cluster, and not a high-performance scientific (Beowulf-style) cluster.

A few caveats before proceeding – much of what I'll reference below will be relative to the Linux-HA[1] framework, but the concepts are easily translated to any other clustering framework one might have in mind.

It's also worth noting that not every application, nor every configuration needs every component.   Adding unnecessary components adds complexity, and complexity is the enemy of reliability.

Many of the components (cluster filesystem, DLM) are primarily needed by cluster-aware applications.  Note that at this time (early 2008) very few applications are cluster-aware.

The Full Cluster Stack Exposed!

Below is a picture of the full cluster stack – which I'll describe in more detail later.  For the most part, the components higher up in the picture build on the components lower down in the picture.  To simplify the drawing, I didn't add all the who-uses-whom lines that one might want to make a detailed study of this subject.

 

Cluster Comm - Intracluster  communications

The most basic component any cluster needs is intracluster communications. There are a variety of different possibilities, but guaranteed packet delivery is a requirement.  Linux-HA has its own custom comm layer for doing this.  It's not perfect, but it works.  At one time in the past, we provided support for the AIS cluster APIs, and if you use OpenAIS today, then you can still have a reasonable cluster using Linux-HA and providing compatible support for the AIS protocols.  As will become clear, it's not a perfect configuration, but it's a reasonable one.  (Of course, like everything, it can always be improved even more)

Very large clusters (hundreds to tens of thousands of nodes) will likely require a different communication protocol, since most guaranteed delivery multicast protocols don't scale that high. 

Nevertheless, in an ideal world, all cluster components and cluster-aware applications would sit on top of the same set of communications protocols.

Membership – who's in the cluster?

Looking to the right of the cluster comm box on our architecture chart, you'll see the membership box.  The next basic function that a cluster has to provide is membership services.  Membership closely related to communication – since a simplistic view of membership is just who we can communicate with.  It is highly desirable that everyone in the cluster be able to communicate with everyone else. It's the job of the membership layer to provide this information to the cluster.

When your communication fails in weird ways, it's the job of the membership layer to present a view of the cluster that makes sense – in spite of the weird kinds of failures that might be going on.

If we eventually wind up with multiple kinds of communications methods, then we'll also have multiple ways of becoming a member.

Linux-HA (with or without OpenAIS) supports the AIS membership APIs.

Like I mentioned for communication, in an ideal world,  all cluster components would use exactly the same membership information.  However, it is important to note that the membership one uses must be computed using the communication method being used by the application.  So, unless every cluster-aware application uses both the common communication method and the common membership, it risks getting its membership out of sync with respect to its communication and other components using other communication methods.  In many cases, this can't be avoided.  Methods for coping with such discrepencies are discussed in more detail at the end of this post.

Fencing

Fencing is the ability to “disable” nodes not currently in our membership without their cooperation  Many of you will remember having discussed this in some detail in an earlier post[2]. As I explained in more detail there, fencing  is vital to ensure safe cluster operation

Our current implementation is STONITH[3]-based - STONITH == Shoot The Other Node In The Head

Quorum

Quorum is a topic we've talked about extensively in a couple of earlier posts [2] [4]. Quorum encapsulates the ability to determine whether the cluster can continue to operate safely or not

Quorum is tied closely to both fencing and membership.  In practice, as we discussed before, it is often highly desirable to implement multiple types of quorum.  Linux-HA currently provides multiple implementations and can provide more through plugins.  Like membership and communication, it is desirable for all cluster components to use the same quorum mechanism.  All the interesting and legal ways that quorum can interact with fencing and membership and the communication layer are too detailed for this posting.

Cluster Filesystems

Cluster filesystems allow multiple machines to sanely mount the same filesystem at the same time.  This is a great boon to parallel applications.  Cluster filesystems typically don't use the network or another server involved when doing bulk I/O. Each node mounting the filesystem is normally expected to have access to the data.  This typically requires a SAN.

Typically, this is done to improve performance, but convenience and manageability are common secondary goals  Cluster filesystems are related to, but distinct from, network filesystems like NFS and CIFS. On Linux, there are several cluster filesystems available including

• Red Hat's GFS

• Oracle's OCFS2

• IBM's GPFS

• etc.

Normally, when they're being used for performance reasons, cluster-aware applications are required.  You can't typically just run 'n' copies of your favorite cluster-ignorant application and have it work.  The filesystem won't scramble the data, but your application typically will.  It goes without saying that high-performance cluster filesystems run in the kernel – unlike all the other items we've talked about before.  Because of the high-performance, it's common for a cluster filesystem to have its own communication and membership code – not using the typically userland communications and membership code.  Since membership isn't high bandwidth or really low latency information, it is possible to feed membership from a user-space membership layer into the kernel.  Of course, then the membership and the communications layer are out of sync.  It is arguable whether this is an improvement or not.

Cluster filesystems typically need cluster lock managers – described in the next section.

DLM - Distributed Lock Manager

A DLM (Distributed Lock Manager) provides locking services across the cluster, and it's an interesting piece of code to implement them – particularly the error recovery. 

To some degree, DLMs are analogous to System V semaphores but – cluster-aware.  In addition, they provide much more sophisticated API and semantics.  Although DLM APIs are fairly well understood, there is no formal standard, so switching from one to another can be annoying.  Red Hat has a reasonable kernel-based DLM which they use with GFS.  DLMs commonly have their own separate communications and membership code.  The comments about getting membership from user-space and having them be potentially different from cluster filesytems also apply here.

Cluster Volume Managers

You might think that you really don't need a cluster-aware volume manager.  Sometimes you might be right.  More often, if you thought that, you'd be wrong.   A cluster volume manager is just like a regular volume manager – only cluster aware.  This is to keep different nodes from getting inconsistent views of the layout of a set of disks or volumes.   The current cluster-aware volume managers are EVMS and CLVM.  Only CLVM is expected to survive into the long term.

The big challenges for cluster volume managers are high-performance mirroring and snapshots.  These operations are potentially very difficult to implement right and fast.  Cluster-aware volume managers often have both kernel and user-space components.  The membership inconsistency issues here are similar to those for cluster filesystems and the DLM.

CRM - Cluster Resource Manager

Every HA cluster has something like a CRM, but they may divide up these functions differently.  Our CRM is a policy-based decision maker for what should run where – handling failed services and failed cluster nodes.

The CRM is similar to UNIX/Linux startup init scripts – it starts everything up – but across a cluster following some policies, and managing failures.

The Linux-HA CRM is arguably the best cluster resource manager around today – at least in terms of flexibility and power.  It has usability issues, and can be extended, but those are solvable.

The Linux-HA CRM function is largely divided between the PE and TE – which are described below.

PE - Policy Engine

The Policy Engine is a key component of the CRM and does two distinct things.

• It determines what should run where (cluster layout)

• It creates a graph of actions of how to get from the current state of affairs to the new desired state

This graph of actions is then given to the TE (described below).

The system would have more flexibility if he PE were split into two parts for these two functions, and supported plugins for the cluster layout function. 

It currently isn't aware of resource cost, nor of absolute resource limits and load balancing considerations, which complicate optimal placement.   Those would be good things to add to it in the future.  Having plugins for doing resource placement would also be a highly useful and desirable thing.

TE - Transition Engine

Receives a graph of actions to perform from the policy engine, then uses the LRM proxy to communicate with the LRMs to carry out the actions

Its main jobs are action sequencing, error detection and reporting

CIB - Cluster Information Base

The CIB manages information on cluster configuration and current status.  The cluster configuration includes the configuration and policies as defined by the system administrator.

Its key difficulty is to keep a consistent copy replicated across the cluster, resolving potential version differences.

All the data it manages is XML, and the CIB has a minimal knowledge of the structure of this XML.

LRM - Local Resource Manager

In the Linux-HA architecture, a local resource manager runs on every machine and carries out the tasks given to it. Everything that gets done gets carried out by the LRM.  Examples are:

• start this resource

• stop this resource

• monitor this resource

• migrate this resource

• etc.

The LRM provides interface matching to the various kinds of resources through Resource Agents.  The Linux-HA LRM supports several classes of Resource Agents.

The LRM is not at all cluster-aware.  It can support an arbitrary number of clients, one of which is the LRM communications proxy (below).

LRM Communications Proxy

The LRM proxy communicates between the CRM and the LRMs on all the various machines.  This function is currently built into the CRM.  This architectural decision was based on expedience more than anything else.

To support larger clusters this needs to be separated out, made more scalable, and more flexible.  This would allow a large number of LRMs to be supported by a small number of LRM proxies.   In large systems, this would probably use the ClusterIP capability to provide load distribution (leveling) across multiple LRM proxies.

Init - Initialization and recovery

This code does really three things:

• Sequences the startup of the cluster components

• Recovers from component failures (restart or reboot)

• Sequences the shutdown of all the various cluster components

This is currently provided by Linux-HA and bundled with the Linux-HA communications code.  This likely needs to be separated out to a separate proxy function (process) in the future.

Infrastructure

The Linux-HA infrastructure libraries (“clplumbing”) does a wide variety of things.  A few samples include:

• Inter-Process Communication

• Process management

• Event management and scheduling

• Many other miscellaneous functions

Surprisingly, these libraries amount to about 20K lines of code.

Quorum Daemon

The quorum daemon is an unusual daemon, because it's the only daemon we have that's intended to run outside the cluster proper.  It is instrumental in solving certain knotty quorum problems – especially for:

• 2-node clusters (very common)

• Split-site (disaster recovery) clusters

This was discussed extensively in previous postings [4] and [5].

Management Daemon

Provides Complete Authenticated Configuration and Status API.  This includes both information contained in the CIB, and also information about the communications configuration and so on.

The management daemon is used by the:

• GUI

• SNMP agent

• CIM agent

Clients are authenticated using PAM, and all communications is via SSL, so its clients can safely be outside cluster, or even outside a firewall.  This daemon should provide different levels of authorization depending on the authenticated user, and should log its actions in a format suitable for Sarbanes-Oxley (SOX) auditing purposes.

GUI

Provides a Graphical User Interface providing configuration and status information.  It also supports creating and configuring the cluster.  Note that at the present time, there are a number of useful cluster configurations it cannot create.

CIM and SNMP agents

The CIM and SNMP agents provide CIM and SNMP management interfaces for systems management tools.  The CIM interface supports status updates and configuration changes, whereas the SNMP interfaces only report status.

Disadvantages of this architecture

For a variety of reasons, kernel space doesn't have access to user-space cluster communications or membership.

As a result, both the DLM and most cluster filesytems implements their own membership and communications.

This is in contradiction to the “ideal world” statements earlier.  This can result in some odd cases where one communication method is working in a particular case, but another method is not.  This results in differences in membership – which can have bad effects.

Why this might not be quite as bad as it seems

One reason why one might not worry about this as much as one might, is because it's a problem which one can't make go away.  A cluster system will always have to interface with software packages which do their own communication, and compute their own membership for a variety of usually good reasons.  As a result, this is a problem which we can't make go away.  Instead we have to deal with it effectively.  There are basically two cases to consider:

1. The “Main” membership thinks that node X should not be in the cluster, whereas the “Other” membership thinks it should be.

2. The “Other” membership thinks that node X should not be in the cluster, whereas the “Main” membership thinks it should be.

Let's take these two cases one at a time:

Case 1:

If the main membership thinks node X is not in the cluster, then it will simply not start any resources on node X.  This takes care of the problem.

Case 2:

If the “Other” membership discovers that a particular node should be dropped from its view of membership, and it can inform the CRM not to start its resources on that machine, then the local view of this membership from the perspective of the resources it deals with is effectively made to exclude these Other-errant nodes.  In the Linux-HA CRM this is easily done having the Other-resources write node attributes to cause those nodes to be excluded, and the rules would then be written to exclude those nodes from consideration for running Other-related resources.

Although Case 2 isn't pretty, it works, and no amount of wishing and hoping is likely to ever make this kind of problem go away in the general case - particularly when one involves proprietary applications  So, even if there is some membership discrepancy, it can is always possible to manage it appropriately assuming you can get a tiny bit of cooperation from the application.

References

[1] http://linux-ha.org/
[2] http://techthoughts.typepad.com/managing_computers/2007/10/split-brain-quo.html
[3] http://linux-ha.org/STONITH
[4] http://techthoughts.typepad.com/managing_computers/2007/10/more-about-quor.html
[5] http://techthoughts.typepad.com/managing_computers/2007/11/quorum-server-i.html

How Managed Virtualization (including HA) conflicts with System Management

Managed Virtualization Versus System Management

In an earlier post[1], I talked about a couple of kinds of virtualization, comparing two of them and highlighting their strengths.  This posting discusses how virtualization can confuse and confound conventional systems management - both automated and manual, and gives some thoughts on how to deal with it.

We all know that virtualization is a GoodThing(TM).  Therefore, it can't really have any disadvantages, can it?  <tongue-in-cheek-off> Unfortunately, it does have disadvantages.  The great strength of virtualization is its ability to break the ties between a service or operating system and the server which implements its service.  Many software systems and a good number of human beings find this confusing.  If I want to reboot a physical server, what services or operating systems will be disrupted by the reboot?

Conversely, if I want to do something to the machine that's running a particular service, which machine do I have to log into?  If you're running both service virtualization (conventional HA like Linux-HA[2]) on top of server virtualization (ala Xen or VMware), then you have a doubly difficult task - first you have to figure out which virtual machine is running a service, then you have to figure out which physical machine is running that particular virtual machine.

This can be really annoying and can easily result in system administrators[3] making mistakes either in the middle of the night, or when under pressure (which all sysadmins know is pretty much all the time).

Remember - Complexity is the Enemy of Reliability.   This is just another example of my favorite phrase at work.

And, if you want to have server monitoring software which tries to figure out whether a service is stopped and have it restart it, then it can also get confused by the fact that all these stupid servers and services are always moving around.  They just won't stay put!  Back in the olden days, you logged into a server and you edited the inittab, and you always knew what hardware it was running on and what server it was.  Now, with virtualization, and especially with virtualization management software, you never know what's where.

A Recipe for Chaos and Conflict

Your HA software and/or your virtualization management software can move things around on you.  Imagine that you have these four kinds of things in your data center:

• High-Availability (HA/service-virtualization) management software

• Virtualization management software

• System management monitoring software

• Human system administrators

This is a recipe for chaos, interspersed with the occasional career-limiting disaster. It's this kind of thing that leads system administrators to pull their hair out, and keep their resumes up to date.  None of these is bad by itself, in fact, each is a GoodThing(TM).  But they don't normally play well with each other. In typical myopic software design fashion, each of these layers is usually unaware of the other layers (except, of course for the last (human) layer - who has to make up for all the poor integration).

In addition, since the software layers typically aren't aware of all this wonderful virtualization going on, they can't really deal with the picture reliably.  They don't know what should be happening where, because it isn't fixed.  The various virtualization management packages keep changing things!

So, what's a body to do?  As far as I know, there are two basic options.

1. Integrate the four layers of management with each other using things like CIM[4] and SNMP[5]

2. Empower your HA software to also manage the server virtualization of your data center

Integration of Layers

Virtually every data center (sadly, pun intended) has a variety of server types and a variety of operating systems, and a variety of management software.  They mostly don't play well with each other.  Almost the only way to get them to play together - even if imperfectly - is to have them talk together using industry standard protocols.

Today, that means using SNMP or CIM.   Here is my personal view on the characteristics of these two protocols for your consideration.

• SNMP - widely deployed - implemented in a truly compatible way, but far too weak for a job this hard.  SNMP is great for grabbing statistics, checking whether a server or router is up and what kind of load it is seeing in great detail.  Anything much beyond this, and the MIBs become 100% vendor-specific - meaning that cross-vendor integration breaks down - basically completely.  For HA clustering or virtualization management or worse yet the combination of the two - forget it.

• CIM - widely deployed in expensive disk subsystems - but rarely deployed outside that.  It has newly developed models for virtualization and clustering, but like most standards they're mostly lowest-common-denominator standards, and unfortunately not widely deployed.  For example, Linux-HA[2] implements CIM, but unfortunately Linux-HA has tremendous power and capability which CIM can't begin to model.  So, this winds up being only possible to model using vendor-specific extensions - greatly weakening the possible integrations.

Now, I'm not saying that these two protocols are useless - far from it. Without open standards like CIM and SNMP, the prospect truly is hopeless.   But I am saying that integrating them in the typical-for-the-industry highly-heterogeneous data center is a challenge, and the more layers there are to integrate, the bigger the challenge.  Since standards necessarily trail industry practice, the more "bleeding edge" the topic (i.e., HA clustering or virtualization) and the more powerful the underlying tool (like Linux-HA), the greater the mismatch.

Here we have two bleeding edge topics and four layers.  Yikes!  Surely there must be some kind of alternative to this somewhat-unattractive mess.

Decrease The Layers and Let Them Manage Themselves

As I mentioned in my earlier virtualization posting, some HA packages (like Linux-HA) can also manage virtualization simultaneously.  So, one way of dealing with this is to let (or extend) your service virtualization product also manage your server virtualization.  One advantage of this approach is that service virtualization software (HA software) is comparatively mature technology, minimizing the risk.

Unfortunately, this doesn't yet go all the way in solving the problem either.  There are a few things that should change to make this really work well. These include

• Support much larger HA clusters - hundreds to thousands of nodes.  In an ideal world, you'd really like fewer of these HA/virtualization clusters as you can get.  Today you'd typically have to have one of these clusters for every 8-32 physical servers - which makes an awfully lot of these things to manage in a data center containing hundreds or thousands of servers.

• Integrate with many virtualization layers - Such a product would need to integrate with Xen, IBM System Z, IBM System P, Linux KVM, VMware, and future virtualization layers like the one promised by Microsoft.   This isn't rocket science, but by the time you're done, it will be some work.

• Support monitoring and controlling services inside the virtual machine - Otherwise you haven't really integrated the two layers - and you wind up running some HA software inside some of the virtual machines.  Again, this isn't rocket science, but it will require some work[1] for each operating system you want to manage services for.

• Integrate with provisioning systems - so that you can add and delete virtual machines and allocate disk to them and their applications with fewer possibilities for error, and more automation.

None of these items are technically difficult, and none of them are prohibitively expensive to implement.  Given that I'm the project leader for Linux-HA, and Linux-HA is one of the most capable HA products around, you might imagine that some of these thoughts are on my mind for our future  ;-).  Of course, that doesn't eliminate the necessity for integration with the remaining layers above, which is why Linux-HA implements both CIM and SNMP.  This allows the virtualization management infrastructure to actively and autonomically manage  servers and services, while letting it bubble up events (especially those it can't automatically recover from) to the management consoles and humans via protocols like SNMP and/or CIM.

Conclusions

Virtualization technologies add complexity to the data center along with the benefits they bring, and in the process may render the existing management facilities less than useful.  However, if HA and Virtualization management are performed by a single entity, and open standards like CIM and SNMP are used, systems can be active the problems can be minimized.

See Also

Preparing for Virtual Management http://www.itbusinessedge.com/blogs/dcc/?p=276

References

[1] http://techthoughts.typepad.com/managing_computers/2007/09/virtualization-.html
[2] http://linux-ha.org/
[3] http://linux-ha.org/SysAdmin
[4] http://www.dmtf.org/standards/cim/
[4] http://en.wikipedia.org/wiki/Common_Information_Model_%28computing%29
[5] http://en.wikipedia.org/wiki/Simple_Network_Management_Protocol

Quorum Server Illustrated - updated

In two earlier posts [1] [2], I gave brief descriptions of the quorum server which seem to have left as much confusion as they provided clarity.  This post is only about the Linux-HA quorum server, and includes illustrations for clarity.

The Linux-HA Quorum API

In the Linux-HA quorum API, you can configure a number of quorum modules which are used as follows.  If a quorum module returns HAVEQUORUM, then the cluster has quorum.  If it returns NOQUORUM then the cluster does not have quorum.  If a quorum module returns QUORUMTIE, then the next quorum module in the list is consulted.  If the final module returns QUORUMTIE, then it is treated as a NOQUORUM event.

The quorum daemon is normally used in conjunction with the nomal arithmetic voting quorum module, so that it is only consulted when the number of nodes in the cluster is exactly half the number of configured modules in the system.  So, it is worth noting that the quorum server will never be consulted if a cluster has an odd number of nodes.

Quorum Server Scenarios

Below, I'll go through the basic quorum server cases so you can see how all this works in more detail - with pictures, even!

Normal Situation - Everything up

In the picture above, everything is normal.  The quorum server is up, and both sites are also up.  Because the cluster has all its nodes up, the quorum server is irrelevant.

Single Site Failure

In the situation above, we show the "New Jersey" site as down.  In this case, the conventional voting quorum has a tie (1/2 - exactly half of the nodes).  In this case the quourm server is consulted.  Since only New York is talking to the quorum server, the quorum server grants quorum to the New York site.

Split Brain Avoided

In the case above, the link between the sites has been lost, but both sites and the quorum server are all up.  In this case, both New York and New Jersey contact the quorum server because each sees 1/2 nodes as being up - resulting in a tie condition.

In this case, the quorum server will choose one of the two sites to provide quorum to, and I assume in this case that New York was chosen.  Because New Jersey  wasn't granted quorum, it will shut its resources down.

What happens when the quorum server goes down?

That is the situation shown above.  Because New York and New Jersey are both up, they have 2/2 votes and both provide service as they should.  This illustrates the point that the quorum server is not a single point of failure.

Multiple Failures -> Loss of Service

In this final case, multiple failures have occurred - both New Jersey and the quorum server are down.  In this case, New York doesn't have quorum, so it shuts down services and none are provide by any node in the cluster.  Of course, this situation can be overridden in the cluster configuration by changing the quorum policy, but from an automated perspective, this is all that can be (should be) done.

Security Concerns

If you want to run your quorum server communications across networks which mig

Alan eats his own cl_respawn dog food. Yum!!

In this posting, I show how to use cl_respawn[1] to monitor my system logging and help keep it running, and along the way, I improved cl_respawn a little as well.  In addition, I explain why I couldn't just use the respawn directive in /etc/inittab[5] (and why you probably can't either).   I first talked about cl_respawn in one of my first blog posts[6].

The problem

When we run our automated CTS[2] tests for Linux-HA[3] we rely on the guaranteed log entry delivery provided by syslog-ng[4].  Basically, we redirect all our logs in a test cluster to a test overseer machine, and then CTS watches this consolidated log for errors and correct behavior.

This is a nice system and it works pretty well, but it relies on the reliability of syslog-ng.  For the most part, that's just fine.  But, sometimes syslog-ng just stops running.  Then the tests show that Heartbeat has failed, but it's really just syslog-ng that's crashed on me.  So, in the past I added some code to CTS to make it test the logging after every error, and then hit the machines over the head with a hammer and restart logging if logging wasn't working.

This was sort-of OK, because it meant subsequent tests would run fine, but the one test would show failed - even though it probably succeeded.  This would be fine, except that one of my machines (my oldest and slowest) had syslog-ng die on it a few times a day.  I don't know why, and as long as I can live with it for my testing, I don't much care.  I just want it to work.  (I know, it's a lousy attitude, but I have way more to do than I can possibly do).

The solution

Then it I had this revolutionary thought - I could use HA software to make my logging highly available!!

Hold the presses, folks, new headline reads
   "HA guru realizes he can use HA software just like he tells everyone else to do!"

To fix this problem all I had to do was change the init script for syslog to use our cool little cl_respawn tool to babysit the syslog-ng service.  Although I could have used Heartbeat to monitor this service, it seemed like overkill and would have conflicted with CTS.

So, I set out to use cl_respawn to restart syslog-ng quickly - minimizing but not eliminating the possibliity of losing important log messages.

When I looked at the init scripts (they're from SUSE Linux), they had these statements in them:

• For starting
  startproc -p ${syslog_pid} ${BINDIR}/${syslog} $params
• For stopping
  killproc -p ${syslog_pid} -TERM ${BINDIR}/${syslog} ; rc_status -v
• For status
  checkproc -p ${syslog_pid}      ${BINDIR}/${syslog{; rc_status -v
• checkproc -p ${syslog_pid}      /usr/bin/cl_respawn; rc_status -v

My first thought was ll I had to do was insert cl_respawn ahead of the ${BINDIR}/syslog and I'd be done.  Well.... not quite...

If I had done that, then the pid file for the service ${syslog_pid} would have pointed not to cl_respawn, but to syslog-ng.  So, when I tried to shut down syslog, cl_respawn would have just respawned it.  OOPS.  Not quite the right effect.

What was necessary was for the syslog pid file to contain the pid of cl_respawn, not the pid of syslog-ng.  One minor problem - the author of cl_respawn didn't deal with pidfiles.  To fix that, I added support for a -p option to tell it the name of the pid file to use.

Now I try it.  Uh-oh... It didn't work.  The logs are quickly filled with attempts to start  ${syslog} and having it fail continually with  socket in use.  What was all that about?

By default, syslog-ng forks itself into the background,  and its parent process exits.  That makes cl_respawn think it's died - so it restarts it - and it fails ad infinitum.  So, I read the man page for syslog-ng and discover the -F option to keep it from forking.  Without that, cl_respawn can't tell when it dies.

Along the way, I read the code, find a couple of other minor bugs and fix them.  I update my init script and now it looks like this:

• For starting
  startproc -p ${syslog_pid} /usr/bin/cl_respawn -p ${syslog_pid} ${BINDIR}/${syslog} -F $params
• For stopping
  killproc -p ${syslog_pid} -TERM /usr/bin/cl_respawn ; rc_status -v
• For status
• checkproc -p ${syslog_pid}      /usr/bin/cl_respawn; rc_status -v

Of course, if you don't run SUSE Linux, then your init scripts will look somewhat different, but I'm sure you'll figure it out.

Why not just use respawn in inittab?

Those of you who know UNIX administration to any degree realize that /etc/inittab[5] has a respawn directive you can give it.  Why wouldn't that do the trick?  The short answer is service dependencies.   The longer answer is below:

• Logging depends on other /etc/init.d services, so you don't want it to start until after those other services (like the network) are started.  The LSB init script system supports these dependencies and starts things in the right order.
• Other services depend on logging.  A number of other services can't start until after logging starts.  If you try and disable the /etc/init.d/syslog service on your machine so you can start it with respawn from /etc/inittab, havoc ensues - because these other services won't start until the /etc/init.d/syslog service is started.  If you disable it, they won't start.
• What fun would that be?  I mean, if we wrote this cl_respawn tool, we probably ought to use it ;-).

What did I learn?

• How to use cl_respawn in real life
• Some missing requirements for cl_respawn
• I was reminded of the advantages of using our own software
• How handy simple little tools like cl_respawn can be

[1]  http://hg.linux-ha.org/dev/file/tip/tools/cl_respawn.c
[2]  http://linux-ha.org/CTS
[3]  http://linux-ha.org/
[4]  http://www.balabit.com/network-security/syslog-ng/opensource-logging-system/"
[5]  http://www.freebsd.org/cgi/man.cgi?query=inittab&manpath=Red+Hat+Linux%2Fi386+9&format=html
[6]  http://techthoughts.typepad.com/managing_computers/2007/09/tools-for-servi.html

More about quorum - updated

In a previous article[1], I talked about quorum, and alluded to some more details about quorum which I'll discuss here in a little more detail.  Let's examine a couple of common quorum tie-breaker methods, and see what's useful, and what's hype, and what's painful to use.

Can the standard voting quorum method fail?
By fail, I mean, can it grant quorum to two partitions simultaneously?  The answer, unfortunately, is yes, even though it seems like a mathematical impossibility.  This is because the world, unfortunately, is more complicated than simple mathematics, and quorum methods don't stand alone.  Quorum methods are tied to membership algorithms.  If a membership method fails, and for a period of time, a given node appears to be in the membership of two partitions, then while that's true, both partitions could legitimately think they have quorum.  Like many possible failures, this one is unlikely, but it is certainly possible.  Sigh...  Paranoia is so depressing.

SCSI Reserve Limitations

In the earlier article, I mentioned that SCSI reserve was often painful to use, without being explicit on why.  Let's explore that now.  SCSI reserve is an operation which happens at the physical disk volume level - that is, at the SCSI LUN  level.  With "dumb" disks this typically corresponds to an entire disk spindle - which nowadays is something like 180G-750G of disk data - which is clearly a significant waste of resources.

However, most people who use shared disks use them in a SAN are using it with a smart SAN disk controller which allows the creation of "logical" volumes which correspond to more much smaller sizes for a single SCSI LUN, using any RAID method you want.  For a disk used just as a quorum disk, you don't need to actually write anything on it, so you want it as small as you can make it.  But, most people probably make it RAID 1, which means that the minimum size is probably something like 2 gigabytes of SAN disk. If you have a large data center, it could easily have 50 clusters in it, and each one requires such a quorum device.  In this case, this makes for a lot of extra volumes, extra administration and possibilities for confusion and human error going on here.  In addition, smaller SAN disk units may have limitations on the total number of disk partitions they can manage.

So, perhaps you think - I'm smarter than that, I'll just use software volume managers to take care of this for me, and avoid all those extra logical disks in my SAN.  Unfortunately, that typically doesn't work.  This is because when you issue the SCSI reserve, it can't reserve a logical partition, only a physical partition, so many logical volume managers block reserve operations.  So, logical volume managers are not much help here.

To make it even more complicated, multi-pathing to disk devices often confuses (i.e., "breaks") disk reservation issues - particularly with SCSI II non-persistent reservations.

Of course, if you're replicating data instead of sharing it, disk reserve operations are of no help at all - since reserving a disk on one disk volume has no effect whatsoever on the other volume.

None of these considerations are changed by what kind of SCSI reserve you issue (persistent or the older non-persistent reserve).  However, there are even more problems that occur when you use the older non-persistent SCSI II reserve (the most commonly available kind), since it isn't persistent, and is broken by a bus reset or a device reset.  So, if you use SCSI II reserve, then you have to continually verify that you still have the reserve.

Count Key Data Disk Reserve

Mainframe disk subsystems support a non SCSI-based disk model, called the extended count key data (ECKD) disks.  These disks also support reserve methods similar to those provided by SCSI.

Quorum Daemon - Helpful or Hype?
Earlier, I said that the Linux-HA[2] quorum daemon[3] can help out here - not only for local shared disk situations, but for disaster-recovery type situations where you're replicating data between sites, and gave a hand-waving style argument on why that's so.  Let's see if we can go past the hand waving into more of the details so you can see what this is, how this works, and decide for yourself it would be helpful to you, or if this is just more hype written by someone who likes his project's work.

If you recall, I also described it as analogous to a software implementation of SCSI reserve.  But, in this case, there are no disks involved, so the hardware and SCSI protocol limitations mentioned above go away.  So, if the hardware has gone away, what kind of software has replaced it, and how does it work?

In the simplest view, the quorum daemon is simple - it takes TCP connections from clients, and when multiple clients both want to have quorum for the same cluster, it grants it to exactly one of them.  So, at this level of detail, it's quite simple, and the logic also straightforward.  But, there's a little more to it when you get into the details, so let's spend a few words describing the next level of detail, for those of you who are still skeptical.

TCP doesn't do a good job of telling you when your peer goes away, so both the client and the server processes send the other side heartbeats.  If the client stops hearing heartbeats, then he notifies his cluster that he no longer has quorum.  When the server stops hearing heartbeats, he takes quorum away from the client, and sends him noquorum messages.  Before switching quorum from one client to the other, the quorum daemon induces a configurable delay to allow the previous quorum owner time to notice that they've lost quorum and to shut down any resources it might be running.

What other interesting design features does it have?  Well, for one, all communication between client and server goes over SSL, with certificate authentication.  The server has a copy of the client's public certificate, and the clients have copies of the server's public certificate - so you don't need to get your certificates from a certificate authority - because we authenticate them by certificate, not through certificate authority (which could be a single point of failure).  Because of the way we use the SSL certificates, both authentication and authorization are bundled together.

Another nice design feature, is that a single quorum daemon doesn't have any fixed upper limit on the number of clusters it can support.  So, a single quorum daemon process should be able to support hundreds of clusters.  This makes it an advantage over just adding a third node to your cluster - because it can server hundreds of clusters, whereas typically a single computer can't join more than one cluster.  Indeed, since this doesn't run the cluster stack for any given cluster, it would be possible for the quorum daemon to work with multiple cluster implementations - as long as they put the hooks in to talk to it.

The combination of these two features allows you to put your quorum daemon on a completely different site, even at a colocation facility, and let your communication flow over the public Internet - securely.  This is really nice economical choice for your split-site DR-style clusters, for companies that don't have 3 or more major data centers.  Of course, if you have dozens of clusters that aren't split-site, you can make a single quorum daemon cluster with fencing to serve all the other clusters in your site.  All without bothering your storage group to create and manage all those little tiny quorum partitions for you.  In fact, without requiring shared storage at all.

Some kinds of software support resource-level quorum, that is, quorum on a resource-level rather than a whole cluster level.  Some of these are called resource-driven clusters.  The quorum daemon idea could also be used for those arrangements as well.

So, what's not to like about the quorum daemon?  Well, you do have to make a certificate for the server, and one for each client.  But, you don't have to pay for them, because they don't need to come from one of the well-known certificate producers.  It would be nice if the quorum daemon would sync its state to a slave copy that could be used in a cluster as a master/slave resource in case the machine running the quorum daemon failed.  You can still run it in a cluster with another machine to take over for it, but the takeover isn't as graceful as it would be if there were a slave copy ready to take over for the master complete with its quorum state.

I've recently learned that HP ServiceGuard[4] has a feature similar to this which they call the quorum server daemon (qs)[5].  The documentation indicates that HP's qs authenticates by IP address and does not use SSL for communication or authentication/authorization.  As a result, there are security concerns associated with it, and qs is probably unsuitable for split-site configurations.

And, even more recently, thanks to Nils Gorrol I've learned that Sun has a similar feature in their sqsd [6] since SunCluster version 3.2.  Looking at the documentation, it looks as though it has similar security issues that HP's qs has for split-site arrangements and potentially insecure networks.

Thanks to all those who keep me on the straight and narrow regarding the facts ;-)

Ping tiebreaker

Some HA systems provide  a ping tiebreaker.  To make this work, you pick a address outside the cluster to ping, and any partition that can ping that address has quorum.  The obvious advantage is that it's very simple to set up - doesn't require any additional servers or shared disk.  The disadvantage (and it's a big one) is that it's very possible for multiple partitions to think they have quorum.  In the case of split-site (disaster recovery) type clusters, it's going to happen fairly often.  If you can use this method for a single site in conjunction with fencing, then it will likely work out quite well.  It's a lot better than no tiebreaker, or one that always says "you have quorum".  Having said that, it's significantly inferior to any of the other methods.

If I've omitted things you want to know about, or this brings up ideas or questions in your mind, or you know of other implementations of the quorum daemon idea, by all means post a reply to this article.

References

[1] http://techthoughts.typepad.com/managing_computers/2007/10/split-brain-quo.html
[2] http://linux-ha.org/
[3] http://www.linux-ha.org/QuorumServerGuide
[4] http://h71036.www7.hp.com/enterprise/cache/6468-0-0-0-121.html
[5] http://www.docs.hp.com/en/B3936-90065/ch03s01.html
[6] http://docs.sun.com/app/docs/doc/819-5360/gbdud?a=view

Split-brain, Quorum, and Fencing - updated

In some ways, an HA system is pretty simple - it starts services, it stops them, and it sees if they and the computers that run them are still running.  But, there are a few bits of important "rocket science" hiding in there among all these apparently simple tasks.  Much of the rocket science that's there centers around trying to solve a single thorny problem - split brain.  The methods that are used to solve this problem are quorum and fencing.  Unfortunately, if you manage an HA system you need to understand these issues.  So this post will concentrate on these three topics: split-brain, quorum, and fencing.

If you have three computers and some way for them to communicate with each other, you can make a cluster out of them and,each can monitor the others to see if their peer has crashed.  Unfortunately, there's a problem here - you can't distinguish a crash of a peer from broken communications with the peer.  All you really know is that you can't hear anything from them.  You're really stuck in a Dunn's law[1] situation - where you really don't know very much, but desperately need to.  Maybe you don't feel too desperate yet.  Perhaps you think that you don't need to be able to distinguish these two cases.  The truth is that sometimes you don't need to, but much of the time you very much need to be able to tell the difference.  Let's see if I can make this clearer with an illustration.

Let's say you have three computers, paul, silas, and mark, and paul and silas can't hear anything from mark and vice versa.  Let's further suppose that mark had a filesystem /importantstuff from a SAN volume mounted on it when we lost contact with it. and that mark is alive but out of contact.  What happens if we just go ahead and mount /importantstuff up on paul? The short answer is that bad things will happen[2]. /importantstuff will be irreparably corrupted as two different computers update the disk independently.  The next question you'll ask yourself is "Where are those backup tapes?". That's the kind of question that's been known to be career-ending.

Split-Brain

This problem of a subset of computers in a cluster beginning to operate autonomously from each other is called Split Brain[3]. In our example above, the cluster has split into two subclusters: {paul, silas} and {mark}, and each subset is unaware of the others.  This is the perhaps most difficult problem to deal with in high-availability clustering.  Although this situation does not occur frequently in practice, it does occur more often than one would guess.  As a result, it's vital that a clustering system have a way to safely deal with this situation.

Earlier I mentioned that there was information you really want to know, but don't know.  Exactly what information did I mean?   What I wanted to know was "is it safe to mount up /importantstuff somewhere else?".  In turn, you could figure that out if you knew the answer to one of these two questions:  "Is mark really dead?" which is one way of figuring out "Is mark going to write on the volume any more?"  But, of course, since we can't communicate with mark, this is pretty hard to figure out.  So, cluster developers came out with a kind of clever way of ensuring that this question can be answered.  We call that answer fencing.

Fencing

Fencing is the idea of putting a fence around a subcluster so that it can't access cluster resources, like  /importantstuff.  If you put a fence between it and its resources, then suddenly you know the answer to the question "Is mark going to write on the volume any more?" - and the answer is no - because that's what the fence is designed to prevent.  So, instead of passively wondering what the answer to the safeness question is, fencing takes action to ensure the "right" answer to the question.

This sort of abstract idea of fencing is fine enough, but how is this fencing stuff actually done? There are basically two general techniques:  resource fencing [4] and node fencing.[5]. 

• Resource fencing is the idea that if you know what resources a node might be using, then you can use some method of keeping it from accessing those resources. For example, if one has a disk which is accessed by a fiber channel switch, then one can talk to the fiber channel switch and tell it to deny the errant node access to the SAN.

• Node fencing is the idea that one can keep a node from accessing all resources - without knowing what kind of resources it might be accessing, or how one might deny access to them.  A common way of doing this is to power off or reset the errant node.  This is a very effective if somewhat inelegant method of keeping it from accessing anything at all.  This technique is also called STONITH[6] - which is a  graphic and colorful acronym standing for Shoot The Other Node In The Head.

With fencing, we can easily keep errant nodes from accessing resources, and we can now keep the world safe for democracy - or at least keep our little corner of it safe for clustering.  An important aspect of good fencing techniques is that they're performed without the cooperation of the node being fenced off, and that they give positive confirmation that the fencing was done.  Since errant nodes are suspect, it's by far better to rely on positive confirmation from a correctly operating fencing component than to rely on errant cluster nodes you can't communicate with to police themselves.

Although fencing is sufficient to ensure safe resource access, it is not typically considered to be sufficient for happy cluster operation because without some other mechanism, there are some behaviors it can get into which can be significantly annoying (even if your data really is safe).  To discuss this, let's return our sample cluster.

Earlier we talked about how paul or silas could use fencing to keep the errant node mark from accessing /importantstuff.  But, what about mark?  If mark is still alive, then it is going to regard paul and silas as errant, not itself.  So, it would also proceed to fence paul and silas - and progress in the cluster would stop.  If it is using STONITH, then one could get into a sort of infinite reboot loop, with nodes declaring each other as errant and rebooting each other, coming back up and doing it all over again.  Although this is kind of humorous the first time you see this in a test environment - in production with important services, the humor of the situation probably wouldn't be your first thought.  To solve this problem, we introduce another new mechanism - quorum.

Quorum

One way to solve the mutual fencing dilemma described above is to somehow select only one of these two subclusters to carry on and fence the subclusters it can't communicate with.  Of course, you have to solve it without communicating with the other subclusters - since that's the problem - you can't communicate with them.  The idea of quorum represents the process of selecting a unique (or distinguished for the mathematically inclined) subcluster.

The most classic solution to selecting a single subcluster is a majority vote.  If you choose a subcluster with more than half of the members in it, then (barring bugs) you know there can't be any other subclusters like this one. So, this is looks like a simple and elegant solution to the problem. For many cases, that's true.  But, what if your cluster only has two nodes in it?  Now,  if you have a single node fail, then you can't do anything - no one has quorum.  If this is the case, then two machines have no advantage over a single machine - it's not much of an HA cluster.  Since 2-node HA clusters are by far the most common size of HA cluster, it's kind of an important case to handle well.  So, how are we going to get out of this problem?

Quorum Variants and Improvements

What you need in this case, is some kind of a 3rd party arbitrator to help select who can fence off the other nodes and allow you to bring up resources - safely.  To solve this problem there is a variety of other methods available to act as this arbitrator - either software or hardware. Although there are several methods available to use as arbitrator, we'll only talk about one each of hardware and software methods: SCSI reserve and Quorum Daemon.

• SCSI reserve:  In hardware, we fall back on our friend SCSI reserve.  In this usage, both nodes try and reserve a disk partition available to both of them, and the SCSI reserve mechanism ensures that only one of the two of them can succeed.  Although I won't go into all the gory details here, SCSI reserve creates its own set of problems including it won't work reliably over geographic distances.  A disk which one uses in this way with SCSI reserve to determine quorum is sometimes called a quorum disk.  Some HA implementations (notably Microsoft's) require a quorum disk.

• Quorum Daemon:  In Linux-HA[7], we have implemented a quorum daemon - whose sole purpose in life is to arbitrate quorum disputes between cluster members.  One could argue that for the purposes of quorum this is basically SCSI reserve implemented in software - and such an analogy is a reasonable one.  However, since it is designed for only this purpose, it has a number of significant advantages over SCSI reserve - one of which is that it can conveniently and reliably operate over geographic distances, making it ideal for disaster recovery (DR) type situations.  I'll cover the quorum daemon and why it's a good thing in more detail in a later posting.  Both HP and Sun have similar implementations, although I have security concerns about them, particularly over long distances.  Other than the security concerns (which might or might not concern you), both HP's and Sun's implementations are also good ideas.

Arguably the best way to use these alternative techniques is not directly as a quorum method, but rather as a way of breaking ties when the number of nodes in a subcluster is exactly half the number of nodes in the cluster.  Otherwise, these mechanisms can become single points of failure - that is, if they fail the cluster cannot recover.

Alternatives to Fencing

There are times when it is impossible to use normal 3rd-party fencing techniques.  For example, in a split-site configuration (a cluster which is split across geographically distributed sites), when inter-site communication fails, then attempts to fence will also fail.  In these cases, there are a few self-fencing alternatives which one can use when the more normal third-party fencing methods aren't available.  These include:

• Node suicide.  If a node is running resources and it loses quorum, then it can power itself off or reboot itself (sort of a self-STONITH).  The remaining nodes wait "long enough" for the other node to notice and kill itself.  The problem is that a node which is sick might not succeed in self-suicide, or might not notice that it had a membership change, or had lost quorum.   It is equally bad if notification of these events is simply delayed "too long".  Since there is a belief that the node in question is, or at least might be, malfunctioning, this is not a trivial question.  In this case, use of hardware or software watchdog timers becomes critical.

• Self-shutdown.  This self-fencing method is a variant on suicide, except that resources are stopped gracefully.  It has many of the same problems, except it is somewhat less reliable because the time to shut down resources can be quite long.  Like the case above, use of hardware or software watchdog timers becomes critical.

Note that without fencing, the membership and quorum algorithms are extremely critical.  You've basically lost a layer of protection, and you've switched from relying on a component which gives positive confirmation to relying on a probably faulty component to fence itself, and then hoping without confirmation that you've waited long enough before continuing.

Summary

Split-brain is the idea that a cluster can have communication failures, which can cause it to split into subclusters.  Fencing is the way of ensuring that one can safely proceed in these cases, and quorum is the idea of determining which subcluster can fence the others and proceed to recover the cluster services.

An Important Final Note

It is fencing which best guarantees the safety of your resources.  Nothing else works quite as well.  If you have fencing in your cluster software, and you have irreparable resources (i.e. that would be irreparably damaged in a split-brain situation), then you must configure fencing.  If your HA software doesn't support (3rd party) fencing, then I suggest that you consider getting a different HA package.

See Also

General cluster concepts[8]

References

[1]   http://linux-ha.org/DunnsLaw
[2]   http://linux-ha.org/BadThingsWillHappen
[3]   http://linux-ha.org/SplitBrain
[4]   http://linux-ha.org/ResourceFencing
[5]   http://linux-ha.org/NodeFencing
[6]   http://linux-ha.org/STONITH
[7]   http://linux-ha.org/
[8]   http://linux-ha.org/ClusterConcepts

How to use a watchdog timer

In an earlier posting[1], I mentioned that explaining how to optimally use a watchdog driver would be a good thing to talk about later. Now seems a good time to talk about that, giving a brief overview of some good techniques for getting the most out out of your watchdog timer.

As was mentioned previously, one can have a software watchdog timer like softdog[2], or a hardware timer, or a watchdog utility like apphbd[3] (application heartbeat daemon). Although each method has its advantages and disadvantages, the methods that an application can use to take best advantage of them are very similar.

The basic idea of using a watchdog is simple:  Periodically send a heartbeat to the watchdog timer.  If the application fails to heartbeat in the specified interval, or exits prematurely, then a recovery action is taken.  So, all your application has to do  is set a timer and tickle the watchdog timer when your timer goes off.  Sounds extremely simple - and for the most part it is.

How to get into trouble with watchdog timers

If your application does disk I/O or grows in size as it runs, or calls functions or systems calls that might block, then the timing of your application can change dramatically when the system is under heavy I/O load or memory pressure.  This can mean that your application is judged to be hung when it's not.  When this happens, the watchdog timer you're using will trigger a recovery action - maybe restarting your application, or rebooting the machine.  For this kind of a situation, this is probably not what you had in mind.  Of course, if you're in an HA environment like Linux-HA[4] where a machine reboot will cause a service failover, this may exactly what's needed to straighten out your problem.  As always, YMMV[5].

When not to use watchdog timers

Watchdog timers need reasonably reliable real-time performance, and an application which you can modify and which runs periodically (or which you're willing to make run periodically).  If you can't modify the application, or it is expected to have extremely erratic real-time performance, it doesn't run continuously, or runs in an environment which services your application erratically, then watchdog timers may not be for you.

Making your watchdog timer do more

Having your application not appear to be dead is sort-of-OK, but not exactly a deep metric for how well your application is behaving. Since this code will run inside your application, and you have to modify your application anyway, you have the opportunity to make this a form of white-box testing [6]. To do this, tie tickling your watchdog timer to good measures of your program's sanity. Below are a few examples of how you might go about doing this.  Not every example is appropriate for every type of application, so take this as food for thought.

• Audit your data structures. If you have several data structures representing work to do, clients, outputs, etc., you can audit your data structures for mutual consistency, and only send out heartbeats when you don't find any errors.  For example, perhaps every piece of work to be done ought to belong to an active client. This is an interesting technique - because it can allow for transient "false positives" or errors that get corrected by the natural flow of the application.  If you audit your data structures once every 10 seconds, but set your heartbeat rate to 30 seconds, then you can have a transient error last up to two iterations without causing a restart.  If it persists beyond that, your watchdog timer will take action.

• Check for work being processed.  If you have work in your input queues, but no work has been completed since the last heartbeat, then suppress sending out a heartbeat until some work actually gets processed.

• Check for old work to be done.  If you have work queues, you can skip heartbeats whenever you have work in your queues which is "too old".  This may be a symptom that your application isn't processing its work, or that it has somehow lost track of this particular piece of work.

Of course, these are just a few simple ideas, but they may spark some better ideas for your particular application.  Anything you can examine periodically to see if your application seems to be doing whatever it is it's supposed to do is a potential candidate for using to control when and whether to tickle your watchdog timer.

Making your watchdog timer more reliable

Something you should really avoid is false positives on watchdog timeouts - especially when the consequence is to reboot the machine. Spurious reboots are typically frowned upon ;-).  This is all a fine idea, but how exactly do you go about doing that?  Here are a few tips to keep in mind.

1. Tune your timeout interval. Some watchdog timers (like apphbd) have a warning level as well as a fatal timing level. Be sure and take advantage of the warning level to help you tune your application's heartbeat interval.  For example, if your application really ought to send out a heartbeat every second, you can set the warning time threshold to 1.5 seconds, and the fatal watchdog timer to a much larger value, say 5 or 10 seconds. That way, as you test, you can see how close you come to the 3 second time limit in your most extreme cases of load.

2. Know your application's expected behavior.  If is single-threaded and does short-lived tasks throughout the day, except for once a day when it produces a summary report, then keep that extra work and the delay it can cause in mind when setting your timer value.

3. Make your application a better real-time citizen. Rather than processing all its work in one huge uninterruptible chunk, make sure you process chunks whose size is bounded by some constant amount of time, and allow interruptions (for processing watchdog timer requests) between these chunks.

4. Consider using the glib mainloop task dispatcher[7]. The mainloop construct is great for event driven programs which don't actually require threading. We use it extensively in Heartbeat [4], and it has worked out very well for us.

5. Consider using threads. Many people swear by threads as the only way to write any reasonable program. Many people (including many of those same people) swear at them[8], When properly used, threads can be helpful in writing programs with better real-time behavior.

6. Consider doing your disk I/O into a separate process (or thread). It's usually disk I/O or memory allocation (implying disk I/O) which is most likely to hang your program and give it unpredictable realtime behavior.

7. Consider using asynchronous I/O. This is another technique for avoiding blocking by disk I/O. It's not terribly portable, and the API seems to me to be a little subject to change, but it's a really nice idea.

8. Consider locking your program into memory. If your program needs about the same amount of real memory as it occupies virtual memory, then the system impact of locking your program into memory may not be high. This is not for everyone - because if everyone did this, then why bother implementing virtual memory?

9. Consider setting your program with a soft-realtime (POSIX realtime) priority. Even more than the previous step, this step is not for everyone. If your program goes into an infinite loop, the entire system stops - end of story. Not good. But, if your program is critical, small, and well-behaved, this can be a reasonable thing to do.

10. Consider running real time Linux [9]. This is an even more drastic step, but if your whole system needs to be real time, some great work has been done here which you might look into.

11. If you're writing in Java, consider real-time Java from IBM[10]. A better recommendation might be to not use Java, but for some people Java is their religion, but if you or your management insists on Java, then this may be just the ticket for you.

In summary, here are general ideas to keep in mind:

• Tie tickling the watchdog timer with the sanity of your application

• Do what is reasonable to improve the predictability of the realtime behavior of your program

Hopefully the tips above will help you do this.

References

[1] http://techthoughts.typepad.com/managing_computers/2007/09/monitoring---a-.html
[2] http://linux-ha.org/softdog
[3] http://linux.die.net/man/8/apphbd
[4] http://linux-ha.org/
[5] http://en.wiktionary.org/wiki/your_mileage_may_vary
[6] http://en.wikipedia.org/wiki/White_box_testing
[7] http://library.gnome.org/devel/glib/unstable/glib-The-Main-Event-Loop.html
[8] http://sourcefrog.net/weblog/software/languages/java/java-threads.html
[9] http://www-03.ibm.com/press/us/en/pressrelease/21232.wss
[10] http://domino.research.ibm.com/comm/research_projects.nsf/pages/metronome.index.html

Virtualization as High Availability (Disaster Recovery) or High-Availability as Virtualization?

Many pundits and other folks like VMware's CEO Diane Greene have touted[1] virtualization as being the "cure" to disaster recovery, many for the past several years.  Disaster recovery can be pretty reasonably viewed as being high-availability over distance, so it makes some sense to see how DR, HA and virtualization fit together.  What's hype here, and what's real?  Let's look and see what we find out.

What is virtualization?

Wikipedia[2]defines virtualization like this:

In computing, virtualization is a broad term that refers to the abstraction of computer resources. One useful definition is "a technique for hiding the physical characteristics of computing resources from the way in which other systems, applications or end users interact with those resources".

At the moment, this term is most commonly used to refer to machine or OS-level virtualization,  storage virtualization and something I'll call container virtualization.  For now, let's ignore storage virtualization.   Machine-level virtualization means basically making it difficult to tell exactly which physical machine an OS is running on at any given point in time.  There are many other kinds of virtualization as well - for example service virtualization.   Service virtualization doesn't hide or abstract physical machines, but instead virtualizes or hides services.  That is, there is no fixed binding between services and physical machines.  This is, in fact, exactly what classic HA software does - software like Linux-HA[3] . What an HA system does is use service virtualization to recover from server failures, and monitors services so that it can restart them - either locally or on another server.

Customers care about services, not about OSes or physical machines, so from their point of view, machine and service virtualization are similar.  Both  provide the potential for recovering from failures of physical servers and OSes. However, since machine virtualization doesn't  have any visibility of the individual services and the dependencies between them, it can't monitor them and restart them if they fail.

But, the ideas of virtualization need management entities to oversee them and decide to move virtual entities, and coordinate the movement of these virtual entities from one place to another.    VMware will soon have their Site Recovery Manager[4], and Linux has Linux-HA[3] (among other solutions).  The purpose of such management entities is to detect and respond to failures and administrative requests by starting, stopping, and migrating virtual entities in their purview to different sets of physical servers.

What does server (or OS-level) virtualization brings to the table here?

Why would I care?  If HA systems produce a result which sounds so far like it would be similar or even superior from the customer's perspective to what OS level virtualization does (at least from an availability perspective), then why should I care?  The answer is not found in responses to failures, but in responses to administrative events.  If you want to migrate a running virtual machine from physical machine A to physical machine B, many server virtualization techniques (including Xen[10] and VMware) provide a new tool - transparent migration.  In transparent migration, the virtual machine image is migrated from physical machine A to physical machine B while processing work continually without appearing to stop.  This is a pretty cool trick, since you can then migrate virtual machines with apparently zero downtime.  Sadly, you can't migrate a crashed OS or an OS from a crashed server in this way.  As a result, it doesn't help at all with unplanned outages.  However, it makes it possible to take a planned outage with very minimal impact.  In a practical sense, if your server is running full-throttle and writing on all the pages in memory very rapidly, the migration will likely impact performance. So, it's best to not to migrate virtual machines during peak load without good reasons.

Can conventional HA software manage transparent migration?

The simple answer is "mostly no".  Most HA software has only three basic operations that they perform on their resources:  Start, Stop and Monitor.  If you map these operations to operate on virtual machines as resources, start means boot the OS, stop means shut it down, and monitor means look and see if it appears to be running correctly.  Note that each of these also operates on only one machine at a time. A migration like described above becomes stop the virtual machine on machine A, and then start it on machine B.  This is hardly transparent, since the stop and start operations create an outage which will range in time from two minutes to 30 or more minutes and completely loses all transient application state.

The problem is in the HA software's model of resources (or services).  The operations which it performs are like this OP(resource, node) for any given OP like start, stop or monitor.  What's needed instead is a operation model which permits OP(resource, fromnode, tonode).  In addition, when one adds dependencies, then the migratable resource can't depend on any non-migratable resource. [It's actually a little more complicated than this, but it would take too much time to explain all the details in this post).

Are there any HA packages that can model transparent migration?

Yes.  Sorry to sound a bit like a broken record[13], but Linux-HA[3] can do that very nicely - and to my knowledge it is the only HA system which can.  It includes support for migratable resources like virtual machines and container virtualization, and handles the complexities of dependencies alluded to above, along with other potential problems not alluded to above.  So, if you use Linux-HA to manage your virtual machines, you get full use of transparent migration without having to figure out some monumental kludge to shoehorn something kind of like transparent migration into your HA package.  It just works.  And, it just works for containers, and resources which support checkpoint restart.  All in all, a nice feature I'd say.

What does transparent migration have to do with disaster recovery?

Nothing.  Well... Almost nothing.  Let me explain...  Since transparent migration only helps move services when the machines are up, that's not normally very useful in a disaster - since presumably a fire or flood or power failure, or building collapse or whatever has rather rudely and typically unexpectedly interrupted service - precluding migration. Why did I say almost nothing above?  Because some kinds of disasters (hurricanes, and floods for example) often give warning in advance - so you can use transparent migration for those cases.  The other reason is that you can test your disaster recovery without any service outages.  Or rather, you can partially test your disaster recovery.  If there are any resources (disk drives for example) which are only needed during boot, then if those are missing, you probably won't notice when you migrate over and back. So, it is still desirable to periodically do non-transparent migrations of your services to make sure things are all still working after the latest configuration changes.  However, you can migrate over and back every weekend, and maybe once a month or once a quarter perform non-transparent migrations just to make sure all your bases are covered.

Can't you do this with an HA cluster split across sites?

Yes.  You can do it with only service virtualization and a conventional HA system, or you can do it with Linux-HA[3] and either service or server virtualization.  In either case, you have to prepare for the replication of data[5] across sites.  Note that making sure all the data can be accessed from multiple physical machines is part of the baseline discipline which is imposed both by virtual machine failover and by service failover.   This discipline is a starting place for any kind of disaster recovery.  This is a valuable discipline to undertake.  So, although this discipline of replicating data and OSes is a good thing, since it's not unique to virtual machines, this part is more hype than reality.  There's nothing special about virtual machines in this respect.  In this respect, any differences that exist, don't exist because of virtualization per se, but how easy the management tools are to use correctly, and how comprehensive they are.

Advantages of service virtualization

• Monitors individual services [6] [7] - can detect and recover from software errors that don't cause OS crashes

• Can recover individual services (more fine-grained)

• Can recover servers and services without time for a reboot  (faster)

Advantages of Server virtualization

• Transparent migration for administrative events is outage-free

• All virtual machines more or less look alike (simplicity)

Best of Both Worlds?

Linux-HA[3] can model virtual machines as resources, and it can model normal services (web server, etc) as resources.  Can it do both at the same time?  Well... Yes and No...  Doing this would require that you have both virtual machines as resources and as cluster members at the same time.  Although you could do this, it might have unpleasant side effects.  However, you could have a cluster of virtual machines and a cluster of real machines - and that would work - but would be two clusters to manage.  To truly get the best of both worlds, you'd have to implement our container resource proposal[8] or something like it.  Of course, since Linux-HA is an open source project, we'd love to see a patch to implement it ;-).

See Also

Virtualization Blog[9],
Dabcc virtualization and server management web site[12],

References

[1]  http://www.byteandswitch.com/document.asp?doc_id=133643
[2] http://en.wikipedia.org/wiki/Virtualization
[3] http://linux-ha.org/
[4] http://www.vmware.com/company/news/releases/srm.html
[5] http://techthoughts.typepad.com/managing_computers/2007/09/automated-disas.html
[6] http://techthoughts.typepad.com/managing_computers/2007/09/monitoring---a-.html
[7] http://techthoughts.typepad.com/managing_computers/2007/09/tools-for-servi.html
[8] http://old.linux-foundation.org/developer_bugzilla/show_bug.cgi?id=1417
[9] http://www.virtualization.info/
[10] http://www.xensource.com/
[11] http://www.vmware.com/
[12] http://www.dabcc.com/
[13] http://en.wikipedia.org/wiki/Gramophone_record

Tools for monitoring services

For the purposes of this posting, I'm concentrating more on things that can tell you if a particular service is working or not working and recover a failed service, and less on a datacenter-wide view of service health.

As I discussed in my previous posting on this topic[1], there are three basic ways to monitor a service:

1. See if the process providing the service is alive
2. Use the service API to determine if the service is well (simplified version: check to see if someone is listening on the port)
3. Instrument the service to periodically heartbeat (check in with) a watchdog program or device

There are a lot of programs to monitor services using these three methods.  Many of the open source programs to provide these kinds of services are listed on the Linux-HA service monitoring page[2].  Rather than reproduce that page here, I'll discuss a few of the better-known programs found there. 

• Monit[3] is very highly thought of by many sysadmins, and is often used for service monitoring and restart.  The information in this post about monit[3] was provided by the most excellent Christian Wilken.
  
  Monit is a monitoring tool which can take the necessary action to ensure service availability. It can monitor services locally or remotely by polling a specific port (type 2 monitoring). It can monitor binary files. I.e. you might want monit to monitor your apache binary file or any other binary on the system. It checks the md5sum and the octal permissions of the file and warns the administrator if something has changed (and unmonitor the service). It can also check the uid and gid of the file.
  
  Through the integrated web interface you can see uptime, cpu load and memory consumption for each service monitored. you can decide what to do when a service fails.
  
  Here's an example: you want to monitor apache (port 80) and if it fails you want to restart the service. If it fails to restart apache (after let's say 3 tries) you can set it up to raise an alarm (and send a mail to the admins). you can also setup the apache service monitoring to depend on apache_bin (the binary apache file). That means if the apache binary has changed the apache service is unmonitored until you take action.
  
  Monit also monitors its own binaries so if you (or someone else for that matter) updates the monit code you will also get an alarm and you have to take action. However, it does not monitor its own operation, or register itself with some other monitoring service like /dev/watchdog or apphbd.   Monit is a very flexible monitoring/automation tool and its config file is simple and easy to learn.  Monit is a simple yet effective tool.
• Mon[4] is another well-thought-of monitoring tool.  It is a straightforward Perl program which has goals very similar to Monit.  Mon monitors services, and then takes actions when services are found to have failed.  For example, it will notify an administrator, or restart a service.  It comes with scripts which are capable of monitoring a large number of services.  It has been around for many years, and has many happy users.  It is simple to configure, normally monitors remotely, supports dependencies between monitors, typically is used as a type 2 monitoring service.  Like most similar services, nothing normally monitors mon's operation to see if it is still running, or operating correctly.
• Nagios[7] is a very flexible system management package which provides for a lot of capabilities for monitoring servers and services  and taking a variety of actions when failures occur.  It is designed to watch and manage your entire data center.  For the most part, it uses SNMP to do that - however it can use other methods for monitoring as well.  One of the more interesting features of Nagios is that it supports service dependencies.  Normally, each service is defined by hand, and it can take a wide variety of actions when things fail, including escalation actions.  However, it does not natively provide process restarting.   Recently Gerard Petersen has documented[8] a way of getting nagios to do this with a consistent set of rules.  Understanding his method of doing this generically requires an extensive knowledge of Nagios and Gerard describes it as "somewhat cumbersome".  Nevertheless, it does add this capability to Nagios in a generic way.  It is a "type 1" monitoring system (done remotely with ssh) - it only checks if the process is alive.  You can tie this capability together with real service functionality checks (type 2) using dependencies.  However, if the service can't be stopped gracefully, and it has to do a kill -9 to stop it, then the service may not work properly afterwards.  This is a pretty unsurprising caveat.  However, it will attempt to kill the process and recover without manual intervention.  Nagios works remotely and performs both type 1 and type 2 checks (remotely), and implements service dependencies.
  
  It is worth noting that this Nagios technique will not work for servers which are part of an HA cluster, because there is no fixed association between a server name and which services are running on it.  You could use it to monitor the HA server processes if you like, but usually the HA software will do that itself.  Note that when you combine management services, you need to make sure you understand which management service is going to control what, know how each affects the other and  keep them strictly out of each other's way.
  
  Nagios does much more than just monitor services and restart daemons, however.  It keeps statistics, and provides historical and near-real-time performance and load graphs, provides an extensive set of alerts, and performs a variety of other system management functions. I don't know if/how the central Nagios processes are monitored.
  
  
• apphbd[9] is part of the Linux-HA[10] suite of programs.  It performs type 3 monitoring of properly instrumented services.  That is, you can write a program which connects to apphbd, and sends it heartbeats periodically.  When your application doesn't send a heartbeat message by the expected time, or exits without informing apphbd of its intention to exit, then apphbd will notify another application of this condition, which will restart your application.  Apphbd normally registers with /dev/watchdog to send heartbeats to the kernel watchdog device.  As a precaution against false reboots, apphbd locks itself into memory and runs with soft realtime priority.
  
  
• Heartbeat[10] from the Linux-HA project is typically thought of as a clustering package.  However, it is perfectly happy to manage a single system - which it thinks of as a cluster with one node.  It is capable of doing both type 1 and type 2 checks, implements service dependencies, and will automatically restart services if they die, including services which depend on failed service.   The configuration is in XML, and isn't difficult but can be tedious.
  
  For the case of monitoring a single machine, I wrote a prototype script[11] which looks at your init.d directories for all set of active services, and creates a cib.xml configuration file for the set of services on your machine.  Heartbeat will then poll to monitor those services and ensure that they are running, and restart them (and any services which depend on them) automatically.  Because the configuration this script creates relies solely on the init scripts, the monitoring is done by using the status actions of the LSB.  As a result, for the most part these are type 1 monitoring.  To do type 2 service monitoring, it is necessary to create an R2-style Heartbeat configuration and make sure the services to be monitored have a repeating monitor action declared for them.  Of course, the script mentioned before makes sure all this happens.
  
  If you wish to do more thorough monitoring, you can convert these scripts to OCF resource agents, modify the configuration and perform any kind of monitoring you wish.  Heartbeat itself can be monitored by SNMP or CIM - which lets you easily include Heartbeat as being monitored over the data center via Nagios, OpenNMS or other package.  In addition, if you wish to protect yourself against OS hangs and crashes and hardware problems, you can easily add servers to your one-node cluster, make some small rule changes, and now your service will continue running even if a server or OS dies.  The process of changing from a 1-node to an n-node cluster isn't at all complicated.  If one has things configured properly, and one uses the autojoin[12] feature.
  
  It is worth noting that the action which Heartbeat takes when a service can't be stopped is quite different from Nagios'.  In Heartbeat, when a resource (service) can't be stopped, Heartbeat would normally reboot the machine.  Although this sounds drastic, it can get most services running even when the problem is in the kernel.  Since normally Heartbeat is managing a cluster, the service would normally be taken over immediately by another node in the cluster.
  
  With regard to monitoring Heartbeat itself: it monitors its own operation, and it can be configured to heartbeat either with a watchdog device like softdog, or with apphbd (discussed above).  When you put it into a cluster, then the nodes of the cluster monitor each other, and use STONITH to reboot machines which have stopped working.  If you use Heartbeat then, you get as complete a set of monitoring as is available.
• cl_respawn is a simple tool which is packaged as part of the Heartbeat[10] package.  cl_respawn takes arguments which tell it how often to heartbeat with apphbd, and the value of a "magic" exit code.  This exit code is one that when the application exits with this return code that cl_respawn does not restart the its child process.  Although it only does type 1 monitoring, it doesn't poll to see if a process has died, ithe OS gives it a signal and it responds immediately.  When you invoke a process with cl_respawn, it is necessary to make sure it does not fork off and run in the background.
• Supervise[13] from D. J. Bernstein's daemontools[14] replaces the usual init.d with a service directory, so that things are installed, stopped and started and otherwise managed in a non-standard way.    However, as supervise is part of this whole process, service monitoring and restarting comes along for free.  When a daemon is started it becomes a child of supervise.  When a daemon dies, supervise is notified and it is automatically respawned immediately without waiting for a poll interval.  Supervise itself does not appear to be monitored for failures by svscan, which appears to be unmonitored.  Daemontools does not appear to support service dependencies.  Like cl_respawn, when you invoke a process with daemontools it is necessary to make sure it does not fork off and run in the background.
• Watchdog drivers can either be implemented in software like softdog[15]  or they can be implemented in hardware.  All the watchdog drivers that ship with Linux share the same API - that implemented by softdog[15].  As a result, a program which works with softdog can work quite nicely with the hardware-based watchdog drivers.  It's worth noting that kernel hangs which completely disable kernel timer services can cause softdog to malfunction. The good news that hangs like this are extremely rare.   If one has a hardware based watchdog device, it takes a hardware failure for these to fail to reset a hung machine.  This too, is extremely rare.  If one is a true belt-and-suspenders sysadmin (with a enough time and budget), one could use both a software and hardware based watchdog
• ldirectord[16] is a specialized monitoring system which is delivered as a subpackage of the Linux-HA suite.  It monitors servers and services in a load balancing cluster and takes dead servers out of the load balancing rotation, and restarts services when they stop responding.  It is a type 2 service, and because it's an HA resource, is normally monitored by Heartbeat itself.
  

See Also
Howtoforge articles on monitoring[17].

Related systems - not quite on topic

Ganglia[5] is a scalable distributed monitoring system for high-performance computing systems such as clusters and Grids. It is based on a hierarchical design targeted at federations of clusters. It relies on a multicast-based listen/announce protocol to monitor state within clusters and uses a tree of point-to-point connections amongst representative cluster nodes to federate clusters and aggregate their state.

Although it doesn't attempt to repair problems, it is widely used in high-performance clusters and might be of interest to people reading this post.

Opennms[6] is a Java-based enterprise grade open source network monitoring platform. It consists of a community supported open-source project as well as a commercial services, training and support organization.

The goal is for OpenNMS to be a truly distributed, scalable platform for all aspects of the FCAPS network management model, and to make this platform available to both open source and commercial applications.

OpenNMS is oriented towards responding to SNMP data and traps, but can be modified to monitor other things.  It does not appear to me to be well-suited to recovering from a service or process hanging, nor for monitoring a single system.  Where it shines is in managing an entire data center with a large number of SNMP-enabled servers and network devices.

References

[1]  http://techthoughts.typepad.com/managing_computers/2007/09/monitoring---a-.html
[2]  http://linux-ha.org/RelatedTechnologies/MonitoringSoftware
[3]  http://www.tildeslash.com/monit/
[4]  http://mon.wiki.kernel.org/index.php/Main_Page
[5]  http://ganglia.info/
[6]  http://www.opennms.org/
[7]  http://www.nagios.org/
[8]  http://www.gp-net.nl/2007/05/12/nagios-remote-eventhandling-for-init-scripts/en/
[9]  http://linux.die.net/man/8/apphbd
[10] http://linux-ha.org/
[11] http://hg.linux-ha.org/dev/raw-file/8c5da9553636/tools/1node2heartbeat
[12] http://linux-ha.org/ha.cf/AutojoinDirective
[13] http://cr.yp.to/daemontools/supervise.html
[14] http://cr.yp.to/daemontools.html
[15] http://en.wikibooks.org/wiki/Linux_Kernel_Drivers_Annotated/Character_Drivers/Softdog_Driver
[16] http://linux.die.net/man/8/ldirectord
[17] http://www.howtoforge.com/taxonomy_menu/1/59