Managing Computers with Automation

The Incredible Power of Asking The Right Questions

There is incredible power in asking the right questions, and following the answers where they lead - especially when they lead to uncomfortable places.

As you probably remember, in March, 2011, a magnitude 9.0 earthquake hit Japan resulting in a massive tsunami which damaged the Fukushima nuclear power plant. Some of the most serious damage occurred because there was no power to cool the reactor.

(read more)

I've moved my blog content to WordPress, which is why the whole article isn't here.

Crypto background for the Assimilation project

Since its inception, the open source Assimilation project has been concerned with security, and paranoid at every opportunity. Like a lot of software, it has serious security concerns simply because of what it’s designed to do.  On the one hand, our nanoprobes run on every server in the enterprise and exercise root privileges – creating a potentially dangerous attack surface.  On the other hand, we incrementally create a high-value database which has fine-grained and up-to-date information about everything in the environment – software versions, ports, services, IP and MAC addresses, known security vulnerabilities – a veritable treasure map for an attacker.    This article details why cryptography is essential for communication in this environment, and some unique aspects of the problem we’re solving that affect how we use it.  It is our hope our readers (this means you!) will give us a thorough flogging review of how we’re using cryptography in our software – in this article and the next.

This blog has been moved: This article can be found here:  http://bit.ly/assimcrypt1

Rules to automatically monitor services using OCF resource agents

In my previous post on automatic service monitoring rules, I explained the general principles behind the Assimilation project's rules for automatically monitoring with LSB-style init scripts based on what we've discovered.  In this post, I'll extend that further and explain how to monitor using Open Cluster Framework (OCF) resource agents - which are far more powerful, but a bit more interesting to configure.

As you will recall from the previous post, LSB resource agents rules were written using JSON with comments.  The rules for recognizing and configuring OCF resource agents are very similar - and are also described in JSON.  Faithful readers may recall that I wrote a blog post on how OCF monitoring works - which may be of interest to review.

With an LSB resource agent, all you had to do was recognized that it controlled this particular service - there is nothing to configure.  Either it matches, and you can issue a status operation, or it doesn't.  With OCF resource agents, they support configuring multiple instances of resources on the same machines, through a series of name/value parameters in the environment.  Some parameters have defaults, but some have to be configured in order to properly monitor an instance.  The challenge for the Assimilation code is to figure out these parameters from the discovery information and plug them into the right places.

As part of the Assimilation project, we've written an neo4j OCF resource agent for monitoring and controlling Neo4j which we'll discuss here.  The LSB init script merely looks to see if the neo4j process is running.  The OCF resource agent goes further - it issues a query to Neo4j and makes sure it gets a sane result returned.  For good measure, it implements multiple levels of monitoring - one with a simpler query and one with a more complex query.  This allows us to detect when services hang or give erratic results - which is far more interesting than merely looking to see if the process is in the process table.

There are a few interesting parameters for this resource agent which we can derive from our discovery information.  These are:

• ipport - the IP:port address that neo4j is listening on for REST queries
• neo4j_home - the root directory of the neo4j installation
• neo4j - the name of this Neo4j instance

For the purposes of monitoring, only the first parameter is of interest.  The other two parameters are necessary in order to know how to properly start and stop the service (which is part of the OCF resource agent specification).  So, let's see how we tell the Assimilation code how to find and fill in these parameters.

{
    "class":    "ocf",
    "type":     "neo4j",
    "provider": "assimilation",
    "classconfig": [
#       OCF parameter   expression-to-evaluate      regular expression
        [null,          "@basename()",               "java$"],
        [null,          "argv[-1]",                 "org\\.neo4j\\.server\\.Bootstrapper$"],
        ["ipport",      "@serviceipport()",          "..."],
        ["neo4j_home",  "@argequals(-Dneo4j.home)", "/"],
        ["neo4j",       "@basename(@argequals(-Dneo4j.home))","."]
	]
}

The last two columns are the same as in the LSB example in my previous blog post, but the first column is new.  It gives the name of the parameter that corresponds to the expression in the second column.  If this expression and regular expression are only used for recognizing the service and doesn't have a corresponding parameter, then we use the JSON tag null to indicate that.  Our previous matching patterns are the same as they were, and don't have a corresponding parameter to go with it.  To fill in the ipport parameter in our example, we call the serviceport() function which examines the IP address/port combinations that neo4j is listening on, and returns a single IP/port that it is listening on.  This function prefers to return ipv4 addresses, and lower numbered ports.  If it is listening on an ANY address it returns the appropriate loopback address (127.0.0.1 or ::1).

This data that serviceipport() returns is derived from data discovered by our tcpdiscovery discovery agent.  The tcpdiscovery agent currently uses netstat to gather this information.  Because of this we know exactly how to monitor this particular instance of neo4j.  The other two parameters use the argequals() function.  This function searches the command line of this service (collected by the tcpdiscovery agent) for a "-Dneo4j.home=something" argument and returns the something part.  Because you're one of my brilliant readers, you doubtless have no figured out that neo4j is invoked with that argument as a parameter - enabling us extract it from the command line.

As time goes on and we start recognizing more and more services, we will no doubt add more functions to simplify writing these rules.  When the Assimilation code is extended to use other monitoring APIs, these same principles will enable us to configure those monitoring methods using nearly identical formats to either the LSB or OCF formats discussed in these two blog posts.

What's your reaction to this?  Do these kinds of rules seem straightforward?  Incomprehensible?  Useful?  A waste of time?

 

Related articles

Neo4j: The 'Thinking in Graphs' Learning Curve

Neo4j 2.0 A Dream Graph Database

Rules to automatically monitor servers using init scripts

From a monitoring perspective, one of the most exciting possibilities in the Assimilation project comes from the integration of monitoring and discovery.

We've recently implemented the rules which will cause services to be automatically monitored once they're discovered.  In other words, you don't have to tell the system to monitor these services, they'll just get monitored automatically.

For example, if you have a rule which describes how to monitor mysql, then whenever it sees the mysql service, it can start monitoring it without human intervention.

In this blog post, I'll describe how to tell the system to monitor services by using LSB-style init scripts.  Later posts will cover more interesting ways of monitoring.

Monitoring with init scripts

Let's start simple - and see how you can define rules that describe how to recognize a service and monitor it with its init script -- /etc/init.d/service-name status.  For this example, let's look at what it takes to recognize that a service is the secure shell, and then to specify that we monitor it via an LSB-style init script.  Although this isn't a really good way to monitor a service, it does detect if it goes away.

The Assimilation Project's tcpdiscovery process discovers a lot about a service - what ports and IP addresses it listens on, what the binary name is, what arguments it was invoked with, what the user id and group id is and a lot of other things as well.

So, if we want to determine that some service we've discovered is an sshd service, we could do is look to see if the pathname of the binary has a basename of 'sshd'.  Then we would know we could monitor it with /etc/init.d/ssh status.  That would normally be considered more reliable than just saying if it's running on port 22, it must be sshd.

In Assimilation (and Pacemaker) terminology, we call this kind of a monitoring method an LSB monitoring method.  Here's how you would define a rule that does what we've been talking about.  You'll notice that this is just JSON (although we allow comment lines).

{
    "class":        "lsb",
    "type":         "ssh",
    "classconfig": [
        ["@basename()",    "sshd$"]
    ]
}

This tells the system that this is an LSB class (init script) monitoring rule, that the service can be monitored via /etc/init.d/sshd (type=sshd).  In the case of LSB scripts, what the rest of the configuration is trying to do is do pattern matching on things we've discovered to see if this looks like something that can be monitored by this particular kind of monitoring. If you're running upstart or systemd for your init system, there are some exceptions which I'm ignoring for the sake of simplicity.

The list of elements in classconfig are tuples containing these elements:
    (expression-to-evaluate, regular-expression-to-match)

So, in this case, we have an expression which calls a monitoring-defined function called basename.  Basename returns the basename of its argument, or if it isn't given an argument, it operates on the pathname of the binary that is providing the service.  In our case, this is "/usr/bin/sshd".  This function then returns "sshd".  This is then matched against the (anchored) regular expression 'sshd$'.  If the regular expression matches, then we know that this service can be monitored by the ssh init script.

The Assimilation system can then start a monitoring action against this service automatically - because now we know how to monitor it.

This is the basic template for monitoring many services using init scripts.  However, when the service is written in Java, or in python, the basename of the binary isn't sufficient to tell us that this is the service that we intended.

Here's an example for monitoring the Neo4j graph database - which is written in Java.

{
  "class":       "lsb",
  "type":        "neo4j-service",
  "classconfig": [
      ["@basename()", "java$"],
      ["argv[-1]",    "org\\.neo4j\\.server\\.Bootstrapper$"]
  ]
}

As before, we match the binary name - in this case against "java" - but we also match the final argument "argv[-1]" against the string org.neo4j.server.Bootstrapper.  The double backslashes are required by JSON.  The [-1] notation refers to the last element of the array - something a bit like argv[len(argv)-1]. You can refer to the next-to-the-last element as argv[-2] and so on.

The name 'argv' is the name of the argument list for the listening process in the neo4j graph node representing this service.  Names like argv are evaluated in the context of both the service node and the server node.  If it is defined as an attribute of either node, the resulting value is used.  If the attribute value is a JSON string, then you can also write expressions that look for values inside the JSON.

Expressions consists of names like 'argv', or argv[1], or function calls like @basename.  The currently-defined set of functions is shown below:

• argequals searches a list for an argument of the form name=value. The name is given by the argument in args, and the list 'arglist' is assumed to be the list of arguments.  If there are two arguments in args, then the first argument is the array value to search in for the name=value string instead of 'arglist'.
• basename returns the basename from a pathname. If no pathname is supplied, then the executable name is assumed.
• dirname returns the directory name from a pathname. If no pathname is supplied, then the executable name is assumed.
• flagvalue returns the value after a flag - for example flagvalue(--debug) would return 5 when examining --debug 5 in the argument list.
• OR returns the value of its first non-NULL argument.
• serviceip searches discovery information for a suitable concrete IP address for a service. The optional argument to this function tells it an expression that will provides the map of IP/port combinations discovered for this service.  The IP address it returns corresponds to the port returned by serviceport().  When possible, it returns an IPv4 address.
• serviceipport searches discovery information for a suitable ip:port combination. The optional argument to this function tells it an expression that provides the map of IP/port discovery information for this service.
  The return value is a legal ip:port combination for the given address type (IPv4 or IPv6).  It defaults to the lowest port number being listened to by the service.  When possible it returns an IPv4 address.
• serviceport searches discovery information for a suitable port for a service.   The optional argument to this function tells it an expression that provides the hash table (dict/map) of IP/port discovery information for this service.  If a service is listening on multiple ports, it defaults to the lowest port number being listened to by the service.

The reasons for some of these functions will become apparent in later blog posts.

As noted earlier, this is not the most interesting kind of monitoring to do (in fact, it's usually very minimal).  Later articles will cover the same ground for more interesting (and more complex) kinds of monitoring.

I have a few questions for you readers out there.  Does this make sense?  Assuming we can handle better monitoring methods (and we can), does the idea of not having to configure monitoring seem worthwhile?

Things I learned at the Open Source Monitoring Conference

I recently spoke at the OSMC (Open Source Monitoring Conference and how that went - and also how an audience full of system administrators reacted to the idea of representing their systems as a graph.  This blog post will cover those two topics.

First all, the Assimilation project's appearance at the OSMC was a smashing success.  I was mobbed after my talk with people wanting to learn more, and contribute to the project.  Now all that has to happen is for all those interested to show up and help out.

Thomas Wildhalm summed up his reaction in this tweet:

There's some serious science spiced up with magic and unicorns at @OSSAlanR 's talk at #osmc What I heard so far: #wow and #awesome

One interesting thing about this conference - it was aimed solely at practictioners of monitoring and system management.  They completely get  that they always describe their systems as a graph - their mental models of systems are graphs - and the network people in the audience even more so.

This is easy to validate - ask anyone involved in system or network management to go to a whiteboard and describe some part of the systems they manage.  They always draw circles and boxes and arrows.  That is -they draw a graph.  In all the time I've been in this business (which has been a while), this has always been the case.  There are even specialized tools like Visio which are part of any network engineer's stock in trade - that is - tools to draw graphs representing system architectures.

In the conference, I followed up with "when the model your software has matches your mental model, it's much easier to understand and manage without screwing it up".  They totally get that.  SysAdmins live their lives getting the hulks of software that have crashed back up on their metaphorical feet.  So understanding how a tool represents their data and how that matters to them is something they completely understand.

I spent about 15 years of my career in high-availability systems.  During that time, when someone had made a true and royal mess of their system, it was almost always the case that some minor thing had gone wrong, and the administrator had tried to "help" the system back to health.  But because their mental model of the system didn't match how the system actually was, they wound up making things worse.  I'm sure most any Admin has a story or two about how they have made things worse at one time or another.

This is why they resonate so much with having a system whose mental model matches theirs - it makes these kinds of embarassments much less likely.

Related articles

Mental and System Models: Class Reflection from 10/16

Why Graph Search Is Moving Databases to a New Level

Does Open Source Rise Spell The End Of Traditional Software Vendors?

The Increasing Usages of Open Source Software

Assimilation Message Formats

I've recently been discussing some nitty gritty details about how packets are laid out on the wire with a member of the Assimilation Project, so it seemed good to explain how our packets are constructed and deconstructed to everyone while I was at it.  This blog post talks about how we send lay out the bytes on the wire, with a bit of why and how.

Type, Length, Value messages

First off, the general low-level format is a variant on Type-Length-Value (TLV) messages.  In the software, we refer to these individual TLV fields as Frames, which are in turn concatenated into a FrameSet.  One or more FrameSets can be put into a packet.  Our packets are UDP (with a user-level retransmission protocol to ensure reliable delivery).  It's also worth remembering that in the Assimlation Project, communication between endpoints can consist of only a handful of packets every few months - or even every few years.  By design, silence is the norm - communication is the unusual case.

Each FrameSet has a type of its own.  There are three "magic" classes of TLV Frames, which if they appear, must appear in this order:

1. SignFrame - Digital Signature - currently not cryptographically secure (that's coming).  This Frame is not optional.  It indicates the type of signature and provides the signature data.
2. CryptFrame - Encryption Frame (optional)- indicating what type of encryption was used to encrypt the frame.
3. CompressFrame - Compression Frame (optional) - indicating what kind of compression was used to compress the frame.

If the packet is neither encrypted nor compressed, then it will only have a digital signature frame before the "payload" frames.

Other classes of frames include the following:

• SeqnoFrame - a sequence number frame (used by the reliable protocol)
• AddrFrame - a network address (any one of several types)
• CstringFrame - a NUL-terminated C-style string
• IntFrame - an integer (2,3,4 or 8 byte integers)
• IpPortFrame - an IP address with a port number (can be IPv4 or IPv6)

There are also many types that correspond to a single Frame subclass.  For example, HOSTNAMEs, JSDISCOVER, DSCJSON, RSCJSON are all TLV types that correspond to CstringFrame objects.  In other words, hostnames, output from discovery operations, discovery requests, and resource descriptions are all represented as strings (CstringFrames) - but each has its own TLV type.

Why Signatures, Encryption and Compression?

Since the nanoprobes run as root, and are capable of doing potentially dangerous operations, it is only prudent that we have cryptographically secure digital signatures (which have not yet been implemented).  It is expected that each role (CMA, nanoprobe) will have its own signature.  Although it is possible for each nanoprobe to have its own unique signature, this causes headaches when nanoprobes send each other heartbeats.  It may eventually be implemented, but initial implementations are likely to be simpler.

The case for encryption is a little less obvious.  Although most data from monitoring and discovery is not likely to be security sensitive, in some environments the data will be considered sensitive by the customer - in which case encryption is a good idea.

Compression is needed because our communication is UDP - which limits message sizes to a little more than 60K bytes.  The most voluminous data is expected to be discovery results - which are JSON.  Initial experiments indicate that the compression ratio for JSON should be about 15:1, which gives about 900K for the maximum uncompressed packet size.  Eventually, we may need to add packet fragmentation to accommodate larger discovery results.

 Packet Construction (Marshalling)

The process of constructing a message byte string from a FrameSet proceeds as follows:

1. A first-to-last pass is made through the Frames to deterimine the message size, by adding up the Frame sizes.
2. Space to hold the packet is allocated
3. A last-to-first pass is made through the Frame, placing the data from each frame in its correct position in the message

The encryption and compression Frames perform unique operations.  Although processing things in reverse order seems odd, the demarshalling will reveal a method to my madness...

When a compression frame is encountered, it replaces the message data with a compressed version of the message.  If the compression method cannot (or is instructed not to) reduce the size of the message, it shifts the message up over the compression frame, effectively deleting it.  It adjusts the size of the message accordingly

When an encryption frame is encountered, it replaces the message data after the encryption Frame with an encrypted verison of the message.  Note that it is important that the packet be compressed before encrypting it.  Encrypted packets don't compress very well.

Packet De-Construction (Demarshalling)

The process of demarshalling a message is the mirror of the marshalling method.

1. A first-to-last pass is made through the message, creating Frames that correspond to the various elements of the message.

The following rules are observed:

• The digital signature is checked - and the role associated with that signature is determined and stored away.  (i.e., is this packet from the CMA, or from a nanoprobe).
• If the next frame is an encryption frame, then the remainder of the message is replaced by a decrypted form of itself.
• If the next frame is a compression frame, then the entire message is replaced by a decompressed version of itself.
• The remaining frames are demarshalled as before.

Although the signature, encryption and compression frames are special cases - by enforcing that they occur in the proper order, the amount of special case code is minimized.

Although this isn't exactly rocket science, by doing things in the right order, the whole process is reasonably straightforward - which in my book is a virtue.

Does this make sense to you?  How should it be improved?  How can I improve this description?

 

Related articles

How to implement "no news is good news" monitoring reliably

How Open Cluster Framework monitoring works

Cryptography - "Encryption and Decryption"

How GPG works: Encrypt

How Open Cluster Framework monitoring works

Another thing that I've taken for granted is that people understand how HA systems like Linux-HA or Pacemaker monitor their resources.  This is important to me in the Assimilation Monitoring Project (AMP) because we use the same techniques.  So this post concentrates on how typical resource agents following the Open Cluster Framework (OCF) API (which we like a lot) work.  Since these OCF resource agents were written for high-availability systems, they tend to be both both flexible and robust.  The OCF API is one of those things that's worked out very well - much better than expected.  Both Pacemaker and AMP implement other APIs as well, but for both this is their preferred APIs.

Years ago, I created and headed a team of people from several companies and several different operating systems to create a standard API for managing high-availability resources.  Resource is a vague and general concept - it's the name we used for the things HA systems manage.  It might be a web server, it might be a piece of hardware, it might be an IP address, a filesystem mount, or even something as abstract as a set of firewall rules.  The key thing is you have to assign a meaning to starting it, to stopping it, and you have to be able to tell if it is started (running) or not (stopped).  If you can meet these criteria, then you can make an OCF resource that models it.  The part we'll explore in this article is "can we tell if it's running properly or not"?.  Like the concept of resource, the idea of whether something is properly running is a very flexible idea, and can mean almost whatever you want it to mean.  The only real constraints are that if the start operation succeeds, then the monitor should normally succeed right after the start.  Conversely, if you have issued a stop, then it should monitor as "not running" until a start is issued.

The key to monitoring anything is to monitor the right thing - whatever that might be - and to not make it too trivial.  For example, looking to see if a daemon is running is usually too trivial for meaningful monitoring.  It's better than nothing, but not a lot... Instead you want to monitor something that is highly correlated to the correct functioning of the resource in question.

For example, in the case of a web server, you want to be able to give it a URL to retrieve, and get the right result.  You want to give it a "normal" web page - whose success is well-correlated to the success of the web server and the application behind it.  But you don't want to have to continually update the monitoring script every time you update the web server, so typically such a resource would return the right successful return code (200), and matches a regular expression.  For the OCF Apache resource, the default regular expression is very simple - it looks for an "</html>" at the end of the received stream.  It's very simple, but if the web server is broken, normally it won't look like that.  Of course, you can easily override it with some other value.  This kind of flexibility is common in OCF resource agents.

Similar things apply to other resource agents.  For example, in the MySQL, DB2 and Oracle resource agents, they perform a simple query - that is cheap and should always succeed.  In both cases, the query is against the authorized users, and the result is counted.  The result should be >= 1.  If it's not, then it failed.  Although it's simple, it's a real query that exercises most layers of the database system, and whose result is easily verified.

In general, you want resource agents that are cheap to run and you can run frequently.  However, there are occasionally needs for more in-depth monitoring actions - things you would like to do, but are too expensive to check frequently.

There is a Filesystem resource agent that mounts a filesystem.  The normal level check is something simple like checking to see if it's still mounted, and if a df command on it succeeds.  This is cheap and can be run frequently.  But what if that filesystem has been corrupted - by hardware failures or software bugs?  A more extensive check that might be worth running once a month would be to snapshot the filesystem and peform a full fsck on the snapshot.  The OCF resource agent API supports these multiple levels of monitoring through the OCF_CHECK_LEVEL parameter.  The invoker of a monitoring action is permitted to say how gnarly the test is supposed to be - and if the resource agent supports that level, it will act accordingly.  If it doesn't support that level, it will assume the next lowest level that it does support.

So, and OCF resource agent specification is flexible in two dimensions - it can take parameters to tell it what the resource is, and details of things like regular expressions, and also can be told how expensive a check it is expected to perform.

No matter what you want to check, it is likely that you can write an OCF resource agent to check it for you - and you can even select which check to perform at run time through the use of the OCF_CHECK_LEVEL parameter.  They can be as lightweight as you want, or as "meaningful" as you want (where you get to define what meaningful means to you).

To top it all off, OCF resource agents also provide metadata describing all the paramters that they take and what they mean - potentially in multiple languages.

One other note about implementations of the OCF standard - all the ones I know about monitor resources on the machines where the resources are.  For example, they execute the monitor action on a machine which is running the database.  They then use other techniques to watch this watcher - as is common in reliable systems.

Since AMP implements the OCF resource agent specification (among others), this also means that it will eventually be able to use this API to start and stop resources as well as monitor them.  Extensions to the API also support migrating virtual machines - and there are resource agents written for most common types of virtual machines.

Related articles

How to implement "no news is good news" monitoring reliably

The network is reliable

How heartbeats fit into hierarchies of watchers - and pings don't - or Who will watch the watchmen?

Somehow people seem to think that heartbeats and pings are the same thing.  They're not at all the same thing.  Heartbeats are typically semi-intelligent elements in a hierarchy of watchers.  Pings can't play the same role.  This blog post talks about why, and how hierarchies of watchers work both in general, and more specifically in the case of the Assimilation Monitoring Project (AMP)- which breaks with tradition in a useful and novel way.

Pings

Let's start with pings. The ping command literal ICMP echo request packet is sent to from one machine to another machine.  The ping is then returned as an ICMP echo reply when it is received.  This might is often done at a very low level in the OS, or might even be done by the NIC itself - which only requires power to the NIC and a network connection.  This means that an ICMP ping doesn't test very much at all.  In addition, response to pings is often disabled for security reasons.  So, this is a very poor method of determining even minimal health of a system.  All you really know for sure is that it has power, and was booted at some time in the past.  Not very helpful.

Heartbeats

Heartbeats have two portions, which are semi-independent.  There is the heartbeat sender, which is typically a user process which sends heartbeats at some interval as instructed.  There is the hearbeat receiver which expects and listens for heartbeats.  If it goes without hearing expected heartbeats for a certain period of time, it will report that fact upstream - in our case to the CMA.

Heartbeats are typically sent by user processes which are running under the OS, and if the OS stops scheduling user processes, heartbeats will stop being sent.  In addition the packets they send go through the normal user process queuing system, and if that fails, heartbeats will not be sent.  Many more things are validated as operational by sending and expecting heartbeats than by pings.

Quis custodiet ipsos custodes?

A key thing about monitoring systems for availability is that everything needs to be watched, and all the watchers need to be watched.  In Latin, this comes out to be the classic question "Quis custodiet ipsos custodes?" - Who will watch the watchmen?  The original question is more commomly thought about in political terms regarding corruption, but applies very well here where any computer can fail, and for reliability, you want to make sure that as many failures as possible are caught - so they can be repaired - whether by human or automatic means.

It is common in highly reliable systems like telephone systems to have a hierarchy of watchers, where the behavior of various elements is observed by other elements through a well-organized hierarchy - until the top level is reached.  These techniques may also be connected to watchdog timers, which reboot the machine unless they are positively acknowledged in a fashion similar to heartbeats.  These can also assist the watching hierarchy when properly used.

Once you reach the top of the hierarchy, hierarchies fail us - and then we wonder who will watch the watcher at the top.  In addition, hierarchies don't scale with low effort - one has to keep growing the hierarchy as the number of systems grows.

Within a single nanoprobe in AMP, the hierarchy is pretty normal - our nanoprobes invoke resource agents which watch the resources that they are assigned to watch.  The nanoprobes themselves set timers to watch the resource agents that are watching the resources.  But now the problem gets more difficult.  Who will watch the nanoprobes?  Although watchdog timers could be of some help in recovering from certain kinds of failures, they don't solve the problem of failure detection, and are a bit drastic for most customers.

In a conventional highly available computer cluster such as Linux-HA or Pacemaker, complex multicast consensus algorithms such as PAXOS or Corosync ensure that failures on other systems can be properly observed.  Unfortunately these algorithms require multicast and don't scale to thousands of systems.

Breaking With Tradition (oh my!)

It is at this point that AMP's architecture breaks with tradition.  We arrange for the overwhelming majority of systems to simply monitor each other by having each system monitored by two other systems, and each system also monitors two other systems.  By connecting the systems in a circle, like people standing in a circle holding hands, we can have each system have two neighbors which it waches, and which also watch it.  Because each system monitored by two other systems, losing a single system doesn't cause us to lose monitoring capability (there's no single point of failure).  This looks a lot like the picture below.

In AMP's montoring scheme, each system only has to know about two other systems, and each system is only known about by two other systems.  By closing the circle, the problem of who watches the watchers is solved in a more scalable way than a hierarchy - and without requiring multicast.

As a result, our central system (the CMA) doesn't have to actively watch these other systems, because they are watching each other.  In our implementation, the CMA does have to lay out the neighbor arrangements, but once set up, the watching takes place without intervention.

And as noted, at the top level, the CMA is small enough to use or get the assistance of standard clustering techniques.

This article covers many of the same issues as the "How to implement 'no news is good news' monitoring reliably" posting from a different perspective.  It seems good to present this information in several ways, because it seems a bit counterintuitive.

 

Related articles

How to implement "no news is good news" monitoring reliably

How to implement "no news is good news" monitoring reliably

Sometimes when you've been doing something for a long time, it's easy to take for granted the things you know.  In recent days, I've run into a several people who think that a "no news is good news" methodology for monitoring can't possibly be reliable.  So, this blog post is about that - how the Assimilation Monitoring Project (AMP) follows a "no news is good news" methodology and still reports failures reliably.

For monitoring, what you want is to know that if something fails, you'll know.  In the worst case, you might not diagnose it correctly, but you'll know that something's wrong.  So, let's look at how this is architected and see if we how everything is monitored so that we can see if we can find a way for failures to happen and slip past our monitoring net and not get reported.

You Can't Tell The Players Without A Program

There are a variety of components that are used in the monitoring process for a server named S.  They are:

• The hardware for S
• The network switch port that S is connected to or cable or NIC in S
• The nanoprobe running on S
• The OS for S
• The agents performing monitoring on S
• The nanoprobes, servers or NICs on S's neighbors - let's call them S+1, and S-1.
• The CMA - running under an HA system similar to Linux-HA or Pacemaker
• The network fabric connecting S and its neighbors to the CMA

So, let's see what happens when each of these fails.

If the hardware for S fails, for example, loses power or crashes, then the nanoprobe on S will stop sending heartbeats and both S+1 and S-1 will report the failure to the CMA.

If the switch port, cable or NIC for S fails, then S+1 and S-1 will report the failure to the CMA, and S will attempt (but fail) to report problems with S+1 and S-1.  The CMA will splice S out of the loop, and S+1 and S-1 will become neighbors.  This can also happen if a firewall blocks our packets.  Except for the nanoprobe running on the CMA server, all communication uses the same port - so, it is unlikely that a firewall rule would block communication to the CMA and not also block heartbeats to S+1 and S-1.

If the nanoprobe running on S fails (crashes), then S+1 and S-1 will report that S has crashed.  Not strictly accurate, but a failure does get reported.

If the OS for S crashes or stops scheduling user processes, then the nanoprobe on S will stop sending heartbeats and both S+1 and S-1 will report the failure to the CMA.

If the agents performing monitoring fail by a timeout, the timeout will be reported to the CMA.

If the agents performing the monitoring fail by a false negative, then a failure will be reported to the CMA.  If they fail and create a false positive, then no failure will be reported to the CMA.  These two cases are most likely cause by a bug in the monitoring agent.

If neighbor S+1 fails, then S-1 is still able to report failures of S.  If both neighbor S+1 and S-1 fail, then temporarily no one is able to report failures of S.  However, an anomaly will be detected, by the CMA because S should have also reported the failure of S+1.  More extensive probing from the CMA can be triggered by the anomaly to reveal the actual situation.  For any given failure of any given S, there should always be two or four servers (depending on neighbor toplogy) reporting that failure.  If the number of nanoprobes reporting the failure is not what it should be, then the CMA can probe systems to discover what the actual situation is.  Note that this is a very rare occurrence, and it is unlikely that the software will diagnose the exact failure correctly, but humans can be notified that something odd has happened.

If the CMA (or the server running it) fails when an HA system is protecting it, then there is a period of time when reports of all types from nanoprobes cannot be received.  Once the CMA restarts, then the IP address will be taken over by the new CMA, and reports will once again be processed.  All communication takes place by reliable UDP, so packets will be retransmitted until the new CMA is ready to process them.  It is important to note that packets are not acknowledged by the CMA until they have been fully processed or stored on disk in a place that won't get lost during a failover. It is worth noting that currently packets are not stored on a permanent queue, but are always fully processed before acknowledgements are sent.

If the network fabric between S and the CMA fails there are a few cases to consider.  If any of our monitored servers is connected to the failed switch, then it will be detected as noted earlier.  If this is a backbone switch to which none of our nanoprobes are directly connected, then this failure may not be detected by the current software.  If any of our heartbeat paths go through this switch, then we will detect something is wrong, but we may not diagnose it correctly.  With a fairly dense coverage of servers in the infrastruture by nanoprobes, the probability of this going unnoticed is low.  Although it might be desirable to do so, I don't know of any systems that test every possible path through the switching hierarchy.

These types of backbone switches also cannot be discovered by our current stealthy CDP or LLDP methods.  Although it is straightforward to create agents to monitor these switches, it does not fit as naturally into the current architecture as do servers, services, and directly connected switches.  This is similar to the case of systems on which we cannot install agents.  They can be monitored, but unlike the more normal cases, it is a completely manual effort to configure them and interpret the results.

With more "noisy" discovery of switches through SNMP, such switches could be automatically configured and the overall topology could be more fully mapped - ensuring that the monitoring agents are not stuck behind the failing switch.  This is not planned for the near future.

In summary, through the judicious use of redundancy, the AMP architecture does a very creditable job of being both quite reliable, and yet sticking to a "no news is good news" philosophy.

Does the GPL really work?

There have been a variety of articles written on why copyright sharing/assignment arrangements (or even project contributor agreements) are evil. For the most part, in the end what is concluded is that someone might (gasp) make a commercial product out of an open source project – with the implication that commercial software is evil and only the purest of copyleft arrangements are acceptable. As though somehow copyleft licensing is defeated by offering the software commercially.

But the truth is, that projects can go directions that disappoint contributors even without these kinds of arrangements, and the world is a complicated and interesting place.  In this complicated and interesting world, things can get messy, and are rarely one-dimensional or black-and-white.

Because the Assimilation Project depends heavily on the Neo4j graph database, which has such a copyright assignment arrangement, I've spent some time thinking about this. I've also talked to some of the people involved (although these are only my own opinions, and not theirs or my employer's).

What exactly is Neo4j doing?

They offer exactly the same software under two license models – commercial and open source. They offer some of their software under the GPL and some of it under the AGPL, and they offer these same versions under more conventional commercial licenses. They don't offer an enhanced version only as commercial software, and a less-capable version as open source. They have built a vibrant and active open source community around this model, and are making progress on their commercial endeavors.

Why would they do such a thing?

Licenses like Apache2 are much more “business-friendly” - but they invite companies who are not major contributors to provide commercial support. Given that there are some voracious competitors in the database space, this seems a very legitimate concern.

Why not just GPL or AGPL?

Many companies have concerns about the GPL and how to manage it in their organizations correctly. The more enterprise-ready versions of their products are AGPL which causes even more concerns about how to manage that correctly. The worst-case downside of the GPL is that your product might need to fall under the GPL. The worst-case of accidentally violating a commercial license is that you write someone a check, and go on your way. Companies are much more used to this latter risk, and the worst-case outcome is rather less scary.

Why not just provide proprietary add-ons with the GPL?

The problem with this approach is that the company typically views that they are only making money from the add-ons, and it becomes more difficult over time to continue to contribute to the base (GPL) project. This is one of the scenarios opponents of contributor agreements are concerned about – at least if the add-ons would not be permissible under the GPL. The company has essentially broken faith with the community. This creates friction with the community who often wants to develop similar capabilities to the commercial version, but the company resists. This often leads to a fork, where both parties lose.

What's the best that could happen?

The company will be able to attract more commercial customers, yet be protected by the GPL's reputation from predatory competitors and create a vibrant and active community. These profits are then poured back into the product, and both its investors and all its users benefit. In addition, it is easier for the commercial entity to offer indemnity to its customers and protect its contributors from lawsuits for things like patent infringement.

What's the worst that could happen?

A well-intentioned company like Neo Technologies might falter and get bought out by Evil Technology, Inc. (ETI). At some later date ETI could decide to stop providing any GPLed versions at all. This is really the core fear in the hearts of many who criticize assignment agreements – and again it breaks faith with the community. One thing that open source communities know how to deal with is breaking of faith. If the community hasn't been actively contributing to the project, then they still have their last GPLed version to create a fork from. The community in turn still has the significant value of this product – which they didn't have to pay for. If the community is strong and active, then a fork will likely become even stronger and leave the commercial version in the dust – with the commercial company being the loser. We saw this kind of thing happen with OpenOffice/StarOffice. When Oracle bought Sun, they refused to work with the community – and broke faith with it. The community rose to the challenge and created LibreOffice – which is now the overwhelming favorite over Open/Star Office.

Conclusion

All in all, it's my personal judgment that the GPL and the community development model does work. Open source software is not harmed by the fact that someone can offer the same software commercially, and any abuses which might occur are limited by the right to fork that is guaranteed by most open source licenses.