Managing Computers with Automation

Rules to automatically monitor services using OCF resource agents

In my previous post on automatic service monitoring rules, I explained the general principles behind the Assimilation project's rules for automatically monitoring with LSB-style init scripts based on what we've discovered.  In this post, I'll extend that further and explain how to monitor using Open Cluster Framework (OCF) resource agents - which are far more powerful, but a bit more interesting to configure.

As you will recall from the previous post, LSB resource agents rules were written using JSON with comments.  The rules for recognizing and configuring OCF resource agents are very similar - and are also described in JSON.  Faithful readers may recall that I wrote a blog post on how OCF monitoring works - which may be of interest to review.

With an LSB resource agent, all you had to do was recognized that it controlled this particular service - there is nothing to configure.  Either it matches, and you can issue a status operation, or it doesn't.  With OCF resource agents, they support configuring multiple instances of resources on the same machines, through a series of name/value parameters in the environment.  Some parameters have defaults, but some have to be configured in order to properly monitor an instance.  The challenge for the Assimilation code is to figure out these parameters from the discovery information and plug them into the right places.

As part of the Assimilation project, we've written an neo4j OCF resource agent for monitoring and controlling Neo4j which we'll discuss here.  The LSB init script merely looks to see if the neo4j process is running.  The OCF resource agent goes further - it issues a query to Neo4j and makes sure it gets a sane result returned.  For good measure, it implements multiple levels of monitoring - one with a simpler query and one with a more complex query.  This allows us to detect when services hang or give erratic results - which is far more interesting than merely looking to see if the process is in the process table.

There are a few interesting parameters for this resource agent which we can derive from our discovery information.  These are:

• ipport - the IP:port address that neo4j is listening on for REST queries
• neo4j_home - the root directory of the neo4j installation
• neo4j - the name of this Neo4j instance

For the purposes of monitoring, only the first parameter is of interest.  The other two parameters are necessary in order to know how to properly start and stop the service (which is part of the OCF resource agent specification).  So, let's see how we tell the Assimilation code how to find and fill in these parameters.

{
    "class":    "ocf",
    "type":     "neo4j",
    "provider": "assimilation",
    "classconfig": [
#       OCF parameter   expression-to-evaluate      regular expression
        [null,          "@basename()",               "java$"],
        [null,          "argv[-1]",                 "org\\.neo4j\\.server\\.Bootstrapper$"],
        ["ipport",      "@serviceipport()",          "..."],
        ["neo4j_home",  "@argequals(-Dneo4j.home)", "/"],
        ["neo4j",       "@basename(@argequals(-Dneo4j.home))","."]
	]
}

The last two columns are the same as in the LSB example in my previous blog post, but the first column is new.  It gives the name of the parameter that corresponds to the expression in the second column.  If this expression and regular expression are only used for recognizing the service and doesn't have a corresponding parameter, then we use the JSON tag null to indicate that.  Our previous matching patterns are the same as they were, and don't have a corresponding parameter to go with it.  To fill in the ipport parameter in our example, we call the serviceport() function which examines the IP address/port combinations that neo4j is listening on, and returns a single IP/port that it is listening on.  This function prefers to return ipv4 addresses, and lower numbered ports.  If it is listening on an ANY address it returns the appropriate loopback address (127.0.0.1 or ::1).

This data that serviceipport() returns is derived from data discovered by our tcpdiscovery discovery agent.  The tcpdiscovery agent currently uses netstat to gather this information.  Because of this we know exactly how to monitor this particular instance of neo4j.  The other two parameters use the argequals() function.  This function searches the command line of this service (collected by the tcpdiscovery agent) for a "-Dneo4j.home=something" argument and returns the something part.  Because you're one of my brilliant readers, you doubtless have no figured out that neo4j is invoked with that argument as a parameter - enabling us extract it from the command line.

As time goes on and we start recognizing more and more services, we will no doubt add more functions to simplify writing these rules.  When the Assimilation code is extended to use other monitoring APIs, these same principles will enable us to configure those monitoring methods using nearly identical formats to either the LSB or OCF formats discussed in these two blog posts.

What's your reaction to this?  Do these kinds of rules seem straightforward?  Incomprehensible?  Useful?  A waste of time?

 

Related articles

Neo4j: The 'Thinking in Graphs' Learning Curve

Neo4j 2.0 A Dream Graph Database

Rules to automatically monitor servers using init scripts

From a monitoring perspective, one of the most exciting possibilities in the Assimilation project comes from the integration of monitoring and discovery.

We've recently implemented the rules which will cause services to be automatically monitored once they're discovered.  In other words, you don't have to tell the system to monitor these services, they'll just get monitored automatically.

For example, if you have a rule which describes how to monitor mysql, then whenever it sees the mysql service, it can start monitoring it without human intervention.

In this blog post, I'll describe how to tell the system to monitor services by using LSB-style init scripts.  Later posts will cover more interesting ways of monitoring.

Monitoring with init scripts

Let's start simple - and see how you can define rules that describe how to recognize a service and monitor it with its init script -- /etc/init.d/service-name status.  For this example, let's look at what it takes to recognize that a service is the secure shell, and then to specify that we monitor it via an LSB-style init script.  Although this isn't a really good way to monitor a service, it does detect if it goes away.

The Assimilation Project's tcpdiscovery process discovers a lot about a service - what ports and IP addresses it listens on, what the binary name is, what arguments it was invoked with, what the user id and group id is and a lot of other things as well.

So, if we want to determine that some service we've discovered is an sshd service, we could do is look to see if the pathname of the binary has a basename of 'sshd'.  Then we would know we could monitor it with /etc/init.d/ssh status.  That would normally be considered more reliable than just saying if it's running on port 22, it must be sshd.

In Assimilation (and Pacemaker) terminology, we call this kind of a monitoring method an LSB monitoring method.  Here's how you would define a rule that does what we've been talking about.  You'll notice that this is just JSON (although we allow comment lines).

{
    "class":        "lsb",
    "type":         "ssh",
    "classconfig": [
        ["@basename()",    "sshd$"]
    ]
}

This tells the system that this is an LSB class (init script) monitoring rule, that the service can be monitored via /etc/init.d/sshd (type=sshd).  In the case of LSB scripts, what the rest of the configuration is trying to do is do pattern matching on things we've discovered to see if this looks like something that can be monitored by this particular kind of monitoring. If you're running upstart or systemd for your init system, there are some exceptions which I'm ignoring for the sake of simplicity.

The list of elements in classconfig are tuples containing these elements:
    (expression-to-evaluate, regular-expression-to-match)

So, in this case, we have an expression which calls a monitoring-defined function called basename.  Basename returns the basename of its argument, or if it isn't given an argument, it operates on the pathname of the binary that is providing the service.  In our case, this is "/usr/bin/sshd".  This function then returns "sshd".  This is then matched against the (anchored) regular expression 'sshd$'.  If the regular expression matches, then we know that this service can be monitored by the ssh init script.

The Assimilation system can then start a monitoring action against this service automatically - because now we know how to monitor it.

This is the basic template for monitoring many services using init scripts.  However, when the service is written in Java, or in python, the basename of the binary isn't sufficient to tell us that this is the service that we intended.

Here's an example for monitoring the Neo4j graph database - which is written in Java.

{
  "class":       "lsb",
  "type":        "neo4j-service",
  "classconfig": [
      ["@basename()", "java$"],
      ["argv[-1]",    "org\\.neo4j\\.server\\.Bootstrapper$"]
  ]
}

As before, we match the binary name - in this case against "java" - but we also match the final argument "argv[-1]" against the string org.neo4j.server.Bootstrapper.  The double backslashes are required by JSON.  The [-1] notation refers to the last element of the array - something a bit like argv[len(argv)-1]. You can refer to the next-to-the-last element as argv[-2] and so on.

The name 'argv' is the name of the argument list for the listening process in the neo4j graph node representing this service.  Names like argv are evaluated in the context of both the service node and the server node.  If it is defined as an attribute of either node, the resulting value is used.  If the attribute value is a JSON string, then you can also write expressions that look for values inside the JSON.

Expressions consists of names like 'argv', or argv[1], or function calls like @basename.  The currently-defined set of functions is shown below:

• argequals searches a list for an argument of the form name=value. The name is given by the argument in args, and the list 'arglist' is assumed to be the list of arguments.  If there are two arguments in args, then the first argument is the array value to search in for the name=value string instead of 'arglist'.
• basename returns the basename from a pathname. If no pathname is supplied, then the executable name is assumed.
• dirname returns the directory name from a pathname. If no pathname is supplied, then the executable name is assumed.
• flagvalue returns the value after a flag - for example flagvalue(--debug) would return 5 when examining --debug 5 in the argument list.
• OR returns the value of its first non-NULL argument.
• serviceip searches discovery information for a suitable concrete IP address for a service. The optional argument to this function tells it an expression that will provides the map of IP/port combinations discovered for this service.  The IP address it returns corresponds to the port returned by serviceport().  When possible, it returns an IPv4 address.
• serviceipport searches discovery information for a suitable ip:port combination. The optional argument to this function tells it an expression that provides the map of IP/port discovery information for this service.
  The return value is a legal ip:port combination for the given address type (IPv4 or IPv6).  It defaults to the lowest port number being listened to by the service.  When possible it returns an IPv4 address.
• serviceport searches discovery information for a suitable port for a service.   The optional argument to this function tells it an expression that provides the hash table (dict/map) of IP/port discovery information for this service.  If a service is listening on multiple ports, it defaults to the lowest port number being listened to by the service.

The reasons for some of these functions will become apparent in later blog posts.

As noted earlier, this is not the most interesting kind of monitoring to do (in fact, it's usually very minimal).  Later articles will cover the same ground for more interesting (and more complex) kinds of monitoring.

I have a few questions for you readers out there.  Does this make sense?  Assuming we can handle better monitoring methods (and we can), does the idea of not having to configure monitoring seem worthwhile?

Assimilation Message Formats

I've recently been discussing some nitty gritty details about how packets are laid out on the wire with a member of the Assimilation Project, so it seemed good to explain how our packets are constructed and deconstructed to everyone while I was at it.  This blog post talks about how we send lay out the bytes on the wire, with a bit of why and how.

Type, Length, Value messages

First off, the general low-level format is a variant on Type-Length-Value (TLV) messages.  In the software, we refer to these individual TLV fields as Frames, which are in turn concatenated into a FrameSet.  One or more FrameSets can be put into a packet.  Our packets are UDP (with a user-level retransmission protocol to ensure reliable delivery).  It's also worth remembering that in the Assimlation Project, communication between endpoints can consist of only a handful of packets every few months - or even every few years.  By design, silence is the norm - communication is the unusual case.

Each FrameSet has a type of its own.  There are three "magic" classes of TLV Frames, which if they appear, must appear in this order:

1. SignFrame - Digital Signature - currently not cryptographically secure (that's coming).  This Frame is not optional.  It indicates the type of signature and provides the signature data.
2. CryptFrame - Encryption Frame (optional)- indicating what type of encryption was used to encrypt the frame.
3. CompressFrame - Compression Frame (optional) - indicating what kind of compression was used to compress the frame.

If the packet is neither encrypted nor compressed, then it will only have a digital signature frame before the "payload" frames.

Other classes of frames include the following:

• SeqnoFrame - a sequence number frame (used by the reliable protocol)
• AddrFrame - a network address (any one of several types)
• CstringFrame - a NUL-terminated C-style string
• IntFrame - an integer (2,3,4 or 8 byte integers)
• IpPortFrame - an IP address with a port number (can be IPv4 or IPv6)

There are also many types that correspond to a single Frame subclass.  For example, HOSTNAMEs, JSDISCOVER, DSCJSON, RSCJSON are all TLV types that correspond to CstringFrame objects.  In other words, hostnames, output from discovery operations, discovery requests, and resource descriptions are all represented as strings (CstringFrames) - but each has its own TLV type.

Why Signatures, Encryption and Compression?

Since the nanoprobes run as root, and are capable of doing potentially dangerous operations, it is only prudent that we have cryptographically secure digital signatures (which have not yet been implemented).  It is expected that each role (CMA, nanoprobe) will have its own signature.  Although it is possible for each nanoprobe to have its own unique signature, this causes headaches when nanoprobes send each other heartbeats.  It may eventually be implemented, but initial implementations are likely to be simpler.

The case for encryption is a little less obvious.  Although most data from monitoring and discovery is not likely to be security sensitive, in some environments the data will be considered sensitive by the customer - in which case encryption is a good idea.

Compression is needed because our communication is UDP - which limits message sizes to a little more than 60K bytes.  The most voluminous data is expected to be discovery results - which are JSON.  Initial experiments indicate that the compression ratio for JSON should be about 15:1, which gives about 900K for the maximum uncompressed packet size.  Eventually, we may need to add packet fragmentation to accommodate larger discovery results.

 Packet Construction (Marshalling)

The process of constructing a message byte string from a FrameSet proceeds as follows:

1. A first-to-last pass is made through the Frames to deterimine the message size, by adding up the Frame sizes.
2. Space to hold the packet is allocated
3. A last-to-first pass is made through the Frame, placing the data from each frame in its correct position in the message

The encryption and compression Frames perform unique operations.  Although processing things in reverse order seems odd, the demarshalling will reveal a method to my madness...

When a compression frame is encountered, it replaces the message data with a compressed version of the message.  If the compression method cannot (or is instructed not to) reduce the size of the message, it shifts the message up over the compression frame, effectively deleting it.  It adjusts the size of the message accordingly

When an encryption frame is encountered, it replaces the message data after the encryption Frame with an encrypted verison of the message.  Note that it is important that the packet be compressed before encrypting it.  Encrypted packets don't compress very well.

Packet De-Construction (Demarshalling)

The process of demarshalling a message is the mirror of the marshalling method.

1. A first-to-last pass is made through the message, creating Frames that correspond to the various elements of the message.

The following rules are observed:

• The digital signature is checked - and the role associated with that signature is determined and stored away.  (i.e., is this packet from the CMA, or from a nanoprobe).
• If the next frame is an encryption frame, then the remainder of the message is replaced by a decrypted form of itself.
• If the next frame is a compression frame, then the entire message is replaced by a decompressed version of itself.
• The remaining frames are demarshalled as before.

Although the signature, encryption and compression frames are special cases - by enforcing that they occur in the proper order, the amount of special case code is minimized.

Although this isn't exactly rocket science, by doing things in the right order, the whole process is reasonably straightforward - which in my book is a virtue.

Does this make sense to you?  How should it be improved?  How can I improve this description?

 

Related articles

How to implement "no news is good news" monitoring reliably

How Open Cluster Framework monitoring works

Cryptography - "Encryption and Decryption"

How GPG works: Encrypt

How heartbeats fit into hierarchies of watchers - and pings don't - or Who will watch the watchmen?

Somehow people seem to think that heartbeats and pings are the same thing.  They're not at all the same thing.  Heartbeats are typically semi-intelligent elements in a hierarchy of watchers.  Pings can't play the same role.  This blog post talks about why, and how hierarchies of watchers work both in general, and more specifically in the case of the Assimilation Monitoring Project (AMP)- which breaks with tradition in a useful and novel way.

Pings

Let's start with pings. The ping command literal ICMP echo request packet is sent to from one machine to another machine.  The ping is then returned as an ICMP echo reply when it is received.  This might is often done at a very low level in the OS, or might even be done by the NIC itself - which only requires power to the NIC and a network connection.  This means that an ICMP ping doesn't test very much at all.  In addition, response to pings is often disabled for security reasons.  So, this is a very poor method of determining even minimal health of a system.  All you really know for sure is that it has power, and was booted at some time in the past.  Not very helpful.

Heartbeats

Heartbeats have two portions, which are semi-independent.  There is the heartbeat sender, which is typically a user process which sends heartbeats at some interval as instructed.  There is the hearbeat receiver which expects and listens for heartbeats.  If it goes without hearing expected heartbeats for a certain period of time, it will report that fact upstream - in our case to the CMA.

Heartbeats are typically sent by user processes which are running under the OS, and if the OS stops scheduling user processes, heartbeats will stop being sent.  In addition the packets they send go through the normal user process queuing system, and if that fails, heartbeats will not be sent.  Many more things are validated as operational by sending and expecting heartbeats than by pings.

Quis custodiet ipsos custodes?

A key thing about monitoring systems for availability is that everything needs to be watched, and all the watchers need to be watched.  In Latin, this comes out to be the classic question "Quis custodiet ipsos custodes?" - Who will watch the watchmen?  The original question is more commomly thought about in political terms regarding corruption, but applies very well here where any computer can fail, and for reliability, you want to make sure that as many failures as possible are caught - so they can be repaired - whether by human or automatic means.

It is common in highly reliable systems like telephone systems to have a hierarchy of watchers, where the behavior of various elements is observed by other elements through a well-organized hierarchy - until the top level is reached.  These techniques may also be connected to watchdog timers, which reboot the machine unless they are positively acknowledged in a fashion similar to heartbeats.  These can also assist the watching hierarchy when properly used.

Once you reach the top of the hierarchy, hierarchies fail us - and then we wonder who will watch the watcher at the top.  In addition, hierarchies don't scale with low effort - one has to keep growing the hierarchy as the number of systems grows.

Within a single nanoprobe in AMP, the hierarchy is pretty normal - our nanoprobes invoke resource agents which watch the resources that they are assigned to watch.  The nanoprobes themselves set timers to watch the resource agents that are watching the resources.  But now the problem gets more difficult.  Who will watch the nanoprobes?  Although watchdog timers could be of some help in recovering from certain kinds of failures, they don't solve the problem of failure detection, and are a bit drastic for most customers.

In a conventional highly available computer cluster such as Linux-HA or Pacemaker, complex multicast consensus algorithms such as PAXOS or Corosync ensure that failures on other systems can be properly observed.  Unfortunately these algorithms require multicast and don't scale to thousands of systems.

Breaking With Tradition (oh my!)

It is at this point that AMP's architecture breaks with tradition.  We arrange for the overwhelming majority of systems to simply monitor each other by having each system monitored by two other systems, and each system also monitors two other systems.  By connecting the systems in a circle, like people standing in a circle holding hands, we can have each system have two neighbors which it waches, and which also watch it.  Because each system monitored by two other systems, losing a single system doesn't cause us to lose monitoring capability (there's no single point of failure).  This looks a lot like the picture below.

In AMP's montoring scheme, each system only has to know about two other systems, and each system is only known about by two other systems.  By closing the circle, the problem of who watches the watchers is solved in a more scalable way than a hierarchy - and without requiring multicast.

As a result, our central system (the CMA) doesn't have to actively watch these other systems, because they are watching each other.  In our implementation, the CMA does have to lay out the neighbor arrangements, but once set up, the watching takes place without intervention.

And as noted, at the top level, the CMA is small enough to use or get the assistance of standard clustering techniques.

This article covers many of the same issues as the "How to implement 'no news is good news' monitoring reliably" posting from a different perspective.  It seems good to present this information in several ways, because it seems a bit counterintuitive.

 

Related articles

How to implement "no news is good news" monitoring reliably

Reliable UDP: The last major Assimilation feature before the first release

I'm still on track for a first release of the Assimilation code by the end of the year.  But there is one last interesting (meaning tricky) feature to write before this release.  All communication is over UDP, which means the OS doesn't guarantee packet delivery.  So we need to do that ourselves.  From an availability perspective, we need to acknowledge packets at the application layer and high-throughput is not an issue, so nothing much is lost  (the reasons for that are worthy of their own post).  The most interesting part of this is that our protocol needs to be resilient to replay attacks.  This post explains what a replay attack is, and how we plan on eliminating them.

First off - what's a replay attack?

A replay attack is when a series of messages are captured then played back later.  Because the messages are identical, without some kind of special attention being paid, these messages will be acted on exactly like the original messages.  Because of its connectionless nature, UDP communications are especially suspecptible to this kind of attack.  Although an adversary can't send arbitrary messages because of digital signatures, if it has a way of capturing the messages, they can make the system misbehave - possibly in seriously bad ways.

How to deal with replay attacks?

The computer science literature has a number of scholarly articles written on replay attacks.  But the essence is simple - recognize that this packet has been acted on before.  There are a variety of different methods proposed in the literature, but for my purposes, I'm going to concentrate on two variants of one method - the one Linux-HA uses.

Since we are implementing our own protocol, we have control over thingsl like sequence numbers.  If you could simply ensure that sequence numbers go up forever, without ever wrapping around, then you could simply let the sequence numbers go up forever and you could easily tell if a particular packet had been seen before.  For a variety of reasons, the Linux-HA heartbeat protocol implements that with a generation number combined with a sequence number.  The  sequence number resets to zero whenever the protocol is restarted, and the generation number is ensured to go up monotonically.

How to manage generation numbers

In Linux-HA, there are two ways to manage generation numbers - one is to store the generation number on persistent storage (i.e., disk) and then to increment it whenever the protocol restarts.  This works great if you have persistent storage (some systems don't), and you never restore from backups.  The other method is to set the generation number to the current time since the UNIX epoch.  This works great as long as you never reboot and set the clock back in time to a time before its value at the previous protocol reset.  The Linux-HA heartbeat protocol implements both of these methods.

For the Assimilation project, I plan on implementing a combination of these two approaches.  Use a number based on the current time, but also store it on disk.  If when you restart the protocol, the current time is less than the previous generation number, set the generation number to one greater than the previous generation number.  This can also fail - but only if you have no persistent storage (or do a restore) and you also set the clock back to before the time it had at the last protocol restart.  This combination is an improvement over either method by itself and the probability of failures is significantly reduced.

Over the next month or so, I'm planning on implementing the reliable UDP protocol including this method of eliminating replay attacks.   As a result, I'm looking for feedback on this.  Do you have any suggestions on how to improve this?  Are there holes in it or use cases that fail that I missed?

PS: The current release implements digital signatures, but doesn't yet implement any cryptographically useful ones.

 

Related articles

Assimilation Project Licensing

Building Secure Web Applications Infographic

Zero Configuration Discovery and Server Monitoring in the Assimilation Monitoring Project

Reliable UDP (RUDP): The Next Big Streaming Protocol? - Streaming Media Magazine

Clocks and Security

Zero Configuration Discovery and Server Monitoring in the Assimilation Monitoring Project

Wouldn't it be wonderful if you could just drop a monitoring package onto your servers with no configuration at all, and it all started up, began monitoring your servers, discovered your services, dependencies and switch port connections all without you doing anything - with practically no load on your monitoring servers, and without setting off any network alarms?

You can now do that with the Assimilation Monitoring Project.  Really.

For the skeptics and geeks out there (you know who you are!), this blog post tells you how it works in some detail.

In earlier posts, stealth discovery of servers, services, dependencies, and switch ports was covered in some detail.  Earlier posts also explained how the hyperscale O(1) monitoring works. So, I won't repeat those things here.

What's new is the zeroconf-like configuration-less startup.  The problem is this:  We have a bunch of nanoprobes on an unknown number of machines, some of which may be using our preferred port for other things, and we have a collective management authority (CMA) - and we need to connect them to each other - without configuring anything specific for a given site.

So, how can this be done?

It first started off with a trip to IANA - to get a reserved multicast address.  I sent an email to IANA requesting that they reserve a specific multicast address for the Assimilation project.  After a a few weeks and half dozen emails back and forth, they agreed that this was the right approach, and they assigned one for the project.  The project is now the proud owner of multicast address 224.0.2.5.  Here's how it's used:

1. When the CMA starts up, it binds to the ANY address and port 1984.  It then also joins the multicast group on 224.0.2.5.  This socket now hears any packets sent to any of the unicast addresses on the machine, or to our multicast address.
2. When a nanoprobe starts up, it tries to bind to the ANY address on UDP port 1984.  If that port is already being used, it asks the OS to assign a port to it to bind with the ANY address.
3. The nanoprobe then sends a single multicast message to our multicast address requesting connection and setup.  This message includes some local network configuration. Nanoprobes do not join the multicast group.
4. The CMA hears this multicast packet from the nanoprobe.  It then records the address and port the nanoprobe sent from in the database, and begins the process of configuring it into the system.  Configuration consists of these steps:
   • Analyze the network information and put it into our Neo4j database.  This includes the entries for all the NICs and IP addresses on the machine.
   • Send it a packet giving it a variety of configuration information - including the IP addresses and ports to use when contacting the CMA for various purposes.
   • Connect it in a neighbor monitoring arrangement as noted in earlier posts.
   • Send it commands to perform and schedule discovery actions.
5. The nanoprobes then obey these commands which then causes all further communication with the CMA to be sent to the the unicast IP addresses that were sent in the configuration packet from the step above.
6. Everyone lives happily ever after.

It is important that the nanoprobes be able to tolerate having the preferred port be unavailable when they start.  One reason is that in a large and diverse customer set, it's hard to predict who might be using this port already.  Another even more important reason is that we want to start a nanoprobe running on the CMA - and the CMA always uses that port.

Here are the requirements for this to work:

• There must be a well-known multicast address which most sites can use.
• There must be a well-known port which is free on the CMA.
• The site must permit multicast.
• Multicast must extend between a CMA and all its nanoprobes - but not to other CMAs.
• On the CMA servers, the nanoprobes have to be started after the CMA.

Obviously not every site can meet all these requirements - but many can.  And for those that can, it makes for a great open-box experience - and definitely a much lower suck factor.  Imagine that - monitoring that doesn't suck!

So, what's your reaction to this?  Will you be able to run it without configuration?  Or will you have to configure in the address of your CMA?  Is this cool?  Or is it just ho-hum?

Assimilation Monitoring LinuxCon Video

I mentioned a few weeks ago that my talk at LinuxCon in San Diego had been very well received.  Thanks to some good friends, we also created a video of the event, and this week I want to point you to the final cut of that video.  This talk is a great introduction to the Assimilation Monitoring Project.

Thanks to Robert Holder, the video includes annotations and highlighted versions of my slides.  He has rendered this video in 720p so you can see everything in its HD glory, and also in 480p for those with more limited network bandwidth.  You can also look at the slides separately (even though they're quite readable in the video).

This includes an overview of many of the aspects of the project covered here in my blog.  Topics covered include:

• Project scope - what problems we address
• Architectural overview
• Why it's incredibly scalable
• Continuous integrated stealth discovery
• How service monitoring works
• Central Authority capabilities
• Nanoprobe (agent) capabilities
• Why Neo4j?
• How discovery works
• Sample discovery data
• Sample Dependency and switch discovery graphs
• Current state, future plans

URL for 720p:  http://bit.ly/AssimMonVid
URL for 480p:  http://bit.ly/UcKKtT

I'd love to hear feedback on the project, on the presentation.  What did you like?  Is this something you'd be interested in?  What did you not like?  What needs improving?  What would you suggest for the project, or (gulp!) the presenter?

Maintainable IP address assignment for clouds and large clusters

This post describes a more maintainable method than normal DHCP for automatically assigning host names and IP addresses to servers which is ideal for cloud computing and large clusters.  It assigns them according to the location of the server, and requires zero administrative effort when you add or replace a server.

DHCP is OK, but it costs time and effort to keep up to date as you install and replace servers.  If you have a million servers, you'd like for the IP addresses to be easily correlated to where the hardware is.  For example, you'd like for the server name to be something like Drone-Rracknum-Sservernum and the IP address to be something like PREFIX.rack-number.server-placement-in-rack.  Then your IP address would tell you immediately where the server is in your data center.  If you have several hundred servers or more, this is a good thing.  But you don't want to hard wire MAC addresses into DHCP configurations or IP addresses into filesystem images, either.  Both are high-maintenance methods.

This post talks about an alternative method of assigning IP addresses, that lets you locate your servers easily, is lower overhead and more maintainable than the usual way of using DHCP.

The assumption of this arrangement is that you have a managed switch or two for each cabinet of servers, and that you maintain the IP addresses of the switches in DHCP.  When you replace a switch, you put its MAC address / IP address pair into DHCP and away you go.   But there are many fewer switches than servers, so this is something like 1/49 or so of the number of servers you have.

Once you have this, then the rest is simple.  You assign the switch IP addresses sparsely, perhaps one per 50 addresses in a subnet.  For example, 10.0.0.1, 10.0.0.51, etc.   You then rewrite the network startup to listen for a link discovery packet (LLDP or CDP).  This packet contains the management address of the switch, and the switch port this NIC is connected to.

If you assign these switch numbers in a way that reflects your hardware layout, then it's simple to decode that back into the placement of the switch in your environment.

Basically, the host computes the host name it gives to DHCP, and lets DHCP do its thing.  The host name should be structured something like this: Drone-Rracknum-Sservernum. Of course, the script to translate the switch IP address to a rack number, and servernum offset is site dependent, but not hard to compute.  If you have your racks arranged in nice neat rows and columns, you could go further and translate the racknum into a row and column or other local designations for the location of the server.

You give this host name to DHCP, and it then assigns you an IP address, DNS servers, and other information from that.  If you wanted to hard-wire your "other information", the IP address itself is also easily computed, and you can avoid DHCP completely.

Now when you replace that computer with a different one, the new one automatically takes on the same IP address as the previous one had without any manual effort.

If you aren't familiar with CDP or LLDP - they provide management address and port information and a lot of other good stuff by periodically sending layer two packets to their connected endpoints.  Most or all modern managed switches support one or both, and there are no known security issues that arise from enabling them.

It is more maintainable - instead of having to update DHCP tables with new MAC addresses for all the servers and switches, you only have to maintain the IP/MAC addresses of the switches, and the hostname/IP address correlations - leaving the MAC addresses out.

Of course, this also assumes that you have a rational cabling system in your cabinets - one where the switch port numbers correspond to positions of the servers in the cabinet.  The main downside is that the systems take longer to come up.  DHCP servers respond quickly, but you will have to wait to hear your link discovery packet.  Of course, you will have to have another method for providing the netmask and the list of DNS servers.  Although the former can be hard-wired, the latter may need to change over time.

Thanks to Narayan Desai of Argonne National Labs who introduced me to CDP and LLDP a few years back, and inspired these ideas.  No doubt people have done this kind of thing before - maybe better.  I mentioned this idea at LinuxCon, and people asked me to write it up.  So here's a post about it.

What's your reaction to this?  Is waiting for an LLDP packet going to take too long?  How do you do this at your data center?  Do you have a better method you follow?

I see dead servers - in O(1) time

The title for this blog post comes from a T-shirt I had made for the Assimilation Project.  I wore a nicer version of it at my recent talk at LinuxCon 2012.

The Assimilation project has some significant and unique claims to scalability.  Some of these have been discussed before.  This blog article will explain the different aspects of the project and how each of them measures up in terms of scalability.

Before we get started, there is some geeky computer science terminology that needs explaining.  When the work that is being done doing something doesn't go up as the number of servers goes up, us geeky-folk call it O(1).  This notation is called big-oh notation.  When the work doubles when you double the number of servers and goes up by 10 when the number of servers goes up by 10 and so on, using big-oh notation, we call that O(n) - because the work goes up as the number (n) of servers goes up.

With that out of the way, let's cover the liveness or membership aspects of the system - those discussed in an earlier post, and do a little work on definitions.  I typically say that the liveness checking is essentially O(1).  That is, the amount of work done by any one server doesn't go up as the number of servers goes up.  The programmers and lawyers out there will have noticed the word "essentially".  The part that is purely O(1) is the part that discovers that nothing is wrong ("well-baby checks").  The work for that part never goes up as servers are added.  However, the work required to process failures does go up as the number of failures goes up.  And the number of failures goes up as you add servers.  As one back-of-the-envelope data point, if you have a million servers, you will have about 500 servers per day fail.  That's a pretty small number of events to process - and many orders of magnitude less than the number of  well-baby-checks that this method avoids sending to the CMA.  As far as I know at the time of this writing, this is a unique attribute of the assimilation project.

Note that this work is still being done, but because of the way this work is distributed, it is not concentrated on a single machine or collection of machines - this is why I say it's scalable.

For security reasons, we control the nanoprobes from the CMA, so when they come online the CMA has to splice them into its rings.  But, it would still need to know it came online, so that part is inevitably O(n) anyway.  Our creating rings out of them increase the work somewhat, but don't change the order of complexity of the system.

What about service failures, or other exceptions not involving servers crashing or rebooting?  Those are still O(n).  Well-baby service checks are performed purely locally.  Until they fail, they follow the "no news is good news" philosophy - so only failures get propagated upstream.   Some other systems do this (so-called passive checks), but in our case, these are the norm, not the exception.  Because they combine with the reliable liveness checks, the no-news-is-good-news for service checks is reliable.

The last item is Stealth discovery^TM. Discovery information is largely O(n) - but based on the number of configuration changes - rather than the number of nodes.  Like most things, practice is more complicated than theory  So, let's look at those practical details.

At the present time, we have decided not to have any persistent discovery information kept on the systems.  So, whenever a system reboots, it doesn't know if a particular discovery item has been announced before or not - so it will send all discovery information once at each reboot.  After it is sent, it then follows the no-news-is-good-news philosophy, and doesn't resend it unless it changes.  Note that this is curable by adding persistent state to the discovery data.  If this is needed it can be added - with some careful checking to deal with things like restoring data from backups, cloning of systems and other common administrative tasks.

It is the difficulty of dealing with this properly (whcih is harder than it seems) that has led me to the current decision.

In summary, the largest amount of effort most monitoring systems perform centrally is now performed in a fair and evenly distributed fashion - resulting in an O(1) designation for these "well-baby checks".  However, there are still a number of things which, are still O(n) - but are drastically reduced by our no-news-is-good-news design philosophy.

Credits:  The words on the T-shirt were inspired by a T-shirt that Splunk gave out a while back.

Does this make sense to you?    Does the pun in the title and shirt hurt your soul, make you groan, or does it make you laugh?

Injecting Nanoprobes into Servers - What's that about?

I've recently had some people who have asked about the how nanoprobes work – are they clients, or what exactly are they? They start out like clients, and behave in some ways like peers, and maybe a bit like servers. So what the heck are they? The simplest explanation is that they are autonomous delegates of the central management authority. Read on to find out more about how this unconventional model works and why this authority model is key to unprecedented scalability and stealth discovery™ in the discovery-driven Assimilation monitoring project.

In the Assimilation Monitoring project, there are two primary kinds of entities, the Central Management Authority (CMA) which there is one of, and nanoprobes, which there are many of – one per operating system image being monitored.

The simplest way to understand the relationship between the CMA and its nanoprobes is that nanoprobes are just extensions of the CMA – with limited authority. They are extended limited trust – enough for them to do the requested tasks – and more than in typical client-server models. Within this view of the world, they are autonomous agents, trusted to do the tasks requested without continual supervision – implementing a no-news-is-good-news philosophy. So if a nanoprobe is requested to monitor a web server every 10 seconds, it will do so silently unless a failure occurs. This is our “no news is good news” philosophy. When nothing is broken, and no configuration is changing, the nanoprobes say nothing to the CMA.

They receive orders from the CMA, and whatever those are, they carry them out. These order include things like requests to perform repetitive discovery actions, monitor services which only involve the current OS image. But they also include some actions which involve other entities in the network. For example, they send heartbeats to peer nanoprobes, and are asked to expect them within a certain interval as well. Similarly, they can be asked to perform monitoring actions that monitor services provided by appliances and other things we can't put nanoprobes in – and it likely won't even know it's a remote thing being monitored. For example, when monitoring a web server, it does an HTTP GET operation on an IP address. Normally, this is a local address, but the monitoring agent doesn't know or care if it's remote. Of course, the CMA cares if it's local or remote, but the nanoprobe doesn't.

This is another important aspect of the authority scheme – nanoprobes have no idea why they're being asked to do anything, and they don't need to. They are policy-free. This is makes them lighter-weight, and easier to keep updated – because they embody less knowledge which might need updating. Given that there may be a million of them, this is a really good idea.

Security of nanoprobes is important – particularly when acting as peers. For this reason, nanoprobes use public keys to sign all their packets, and the actions peers can perform are limited. Currently they only send heartbeats to each other. This is an action with minimal security implications. They only obey commands from the CMA, whose public key each one knows.

Nanoprobe startup resembles DHCP. It starts up, sends a “phone home” message to the CMA announcing that it's here, and what it's local network configuration is, and waits for orders from the CMA. The CMA then sends it a configuration packet, indicating that it heard the request for configuration, and a series of other packets telling it who to send heartbeats to, and what discovery actions to run. The default on the initial phone home message will be to send a multicast packet to a reserved multicast address. For many sites, this means that the Assimilation system will require no configuration at all to get started. For sites that don't support multicast like clouds, the address of the CMA will have to be configured for startup.

Because we have continually running active agents on every monitored machine, we can monitor things besides services – we can monitor anything for which an agent has been written. This is how stealth discovery works. Our active agents can monitor the set of ports being listened to and notify the CMA when new services are added, or old ones go away – just by using netstat. Because we run on the machines, our nanoprobes can discover anything internal to the machine, and monitor when those configuration things they've discovered change. Since the output of discovery agents is JSON, it's easy to see if it's changed, and in the spirit of no news is good news, only send things to the CMA when they change. In some sense, discovery is simply monitoring the metadata of the data center.

Since discovery is just running scripts that produce JSON, it's quite simple to discover more or less anything on a server. This might be hardware configuration, software configuration, ARP neighbors, etc. OF course, the CMA puts this information into the Neo4j graph database. These kinds of scripts are what SysAdmins are typically quite good at writing, so our hope is that we can create a community of people collaborating on stealth discovery – similar to the community around OCF resource agents. One good sign, Ted Zlatanov of the CFEngine3 project has decided to collaborate with us on this effort.

In many ways, this autonomous delegation methodology is exactly what makes the Assimilation project able to do what it does. This treating of discovery as the monitoring of metadata makes this work elegantly, and is the key to our approach. In my view, discovery and monitoring put together into discovery-driven monitoring truly is greater than that of the simple sum of its parts.

What's your take on all this? Does this slightly oddball way of working make sense to you? What questions do you have about it? Does this all hang together for you, or just sound like another self-congratulatory load of organic plant fertilizer?