What is Stealth Discovery?

One of the most interesting features in the Assimilation Monitoring Project is that it includes Continuous Integrated Stealth Discovery^TM as part of the monitoring operation.  So, what exactly is that, and why do I think is it so cool?  Well, let's start with what it is, and hopefully it will become clear along the way why it's well worthwhile...

• Continuous - it runs all the time - discovering new and changed things in minutes to a few hours.  It is always up to date.
• Integrated - discovery is fully integrated with monitoring.  No separate configuration, infrastructure or software is required.  Discovery happens automatically along with monitoring.  Because it is integrated, it follows the Assimilation monitoring methodology of "no news is good news".  If nothing changes, nothing gets reported - which is a big step on the road to scalability.
• Stealth - it doesn't send out any probe packets.  No port scanning, no pings.  No network packets to discovery anything.  No need to get permission from the network security team :-D.
• Discovery - it discovers things in your environment - things like
  • Servers (IP addresses and MAC addresses)
  • Switches and switch ports (including settings)
  • Services offered
  • Services consumed
  • Dependencies between services and servers
  • Local network configuration

Since this is stealthy and low overhead, it can run often - effectively continuously.  So within a few minutes of bringing up a new server or service, the system knows about it.  If that server or service isn't being monitored, it still knows about it, and can tell you that what's not being monitored - closing the quality gap for your monitoring.  It's hard to fix a service that's not monitored.  Knowing for sure that everything is being monitored can eliminate some of those embarassing career-limiting events.

In addition, it gathers a lot of detailed information about the things it is monitoring or might  monitor - MAC addresses, what IP ports go with which binaries, what options they were invoked with, what user ids and group ids they are running under.  Because discovery is integrated with monitoring, this information is readily available in the monitoring environment as well.

If you have LLDP or CDP enabled (default in most switches), then it also knows which servers are plugged into which switches, and which ports on those switches, and the switch settings for those ports.  Since it knows the corresponding settings on the OS side, it is a straightforward matter to compare them for consistency.

There are also many possibilities for making configuration easy.  Since it knows exactly which services are running on which ports, it is planned for it to suggest how to monitor it.  If you like the suggestion, click OK and away you go.

Continuous Integrated Stealth Discovery creates a rich set of data accurately describing your systems current configurations in great detail.  It makes it easy to find those systems and services that might have fallen through the cracks in the past.  It will simplify initial and ongoing configuration by suggesting how to monitor services it has seen before.

As far as data format, the data is collected either as binary packets (LLDP or CDP only) or for every other kind of discovery, as JSON objects.

As noted in my earlier post, this discovery information will be collected into a set of dependency graphs in Neo4J.

A good question for a project which is still very much a work-in-progress is "How much of this really works now?"  The answer in early June 2012 is: the data collection architecture and code for discovery is there and it works.  The code for putting this data into a permanent database (the Neo4J graph database) is currently being written.  The code for collecting many (but not all) of the specific things described above has been written and works.  Nothing that looks like a GUI has been started.

It's not all there, but it's getting there...