Managing Computers with Automation

An Assimilation type schema in Neo4j

This week I want to talk about an aspect of the Assimilation database schema which is somewhat controversial, an aspect of the schema for which the jury is still out.

I chose to represent the Assimilation node type hierarchy with relationships which currently serve no purpose other than to represent the types of nodes in the database. This post will talk about why I put the type hierarchy in, and why it might be a good idea, or maybe not.

Originally, I had not put the type hierarchy in, as I saw no reason for it - each node is labelled with its type internally anyway through a nodetype attribute, and most of my nodes are represented in separate indexes, one per nodetype.  For example, Drones are indexed in a Drone table indexed by its name (or designation if you prefer), NICs in a NIC table, indexed by MAC address, IP addresses, indexed by canonical-form IP address and so on. Moreover, each node as a field in it called nodetype, which tells the type of the node.

Given the nodetype field and all these indexes, and the fact that my code doesn't use them why would I do this?  The first answer is not very compelling - I saw some other Neo4j applications which did it, and since I was (and still am) a Neo4j newbie, who is still learning to think in graphs, I figured it must be a good idea - and maybe I would learn something along the way.

Below you'll see an example of what this looks like with one node of each type relating to its nodetype node. Each of the objects (NIC, IPaddr, Switch, Drone, etc) has an IS_A relationship with its nodetype. Each nodetype also has an IS_A relationship with node zero – sometimes called the reference node in Neo4j.

 

As you can see, this seems a bit redundant -- after all I have the indexes of everything anyway.  My original real reason for putting these in from my perspective is simple - debugging.

Neo4j has a shell, and the way the shell operates, it's easy to follow relationships to see how the program is creating the nodes and relationships - if you start from node zero and go from there.  How this works is quite nice and makes seeing what the CMA did quite easy.  The Neo4j shell is somewhat reminiscent of a normal UNIX-style shell - with each node in the graph being treated analogously a directory or folder. So if I've added new relationships or node types, I can conveniently find any type of node through the reference of node zero.

There's also another reason as well...  Hypothetically, if I wanted to look for some attribute of a node that there was no index for, the entire query would run more quickly (and be easier to write) if I do it all in terms of relationships rather than walking through the indexes first. So, I can imagine it being useful in the future – but right now, I'm paying the cost of creating all these IS_A relationships – which I currently use primarily for debugging.

So, I get some debugging advantage, but I pay a price. In the future, I might get some performance advantages from it, but that day hasn't come yet – and may never come. So, is it a good idea – at this point in time, I'm not yet concerned about performance, and the debugging simplicity I get makes it worthwhile to me – for now. If I want to remove them later, it's all done in one place, so it would be easy to change.

I know that some of the people who read this blog know a lot more about Neo4j than I do. So, what do you think? Does this seem reasonable to you?

Discovering Switches: It's amazing what you can learn just by listening...

We recently added code to discover switches, switch ports and settings - all in the Stealth Discovery^TM way - without sending out any packets at all!  So now you know which switches and which switch ports every monitored server is plugged into.  As a bonus we pick up some interesting configuration information on your switch and your particular switch port - just by perking our ears up and listening...  Now when you send someone to the closet to do something to your switch port, there is no doubt which port is yours - regardless of that little mistake in the cross-connects, or that tiny error in documentation.  [Anyone want to write a cool iPad switch mapping app for this?]

This discovery info is also helps our highly scalable monitoring protocol keep 96% of our heartbeat packets off your backbone network as described on the web site, and our first article on the subject.  Of course, if another monitoring system wanted to know your switch connectivity, it would likely ask you to enter it.  But that would suck - so we discover it instead.  Because we (re-)discover it continually, it stays up to date (within a minute or so), and we can use that discovered information in several ways.

Since most people don't have a clue how to do this let's talk about it...  All modern managed switches implement one or both of two link level discovery protocols - LLDP or CDP.  These protocols send out unsolicited packets on each switch port about once a minute that tell you things like what switch is your NIC connected to, which port on that switch, what it's IP and MAC addresses are, and other configuration infomation (VLAN, MTU, speed, other things).

So we just listen for those packets and when they change, send them upstream to the CMA which then translates them into JSON and puts them into the database - nodes, relationships and all.  So what does that look like when you put it into the database?

If you look on the right, you'll see it introduces a new kind of node - a Switch, and a new relationship "wiredto" which indicates a physical connection between two NICs.  In this case, one NIC is in the switch and one is in the host.  If you change which port or switch a host is plugged into, we will update the database within a minute or so of its happening.  This is Way Cool!

The switch NIC has some interesting information about it - it knows which switch port it is, and pulls some descriptive information about the port from the switch.  If you look further, you'll see that we know the switch's administrative address, and the corresponding MAC address - which in this case is the same as the switch's name.  Note that at least in this case, this isn't a physical NIC, but a logical one - there is no corresponding jack on the switch.

Since all this information shows up in the database, we can take the switch dependency into account when monitoring, produce a port map for a switch, and of course, properly create the switch-level rings for our monitoring protocol - all without manual configuration!  Since we know the switch's administrative address, we can easily monitor it - either continually or when we suspect it of flaking out - again without any manual configuration.

Is this something you'd like to see at your site?  Do you know if your site enables LLDP or CDP?  What would you do with this information if you had it?  What dowsides to this do you envision?

Clients, Servers and Dependencies, Oh My!

One of the things that people have gotten most excited about in the Assimilation Monitoring Project in the area of discovery is the discovery of clients, servers and particularly dependencies.

That code is now in the Assimilation code base.  We discover client processes, server processes, and their interconnections.  In this post, I'll explore how this works and what this looks like in the Neo4j graph database.  These dependencies are discovered using Stealth Discovery^TM methods - without port scanning or packet sniffing - .

If a server process goes down, its clients will most likely won't function correctly - so there's a dependency of the client on the server.  Imagine your NOC suddenly lights up with a dozen services broken in a cascading failure.  To find the root cause, find failed services which aren't clients of other failed services - by following the dependencies.  Because we discover these client->server dependencies (and a number of other types) automatically, the Assimilation code will be able to help you or your DevOps team quickly find the root cause.

On the right is a capture of some of the data from the new client/server discovery code on my test machine.  For this example, I did an ssh from 'servidor' to 'servidor'.  In the upper right is a green box labelled 'tcp-process/ssh'.  This is the client process of the connection.  It in turn connects to the violet 10.10.10.5:22 IP:tcpport node by a 'tcpclient' relationship.  This port in turn is listened to via the 'tcpservice' relationship by the sshd server process in the green box labelled 'tcp-process/sshd'.  As you can see from the graph, both of them are 'runningon' Drone servidor.

So, you can see that if you have a problem with the 'ssh' client, it could be caused by a problem with the 'sshd' service - just by following the tcpclient->tcpservice links.  Of course, if the server it's 'runningon' fails, then things will also fail.  Similarly by following the tcpclient->baseip->ipowner, relationships you can see that failure of the NIC eth0 can also cause problems.  It's these kinds of complex dependency relationships where Neo4j shines.

The sshd process is also listening to port 22 on 10.10.10.200, but no one is connecting to it by that address.  It turns out sshd listens to both the IPv4 and the IPv6 ANY addresses.  Some applications (like sshd) listen on all addresses, some on just a few, and some (like BIND) look to see what all addresses exist when they start and bind to each one individually (funky, eh?).  Sshd is a little weird because Linux has a dual IPv4/IPv6 stack and there's no need to listen to both the IPv6 :: and the IPv4 0.0.0.0 ANY addresses, but apparently sshd has two separate sockets - one for IPv4 and one for IPv6 clients.

The source of this data in the database is from our  tcplisteners and tcpclients discovery agents - shell scripts that use netstat and /proc to grab cool stuff...

So, what's your take on this?  Will it be useful to you?  What do you think we missed?  What would be your favorite thing for us to discover?

Assimilation Ring Neo4j Schema

In this exciting episode of the Assimilation Project Neo4j schema series, I'll go over how the monitoring ring structures (which are key to its scalability) are represented in Neo4j.  This is in some ways pretty simple, and some ways kind of clever.  Either way it is one of the more critical parts of the schema to update correctly.  Below is an example of a 3-node ring - after three nodes have joined, and been connected up in a ring.  This particular ring was produced by one of my test cases.

Some of you who read the Neo4j mailing list will remember my struggles with coming up with a schema for this - mentioning I was going to have to think harder finishing with something like "I hate that".  The schema below is the result of this harder thinking.  Although there was some thinking involved, it wasn't so much harder as it was retraining my brain to think of how to express my problem within the power and limitations of a graph database, and in particular Neo4j.  Although I didn't initially like the arbitrarily-named ring relationships, the idea has grown on me and I kind of like it by now.  If you don't like it, then by all means let me know, and tell me how I should have done it in the comments.

Fans from my previous post will note the introduction of a new entity - a Ring - which I have named "The_One_Ring" (with a nod and apologies to J. R. R. Tolkien).  Each of the rings has a RingMember_The_One_Ring relationship pointing to the ring(s) they belong to.  If a Drone belongs to more than one ring, then it will have more than one RingMember_* relationship.  Note that this is where the idea of a formal schema gets stretched a little thin.  There is no pre-set limit on the types (or names) of RingMember_* relationships that might exist - you'll have one for every subnet, and one for every switch.  But because Neo4j is schemaless, and our code is flexible and consistent, no one cares.

Assimilation Ring Structure in Neo4j

Then, there are the RingNext_The_One_Ring relationships - which are always cyclic.  Each member that has a RingMember_The_One_Ring relationship also has exactly one RingNext_The_One_Ring relationship as well.  These RingNext_* relationships will always form a cycle.   These RingNext_* relationships represent exactly the monitoring rings which are described on the Assimilation web site, and appear in our earliest blog post on the subject.  The only difference is that in the packet world, we have to construct separate links in each direction, and in Neo4j, there is no reason to create two relationships, since you can traverse the relationships in either direction.  However, they aren't truly bi-directional links, since there is an ordering of nodes on the ring.

Whenever a node is added to a monitoring ring, the corresponding RingNext_* relationships are added to the Neo4j model of the world at the same time.

There are two deficiencies in the current ring management code - which will be remedied as time permits.  These are:

• The RingMember_* relationships ought to have an IPaddr attribute which says which IP address of the server is used in this particular ring.  Because servers may have multiple interfaces, many rings have network topology affinity, and interfaces and IP addresses  can change over time, it would be good to know for sure which address was used when it joined the ring.  This is simple to cure, but it's not my priority this week ;-)
• The maintenance of rings both in the database and in the network needs to be protected with transactions.  It is essential that these rings are never corrupted, even if the server crashes while updating them. It is also vital that the database and the network be in sync.  You definitely don't want to have to do the equivalent of an fsck on the network and the database to figure out what was broken because of the crash.  That would get a significantly high reading on the old suck-o-meter.  This one is not nearly as simple to cure.
  
  Neo4j has transactions, but the REST interface I'm currently using doesn't support them easily.  In addition, it is also important that updates to the rings in the network match the database.  If the Collective Management Authority (CMA) crashes while doing a ring update, it is essential that we know that the update was in progress when the system crashed so we can complete it.  For this purpose, Neo4j transactions are not directly helpful, since they can't repair the rings out in the network when they are completed after a crash.  Most of the other updates to the database are idempotent - that is, they can be repeated with no ill-effects.  However, maintaining a linked list in the database and in the network requires transactional semantics - beyond the obvious database transaction paradigm.  This will an interesting puzzle to solve and the right solution will require some thought.  Although it might make my head hurt, it will be a good kind of hurt...

You may recall from my previous post I mentioned that it could have easily been represented in a hierarchical database.  Now, this part of the graph doesn't look hierarchical at all - and it's not even a DAG (Directed Acyclic Graph).  There is always a cycle in it - and in a larger data center with multiple levels of rings, there will be many cycles by design.  One could represent it with order in a relational database, but it would be painful.  Although Neo4j doesn't enforce my schema, it does enforce something very useful - nodes cannot be removed and leave dangling relationships.  Although you can write something similar in relational database constraints, it is expensive to enforce - lots of looking up nodes in secondary indices before deleting and so on.  Definitely slow and messy.

There is still a lot of things to cover - some that I've already put in the code, and some that are coming soon on my radar screen.  Stay tuned for another exciting episode in Assimilation Schema for Neo4j next week.

 

Neo4j Server Schema for the Assimilation Project

In an earlier posting, I explained why I was looking at using the Neo4J graph database for the Assimilation Project.  In the meantime, I made the decision, created a schema for it, and wrote the code to create the database.  Although Neo4J is schemaless, every database needs a schema - a set of organizing principles - even if they are pretty flexible and aren't enforced by the database itself.

In this installment, I'll cover the basic schema of how I currently store information about servers in Neo4J.  In later installments I'll cover other parts of the schema - like how the rings are represented, how services and dependencies are represented and so on.

Along the way, I'll risk exposing my ignorance of how to store data "properly"

Server component graph

or "efficiently" in Neo4J.  And certainly, the Assimilation CMA code doesn't yet make any pretense of being efficient or fast.  Right now I just want it to work ;-)

 

For the server-only part of the graph, there are three basic kinds of entities represented.  These are:

• Servers (operating system images)
• NICs
• IP addresses

It's pretty easy to see that this particular part of the system is hierarchical.  If this were all there were to the overall schema, something like MongoDB might be a better choice.  But it's not all there is to a data center...

On the right is an example taken from running tests on my home desktop machine - inappropriately named servidor...

Since this is the Assimilation system, and we monitor the servers with "nanoprobes" it seemed appropriate to name servers "Drones"

This part of the system has three indexes to help find these components.  They are:

• Index of server names
• Index of MAC addresses (which point to NICs)
• Index of IP addresses

 As you can see I've created a variety of relationships between these various components. These are:

• nicowner - from a NIC to the Drone (server) it's installed in
• ipowner - from an IP address to the NIC that it's associated with
• iphost - from an IP address to the host that its NIC is installed in
• primary IP from a Drone to its "primary" IP address

One thing to note about Neo4J - you can follow a link in either direction.  So, for example, if you wanted to follow the nicowner link from the Drones to its NICs this is perfectly easy to do so.  Another sort of obvious thing is that the iphost links are redundant with the ipowner/nicowner path - but having them made writing the software a bit simpler - so I put them in.  I might change my mind in the future, but they're there for now.

There is more to the graphs that the CMA generate now, and will be even more in the future - and those will be covered in later blog posts.

As a closing note, I drew this graph with a little Python program I wrote that uses Nigel Small's excellent py2Neo bindings for Neo4j - which I also use in the Assimilation CMA.   My program is about 50 lines of code and feeds into the Graphviz's cool and useful dot program - I suspect I'll get to know it a lot better after I've worked with graphs for a while ;-).  I became aware of it because it draws all the source dependency graphs that Doxygen generates for the Assimilation web site.  Dependencies - graphs -- what an obvious combination.

 As a post-postcript, I've also started a mailing list for the Assimilation Project - which you can sign up for here:  http://lists.community.tummy.com/cgi-bin/mailman/listinfo/assimilation