Announcing the 0.1 er, 0.2 – 0.3 … um 0.4?, no wait – 1.51 release of DPDiscover for Cacti!  See the CAVEATS below.  You can download the latest version or clone the repository using git.

Cacti is a graphing solution that I’m putting a lot of time into in an effort to establish it as a replacement for a largely homegrown MRTG based system.  Cacti introduces some management and interface automation that we’ve been lacking in our MRTG implementation.  While not the perfect solution out of the box by any means, I have found it very easy to establish templates for data that Cacti might not graph out of the box, and even add desired queries (even those that are SSH script based) to Cacti.

DPDiscover

DPDiscover is the first of three plugins I see myself putting time into (and, ironically, is the least useful to my place of employment in the long run).  However, I’ve seen people asking for the capability on various Cacti forums, and our environment has some of the prerequisites needed for developing such a plugin, so I’m putting an unhealthy amount of time into it.  The second plugin I’m going to develop will (hopefully, if I can get it coded and running) collect information from multiple Cacti installs and simply replace the existing “Graph Tree” with one that will allow someone to browse the tree and find devices that are being polled by other Cacti installs (making the URL for the device take you to that other Cacti install).  The last plugin on my list is going to be for in-house use only, interfacing with our existing home grown management system to add devices to a given Cacti install.

DPDiscover is heavily based off of the original Cacti Auto Discovery plugin, with a few caveats:

• Auto Discovery included code to auto-add graphs and tree entries.  I stripped that out for DPDiscover; if you need that functionality, use Autom8.
• DPDiscover is heavily DNS dependent.  You should have all of your equipment available via either A or AAAA records.
• DPDiscovery, therefore, should be (though this may take a version or two for me to get properly tested) IPv6 compatible (we have a significant amount of equipment that is IPv6 accessible only).
• Auto Discovery simply swept IP(v4) ranges using a ping method to probe and then attempt to add equipment to Cacti.  DPDiscover works completely differently: it pulls the existing Cacti hosts (we’ll call them ‘parents’), uses SNMP to look for LLDP, CDP, and FDP information on those hosts, attempts to look up addresses for newly detected hosts, and then attempts to probe the new equipment via SNMP.  If successful, and the device in question matches to a Host Template properly, the new host is added and in turn also becomes a ‘parent’ so that its discovery protocol information can be pulled and further hosts detected and probed.
• All hosts (known, added, and detected) are kept track of and stored in a separate table for auditing purposes.  The option to add a device from the auditing page will be available.
• The option of using a known “parent’s” SNMP information in an attempt to probe a device exists.  There’s even a string based “filter” that can be used – if the SNMP information for the parent doesn’t contain the filter, it isn’t used.

I’ve tried to code DPDiscover to be different enough from Auto Discovery so that there shouldn’t be much in the way of collisions.  However, having both installed is not something I’ve tested for yet, so, you know, bad things could happen, and given that it’s all GPL, it’s all on you.

Instructions/Installation

To install DPDiscover, just download the most recent version of the plugin, untar it into your Cacti “plugins” directory (it should create a “dpdiscover” directory).  That’s pretty much it.

Usage:

1. In Cacti under “Settings” – “Misc”, there should now be a “DPDiscover” section.  Review all available options; explanations of each setting’s purpose should be right there for you to review.
2. DPDiscover uses hosts that already exist in Cacti in order to find other hosts, so you’ll need at least one parent available (manually entered into Cacti) for DPDiscover to start its search from.  Ideally, this should be a central piece of equipment, such as a backbone or node router/switch.  It must participate with its neighbors in LLDP, CDP, or FDP, so servers aren’t usually a good candidate for discovery purposes.
3. In the “Console” menu under “Templates”, there should be an entry called “DPDiscover Templates” (if you are not the user “Admin”, you might have to grant yourself access in the “User Management” section to the “View Host DPDiscover” “Realm”).  DPDiscover attempts to pull a “System Description” (OID .1.3.6.1.2.1.1.1.0) and match some substring of that up with a Host Template.  So, you’ll need host templates, and then some “DPDiscover Templates” that match those host templates to a “System Description”.  Depending on your host templates, you may need to be fairly specific.  “System Description”s of devices detected and successfully probed (added or not) as well as the those of “known” devices should be found in the “discover” tab (though be wary that HTML displays may hide multiple spaces within a given string). Failure to configure Host Templates and the DPDiscover Templates association will cause DPDiscover to exit (or possibly bug out).

Let a discovery run and see what happens …

Debugging

Running on Linux? Here’s some handy information on how to run the plugin in debug mode – from the command line.

1. Install the plugin as normal.  However, leave the “Poller Frequency” disabled.
2. Go to your linux command line and preferably change directory into the dpdiscover directory.
3. Run “php -q ./findhosts.php -d”.  Watch the info fly by.

You will get a confirmation email at the end (if you’ve configured an address) but no changes will be made to your Cacti database (no devices added, no updates sent to the plugin_dpdiscover_hosts table).  If you come across something that looks like a bug, please let me know.

CAVEATS

For 1.51:

• I woke up the next morning and figured out how to fix the third issue Fabrice had commented on, which was: When you add a device using the “discover” tab, its protocol doesn’t change, and it continues to offer the “Add” button.  Providing you use the “description” and “hostname” values DPDiscover thinks the new device should use, it will change the “protocol” and “Add” button to the text “ADDED”.

For 1.50:

• Fabrice35 on GitHub has quickly become my debugger for the “discover” tab. He encountered issues with some values on the “add” screen being zero (instead of pulling from the default settings) as well as the “description” field not following the DPDiscover settings (“FQDN” or not). In addition to these issues, I figured the IP/Hostname setting in DPDiscover should also be checked before passing values to the “add” function in Cacti. These issues are marked as resolved in 1.50.

For 1.49:

• Added filter to “discover” tab for “Protocol”, so you can view just the entries in the database discovered by a particular protocol.

For 1.48:

• Someone actually used the “discover” tab,  which is not as tested nor as retouched from the original “Auto Discovery” code as everything else is.  Turns out there was a couple of bugs which I was able to reproduce and fix.

For 1.47:

• A bug in all (?) previous versions becomes obvious when running from the command line using PHP 5.4.x.  The offending line wasn’t really needed/appropriate (as far as I can determine at this time), so it was removed (commented out in this version – will likely get removed later, or added back in a more sane fashion if it turns out it actually was needed).

For 1.46:

• DPDiscover (if you read the docs) requires Host Templates, and then for the Cacti Admin to associate those Host Templates with text of some sort so that, when a new device is discovered, it can be added with the proper host template.  I failed to properly check for the existence of these associations, and that would cause a small bug to appear.  This version resolves that issue.

For 1.45:

• Timing is a thing I can get anal about.  The nature of Cacti, crontabs, and polling would lead to “drift”.  The development site has multiple Cacti installs, all polling at 1 minute intervals, and DPDiscover would run daily, originally scheduled for 8:00am.  Drift would cause subsequent runs on a given server to start later and later, as the “start” time recorded in the database wouldn’t be “exactly” at 8:00:00 (to the second).  This would cause (little by little) the discovery to be pushed to later and later polls.  The solution I settled on is to “fudge” the start time by recording it not as the “actual” start time, but the start time pushed back to the minute (exact – 00 seconds) of the “base start time” (IE, with 8:00am as our start time, the time recorded in the database would be whatever hour of the day the job actually started, but with 00 minutes and 00 seconds before converting to the time int that is stored in the database).  So, if you have your jobs running every 4 hours, and a base start time of 8:00am, the jobs should record start times of 8:00:00am, 12:00:00pm, 4:00:00pm, etc. and should start somewhere close to the appropriate future (whenever polling is triggered, after the next appropriate start time – which, for a 5 minute polling interval, could be something like 8:04:03 depending on how long everything takes to run).

For 1.44:

• Sanity checks were added to ensure a couple of things required for proper functioning of DPDiscover existed prior to the sweeping of the network.

For 1.43:

• Disabled hosts are ignored; while read in from the database for some purposes, I’ve attempted to avoid having them be polled or probed for SNMP values.
• An issue exists where in this version and all versions prior, DPDiscover trusted that you knew what you were doing and wouldn’t check to see that you provided all needed information before running.  So, if you’re missing parent hosts or host template mappings, things won’t exactly work properly.

For 1.42:

• Not published outside of GitHub, this was an attempt to change the timing for polls so that they didn’t shift over time.

For 1.41:

• You can configure DPDiscover to not fix names.  If a device reports its name, and that name resolves to an IP, but the name on the device is not considered “authoritative”, then you’ll want to turn name changing off.

For 1.40:

• Code included will detect and fix a name change; if discovery reports a name that resolves to an IP, and the name is considered “unknown” but the IP is linked to a “known” host, the name of the known host will be changed to the discovered name.
• Some Brocade devices in the development environment, when FDP information was gathered via SNMPv2, would appear to “time out” and not successfully report any FDP information (this is due to a get_bulk_request issue that is probably software revision dependent).  A “Settings” option, if active (it is not by default), will trigger code that will rewalk the device (providing LLDP and CDP either aren’t being used or similarly fail) using FDP and SNMPv1 in a last ditch effort to attempt to find and add devices.

For 1.30:

• The “Clear DPDiscover Results” “Utilities” option should work now.
• The “upgrade” function in “setup.php” should be run by whatever program might need the system to be updated.  This includes dpdiscover.php (the report page) and findhosts.php (the poller).
• sysUptime in the hosts table has been changed to a bigint(64) in an attempt to get more accurate information regarding a device’s uptime.  sysUptime (and other SNMP values such as location, etc.) should be available for properly responding known, added, and devices that don’t match a template but appear to have a valid SNMP connection option.  Note that if a device only responds to SNMP v1, sysUptime will not be accurate beyond 228 days or so.
• A bug was patched in dpdiscover.php – I don’t run with errors shown on the page, but it turns out the reference to a value indexed (community string) wasn’t right.
• The Autom8 bug that I discovered lies with Autom8 only, and only if you’re playing with a Cacti database type of “mysqli”.  Using “mysql”?  You should be fine.

For 1.26:

• Where the heck did the setting for FDP go?  Well, it’s back now.  Not sure if the source I provided for 1.25 had it, but it disappeared from the git repo somewhere along the line.
• There appears to be an interoperability bug between DPDiscover and Autom8.  If Autom8’s “Tree Item Creation” is on, it appears as if Autom8 adds devices added by DPDiscover to all trees there are rules for.  I’m looking at a workaround but so far the only one I can figure is that you turn off “Tree Item Creation” if DPDiscover is going to run, and only use it manually (turn it on, then choose all devices and run Autom8 rules on them, and then turn it back off).

For 1.25:

• I changed the IP change code to use a more appropriate method of changing the IP.  It looks to be working properly – so much so that I was unaware that a live test took place.  See the caveats for 1.2 below for the continued disclaimers (though I think the code is a bit better than before).

For 1.2:

• Code was introduced to detect IP changes.  If a known host fails to connect properly, and both “Use IP for Hostname” and “Fix IP for Hostname” are active, DPDiscover attempts a DNS look up.  If a single IP is returned, and it is not equal to what was pulled from the ‘host’ table, the IP is SNMP probed.  If successful, DPDiscover continues on its way, using the new IP, after updating the ‘host’ and ‘poller_item’ tables appropriately.  HOWEVER, this is only the intention.  It has not successfully been tested.  Initial test did not include the ‘poller_item’ table and resulted in pulls failing until the setting was “saved” on the Device page for the test device.  It is very possible there is another table that may need updating to the new IP.
• Be sure to upgrade as instructed under the “1.1” caveats, and check your setting for “Fix IP for Hostname”.

For 1.1:

• New option: Include “skipped” or “excluded” hosts in the email report (or not).
• Upgrade from all versions: disable plugin, overwrite files, enable plugin.  This should not wipe any settings or tables and should (if upgrading from 0.1) change settings appropriately.

For 1.0:

• There is code to cover the possibility of using an IPv6 only available hostname as the “hostname” instead of the IP.  However, none of it has been tested.  I leave that to someone who’d rather have Cacti do the work of asking DNS to resolve IPs than just having Cacti go straight to the IP.
• FDP is now available as a discovery protocol but is disabled by default.  Visit your settings to turn it on.

For 0.4:

• IPv6 has been tested for “IP as Hostname” – IPs are added using “udp6:[IPv6::IP]”.  FQDN as hostname?  Probably very buggy if only an IPv6 address exists for the host.  This may also be very dependent on what version of SNMP you’re using (modern versions of Net-SNMP should be okay).
• You can now choose whether to use a particular discovery protocol or not.  Be aware that LLDP aware Cisco devices will be scanned both for LLDP and CDP, but this should be safe; the CDP scanner should only add devices not detected via LLDP.
• Yeah, still no FDP.  Maybe in 0.5?  I actually have Cacti on a mon-box that scans Brocade equipment now.
• Upgrading from everything except 0.1 should simply be a matter of overwriting the existing files, and then going in to “Settings” – “Misc”, verifying the settings for using a particular discovery protocol, and then “SAVE”ing the settings.

For 0.3:

• Many of the same caveats exist for 0.3 as did 0.2; 0.3 was an actual bugfix release.  Fixed in the release:
  • A failure to use the proper setting name in findhosts.php would, if configured to use parent SNMP values, fail to filter properly (resulting in always using the parent’s values).  I’m actually lucky this didn’t result in several pieces of equipment being accidentally added to the wrong Cacti server.
  • Our first install of the plugin was on CentOS 6.x; the second, on CentOS 5.x.  Turns out there’s a significant PHP difference in the two, so we have a new requirement: you must be using a version of PHP >= 5.2.0 (earlier with the “filter” PECL module may suffice).  CentOS 5.x users: uninstall all “php” packages and replace them with the “php53” ones.
• IPv6 has been tested and has failed.  0.4 should fix this; when the DNS query finds an IPv6 IP, it will wrap it like: udp6:[IPv6::IP]  Seems Cacti requires this, anyway, providing you’re using an SNMP version that understand this (like Net-SNMP).
• I had switched the parent port detection for LLDP to use the PortId (instead of PortDesc).  Unfortunately, I was reminded that the reason why I used PortDesc originally was that the Brocades use the port’s MAC address as the Id.  So, if a MAC address is detected, PortDesc will be used instead.  We can’t use just PortDesc because a recent Nexus 7000 OS started using, if configured, the port’s “description” (or ifAlias) for PortDesc.  Older versions of the Nexus 7000 OS didn’t publish either OID, so code is included to fall back to ifName.  It’s important to be aware of all this because this may become an ongoing issue when it comes to parent port names.

For 0.2:

• “discover” tab should be a bit more functional and clean.  The “ADD” function has been tested (once), but could still be buggy.
• Email reports work, and should include hosts with IPs that are not “known” and weren’t added (probably due to being unable to matched to a host template).
• Naming is more standard!  We are officially “DPDiscover” or “dpdiscover” (no “y”).  There was a lot of cleanup for this.
• Upgrading from 0.1:
  1. Oh, I’m sorry … well, “disable” and “uninstall” plugin using the Plugins interface in settings.
  2. ungzip/detar the file and put it in the plugins directory as “dpdiscover”.  Feel free to delete the “dpdiscovery” directory.
  3. “install” and “activate” the plugin.  This will unfortunately wipe out your DPDiscover Templates, so you’ll need to add those in.  However, it should go through the “settings” for “dpdiscovery”, delete them, but add them back in for “dpdiscover”.
• If you don’t see the “tab” at the top and “config.php” has the right setting in it, and you also don’t see “DPDiscover Templates” on the left menu, I’m guessing you’re an admin user, but not “admin”.  Grant yourself privileges in your “User Management” – user – “Realm Permissions”.  I can’t fix this; this is an issue with non “admin” users.

For 0.1:

• The “discovery” tab needs clean up and is just barely functional.  The “ADD” function from that screen hasn’t been touched/tested, and I need to remove the button when the device listed is a “known” device.
• The code in general is probably in need of much cleanup.
• It’s only been successfully run once on my own system.  You should probably run it from a command line with a “-d” argument and look at the debug data.  No changes to any database should be done when using a “-d”, but it will spit out a crap-ton of debugging information.
• The mail report function has not been successfully tested.  On my test run, email bugged out.  I figured out at least one thing that would make it bug and patched it.
• This version performs only LLDP and CDP searches (in that order).
• IPv6 compatibility was coded in but has not been tested yet.

Please submit bugs using (preferably) GitHub, (if you don’t want to sign up there) here, or (if you don’t want to sign up here) on the Cacti Forums posting for the plugin (and if you don’t want to sign up there either, then, well, you’re on your own).