[Okta Webinar] Learn how to a build a cloud-first strategyRegister Now

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 1790
  • Last Modified:

Problem with (someone else's) Perl code...

Hello,

I am using a monitoring package called BigBrother (written in C I think).  Anyway, one of the Open Source add-ons is written in Perl. Basically this perl script uses Net::SNMP to get some stats from Cisco devices, check the results, and return a status to the BigBrother server.  Forgetting the BigBrother part, all this script does is check the SNMP values, and execute a command that reports the status.

Status for BigBrother is based on color...
Red = Bad
Yellow = Warning
Green = OK
Clear = No results

As an example, this script will check the fans on a Cisco Router.  If all the fans return OK, then the result is green.  Another example, it will check the power supplies.  If all the power supplies are OK, it returns green.  

Here's the problem.  The code checks each power supply, then looks for the "worst color".  So if one power supply is bad, and three more are green, the overall alert should be red.  

Something is wrong with this routine.  It's returning green, even when one power supply is red.

This is someone else's code, and I tried to get in touch with them, but have gotten no answers.  Could anyone take a look at the code and see if you can find a problem.  I'll post the code below...

Thanks!
Shane
0
shanepresley
Asked:
shanepresley
  • 6
  • 4
1 Solution
 
shanepresleyAuthor Commented:
Full source and documentation at http://www.deadcat.net/viewfile.php?fileid=675

#!/usr/local/bin/perl

# Monolithic SNMP gatherer for BigBrother.

use strict;
use Carp;
use FileHandle;
use FindBin qw($Bin $Script);
use Net::SNMP qw(:snmp);
use vars qw($VERSION);
( $VERSION ) = '$Revision: 1.77 $' =~ /\$Revision:\s+([^\s]+)/;

my $debug = $ENV{'DEBUG'};

our $is_commandline = 0; # &env_check sets it to true if it looks like
                         # I was not called by BigBrother.
                         # In this case, I assume I'm testing and
                         # print to stdout instead of
                         # sending to BB.

my $defbbtmp = '/tmp';
my $defbbhome = '/usr/local/bb/bb';
my $defbb = "$defbbhome/bin/bb";
my $defbbdisp = '127.0.0.1';
my $defmachine = 'localhost';

# For SNMP
my $defretries = 3;
my $deftimeout = 5;
my %snmpoids = (
  'sysDescr'      => '.1.3.6.1.2.1.1.1.0',
  'sysObjectID'   => '.1.3.6.1.2.1.1.2.0',
  'sysUpTime'     => '.1.3.6.1.2.1.1.3.0',
  'sysName'       => '.1.3.6.1.2.1.1.5.0',
  'ifDescr'       => '.1.3.6.1.2.1.2.2.1.2',
  'ifSpeed'       => '.1.3.6.1.2.1.2.2.1.5',
  'ifOperStatus'  => '.1.3.6.1.2.1.2.2.1.8',
  'tcp'                => '.1.3.6.1.2.1.6',
  'tcpCurrEstab'       => '.1.3.6.1.2.1.6.9',
  'tcpConnRemAddress'  => '.1.3.6.1.2.1.6.13.1.4',
  'bgpLocalAs'      => '.1.3.6.1.2.1.15.2.0',
  'bgpPeerState'    => '.1.3.6.1.2.1.15.3.1.2',
  'bgpPeerAdminStatus'    => '.1.3.6.1.2.1.15.3.1.2',
  'bgpPeerRemoteAs' => '.1.3.6.1.2.1.15.3.1.9',
  'bgpPeerFsmEstablishedTime'  => '.1.3.6.1.2.1.15.3.1.16',
  # hrStorage used mostly by the UCD/NET daemon
  # Memory and disk information are stored in the same tree
  # UCD/NET also uses separate dskTable and memory tables
  'host'                     => '.1.3.6.1.2.1.25',
  'hrStorage'                => '.1.3.6.1.2.1.25.2',
  'hrStorageTypes'           => '.1.3.6.1.2.1.25.2.1',
  'hrStorageOther'           => '.1.3.6.1.2.1.25.2.1.1',
  'hrStorageRam'             => '.1.3.6.1.2.1.25.2.1.2',
  'hrStorageVirtualMemory'   => '.1.3.6.1.2.1.25.2.1.3',
  'hrStorageFixedDisk'       => '.1.3.6.1.2.1.25.2.1.4',
  'hrStorageRemovableDisk'   => '.1.3.6.1.2.1.25.2.1.5',
  'hrStorageFloppyDisk'      => '.1.3.6.1.2.1.25.2.1.6',
  'hrStorageCompactDisc'     => '.1.3.6.1.2.1.25.2.1.7',
  'hrStorageRamDisk'         => '.1.3.6.1.2.1.25.2.1.8',
  'hrStorageFlashMemory'     => '.1.3.6.1.2.1.25.2.1.9',
  'hrStorageNetworkDisk'     => '.1.3.6.1.2.1.25.2.1.10',
  'hrStorageEntry'           => '.1.3.6.1.2.1.25.2.3.1',
  'hrStorageType'            => '.1.3.6.1.2.1.25.2.3.1.2',
  'hrStorageDescr'           => '.1.3.6.1.2.1.25.2.3.1.3',
  'hrStorageAllocationUnits' => '.1.3.6.1.2.1.25.2.3.1.4',
  'hrStorageSize'            => '.1.3.6.1.2.1.25.2.3.1.5',
  'hrStorageUsed'            => '.1.3.6.1.2.1.25.2.3.1.6',
  'ifXEntry'       => '.1.3.6.1.2.1.31.1.1.1',
  'ifAlias'        => '.1.3.6.1.2.1.31.1.1.1.18',
  'enterprises'                => '.1.3.6.1.4.1',
  'cisco'                      => '.1.3.6.1.4.1.9',
  'busyPer'                    => '.1.3.6.1.4.1.9.2.1.56.0',
  'avgBusy1'                   => '.1.3.6.1.4.1.9.2.1.57.0',
  'avgBusy5'                   => '.1.3.6.1.4.1.9.2.1.58.0',
  'ciscoEnvMonTemperatureStatusTable' => '.1.3.6.1.4.1.9.9.13.1.3',
  'ciscoEnvMonTemperatureStatusEntry' => '.1.3.6.1.4.1.9.9.13.1.3.1',
  'ciscoEnvMonTemperatureStatusDescr' => '.1.3.6.1.4.1.9.9.13.1.3.1.2',
  'ciscoEnvMonTemperatureStatusValue' => '.1.3.6.1.4.1.9.9.13.1.3.1.3',
  'ciscoEnvMonTemperatureThreshold'   => '.1.3.6.1.4.1.9.9.13.1.3.1.4',
  'ciscoEnvMonTemperatureLastShutdown'=> '.1.3.6.1.4.1.9.9.13.1.3.1.5',
  'ciscoEnvMonTemperatureState'       => '.1.3.6.1.4.1.9.9.13.1.3.1.6',
  'ciscoEnvMonFanStatusDescr'         => '.1.3.6.1.4.1.9.9.13.1.4.1.2',
  'ciscoEnvMonFanState'               => '.1.3.6.1.4.1.9.9.13.1.4.1.3',
  'ciscoEnvMonSupplyStatusDescr'      => '.1.3.6.1.4.1.9.9.13.1.5.1.2',
  'ciscoEnvMonSupplyState'            => '.1.3.6.1.4.1.9.9.13.1.5.1.3',
  'ciscoEnvMonSupplySource'           => '.1.3.6.1.4.1.9.9.13.1.5.1.4',
  'ciscoMemoryPoolMIB'                => '.1.3.6.1.4.1.9.9.48',
  'ciscoMemoryPoolObjects'            => '.1.3.6.1.4.1.9.9.48.1',
  'ciscoMemoryPoolTable'              => '.1.3.6.1.4.1.9.9.48.1.1',
  'ciscoMemoryPoolEntry'              => '.1.3.6.1.4.1.9.9.48.1.1.1',
  'ciscoMemoryPoolType'               => '.1.3.6.1.4.1.9.9.48.1.1.1.1',
  'ciscoMemoryPoolName'               => '.1.3.6.1.4.1.9.9.48.1.1.1.2',
  'ciscoMemoryPoolAlternate'          => '.1.3.6.1.4.1.9.9.48.1.1.1.3',
  'ciscoMemoryPoolValid'              => '.1.3.6.1.4.1.9.9.48.1.1.1.4',
  'ciscoMemoryPoolUsed'               => '.1.3.6.1.4.1.9.9.48.1.1.1.5',
  'ciscoMemoryPoolFree'               => '.1.3.6.1.4.1.9.9.48.1.1.1.6',
  'ciscoMemoryPoolLargestFree'        => '.1.3.6.1.4.1.9.9.48.1.1.1.7',
  'c2900PortIfIndex'           => '.1.3.6.1.4.1.9.9.87.1.4.1.1.25.0',
  'c2900PortDuplexState'       => '.1.3.6.1.4.1.9.9.87.1.4.1.1.31.0',
  'c2900PortDuplexStatus'      => '.1.3.6.1.4.1.9.9.87.1.4.1.1.32.0',
  'c2900PortAdminSpeed'        => '.1.3.6.1.4.1.9.9.87.1.4.1.1.33.0',
  'ciscoProcessMIB'            => '.1.3.6.1.4.1.9.9.109',
  'cpmCPUTotal5sec'            => '.1.3.6.1.4.1.9.9.109.1.1.1.1.3',
  'cpmCPUTotal1min'            => '.1.3.6.1.4.1.9.9.109.1.1.1.1.4',
  'cpmCPUTotal5min'            => '.1.3.6.1.4.1.9.9.109.1.1.1.1.5',
  'cpmProcess'                 => '.1.3.6.1.4.1.9.9.109.1.2',
  'cpmProcessEntry'            => '.1.3.6.1.4.1.9.9.109.1.2.1.1',
  'cpmProcessName'             => '.1.3.6.1.4.1.9.9.109.1.2.1.1.2',
  'cpmProcessuSecs'            => '.1.3.6.1.4.1.9.9.109.1.2.1.1.4',
  'cpmProcessAverageUSecs'     => '.1.3.6.1.4.1.9.9.109.1.2.1.1.6',
  'cpmProcessExtEntry'         => '.1.3.6.1.4.1.9.9.109.1.2.2.1',
  'cpmProcExtPriority'         => '.1.3.6.1.4.1.9.9.109.1.2.2.1.8',
  'cpmProcessExtRevEntry'      => '.1.3.6.1.4.1.9.9.109.1.2.3.1',
  'cpmProcExtPriorityRev'      => '.1.3.6.1.4.1.9.9.109.1.2.3.1.8',
  'ciscoFirewallMIB'           => '.1.3.6.1.4.1.9.9.147',
  'cfwHardwareStatusValue'     => '.1.3.6.1.4.1.9.9.147.1.2.1.1.1.3',
  'cfwHardwareStatusDetail'    => '.1.3.6.1.4.1.9.9.147.1.2.1.1.1.4',
  'cfwBufferStatValue'         => '.1.3.6.1.4.1.9.9.147.1.2.2.1.1.4',
  'cfwConnectionStatValue'     => '.1.3.6.1.4.1.9.9.147.1.2.2.2.1.5',
  # Original extension by Craig Cook uses things like cpqHoCpuUtilMin.0
  # even though CPU items are in a table and table entries usually don't
  # get a zero stuck at the end. I don't have a Compaq to test so I must
  # be careful, prepare for both possibilities with Compaq items.
  'compaq'                       => '.1.3.6.1.4.1.232',
  'cpqHeCorrMemLogStatus'        => '.1.3.6.1.4.1.232.6.2.3.1.0',
  'cpqHeCorrMemTotalErrs'        => '.1.3.6.1.4.1.232.6.2.3.3.0',
  'cpqHeCorrMemErrorCntThresh'   => '.1.3.6.1.4.1.232.6.2.3.5.0',
  'cpqHeThermalCondition'        => '.1.3.6.1.4.1.232.6.2.6.1.0',
  'cpqHeThermalTempStatus'       => '.1.3.6.1.4.1.232.6.2.6.3.0',
  'cpqHeThermalSystemFanStatus'  => '.1.3.6.1.4.1.232.6.2.6.4.0',
  'cpqHeThermalCpuFanStatus'     => '.1.3.6.1.4.1.232.6.2.6.5.0',
  'cpqHeTemperatureIndex'        => '.1.3.6.1.4.1.232.6.2.6.8.1.2',
  'cpqHeTemperatureLocale'       => '.1.3.6.1.4.1.232.6.2.6.8.1.3',
  'cpqHeTemperatureCelsius'      => '.1.3.6.1.4.1.232.6.2.6.8.1.4',
  'cpqHeTemperatureThreshold'    => '.1.3.6.1.4.1.232.6.2.6.8.1.5',
  'cpqHeTemperatureCondition'    => '.1.3.6.1.4.1.232.6.2.6.8.1.6',
  'cpqHeEventLogSupported'       => '.1.3.6.1.4.1.232.6.2.11.1',
  'cpqHeEventLogCondition'       => '.1.3.6.1.4.1.232.6.2.11.2',
  'cpqHeEventLogEntry'           => '.1.3.6.1.4.1.232.6.2.11.3.1',
  'cpqHeEventLogEntryNumber'     => '.1.3.6.1.4.1.232.6.2.11.3.1.1',
  'cpqHeEventLogEntrySeverity'   => '.1.3.6.1.4.1.232.6.2.11.3.1.2',
  'cpqHeEventLogEntryCount'      => '.1.3.6.1.4.1.232.6.2.11.3.1.5',
  'cpqHeEventLogInitialTime'     => '.1.3.6.1.4.1.232.6.2.11.3.1.6',
  'cpqHeEventLogUpdateTime'      => '.1.3.6.1.4.1.232.6.2.11.3.1.7',
  'cpqHeEventLogErrorDesc'       => '.1.3.6.1.4.1.232.6.2.11.3.1.8',
  'cpqHeEventLogFreeFormData'    => '.1.3.6.1.4.1.232.6.2.11.3.1.9',
  'cpqHostOs'                    => '.1.3.6.1.4.1.232.11',
  'cpqHoCpuUtilEntry'            => '.1.3.6.1.4.1.232.11.2.3.1.1',
  'cpqHoCpuUtilMin'              => '.1.3.6.1.4.1.232.11.2.3.1.1.2',
  'cpqHoCpuUtilFiveMin'          => '.1.3.6.1.4.1.232.11.2.3.1.1.3',
  'cpqHoCpuUtilThirtyMin'        => '.1.3.6.1.4.1.232.11.2.3.1.1.4',
  'cpqHoCpuUtilHour'             => '.1.3.6.1.4.1.232.11.2.3.1.1.5',
  'cpqHoFileSysEntry'            => '.1.3.6.1.4.1.232.11.2.4.1.1',
  'cpqHoFileSysIndex'            => '.1.3.6.1.4.1.232.11.2.4.1.1.1',
  'cpqHoFileSysDesc'             => '.1.3.6.1.4.1.232.11.2.4.1.1.2',
  'cpqHoFileSysSpaceTotal'       => '.1.3.6.1.4.1.232.11.2.4.1.1.3',
  'cpqHoFileSysSpaceUsed'        => '.1.3.6.1.4.1.232.11.2.4.1.1.4',
  'cpqHoFileSysPercentSpaceUsed' => '.1.3.6.1.4.1.232.11.2.4.1.1.5',
  'cpqHoFileSysAllocUnitsTotal'  => '.1.3.6.1.4.1.232.11.2.4.1.1.6',
  'cpqHoFileSysAllocUnitsUsed'   => '.1.3.6.1.4.1.232.11.2.4.1.1.7',
  'cpqHoPhysicalMemorySize'      => '.1.3.6.1.4.1.232.11.2.13.1.0',
  'cpqHoPhysicalMemoryFree'      => '.1.3.6.1.4.1.232.11.2.13.2.0',
  'cpqHoPagingMemorySize'        => '.1.3.6.1.4.1.232.11.2.13.3.0',
  'cpqHoPagingMemoryFree'        => '.1.3.6.1.4.1.232.11.2.13.4.0',
  'upsBasicIdentModel'             => '.1.3.6.1.4.1.318.1.1.1.1.1.1.0',
  'upsBasicBatteryStatus'          => '.1.3.6.1.4.1.318.1.1.1.2.1.1.0',
  'upsBasicBatteryTimeOnBattery'   => '.1.3.6.1.4.1.318.1.1.1.2.1.2.0',
  'upsAdvBatteryCapacity'          => '.1.3.6.1.4.1.318.1.1.1.2.2.1.0',
  'upsAdvBatteryTemperature'       => '.1.3.6.1.4.1.318.1.1.1.2.2.2.0',
  'upsAdvBatteryRunTimeRemaining'  => '.1.3.6.1.4.1.318.1.1.1.2.2.3.0',
  'upsAdvBatteryReplaceIndicator'  => '.1.3.6.1.4.1.318.1.1.1.2.2.4.0',
  'upsAdvInputLineVoltage'         => '.1.3.6.1.4.1.318.1.1.1.3.2.1.0',
  'upsAdvInputLineFailCause'       => '.1.3.6.1.4.1.318.1.1.1.3.2.5.0',
  'upsBasicOutputStatus'           => '.1.3.6.1.4.1.318.1.1.1.4.1.1.0',
  'upsAdvOutputVoltage'            => '.1.3.6.1.4.1.318.1.1.1.4.2.1.0',
  'upsAdvOutputLoad'               => '.1.3.6.1.4.1.318.1.1.1.4.2.3.0',
  'upsBasicBatteryLastReplaceDate' => '.1.3.6.1.4.1.318.1.1.1.2.1.3.0',
  'netapp'                      => '.1.3.6.1.4.1.789',
  'miscNfsOps'                  => '.1.3.6.1.4.1.789.1.2.2.1.0',
  'miscNetRcvdKB'               => '.1.3.6.1.4.1.789.1.2.2.2.0',
  'miscNetSentKB'               => '.1.3.6.1.4.1.789.1.2.2.3.0',
  'miscGlobalStatus'            => '.1.3.6.1.4.1.789.1.2.2.4.0',
  'miscHighNfsOps'              => '.1.3.6.1.4.1.789.1.2.2.5.0',
  'miscLowNfsOps'               => '.1.3.6.1.4.1.789.1.2.2.6.0',
  'miscHighNetRcvdBytes'        => '.1.3.6.1.4.1.789.1.2.2.11.0',
  'miscLowNetRcvdBytes'         => '.1.3.6.1.4.1.789.1.2.2.12.0',
  'miscHighNetSentBytes'        => '.1.3.6.1.4.1.789.1.2.2.13.0',
  'miscLowNetSentBytes'         => '.1.3.6.1.4.1.789.1.2.2.14.0',
  'envOverTemperature'          => '.1.3.6.1.4.1.789.1.2.4.1.0',
  'envFailedFanCount'           => '.1.3.6.1.4.1.789.1.2.4.2.0',
  'envFailedFanMessage'         => '.1.3.6.1.4.1.789.1.2.4.3.0',
  'envFailedPowerSupplyCount'   => '.1.3.6.1.4.1.789.1.2.4.4.0',
  'envFailedPowerSupplyMessage' => '.1.3.6.1.4.1.789.1.2.4.5.0',
  'cpuBusyTime'                => '.1.3.6.1.4.1.789.1.2.1.2.0',
  'cpuBusyTimePerCent'         => '.1.3.6.1.4.1.789.1.2.1.3.0',
  'cpuIdleTimePerCent'         => '.1.3.6.1.4.1.789.1.2.1.5.0',
  'qrEntry'                    => '.1.3.6.1.4.1.789.1.4.3.1',
  'qrIndex'                    => '.1.3.6.1.4.1.789.1.4.3.1.1',
  'qrType'                     => '.1.3.6.1.4.1.789.1.4.3.1.2',
  'qrId'                       => '.1.3.6.1.4.1.789.1.4.3.1.3',
  'qrKBytesUsed'               => '.1.3.6.1.4.1.789.1.4.3.1.4',
  'qrKBytesLimit'              => '.1.3.6.1.4.1.789.1.4.3.1.5',
  'qrFilesUsed'                => '.1.3.6.1.4.1.789.1.4.3.1.6',
  'qrFileLimit'                => '.1.3.6.1.4.1.789.1.4.3.1.7',
  'qrPathName'                 => '.1.3.6.1.4.1.789.1.4.3.1.8',
  'dfEntry'                    => '.1.3.6.1.4.1.789.1.5.4.1',
  'dfIndex'                    => '.1.3.6.1.4.1.789.1.5.4.1.1',
  'dfFileSys'                  => '.1.3.6.1.4.1.789.1.5.4.1.2',
  'dfKBytesTotal'              => '.1.3.6.1.4.1.789.1.5.4.1.3',
  'dfKBytesUsed'               => '.1.3.6.1.4.1.789.1.5.4.1.4',
  'dfKBytesAvail'              => '.1.3.6.1.4.1.789.1.5.4.1.5',
  'dfPerCentKBytesCapacity'    => '.1.3.6.1.4.1.789.1.5.4.1.6',
  'dfInodesUsed'               => '.1.3.6.1.4.1.789.1.5.4.1.7',
  'dfInodesFree'               => '.1.3.6.1.4.1.789.1.5.4.1.8',
  'dfPerCentInodeCapacity'     => '.1.3.6.1.4.1.789.1.5.4.1.9',
  'dfMountedOn'                => '.1.3.6.1.4.1.789.1.5.4.1.10',
  'dfMaxFilesAvail'            => '.1.3.6.1.4.1.789.1.5.4.1.11',
  'dfMaxFilesUsed'             => '.1.3.6.1.4.1.789.1.5.4.1.12',
  'dfMaxFilesPossible'         => '.1.3.6.1.4.1.789.1.5.4.1.13',
  'dfHighTotalKBytes'          => '.1.3.6.1.4.1.789.1.5.4.1.14',
  'dfLowTotalKBytes'           => '.1.3.6.1.4.1.789.1.5.4.1.15',
  'dfHighUsedKBytes'           => '.1.3.6.1.4.1.789.1.5.4.1.16',
  'dfLowUsedKBytes'            => '.1.3.6.1.4.1.789.1.5.4.1.17',
  'dfHighAvailKBytes'          => '.1.3.6.1.4.1.789.1.5.4.1.18',
  'dfLowAvailKBytes'           => '.1.3.6.1.4.1.789.1.5.4.1.19',
  'foundry'                    => '.1.3.6.1.4.1.1991',
  'snChasActualTemperature'    => '.1.3.6.1.4.1.1991.1.1.1.1.18.0',
  'snChasWarningTemperature'   => '.1.3.6.1.4.1.1991.1.1.1.1.19.0',
  'snChasShutdownTemperature'  => '.1.3.6.1.4.1.1991.1.1.1.1.20.0',
  'snChasPwrSupplyDescription' => '.1.3.6.1.4.1.1991.1.1.1.2.1.1.2',
  'snChasPwrSupplyOperStatus'  => '.1.3.6.1.4.1.1991.1.1.1.2.1.1.3',
  'snChasFanDescription'       => '.1.3.6.1.4.1.1991.1.1.1.3.1.1.2',
  'snChasFanOperStatus'        => '.1.3.6.1.4.1.1991.1.1.1.3.1.1.3',
  'snAgGblCpuUtilData'         => '.1.3.6.1.4.1.1991.1.1.2.1.35.0',
  'snAgGblCpuUtil1SecAvg'      => '.1.3.6.1.4.1.1991.1.1.2.1.50.0',
  'snAgGblCpuUtil5SecAvg'      => '.1.3.6.1.4.1.1991.1.1.2.1.51.0',
  'snAgGblCpuUtil1MinAvg'      => '.1.3.6.1.4.1.1991.1.1.2.1.52.0',
  'snAgGblDynMemUtil'          => '.1.3.6.1.4.1.1991.1.1.2.1.53.0',
  'snAgGblDynMemTotal'         => '.1.3.6.1.4.1.1991.1.1.2.1.54.0',
  'snAgGblDynMemFree'          => '.1.3.6.1.4.1.1991.1.1.2.1.55.0',
  'snSwPortInfoChnMode'                 => '.1.3.6.1.4.1.1991.1.1.3.3.1.1.4',
  'snSwPortInfoSpeed'                   => '.1.3.6.1.4.1.1991.1.1.3.3.1.1.5',
  'snSwPortName'                        => '.1.3.6.1.4.1.1991.1.1.3.3.1.1.24',
  'snSwPortIfIndex'                     => '.1.3.6.1.4.1.1991.1.1.3.3.1.1.38',
  'snL4VirtualServerIndex'              => '.1.3.6.1.4.1.1991.1.1.4.2.1.1.1',
  'snL4VirtualServerName'               => '.1.3.6.1.4.1.1991.1.1.4.2.1.1.2',
  'snL4VirtualServerVirtualIP'          => '.1.3.6.1.4.1.1991.1.1.4.2.1.1.3',
  'snL4VirtualServerAdminStatus'        => '.1.3.6.1.4.1.1991.1.1.4.2.1.1.4',
  'snL4RealServerIndex'                 => '.1.3.6.1.4.1.1991.1.1.4.3.1.1.1',
  'snL4RealServerName'                  => '.1.3.6.1.4.1.1991.1.1.4.3.1.1.2',
  'snL4RealServerIP'                    => '.1.3.6.1.4.1.1991.1.1.4.3.1.1.3',
  'snL4RealServerAdminStatus'           => '.1.3.6.1.4.1.1991.1.1.4.3.1.1.4',
  'snL4VirtualServerPortIndex'          => '.1.3.6.1.4.1.1991.1.1.4.4.1.1.1',
  'snL4VirtualServerPortServerName'     => '.1.3.6.1.4.1.1991.1.1.4.4.1.1.2',
  'snL4VirtualServerPortPort'           => '.1.3.6.1.4.1.1991.1.1.4.4.1.1.3',
  'snL4VirtualServerPortAdminStatus'    => '.1.3.6.1.4.1.1991.1.1.4.4.1.1.4',
  'snL4RealServerPortIndex'             => '.1.3.6.1.4.1.1991.1.1.4.5.1.1.1',
  'snL4RealServerPortServerName'        => '.1.3.6.1.4.1.1991.1.1.4.5.1.1.2',
  'snL4RealServerPortPort'              => '.1.3.6.1.4.1.1991.1.1.4.5.1.1.3',
  'snL4RealServerPortAdminStatus'       => '.1.3.6.1.4.1.1991.1.1.4.5.1.1.4',
  'snL4RealServerStatusIndex'           => '.1.3.6.1.4.1.1991.1.1.4.8.1.1.1',
  'snL4RealServerStatusName'            => '.1.3.6.1.4.1.1991.1.1.4.8.1.1.2',
  'snL4RealServerStatusState'           => '.1.3.6.1.4.1.1991.1.1.4.8.1.1.9',
  'snL4RealServerPortStatusIndex'       => '.1.3.6.1.4.1.1991.1.1.4.10.1.1.1',
  'snL4RealServerPortStatusPort'        => '.1.3.6.1.4.1.1991.1.1.4.10.1.1.2',
  'snL4RealServerPortStatusServerName'  => '.1.3.6.1.4.1.1991.1.1.4.10.1.1.3',
  'snL4RealServerPortStatusState'       => '.1.3.6.1.4.1.1991.1.1.4.10.1.1.5',
  'snL4BindIndex'                       => '.1.3.6.1.4.1.1991.1.1.4.6.1.1.1',
  'snL4BindVirtualServerName'           => '.1.3.6.1.4.1.1991.1.1.4.6.1.1.2',
  'snL4BindVirtualPortNumber'           => '.1.3.6.1.4.1.1991.1.1.4.6.1.1.3',
  'snL4BindRealServerName'              => '.1.3.6.1.4.1.1991.1.1.4.6.1.1.4',
  'snL4BindRealPortNumber'              => '.1.3.6.1.4.1.1991.1.1.4.6.1.1.5',
  'snBgp4GenLocalAs'                    => '.1.3.6.1.4.1.1991.1.2.11.1.28.0',
  'snBgp4NeighGenCfgNeighIp'            => '.1.3.6.1.4.1.1991.1.2.11.6.1.1.1',
  'snBgp4NeighGenCfgRemoteAs'           => '.1.3.6.1.4.1.1991.1.2.11.6.1.1.7',
  'snBgp4NeighGenCfgShutdown'           => '.1.3.6.1.4.1.1991.1.2.11.6.1.1.16',
  'snBgp4NeighGenCfgDesc'               => '.1.3.6.1.4.1.1991.1.2.11.6.1.1.20',
  'snBgp4NeighOperStatusIndex'          => '.1.3.6.1.4.1.1991.1.2.11.15.1.1.1',
  'snBgp4NeighOperStatusIp'             => '.1.3.6.1.4.1.1991.1.2.11.15.1.1.2',
  'snBgp4NeighOperStatusRemoteAs'       => '.1.3.6.1.4.1.1991.1.2.11.15.1.1.3',
  'snBgp4NeighOperStatusBgpType'        => '.1.3.6.1.4.1.1991.1.2.11.15.1.1.4',
  'snBgp4NeighOperStatusState'          => '.1.3.6.1.4.1.1991.1.2.11.15.1.1.5',
  'snBgp4NeighborSummaryIndex'          => '.1.3.6.1.4.1.1991.1.2.11.17.1.1.1',
  'snBgp4NeighborSummaryIp'             => '.1.3.6.1.4.1.1991.1.2.11.17.1.1.2',
  'snBgp4NeighborSummaryState'          => '.1.3.6.1.4.1.1991.1.2.11.17.1.1.3',
  'snBgp4NeighborSummaryStateChgTime'   => '.1.3.6.1.4.1.1991.1.2.11.17.1.1.4',
  'snSI'                                => '.1.3.6.1.4.1.1991.1.3.3.1',
  'snSIXL'                              => '.1.3.6.1.4.1.1991.1.3.3.2',
  # The UCD/NET-SNMP daemon uses many items in the mib-2.host branch
  'ucdavis'                   => '.1.3.6.1.4.1.2021',
  'dskEntry'                  => '.1.3.6.1.4.1.2021.9.1',
  'dskPath'                   => '.1.3.6.1.4.1.2021.9.1.2',
  'dskDevice'                 => '.1.3.6.1.4.1.2021.9.1.3',
  'dskTotal'                  => '.1.3.6.1.4.1.2021.9.1.6',
  'dskAvail'                  => '.1.3.6.1.4.1.2021.9.1.7',
  'dskUsed'                   => '.1.3.6.1.4.1.2021.9.1.8',
  'dskPercent'                => '.1.3.6.1.4.1.2021.9.1.9',
  'dskPercentInode'           => '.1.3.6.1.4.1.2021.9.1.10',
  'laLoadFloat'               => '.1.3.6.1.4.1.2021.10.1.6',
  'laLoadFloat.1'             => '.1.3.6.1.4.1.2021.10.1.6.1',
  'laLoadFloat.2'             => '.1.3.6.1.4.1.2021.10.1.6.2',
  'laLoadFloat.3'             => '.1.3.6.1.4.1.2021.10.1.6.3',
  'ssCpuUser'                 => '.1.3.6.1.4.1.2021.11.9.0',
  'ssCpuSystem'               => '.1.3.6.1.4.1.2021.11.10.0',
  'ssCpuIdle'                 => '.1.3.6.1.4.1.2021.11.11.0',
  'ucdVersionTag'             => '.1.3.6.1.4.1.2021.100.2.0',
  'netscreen'                 => '.1.3.6.1.4.1.3224',
  'nsSetGenSwVer'             => '.1.3.6.1.4.1.3224.7.1.5.0',
  'nsResCpuLast1Min'          => '.1.3.6.1.4.1.3224.16.1.2.0',
  'nsResCpuLast5Min'          => '.1.3.6.1.4.1.3224.16.1.3.0',
  'nsResCpuLast15Min'         => '.1.3.6.1.4.1.3224.16.1.4.0',
);

my %enterprisenumbers = (
  9 => 'cisco',
  232 => 'compaq',
  318 => 'apc',
  529 => 'ascend',
  789 => 'netapp',
  1991 => 'foundry',
  2021 => 'ucdavis',
  3224 => 'netscreen',
);

my %snmpsyntaxdb = (
  # .1.3.6.1.2.1.2.2.1.8
  'ifOperStatus' => {
    '1' => 'up',
    '2' => 'down',
    '3' => 'testing',
    '4' => 'unknown',
    '5' => 'dormant',
    '6' => 'notPresent',
    '7' => 'lowerLayerDown',
  },
  # .1.3.6.1.2.1.15.3.1.2
  'bgpPeerState' => {
    '0' => 'nostate',
    '1' => 'idle',
    '2' => 'connect',
    '3' => 'active',
    '4' => 'opensent',
    '5' => 'openconfirm',
    '6' => 'established',
  },
  # .1.3.6.1.2.1.15.3.1.3
  'bgpPeerAdminStatus' => {
    '1' => 'stop',
    '2' => 'start',
  },
  # .1.3.6.1.4.1.9.9.13.1.3.1.6
  'ciscoEnvMonTemperatureState' => {
    '1' => 'normal',
    '2' => 'warning',
    '3' => 'critical',
    '4' => 'shutdown',
    '5' => 'notPresent',
    '6' => 'notFunctioning',
  },
  # .1.3.6.1.4.1.9.9.13.1.4.1.3
  'ciscoEnvMonFanState' => {
    '1' => 'normal',
    '2' => 'warning',
    '3' => 'critical',
    '4' => 'shutdown',
    '5' => 'notPresent',
    '6' => 'notFunctioning'
  },
  # .1.3.6.1.4.1.9.9.13.1.5.1.3
  'ciscoEnvMonSupplyState' => {
    '1' => 'normal',
    '2' => 'warning',
    '3' => 'critical',
    '4' => 'shutdown',
    '5' => 'notPresent',
    '6' => 'notFunctioning',
  },
  # .1.3.6.1.4.1.9.9.13.1.5.1.4
  'ciscoEnvMonSupplySource' => {
    '1' => 'unknown',
    '2' => 'ac',
    '3' => 'dc',
    '4' => 'externalPowerSupply',
    '5' => 'internalRedundant',
  },
  # .1.3.6.1.4.1.9.9.87.1.4.1.1.31.0
  'c2900PortDuplexState' => {
    '1' => 'full',
    '2' => 'half',
    '3' => 'auto',
  },
  # .1.3.6.1.4.1.9.9.87.1.4.1.1.32.0
  'c2900PortDuplexStatus' => {
    '1' => 'full',
    '2' => 'half',
  },
  # .1.3.6.1.4.1.9.9.87.1.4.1.1.33.0
  'c2900PortAdminSpeed' => {
    '1' => 'auto',
    '10000000' => '10000000',
    '100000000' => '100000000',
    '155520000' => '155520000',
  },
  # .1.3.6.1.4.1.9.9.109.1.2.2.1.8
  'cpmProcExtPriority' => {
    '1' => 'critical',
    '2' => 'high',
    '3' => 'normal',
    '4' => 'low',
    '5' => 'notAssigned',
  },
  # .1.3.6.1.4.1.9.9.109.1.2.3.1.8
  'cpmProcExtPriorityRev' => {
    '1' => 'critical',
    '2' => 'high',
    '3' => 'normal',
    '4' => 'low',
    '5' => 'notAssigned',
  },
  # .1.3.6.1.4.1.9.9.147.1.2.1.1.1.3
  'cfwHardwareStatusValue' => {
    '1' => 'other',
    '2' => 'up',
    '3' => 'down',
    '4' => 'error',
    '5' => 'overTemp',
    '6' => 'busy',
    '7' => 'noMedia',
    '8' => 'backup',
    '9' => 'active',
    '10' => 'standby',
  },
  # .1.3.6.1.4.1.232.6.2.6.1.0
  'cpqHeThermalCondition' => {
    '1' => 'other',
    '2' => 'ok',
    '3' => 'degraded',
    '4' => 'failed',
  },
  # .1.3.6.1.4.1.232.6.2.6.3.0
  'cpqHeThermalTempStatus' => {
    '1' => 'other',
    '2' => 'ok',
    '3' => 'degraded',
    '4' => 'failed',
  },
  # .1.3.6.1.4.1.232.6.2.6.4.0
  'cpqHeThermalSystemFanStatus' => {
    '1' => 'other',
    '2' => 'ok',
    '3' => 'degraded',
    '4' => 'failed',
  },
  # .1.3.6.1.4.1.232.6.2.6.5.0
  'cpqHeThermalCpuFanStatus' => {
    '1' => 'other',
    '2' => 'ok',
    '4' => 'failed',
  },
  # 1.3.6.1.4.1.232.6.2.6.8.1.3'
  'cpqHeTemperatureLocale' => {
    '1' => 'other',
    '2' => 'unknown',
    '3' => 'system',
    '4' => 'systemBoard',
    '5' => 'ioBoard',
    '6' => 'cpu',
    '7' => 'memory',
    '8' => 'storage',
    '9' => 'removableMedia',
    '10' => 'powerSupply',
    '11' => 'ambient',
    '12' => 'chassis',
    '13' => 'bridgeCard',
  },
  # 1.3.6.1.4.1.232.6.2.6.8.1.6
  'cpqHeTemperatureCondition' => {
    '1' => 'other',
    '2' => 'ok',
    '3' => 'degraded',
    '4' => 'failed',
  },
  # .1.3.6.1.4.1.232.6.2.11.1
  'cpqHeEventLogSupported' => {
    '1' => 'other',
    '2' => 'notSupported',
    '3' => 'supported',
  },
  # .1.3.6.1.4.1.232.6.2.11.2
  'cpqHeEventLogCondition' => {
    '1' => 'other',
    '2' => 'ok',
    '3' => 'degraded',
    '4' => 'failed',
  },
  # .1.3.6.1.4.1.232.6.2.11.3.1.2
  'cpqHeEventLogEntrySeverity' => {
    '2' => 'informational',
    '3' => 'infoWithAlert',
    '6' => 'repaired',
    '9' => 'caution',
    '15' => 'critical',
  },
  # .1.3.6.1.4.1.318.1.1.1.2.1.1.0
  'upsBasicBatteryStatus' => {
    '1' => 'unknown',
    '2' => 'batteryNormal',
    '3' => 'batteryLow',
  },
  # .1.3.6.1.4.1.318.1.1.1.4.1.1.0
  'upsBasicOutputStatus' => {
    '1' => 'unknown',
    '2' => 'onLine',
    '3' => 'onBattery',
    '4' => 'onSmartBoost',
    '5' => 'timedSleeping',
    '6' => 'softwareBypass',
    '7' => 'off',
    '8' => 'rebooting',
    '9' => 'switchedBypass',
    '10' => 'hardwareFailureBypass',
    '11' => 'sleepingUntilPowerReturn',
    '12' => 'onSmartTrim',
  },
  # .1.3.6.1.4.1.318.1.1.1.2.2.4.0
  'upsAdvBatteryReplaceIndicator' => {
    '1' => 'noBatteryNeedsReplacing',
    '2' => 'batteryNeedsReplacing',
  },
  # .1.3.6.1.4.1.318.1.1.1.3.2.5.0
  'upsAdvInputLineFailCause' => {
    '1' => 'noTransfer',
    '2' => 'highLineVoltage',
    '3' => 'brownout',
    '4' => 'blackout',
    '5' => 'smallMomentarySag',
    '6' => 'deepMomentarySag',
    '7' => 'smallMomentarySpike',
    '8' => 'largeMomentarySpike',
    '9' => 'selfTest',
    # In the MIB file itself the following is spelled as rateOfVoltageChnage
    '10' => 'rateOfVoltageChange',
  },
  # .1.3.6.1.4.1.789.1.2.2.4.0
  'miscGlobalStatus' => {
    '1' => 'other',
    '2' => 'unknown',
    '3' => 'ok',
    '4' => 'nonCritical',
    '5' => 'critical',
    '6' => 'nonRecoverable',
  },
  # .1.3.6.1.4.1.789.1.2.4.1.0
  'envOverTemperature' => {
    '1' => 'no',
    '2' => 'yes',
  },
  # .1.3.6.1.4.1.1991.1.1.1.2.1.1.3
  'snChasPwrSupplyOperStatus' => {
    '1' => 'other',
    '2' => 'normal',
    '3' => 'failure',
  },
  # .1.3.6.1.4.1.1991.1.1.1.3.1.1.3
  'snChasFanOperStatus' => {
    '1' => 'other',
    '2' => 'normal',
    '3' => 'failure',
  },
  # .1.3.6.1.4.1.1991.1.1.3.3.1.1.4
  'snSwPortInfoChnMode' => {
    '0' => 'none',
    '1' => 'half',
    '2' => 'full',
  },
  # .1.3.6.1.4.1.1991.1.1.3.3.1.1.5
  'snSwPortInfoSpeed' => {
    '0' => 'none',
    '1' => 'auto',
    '2' => '10000000',
    '3' => '100000000',
    '4' => '1000000000',
    '5' => '45000000',
    '6' => '155000000',
    '7' => '10000000000',
  },
  # .1.3.6.1.4.1.1991.1.1.4.2.1.1.4
  'snL4VirtualServerAdminStatus' => {
     '0' => 'disabled',
     '1' => 'enabled',
  },
  # .1.3.6.1.4.1.1991.1.1.4.3.1.1.4
  'snL4RealServerAdminStatus' => {
     '0' => 'disabled',
     '1' => 'enabled',
  },
  # .1.3.6.1.4.1.1991.1.1.4.4.1.1.4
  'snL4VirtualServerPortAdminStatus' => {
     '0' => 'disabled',
     '1' => 'enabled',
  },
  # .1.3.6.1.4.1.1991.1.1.4.5.1.1.4
  'snL4RealServerPortAdminStatus' => {
     '0' => 'disabled',
     '1' => 'enabled',
  },
  # .1.3.6.1.4.1.1991.1.1.4.8.1.1.9
  'snL4RealServerStatusState' => {
     '0' => 'serverdisabled',
     '1' => 'serverenabled',
     '2' => 'serverfailed',
     '3' => 'servertesting',
     '4' => 'serversuspect',
     '5' => 'servershutdown',
     '6' => 'serveractive',
  },
  # .1.3.6.1.4.1.1991.1.1.4.10.1.1.5
  'snL4RealServerPortStatusState' => {
     '0' => 'disabled',
     '1' => 'enabled',
     '2' => 'failed',
     '3' => 'testing',
     '4' => 'suspect',
     '5' => 'shutdown',
     '6' => 'active',
     '7' => 'unbound',
     '8' => 'awaitUnbind',
     '9' => 'awaitDelete',
  },
  # .1.3.6.1.4.1.1991.1.2.11.6.1.1.16
  'snBgp4NeighGenCfgShutdown' => {
    '0' => 'disabled',
    '1' => 'enabled',
  },
  # .1.3.6.1.4.1.1991.1.2.11.15.1.1.4
  'snBgp4NeighOperStatusBgpType' => {
    '0' => 'ebgp',
    '1' => 'ibgp',
  },
  # .1.3.6.1.4.1.1991.1.2.11.15.1.1.5
  'snBgp4NeighOperStatusState' => {
    '0' => 'noState',
    '1' => 'idle',
    '2' => 'connect',
    '3' => 'active',
    '4' => 'openSent',
    '5' => 'openConfirm',
    '6' => 'established',
  },
  # .1.3.6.1.4.1.1991.1.2.11.17.1.1.3
  'snBgp4NeighborSummaryState' => {
    '0' => 'noState',
    '1' => 'idle',
    '2' => 'connect',
    '3' => 'active',
    '4' => 'openSent',
    '5' => 'openConfirm',
    '6' => 'established',
  },
);

my $z = 0;
my %cpustatusorder = (
    'cpu' => $z++,
    'cpu1sec' => $z++,
    'cpu5sec' => $z++,
    'cpu1min' => $z++,
    'cpu5min' => $z++,
    'cpu15min' => $z++,
    'la1min' => $z++,
    'la5min' => $z++,
    'la15min' => $z++,
    'cpuuser' => $z++,
    'cpusystem' => $z++,
    'cpubusy' => $z++,
    'cpuidle' => $z++,
);

my %cpustatusnames = (
    'cpu' => 'CPU Usage',
    'cpu1sec' => 'CPU Usage, 1 Second Average',
    'cpu5sec' => 'CPU Usage, 5 Second Average',
    'cpu1min' => 'CPU Usage, 1 Minute Average',
    'cpu5min' => 'CPU Usage, 5 Minute Average',
    'cpu15min' => 'CPU Usage, 15 Minute Average',
    'la1min' => 'Load Average, 1 Minute Average',
    'la5min' => 'Load Average, 5 Minute Average',
    'la15min' => 'Load Average, 15 Minute Average',
    'cpuuser' => 'CPU Time in User Mode',
    'cpusystem' => 'CPU Time in System Mode',
    'cpubusy' => 'CPU Time in Busy Mode',
    'cpuidle' => 'CPU Time in Idle Mode',
);


my $cputhreshdb_ref = undef;
my $cpu_defyellow = 80; # percent
my $cpu_defred = 90; # percent
my $la_defyellow = exists($ENV{'CPUWARN'}) ? $ENV{'CPUWARN'} : 150; # Load average x 100
   $la_defyellow = sprintf("0.2%f",$la_defyellow / 100);
my $la_defred = exists($ENV{'CPUPANIC'}) ? $ENV{'CPUPANIC'} : 300; # Load average x 100
   $la_defred = sprintf("0.2%f",$la_defred / 100);

my $show_cisco_top = 0; # Show similated Cisco top in CPU output

# For Compaq temperature checks
my $compaq_pct_temp_warn = 80;
my $compaq_pct_temp_panic = 90;

# For disk checks
my $diskthreshdb_ref = undef;
my $disk_defyellow = exists($ENV{'DFWARN'}) ? $ENV{'DFWARN'} : 90;
my $disk_defred    = exists($ENV{'DFWARN'}) ? $ENV{'DFWARN'} : 90;

# Turn on or off depending on whether or not you have more servers defined
# than BB can display in one page. TODO: Automate this decision.
# bindtree is the minimum necessary.
my $l4_show_realservers = 0;
my $l4_show_virtservers = 0;
my $l4_show_bindtree = 1;
my $l4_bbhosts_ref = undef; # Database of bb-hosts contents for L4 switch monitor-by-proxy.
                            # Uses the L4 real server states as test results for tests that
                            # BigBrother is not already performing itself.


&main();
exit;


#--------------------
# Prevent stupid typos with cruel autoload!
#--------------------

sub AUTOLOAD {
  use vars qw($AUTOLOAD);
  warn("package ".(caller(1))[0].", file ".(caller(1))[1].", line ".(caller(1))[2]);
  warn(", subname ".(caller(1))[3].", called the following function which doesn't exist: $AUTOLOAD\n");
  exit;
}

sub main {
  if ($ARGV[0] eq '-v') {
    die("$Bin/$Script: $VERSION\n");
  }
  print("Starting...\n") if $debug;

  print("Environment checking\n") if $debug;
  unless (&env_check()) {
    print(localtime().": ".(caller(0))[3].": Environment check failed\n");
    exit 1;
  }

  # Read in bb-xsnmptab
  print("Reading bb-xsnmptab\n") if $debug;
  my $commdb_ref = undef;
  my $testdb_ref = undef;
  unless (($commdb_ref,$testdb_ref) = &read_xsnmptab()) {
    print(localtime().": ".(caller(0))[3].": Unable to get community and test list.\n");
    exit 1;
  }

  # Read in bb-cputab.
  print("Reading bb-cputab\n") if $debug;
  $cputhreshdb_ref = &read_cpu_tab(keys(%$commdb_ref));
  unless (defined($cputhreshdb_ref)) {
    print(localtime().": ".(caller(0))[3].": Unable to read bb-cputab.\n");
    exit 1;
  }

  # Read in bb-dftab
  print("Reading bb-dftab\n") if $debug;
  $diskthreshdb_ref = &read_df_tab(keys(%$commdb_ref));
  unless (defined($diskthreshdb_ref)) {
    print(localtime().": ".(caller(0))[3].": Unable to read bb-dftab.\n");
    exit 1;
  }

  # Read bb-hosts for L4 switch monitor-by-proxy
  print("Reading bb-hosts\n") if $debug;
  unless ($l4_bbhosts_ref = &l4_read_bbhosts()) {
    print(localtime().": ".(caller(0))[3].": Unable to read bb-hosts.\n");
    exit 1;
  }

  # Initialize bb-combo message
  $ENV{'BBCOMBOID'} = $$;
  if ($is_commandline) {
    print("$ENV{'BBHOME'}/bin/bb-combo.sh start\n");
  } else {
    system("$ENV{'BBHOME'}/bin/bb-combo.sh start");
  }

  while (my($host,$community) = each(%$commdb_ref)) {
    # Make only one session
    my($snmpsession,$snmperror) = Net::SNMP->session(
      -hostname => $host,
      -community => $community,
      -debug => $debug,
      -retries => $defretries,
      -timeout => $deftimeout,
      -translate => [ -timeticks => 0x0 ],
    ); # Net::SNMP->session

    if (! defined($snmpsession)) {
      print((caller(0))[3].": SNMP session failed for '$host': $snmperror\n");
      next;
    }

    # Determine properties used in all tests

    # Send a query to determine what brand the machine is.
    print("  Checking enterprise and uptime for $host\n") if $debug;
    my $objectid = undef;
    my $enterprise = undef;
    my $uptime = undef;

    if (my $result = $snmpsession->get_request(-varbindlist =>
                       [$snmpoids{'sysObjectID'},$snmpoids{'sysUpTime'}])) {
      $uptime = $result->{$snmpoids{'sysUpTime'}};
      $objectid = $result->{$snmpoids{'sysObjectID'}};
      (my $brandnum = substr($objectid,length($snmpoids{'enterprises'})+1)) =~ s/\..*$//;
      if (exists($enterprisenumbers{$brandnum})) {
        $enterprise = $enterprisenumbers{$brandnum};
      } else {
        print("Unknown enterprise number '$brandnum' in ObjectID '$objectid'\n");
        next;
      }
    } else {
      print((caller(0))[3].": SNMP get request failed for '$host': ".$snmpsession->error()."\n");
      return;
    }

    # Determine what OS version the machine is.
    my $version = undef;
    unless ($version = &detect_version($host,$snmpsession,$enterprise)) {
      print("Unable to detect version for host '$host' enterprise '$enterprise'\n");
      next;
    }
    print("    version $version\n") if $debug;

    # Run individual tests
    while (my($test) = each(%{$testdb_ref->{$host}})) {
      # Harmless skips
      if ($test eq 'l4') {
        next unless &is_l4switch($host,$snmpsession);
      }
      my $message = '';

      # Run the appropriate test/determine colors
      unless ($message = &create_message($host,$snmpsession,$test,$enterprise,$version,$uptime)) {
        print("Unable to create message for host '$host' enterprise '$enterprise' version '$version' test '$test'");
        next;
      }

      $message .= "\n<!-- Enterprise: $enterprise , Version: $version -->\n";
      # Add version credit
      $message .= "$Script Version: $VERSION\n";

      # Add to combo message
      if ($is_commandline) {
        print("$ENV{'BBHOME'}/bin/bb-combo.sh add \"$message\"\n");
      } else {
        system("$ENV{'BBHOME'}/bin/bb-combo.sh add \"$message\"");
      }
    } # while (my($test) = each(%{$testdb_ref->{$host}}))
  } # while (my($host,$community) = each(%$commdb_ref))

  # Send combo message
  if ($is_commandline) {
    print("$ENV{'BBHOME'}/bin/bb-combo.sh end\n");
    # Print time elapsed since the start of this extension
    printf("%d seconds elapsed\n",time() - $^T);
  } else {
    system("$ENV{'BBHOME'}/bin/bb-combo.sh end");
  }

  return 1;
} # sub main


##### TESTS #####

#----
# $message = &create_message($host,$snmpsession,$test,$enterprise,$version,$uptime)
#
# Create a message for transmission to BigBrother
#----

sub create_message {
  my($host,$snmpsession,$test,$enterprise,$version,$uptime) = @_;

  if ($test eq 'bgp') {
    return &bgp_message($host,$snmpsession,$enterprise,$version);
  } elsif ($test eq 'cpu') {
    return &cpu_message($host,$snmpsession,$enterprise,$version,$uptime);
  } elsif ($test eq 'disk') {
    return &disk_message($host,$snmpsession,$enterprise,$version);
  } elsif ($test eq 'fans') {
    return &fans_message($host,$snmpsession,$enterprise,$version);
  } elsif ($test eq 'l4') {
    return &l4_message($host,$snmpsession,$enterprise,$version);
  } elsif ($test eq 'memory') {
    return &memory_message($host,$snmpsession,$enterprise,$version);
  } elsif ($test eq 'ports') {
    return &ports_message($host,$snmpsession,$enterprise,$version);
  } elsif ($test eq 'power') {
    return &power_message($host,$snmpsession,$enterprise,$version);
  } elsif ($test eq 'temperature') {
    return &temperature_message($host,$snmpsession,$enterprise,$version);
  } elsif ($test eq 'uptime') {
    return &uptime_message($host,$snmpsession,$enterprise,$version,$uptime);
  } else {
    print("Unknown test '$test'\n");
    return;
  }

  return;
} # sub create_message


##### BGP #####

#----
# &bgp_message($host,$snmpsession,$enterprise,$version)
#
# Create a BigBrother message about the BGP state of a host
#----

sub bgp_message {
  my($host,$snmpsession,$enterprise,$version) = @_;
  (my $commahost = $host) =~ s/\./\,/g;
  my $test = 'bgp';
  my $message = '';

  # Cisco uses generic mib-2 BGP MIB but Foundry does not.
  # Should be able to check whether or not I can use the standard MIB.
  my $is_standard = 0;
  my $localas = undef;

  unless (($enterprise eq 'cisco') || ($enterprise eq 'foundry')) {
    # See whether or not I have the standard MIB-2 bits
    if (my $result = $snmpsession->get_request(-varbindlist => [$snmpoids{'bgpLocalAs'}])) {
      if (exists($result->{$snmpoids{'bgpLocalAs'}})) {
        $is_standard = 1;
        $localas = $result->{$snmpoids{'bgpLocalAs'}};
      }
    }
  } # unless (($enterprise eq 'cisco') || ($enterprise eq 'foundry'))

  my %asdb = ();
  my %statedb = ();
  my %timedb = ();
  my %shutdowndb = (); # Only used by Cisco for now
  my %descrdb = (); # Only used by Foundry for now

  if (($enterprise eq 'cisco') || ($is_standard)) {
    unless (defined($localas)) {
     if (my $result = $snmpsession->get_request(-varbindlist => [$snmpoids{'bgpLocalAs'}])) {
       $localas = $result->{$snmpoids{'bgpLocalAs'}};
     } else {
        print(localtime().": ".(caller(0))[3].": Unable to get local AS for $host: ".$snmpsession->error()."\n");
        return;
      }
    }

    if (my $result = $snmpsession->get_table(-baseoid => $snmpoids{'bgpPeerRemoteAs'})) {
      while (my($oid,$asnum) = each(%$result)) {
        $asdb{substr($oid,1+length($snmpoids{'bgpPeerRemoteAs'}))} = $asnum;
      }
    } else {
      print(localtime().": ".(caller(0))[3].": Unable to get remote ASs for $host: ".$snmpsession->error()."\n");
      return;
    }

    if (my $result = $snmpsession->get_table(-baseoid => $snmpoids{'bgpPeerState'})) {
      while (my($oid,$statenum) = each(%$result)) {
        $statedb{substr($oid,1+length($snmpoids{'bgpPeerState'}))} = $snmpsyntaxdb{'bgpPeerState'}{$statenum};
      }
    } else {
      print(localtime().": ".(caller(0))[3].": Unable to get peer states for $host: ".$snmpsession->error()."\n");
      return;
    }

    if (my $result = $snmpsession->get_table(-baseoid => $snmpoids{'bgpPeerAdminStatus'})) {
      while (my($oidnum,$statenum) = each(%$result)) {
        $shutdowndb{substr($oidnum,1+length($snmpoids{'bgpPeerAdminStatus'}))} =
                                 $snmpsyntaxdb{'bgpPeerAdminStatus'}{$statenum};
      }
    } else {
      print(localtime().": ".(caller(0))[3].": Unable to get peer admin statuses for $host: ".$snmpsession->error()."\n");
      return;
    }

    if (my $result = $snmpsession->get_table(-baseoid =>
                                             $snmpoids{'bgpPeerFsmEstablishedTime'})) {
      while (my($oid,$seconds) = each(%$result)) {
        $timedb{substr($oid,1+length($snmpoids{'bgpPeerFsmEstablishedTime'}))} = $seconds;
      }
    } else {
      print(localtime().": ".(caller(0))[3].": Unable to get peer established times for $host: ".$snmpsession->error()."\n");
      return;
    }

  } elsif ($enterprise eq 'foundry') {
    unless (defined($localas)) {
      # "Useless use of private variable in void context at bb-xsnmp.pl line 904."
      # Unless I use 'my' here.
      if (my $result => $snmpsession->get_request(-varbindlist =>
                                                  [$snmpoids{'snBgp4GenLocalAs'}])) {
        $localas = $result->{$snmpoids{'snBgp4GenLocalAs'}};
      } else {
        print(localtime().": ".(caller(0))[3].": Unable to get local AS for $host: ".$snmpsession->error()."\n");
        return;
      }
    }

    # Shutdown state, IP works as index.
    # This doesn't mean what I thought it did though. Disabling for Foundry.
    # Doesn't seem to be a way to get BGP peer admin state for Foundry.
#    if (my $result = $snmpsession->get_table(-baseoid => $snmpoids{'snBgp4NeighGenCfgShutdown'})) {
#      while (my($oidnum,$statenum) = each(%$result)) {
#        $shutdowndb{substr($oidnum,1+length($snmpoids{'snBgp4NeighGenCfgShutdown'}))} =
#                                 $snmpsyntaxdb{'snBgp4NeighGenCfgShutdown'}{$statenum};
#      }
#    } else {
#      print(localtime().": ".(caller(0))[3].": Unable to get shutdown states for $host: ".$snmpsession->error()."\n");
#      return;
#    }

    # Neighbor description
    if (my $result = $snmpsession->get_table(-baseoid => $snmpoids{'snBgp4NeighGenCfgDesc'})) {
      while (my($oidnum,$descr) = each(%$result)) {
        $descrdb{substr($oidnum,1+length($snmpoids{'snBgp4NeighGenCfgDesc'}))} = $descr;
      }
    } else {
      print(localtime().": ".(caller(0))[3].": Unable to get neighbor descriptions for $host: ".$snmpsession->error()."\n");
      return;
    }

    # Remote AS
    if (my $result = $snmpsession->get_table(-baseoid => $snmpoids{'snBgp4NeighGenCfgRemoteAs'})) {
      while (my($oidnum,$remoteas) = each(%$result)) {
        $asdb{substr($oidnum,1+length($snmpoids{'snBgp4NeighGenCfgRemoteAs'}))} = $remoteas;
      }
    } else {
      print(localtime().": ".(caller(0))[3].": Unable to get neighbor remote ASs for $host: ".$snmpsession->error()."\n");
      return;
    }

    # Need index -> remote ip, state, and time
    my %index2ip = ();
    if (my $result = $snmpsession->get_table(-baseoid =>
                                             $snmpoids{'snBgp4NeighborSummaryIp'})) {
      while (my($oidnum,$ipaddr) = each(%$result)) {
        (my $index = $oidnum) =~ s/^.*\.//;
        $index2ip{$index} = $ipaddr;
      }
    } else {
      print(localtime().": ".(caller(0))[3].": Unable to get neighbor IP index for $host: ".$snmpsession->error()."\n");
      return;
    }

    # Remote IP
    if (my $result = $snmpsession->get_table(-baseoid =>
                                             $snmpoids{'snBgp4NeighborSummaryIp'})) {
      while (my($oidnum,$ipaddr) = each(%$result)) {
        (my $index = $oidnum) =~ s/^.*\.//;
        $index2ip{$index} = $ipaddr;
      }
    } else {
      print(localtime().": ".(caller(0))[3].": Unable to get neighbor IP index for $host: ".$snmpsession->error()."\n");
      return;
    }

    # Remote State
    if (my $result = $snmpsession->get_table(-baseoid =>
                                             $snmpoids{'snBgp4NeighborSummaryState'})) {
      while (my($oidnum,$statenum) = each(%$result)) {
        (my $index = $oidnum) =~ s/^.*\.//;
        my $ipaddr = $index2ip{$index};
        my $state = $snmpsyntaxdb{'snBgp4NeighborSummaryState'}{$statenum};
        $statedb{$ipaddr} = $state;
      }
    } else {
      print(localtime().": ".(caller(0))[3].": Unable to get neighbor states for $host: ".$snmpsession->error()."\n");
      return;
    }

    # Time since last change
    if (my $result = $snmpsession->get_table(-baseoid =>
                                         $snmpoids{'snBgp4NeighborSummaryStateChgTime'})) {
      while (my($oidnum,$timeup) = each(%$result)) {
        (my $index = $oidnum) =~ s/^.*\.//;
        my $ipaddr = $index2ip{$index};
        $timedb{$ipaddr} = $timeup;
      }
    } else {
      print(localtime().": ".(caller(0))[3].": Unable to get neighbor state change times for $host: ".$snmpsession->error()."\n");
      return;
    }

  } else {
    print(localtime().": ".(caller(0))[3].": Unknown enterprise '$enterprise' for host '$host'\n");
    return;
  }

  # Look through states and find a color for each peer and an overall color.
  # Include description and shutdown if they exist.
  # 'One line' IP address sort! Woohoo!
  my @peerlist = sort {
    unless ((split(/\./,$a))[0] <=> (split(/\./,$b))[0]) {
      unless (((split(/\./,$a))[1] <=> (split(/\./,$b))[1])) {
        unless (((split(/\./,$a))[2] <=> (split(/\./,$b))[2])) {
          (split(/\./,$a))[3] <=> (split(/\./,$b))[3]
        }
      }
    }
  } keys(%statedb);

  my $routercolor = 'green';
  # Turn $routercolor red if all external peers are down.
  # Turn $routercolor yellow if any internal peers are down
  # or if any external peers are down for longer than fifteen minutes.
  my %colordb = ();
  my $has_externalup = 0; # Set to 1 if any external BGP peers are up.
  foreach my $ipaddr (@peerlist) {
    my $state = $statedb{$ipaddr};
    my $remoteas = $asdb{$ipaddr};
    my $statetime = $timedb{$ipaddr};
    my $descr = exists($descrdb{$ipaddr}) ? $descrdb{$ipaddr} : undef;
    my $shutdown = exists($shutdowndb{$ipaddr}) ? $shutdowndb{$ipaddr} : undef;
    if ($state eq 'established') {
      $colordb{$ipaddr} = 'green';
      $has_externalup = 1;
    } elsif ((defined($shutdown)) && ($shutdown eq 'stop')) {
      $colordb{$ipaddr} = 'clear';
    # } elsif (($state eq 'active') && ($statetime == 0)) {
    } elsif ($statetime == 0) {
      # Means it's never been up
      $colordb{$ipaddr} = 'clear';
    } elsif ($remoteas eq $localas) {
      # Bad news if an IBGP session is down
      $colordb{$ipaddr} = 'red';
      $routercolor = 'red';
    } elsif ($statetime < 900) {
      # Allow 15 minutes of downtime
      $colordb{$ipaddr} = 'yellow';
    } else {
      # More than that, something is wrong
      $colordb{$ipaddr} = 'red';
      $routercolor = ($routercolor eq 'green') ? 'yellow' : $routercolor;
    }

    if (! $has_externalup) {
      $routercolor = 'red';
    }

    $message .= '&'.$colordb{$ipaddr}.'  '.$ipaddr.'   AS '.$remoteas;
    if ((defined($descr)) && (length($descr) > 0)) {
      $message .= "  '$descr'";
    }
    $message .= '  '.$state;

    my $secs = $statetime;
    my $weeks = int($secs / 604800);
    $secs = $secs - $weeks * 604800;
    my $days = int($secs / 86400);
    $secs = $secs - $days * 86400;
    my $hours = int($secs / 3600);
    $secs = $secs - $hours * 3600;
    my $minutes = int($secs / 60);
    $secs = $secs - $minutes * 60;
    my $timestr = '';
    if ($weeks >= 2) {
      $timestr .= "$weeks weeks, ";
    } elsif ($weeks == 1) {
      $timestr .= "1 week, ";
    }

    if ($days >= 2) {
      $timestr .= "$days days, ";
    } elsif ($days == 1) {
      $timestr .= "1 day, ";
    }
    $timestr .= sprintf("%02d:%02d:%02d",$hours,$minutes,$secs);
    $message .= '  '.$timestr;
    $message .= "\n";
  }

  $message = "status $commahost.$test $routercolor ".scalar(localtime)."\n\n$message";
 
  return $message;  
} # sub bgp_message


##### CPU #####

#----
# &cpu_message($host,$snmpsession,$enterprise,$version,$uptime);
#
# Check CPU status and compose into a message
#----

sub cpu_message {
  my($host,$snmpsession,$enterprise,$version,$uptime) = @_;

  print(localtime().": ".(caller(0))[3].": Checking host '$host'\n") if $debug;
  if (! defined($uptime)) {
    print(localtime().": ".(caller(0))[3].": Uptme for '$host' is undefined\n");
    return;
  }
  if ($uptime == 0) {
    print(localtime().": ".(caller(0))[3].": Uptme for '$host' is zero\n");
    return;
  }

  # Find one or more of the following:
  # cpu - All Foundry devices have this. Newer ones have the time-based
  #       averages that should be used instead.
  #       This has been deprecated in new Foundries and should be ignored when
  #       the others are available. Docs for 07.6.04 say reading it resets all
  #       the CPU counters.
  # cpuNsec - Foundry routers
  # cpu1min - Many routers
  # cpu5min - Many routers
  # cpu15min - Many routers
  # la1min - UCD
  # la5min - UCD
  # la10min - UCD-SNMP mib files say this. Not used.
  # la15min - UCD-SNMP config files and snmpwalk say this. Using this.
  # UCD-SNMP also has the following which are also available through vmstat:
  # cpuuser
  # cpusystem
  # cpuidle

  my $cpudb_ref = undef;
  unless ($cpudb_ref = &get_cpu_status($host,$snmpsession,$enterprise,$version)) {
    print(localtime().": ".(caller(0))[3].": Unable to get CPU status for host '$host' enterprise '$enterprise' version $version\n");
    next;
  }

  # Check uptime, hundredths of seconds
  my $uptimecolor = 'green';
  if ((($uptime / 100) / 60) < $ENV{'WARNMINSONREBOOT'}) {
    $uptimecolor = $ENV{'WARNCOLORONREBOOT'};
  }

  # Determine colors
  my $colordb_ref = &check_cpu_colorstatus($host,$enterprise,$version,$cpudb_ref,$cputhreshdb_ref->{$host});
  unless (defined($colordb_ref)) {
    print("Unable to determine colors for host '$host' enterprise '$enterprise' version '$version'\n");
    next;
  }

  # Find worst color
  my @colorlist = ($uptimecolor);
  if (exists($colordb_ref->{'cpu5min'})) { # ciscos
    push(@colorlist,$colordb_ref->{'cpu5min'});
  } elsif (exists($colordb_ref->{'cpu1min'})) { # foundries
    push(@colorlist,$colordb_ref->{'cpu1min'});
  } elsif (exists($colordb_ref->{'cpu'})) { # old foundries
    push(@colorlist,$colordb_ref->{'cpu'});
  } elsif (exists($colordb_ref->{'la5min'})) { # ucd-snmp
    push(@colorlist,$colordb_ref->{'la5min'});
  } elsif (exists($colordb_ref->{'cpubusy'})) { # netapps
    push(@colorlist,$colordb_ref->{'cpubusy'});
  }
  my $worstcolor = &color_compare(@colorlist);

  # Create message
  my $message = '';
  my $test = 'cpu';

  (my $commahost = $host) =~ s/\./\,/g;
  my $upword = '';
  if (($uptime / 100) < 120) {
    $upword = sprintf("%3.2f seconds",$uptime / 100);
  } elsif ((($uptime / 100) / 60) < 120) {
    $upword = sprintf("%3.2f minutes",(($uptime / 100) / 60));
  } elsif (((($uptime / 100) / 60 ) / 60) < 48) {
    $upword = sprintf("%3.2f hours",((($uptime / 100) / 60) / 60));
  } else {
    $upword = sprintf("%3.2f days",(((($uptime / 100) / 60) / 60) / 24));
  }

  $message = "status $commahost.$test $worstcolor ".scalar(localtime)." up: $upword";
  if (exists($cpudb_ref->{'la5min'})) {
    $message .= sprintf(", load=%0.2f",$cpudb_ref->{'la5min'});
  } elsif (exists($cpudb_ref->{'cpu5min'})) {
    $message .= sprintf(", CPU Usage=%3.0f\%",$cpudb_ref->{'cpu5min'});
  } elsif (exists($cpudb_ref->{'cpubusy'})) {
    $message .= sprintf(", CPU Usage=%3.0f\%",$cpudb_ref->{'cpubusy'});
  } elsif (exists($cpudb_ref->{'cpu'})) {
    $message .= sprintf(", CPU Usage=%3.0f\%",$cpudb_ref->{'cpu'});
  }
  $message .= "\n\n";
  if ((($uptime / 100) / 60) < $ENV{'WARNMINSONREBOOT'}) {
    $message .= "Warning: Machine recently rebooted\n\n";
  }

  my @cpuparams = sort { $cpustatusorder{$a} <=> $cpustatusorder{$b} } keys(%$cpudb_ref);
  my $has_la = 0;
  my $has_cpu = 0;
  foreach my $cpu (@cpuparams) {
    $message .= '&'.$colordb_ref->{$cpu};
    $message .= '  ';
    $message .= "$cpustatusnames{$cpu}: ";
    if ($cpu =~ /^la/) {
      # Assume load average
      $has_la = 1;
      $message .= sprintf("%0.2f",$cpudb_ref->{$cpu});
    } elsif ($cpu =~ /^cpu/) {
      # Assume percentage
      $has_cpu = 1;
      $message .= sprintf("%3.0f\%",$cpudb_ref->{$cpu});
    } else {
      print("Unknown parameter for host '$host': $cpu\n");
    }
    $message .= "\n";
  } # foreach my $cpu (@cpuparams)

  $message .= "\n";
  if ($has_cpu) {
    $message .= "&yellow CPU Usage Threshold: ".$cputhreshdb_ref->{$host}{'cpu'}{'yellow'}."\%\n";
    $message .= "&red CPU Usage Threshold: ".$cputhreshdb_ref->{$host}{'cpu'}{'red'}."\%\n";
  }
  if ($has_la) {
    $message .= sprintf("&yellow Load Average Threshold: %0.2f\n",$cputhreshdb_ref->{$host}{'la'}{'yellow'});
    $message .= sprintf("&red Load Average Threshold: %0.2f\n",$cputhreshdb_ref->{$host}{'la'}{'red'});
  }

  if (($enterprise eq 'cisco') && ($show_cisco_top)) {
    # Try to build a top-like listing of active processes,
    # much like the origial SNMP2BB extension.
    # Walk cpmProcessEntry
    # Skip peacefully if anything vital is missing. Old ciscos don't even have cpmProcessEntry.
    my %procpridb = ();
    my %procusecdb = ();
    my %procavgusecdb = ();
    my %procnamedb = ();
    if ((my $result = $snmpsession->get_table(-baseoid => $snmpoids{'cpmProcessEntry'})) &&
        (my $priresult = $snmpsession->get_table(-baseoid => $snmpoids{'cpmProcExtPriority'}))) {
      $message .= "</pre>\n";
      my $qprocname = quotemeta($snmpoids{'cpmProcessName'});
      my $qprocusec = quotemeta($snmpoids{'cpmProcessuSecs'});
      my $qprocavgusec = quotemeta($snmpoids{'cpmProcessAverageUSecs'});
      my $qprocpri = quotemeta($snmpoids{'cpmProcExtPriority'});
      while (my($key,$value) = each(%$result)) {
        if ($key =~ /^$qprocname\.1\.(\d+)/) {
          my $pid = $1;
          $procnamedb{$pid} = $value;
        } elsif ($key =~ /^$qprocusec\.1\.(\d+)/) {
          my $pid = $1;
          $procusecdb{$pid} = $value;
        } elsif ($key =~ /^$qprocavgusec\.1\.(\d+)/) {
          my $pid = $1;
          $procavgusecdb{$pid} = $value;
        } # if ($key =~ /^$qprocname\.1\.(\d+)/)
      } # while (my($key,$value) = each(%$result))
      while (my($key,$value) = each(%$priresult)) {
        if ($key =~ /^$qprocpri\.1\.(\d+)/) {
          my $pid = $1;
          unless (exists($snmpsyntaxdb{'cpmProcExtPriority'}{$value})) {
            print("Cisco Host $host Value $value corresponds to no known cpmProcExtPriority\n");
            return;
          }
          unless (defined($snmpsyntaxdb{'cpmProcExtPriority'}{$value})) {
            print("Cisco Host $host Value $value corresponds to no defined cpmProcExtPriority\n");
            return;
          }
          $procpridb{$pid} = $snmpsyntaxdb{'cpmProcExtPriority'}{$value};
        }
      } # while (my($key,$value) = each(%$priresults))

      # Filter out the inactive processes.
      # use cpmProcessuSecs if needed, cpmProcessAverageUSecs not always present
      while (my($pid,$pname) = each(%procnamedb)) {
        if (exists($procavgusecdb{$pid})) {
          if ($procavgusecdb{$pid} == 0) {
            delete($procnamedb{$pid});
          }
        } elsif (exists($procusecdb{$pid})) {
          if ($procusecdb{$pid} == 0) {
            delete($procnamedb{$pid});
          } # if ($procusecdb{$pid} == 0)
        } # if (exists($procavgusecdb{$pid}))
      } # while (my($pid,$pname) = each(%procnamedb))
 
      # Sort by cpmProcessAverageUSecs/cpmProcessuSecs, descending order
      my @pidlist = ();
      if (scalar(keys(%procavgusecdb)) <= 0) {
        @pidlist = sort { $procusecdb{$b} <=> $procusecdb{$a} } keys(%procnamedb);
      } else {
        @pidlist = sort { ($procavgusecdb{$b} <=> $procavgusecdb{$a}) || ($procusecdb{$b} <=> $procusecdb{$a}) } keys(%procnamedb);
      }
 
      # Print table in the following form:
      # PID     PRI             Time    Name
      $message .= "<!-- Priority keys: ".(scalar(keys(%procpridb)))." -->";
      $message .= "<!-- Priority key results: ".(scalar(keys(%$priresult)))." -->";
      my($samplekey,$samplevalue) = each(%$priresult);
      $message .= "<!-- Sample key results: ".$samplekey." : ".$samplevalue." -->";
      $message .= "<table border=1>\n";
      $message .= "<tr><th>PID</th><th>PRI</th><th>Time(usec)</th><th>Name</th></tr>\n";
      foreach my $pid (@pidlist) {
        my $pname = $procnamedb{$pid};
        my $ppri = (exists($procpridb{$pid})) ? $procpridb{$pid} : 'n/a';
        my $ptime = (exists($procavgusecdb{$pid})) ? $procavgusecdb{$pid} : $procusecdb{$pid};
        $message .= "<tr><td>$pid</td><td>$ppri</td><td>$ptime</td><td>$pname</td></tr>\n";
      }
      $message .= "</table>\n";
      $message .= "<pre>\n";
    } else {
      # The cisco in question is too old to present the requested MIB
      # print("Unable to get process names for Cisco $host: ".$snmpsession->error()."\n");
      # return;
    }

  } # if ($enterprise eq 'cisco') and if I really want to print a top-like table

  return $message;
} # sub cpu_message



#----
# &read_cpu_tab(@hostlist)
#
# Read in the bb-cputab.
# Require a list of wanted hosts.
# Ignore the 'localhost' option in bb-cputab since I'm
# looking at someone else's CPU.
#   $cputhreshdb_ref->{$host}{'la'}{'red'} = '2.00'; # Forget the *100 business
#   $cputhreshdb_ref->{$host}{'cpu'}{'yellow'} = '60';
#----

sub read_cpu_tab {
  my(@hostlist) = @_;
  my %hostdb = map { $_ => 1 } @hostlist;

  my %threshdb = ();
  unless (-f "$ENV{'BBHOME'}/etc/bb-cputab") {
    # bb-cputab doesn't exist, which is the default installation and not
    # necessarily an error
    return \%threshdb;
  }

  my $infh = new FileHandle;
  if ($infh->open("< $ENV{'BBHOME'}/etc/bb-cputab")) {
    while (my $line = $infh->getline()) {
      chomp($line);
      $line =~ s/\#.*$//g;
      $line =~ s/\s+$//g;
      next if $line =~ /^\s*\#/;
      next if $line !~ /\S/;
      if ($line =~ /^\s*(\S+)\s*\:[^\:]*\:\s*([^\:]+)\s*\:\s*([^\:]+)$/) {
        my($host,$yellowlimits,$redlimits) = ($1,$2,$3);
        unless (exists($hostdb{$host})) {
          next;
        } # unless (exists($hostdb{$host}))
        my @yellows = split(' ',$yellowlimits);
        my @reds = split(' ',$redlimits);

        foreach my $yellow (@yellows) {
          if ($yellow =~ /^([\d\.]+)\%$/) {
            # assume cpu percentage
            $threshdb{$host}{'cpu'}{'yellow'} = $1;
          } elsif ($yellow =~ /^(\d+\.\d\d)$/) {
            # assume dotted load average
            $threshdb{$host}{'la'}{'yellow'} = $1;
          } elsif ($yellow =~ /^(\d+)$/) {
            # assume load average x 100
            $threshdb{$host}{'la'}{'yellow'} = sprintf("%0.2f",$1 / 100);
          } else {
            # unknown
            print("Odd yellow threshold for host '$host' in bb-cputab:'$yellow'\n");
          } # if ($yellow =~ /^([\d\.]+)\%$/)
        } # foreach my $yellow (@yellows)

        foreach my $red (@reds) {
          if ($red =~ /^([\d\.]+)\%$/) {
            # assume cpu percentage
            $threshdb{$host}{'cpu'}{'red'} = $1;
          } elsif ($red =~ /^(\d+\.\d\d)$/) {
            # assume dotted load average
            $threshdb{$host}{'la'}{'red'} = $1;
          } elsif ($red =~ /^(\d+)$/) {
            # assume load average x 100
            $threshdb{$host}{'la'}{'red'} = sprintf("%0.2f",$1 / 100);
          } else {
            # unknown
            print("Odd red threshold for host '$host' in bb-cputab:'$red'\n");
          } # if ($red =~ /^([\d\.]+)\%$/)
        } # foreach my $yellow (@yellows)

      } else {
        print("Odd line in bb-cputab:'$line'\n");
      } # if ($line =~ /^\s*(\S+)\s*\:[^\:]*\:\s*(\d+)\s*\:\s*(\d+)\s*$/)
    } # while (my $line = $infh->getline())
    $infh->close;
  } else {
    print("Unable to open bb-cputab: $!\n");
    return;
  }

  # Fill in the blanks
  foreach my $host (@hostlist) {
    unless (exists($threshdb{$host}{'la'}{'yellow'})) {
      $threshdb{$host}{'la'}{'yellow'} = $la_defyellow;
    }
    unless (exists($threshdb{$host}{'la'}{'red'})) {
      $threshdb{$host}{'la'}{'red'} = $la_defred;
    }
    unless (exists($threshdb{$host}{'cpu'}{'yellow'})) {
      $threshdb{$host}{'cpu'}{'yellow'} = $cpu_defyellow;
    }
    unless (exists($threshdb{$host}{'cpu'}{'red'})) {
      $threshdb{$host}{'cpu'}{'red'} = $cpu_defred;
    }
  }

  # Return results
  return \%threshdb;
} # sub read_cpu_tab



#----
# &read_df_tab(@hostlist)
#
# Read in the bb-dftab.
# Require a list of wanted hosts.
#   $diskthreshdb_ref->{$host}{'/var'}{'red'} = '95';
#   $diskthreshdb_ref->{$host}{'/usr'}{'yellow'} = '90';
# 'localhost' and unspecified hosts ignored.
# Spaces eliminated during processing.
#----

sub read_df_tab {
  my(@hostlist) = @_;
  my %hostdb = map { $_ => 1 } @hostlist;

  my %threshdb = ();
  unless (-f "$ENV{'BBHOME'}/etc/bb-dftab") {
    # bb-dftab doesn't exist, which is the default installation and not
    # necessarily an error
    return \%threshdb;
  }

  my $infh = new FileHandle;
  if ($infh->open("< $ENV{'BBHOME'}/etc/bb-dftab")) {
    while (my $line = $infh->getline()) {
      chomp($line);
      $line =~ s/\#.*$//g;
      $line =~ s/\s+$//g;
      next if $line =~ /^\s*\#/;
      next if $line !~ /\S/;
      $line =~ s/\s//g;
      # Look for specific hosts:
      # host:partition:warn%:panic%
      if ($line =~ /^([^:]+):([^:]+):(\d+):(\d+)$/) {
        my($hostname,$partition,$warn,$panic) = ($1,$2,$3,$4);
        next unless exists($hostdb{$hostname});
        $threshdb{$hostname}{$partition}{'red'} = $panic;
        $threshdb{$hostname}{$partition}{'yellow'} = $warn;
      } # if ($line =~ /^([^:]+):([^:]+):(\d+):(\d+)$/)
    } # while (my $line = $infh->getline())
  } else { # if ($infh->open("< $ENV{'BBHOME'}/etc/bb-dftab"))
    print("Unable to open bb-dftab: $!\n");
    return;
  }

  return \%threshdb;
} # sub read_df_tab

#----
# &get_cpu_status($host,$snmmpsession,$enterprise,$version);
#
# Get all kinds of status possible via SNMP
#----

sub get_cpu_status {
  my($host,$snmpsession,$enterprise,$version) = @_;
  my %cpudb = ();
  my %oiddb = ();

  if ($enterprise eq 'cisco') {
    $oiddb{'cpu5sec'} = $snmpoids{'busyPer'};
    $oiddb{'cpu1min'} = $snmpoids{'avgBusy1'};
    $oiddb{'cpu5min'} = $snmpoids{'avgBusy5'};
  } elsif ($enterprise eq 'foundry') {
    if ($version >= 7.1) {
      $oiddb{'cpu1sec'} = $snmpoids{'snAgGblCpuUtil1SecAvg'};
      $oiddb{'cpu5sec'} = $snmpoids{'snAgGblCpuUtil5SecAvg'};
      $oiddb{'cpu1min'} = $snmpoids{'snAgGblCpuUtil1MinAvg'};
    } else {
      $oiddb{'cpu'} = $snmpoids{'snAgGblCpuUtilData'};
    }
  } elsif ($enterprise eq 'netapp') {
    $oiddb{'cpubusy'} = $snmpoids{'cpuBusyTimePerCent'};
    $oiddb{'cpuidle'} = $snmpoids{'cpuIdleTimePerCent'};
  } elsif ($enterprise eq 'netscreen') {
    $oiddb{'cpu1min'} = $snmpoids{'nsResCpuLast1Min'};
    $oiddb{'cpu5min'} = $snmpoids{'nsResCpuLast5Min'};
    $oiddb{'cpu15min'} = $snmpoids{'nsResCpuLast15Min'};
  } elsif ($enterprise eq 'ucdavis') {
    $oiddb{'cpuuser'}   = $snmpoids{'ssCpuUser'};
    $oiddb{'cpusystem'} = $snmpoids{'ssCpuSystem'};
    $oiddb{'cpuidle'}   = $snmpoids{'ssCpuIdle'};
    # Get the float versions, easiet to handle as numbers
    $oiddb{'la1min'} = $snmpoids{'laLoadFloat.1'};
    $oiddb{'la5min'} = $snmpoids{'laLoadFloat.2'};
    $oiddb{'la15min'} = $snmpoids{'laLoadFloat.3'};
  } else {
    print(localtime().": ".(caller(0))[3].": Unknown enterprise '$enterprise' for host '$host'\n");
    return;
  }
  # Get all the requested data
  if (my $result = $snmpsession->get_request(-varbindlist => [values(%oiddb)])) {
    while (my($oidname,$oidnumber) = each(%oiddb)) {
      $cpudb{$oidname} = $result->{$oidnumber};
    }
  } else {
    print("Unable to get anything for host $host enterprise $enterprise version $version: ".$snmpsession->error()."\n");
    return;
  }

  return \%cpudb;
} # sub get_cpu_status

#----
# $colordb_ref = &check_cpu_colorstatus($host,$enterprise,$version,
#                                    $cpudb_ref,$threshdb_ref->{$host})
#
# Determine colors for each of the CPU parameters given and possible.
#----

sub check_cpu_colorstatus {
  my($host,$enterprise,$version,$cpudb_ref,$thresh_ref) = @_;

  my %colordb = ();

  while (my($param,$value) = each(%$cpudb_ref)) {
    if ($param =~ /^la/) {
      # Load average
      if ($value > $thresh_ref->{'la'}{'red'}) {
        $colordb{$param} = 'red';
      } elsif ($value > $thresh_ref->{'la'}{'yellow'}) {
        $colordb{$param} = 'yellow';
      } else {
        $colordb{$param} = 'green';
      }
    } elsif ($param eq 'cpuidle') {
      if (100 - $value > $thresh_ref->{'cpu'}{'red'}) {
        $colordb{$param} = 'red';
      } elsif (100 - $value > $thresh_ref->{'cpu'}{'yellow'}) {
        $colordb{$param} = 'yellow';
      } else {
        $colordb{$param} = 'green';
      }

    } elsif ($param =~ /^cpu/) {
      if ($value > $thresh_ref->{'cpu'}{'red'}) {
        $colordb{$param} = 'red';
      } elsif ($value > $thresh_ref->{'cpu'}{'yellow'}) {
        $colordb{$param} = 'yellow';
      } else {
        $colordb{$param} = 'green';
      }
    } else {
      # Unknown
      print("Unknown paramater '$param' for host '$host'");
    }
  } # while (my($param,$value) = each(%$cpudb_ref))

  while (my($cpu,$value) = each(%$cpudb_ref)) {
    unless (exists($colordb{$cpu})) {
      $colordb{$cpu} = 'clear';
    }
  }

  return \%colordb;
} # sub check_cpu_colorstatus




##### FANS #####

#----
# &fans_message($host,$community,$enterprise,$version);
#
# Check fan status and compose a message accordingly
#----

sub fans_message {
  my($host,$snmpsession,$enterprise,$version) = @_;
  (my $commahost = $host) =~ s/\./\,/g;
  my $message = '';
  my $test = 'fans';

  if ($enterprise eq 'cisco') {

    # walk ciscoEnvMonFanStatusDescr/.1.3.6.1.4.1.9.9.13.1.4.1.2 to get names
    my %descrdb = ();
    if (my $result = $snmpsession->get_table(-baseoid =>
                                             $snmpoids{'ciscoEnvMonFanStatusDescr'})) {
      while (my($key,$value) = each(%$result)) {
        $descrdb{substr($key,length($snmpoids{'ciscoEnvMonFanStatusDescr'})+1)} = $value;
      }
    } else {
      print("Unable to get fan names for Cisco $host: ".$snmpsession->error()."\n");
      return;
    }

    # walk ciscoEnvMonFanState/.1.3.6.1.4.1.9.9.13.1.4.1.3 to get statuses
    my %statedb = ();
    if (my $result = $snmpsession->get_table(-baseoid =>
                                             $snmpoids{'ciscoEnvMonFanState'})) {
      while (my($key,$value) = each(%$result)) {
        $statedb{substr($key,length($snmpoids{'ciscoEnvMonFanState'})+1)} =
                               $snmpsyntaxdb{'ciscoEnvMonFanState'}{$value};
      }
    } else {
      print("Unable to get fan statuses for Cisco $host: ".$snmpsession->error()."\n");
      return;
    }

    # determine colors for each fan
    my %fancolors = ();
    while (my($index,$state) = each(%statedb)) {
      if ($state eq 'normal') {
        $fancolors{$index} = 'green';
      } elsif ($state eq 'notPresent') {
        $fancolors{$index} = 'clear';
      } elsif ($state eq 'warning') {
        $fancolors{$index} = 'yellow';
      } else {
        $fancolors{$index} = 'red';
      }
    } # while (my($index,$state) = each(%statedb))

    # find worst color
    my $worstcolor = &color_compare(values(%fancolors));

    # compose message
    $message = "status $commahost.$test $worstcolor ".scalar(localtime)."\n\n";
    my @indices = sort { $a <=> $b } keys(%statedb);
    foreach my $index (@indices) {
      $message .= '&'.$fancolors{$index}.' '.$descrdb{$index}.': '.$statedb{$index}."\n";
    }

  } elsif ($enterprise eq 'foundry') {
    # walk snChasFanDescription/.1.3.6.1.4.1.1991.1.1.1.3.1.1.2 to get names
    my %descrdb = ();
    if (my $result = $snmpsession->get_table(-baseoid =>
                                             $snmpoids{'snChasFanDescription'})) {
      while (my($key,$value) = each(%$result)) {
        $descrdb{substr($key,length($snmpoids{'snChasFanDescription'})+1)} = $value;
      }
    } else {
      print("Unable to get fan names for Foundry $host: ".$snmpsession->error()."\n");
      return;
    }

    # walk snChasFanOperStatus/.1.3.6.1.4.1.1991.1.1.1.3.1.1.3 to get statuses
    my %statedb = ();
    if (my $result = $snmpsession->get_table(-baseoid =>
                                             $snmpoids{'snChasFanOperStatus'})) {
      while (my($key,$value) = each(%$result)) {
        $statedb{substr($key,length($snmpoids{'snChasFanOperStatus'})+1)} =
                 $snmpsyntaxdb{'snChasFanOperStatus'}{$value};
      }
    } else {
      print("Unable to get fan statuses for Foundry $host: ".$snmpsession->error()."\n");
      return;
    }

    # Determine colors for each fan
    my %fancolors = ();
    while (my($index,$state) = each(%statedb)) {
      if ($state eq 'normal') {
        $fancolors{$index} = 'green';
      } else {
        $fancolors{$index} = 'red';
      }
    } # while (my($index,$state) = each(%statedb))

    # Determine worst color
    my $worstcolor = &color_compare(values(%fancolors));

    # Compose message
    $message = "status $commahost.$test $worstcolor ".scalar(localtime)."\n\n";
    my @indices = sort { $a <=> $b } keys(%statedb);
    foreach my $index (@indices) {
      $message .= '&'.$fancolors{$index}.' '.$descrdb{$index}.': '.$statedb{$index}."\n";
    }

  } elsif ($enterprise eq 'netapp') {
    # get envFailedFanCount/.1.3.6.1.4.1.789.1.2.4.2.0
    # get envFailedFanMessage/.1.3.6.1.4.1.789.1.2.4.3.0
    my $fancount = '';
    my $fanmessage = '';
    if (my $result = $snmpsession->get_request(-varbindlist =>
                                               [$snmpoids{'envFailedFanCount'},
                                                $snmpoids{'envFailedFanMessage'}])) {
      $fancount = $result->{$snmpoids{'envFailedFanCount'}};
      $fanmessage = $result->{$snmpoids{'envFailedFanMessage'}}
    } else {
      print("Unable to get count of failed fans and message for netapp $host: ".$snmpsession->error()."\n");
      return;
    }

    # Determine color
    my $worstcolor = ($fancount > 0) ? 'red' : 'green';

    # Compose message
    $message = "status $commahost.$test $worstcolor ".scalar(localtime)."\n\n";
    $message .= "$fanmessage\n";

  } else {
    print(localtime().": ".(caller(0))[3].": Unknown enterprise '$enterprise' for host '$host'\n");
    return;
  } # if ($enterprise eq 'cisco')

  return $message;

} # sub fans_message

##### L4, CURRENTLY ONLY FOUNDRY SERVERIRONS SUPPORTED #####

#----
# &l4_message($host,$snmpsession,$enterprise,$version);
#
# Compose a message for a layer 4 load balancing device.
# Currently only Foundry ServerIrons supported.
#----

sub l4_message {
  my($host,$snmpsession,$enterprise,$version) = @_;
  my $test = 'l4';

  # SNMPL4 CHECK - GATHER ALL DATA HERE
  my %snmpl4db = ();
  my %oiddb = ();
  if ($enterprise eq 'foundry') {
    while (my($oidtext,$oidnum) = each(%snmpoids)) {
      next unless $oidtext =~ /^snL4/;
      $oiddb{$oidtext} = $oidnum;
    }
  } else {
    print(localtime().": ".(caller(0))[3].": Unknown enterprise '$enterprise'\n");
    return;
  }

  while (my($oidtext,$oidnum) = each(%oiddb)) {
    if (my $result = $snmpsession->get_table(-baseoid => $oidnum)) {
      my $qoidnum = quotemeta($oidnum);
      while (my($fulloid,$value) = each(%$result)) {
        (my $shortoid = $fulloid) =~ s/^$qoidnum\.//;
        $snmpl4db{$oidtext}{$shortoid} = $value;
      }
    } else {
      print(localtime().": ".(caller(0))[3].": Failed to get tree for $oidtext from $host: ".$snmpsession->error()."\n");
      return;
    }
  } # while (my($oidtext,$oidnum) = each(%oiddb))

  # VIRTUAL SERVERS
  my %vn2ip = ();
  my %vip2n = ();
  my %vn2as = ();
  while (my($indexoid,$index) = each(%{$snmpl4db{'snL4VirtualServerIndex'}})) {
    my $virtname = $snmpl4db{'snL4VirtualServerName'}{$index};
    my $ipaddr = $snmpl4db{'snL4VirtualServerVirtualIP'}{$index};
    my $adminstatus = $snmpsyntaxdb{'snL4VirtualServerAdminStatus'}{$snmpl4db{'snL4VirtualServerAdminStatus'}{$index}};
    $vn2ip{$virtname} = $ipaddr;
    $vip2n{$ipaddr} = $virtname;
    $vn2as{$virtname} = $adminstatus;
  } # while (my($indexoid,$index) = each(%{$snmpl4db{'snL4VirtualServerIndex'}}))
  my %vspas = ();
  while (my($indexoid,$index) = each(%{$snmpl4db{'snL4VirtualServerPortIndex'}})) {
    my $servername = $snmpl4db{'snL4VirtualServerAdminStatus'}{$index};
    my $port = $snmpl4db{'snL4VirtualServerVirtualIP'}{$index};
    if ($port eq '65535') { $port = 'default' }
    my $adminstatus = $snmpsyntaxdb{'snL4VirtualServerPortAdminStatus'}{$snmpl4db{'snL4VirtualServerPortAdminStatus'}{$index}};
    $vspas{$servername}{$port} = $adminstatus;
  } # while (my($indexoid,$index) = each(%{$snmpl4db{'snL4VirtualServerPortIndex'}}))

  # REAL SERVERS
  my %rn2ip = (); # Name to IP
  my %rip2n = (); # IP to Name
  my %rsas = (); # Real server admin status
  while (my($absoid,$index) = each(%{$snmpl4db{'snL4RealServerIndex'}})) {
    my $realname = $snmpl4db{'snL4RealServerName'}{$index};
    my $realip = $snmpl4db{'snL4RealServerIP'}{$index};
    my $adminstatus = $snmpsyntaxdb{'snL4RealServerAdminStatus'}{$snmpl4db{'snL4RealServerAdminStatus'}{$index}};
    $rn2ip{$realname} = $realip;
    $rip2n{$realip} = $realname;
    $rsas{$realname} = $adminstatus;
  } # while (my($absoid,$index) = each(%{$snmpl4db{'snL4RealServerIndex'}}))

  my %rsss = (); # Real server status state
  while (my($indexoid,$index) = each(%{$snmpl4db{'snL4RealServerStatusIndex'}})) {
    my $realname = $snmpl4db{'snL4RealServerStatusName'}{$index};
    my $statusstate = $snmpsyntaxdb{'snL4RealServerStatusState'}{$snmpl4db{'snL4RealServerStatusState'}{$index}};
    $rsss{$realname} = $statusstate;
  } # while (my($indexoid,$index) = each(%{$snmpl4db{'snL4RealServerStatusIndex'}}))
 
  my %rspas = (); # Real server port admin status
  while (my($indexoid,$index) = each(%{$snmpl4db{'snL4RealServerPortIndex'}})) {
    my $realname = $snmpl4db{'snL4RealServerPortServerName'}{$index};
    my $realport = $snmpl4db{'snL4RealServerPortPort'}{$index};
    if ($realport eq '65535') { $realport = 'default' }
    my $adminstatus = $snmpsyntaxdb{'snL4RealServerPortAdminStatus'}{$snmpl4db{'snL4RealServerPortAdminStatus'}{$index}};
    $rspas{$realname}{$realport} = $adminstatus;
  }

  my %rspss = (); # Real server port status state
  while (my($indexoid,$index) = each(%{$snmpl4db{'snL4RealServerPortStatusIndex'}})) {
    my $realname = $snmpl4db{'snL4RealServerPortStatusServerName'}{$index};
    my $realport = $snmpl4db{'snL4RealServerPortStatusPort'}{$index};
    if ($realport eq '65535') { $realport = 'default' }
    my $statusstate = $snmpsyntaxdb{'snL4RealServerPortStatusState'}{$snmpl4db{'snL4RealServerPortStatusState'}{$index}};
    $rspss{$realname}{$realport} = $statusstate;
  }

  # check-as-proxy: look for real servers which are listed by name in bb-hosts
  # but don't have the check enabled for which they're offering services
  # via the L4 load balancer.
  my @realhostlist = sort(keys(%rsas));
  foreach my $hostname (@realhostlist) {
    if (exists($l4_bbhosts_ref->{$hostname})) {
      (my $commahost = $hostname) =~ s/\./\,/g;
      foreach my $realport (keys(%{$rspas{$hostname}})) {
        next if $realport eq 'default';
        my $portname = scalar(getservbyport($realport,'tcp'));
        unless (defined($portname)) {
          print(localtime().": ".(caller(0))[3].": Port number $realport has no defined service.\n");
          next;
        }
        if (exists($l4_bbhosts_ref->{$hostname}{$portname})) {
          next; # BigBrother is already monitoring it.
        }
        # If I get this far, I have a service that needs to be reported.
        my $color = 'green';
        my $message = "Port $realport service $portname is funtioning normally";
        if ($rspas{$hostname}{$realport} eq 'disabled') {
          $color = 'clear';
          $message = "Port $realport service $portname has been disabled on the ServerIron";
        } elsif ($rspss{$hostname}{$realport} eq 'disabled') {
          $color = 'clear';
          $message = "Port $realport service $portname has been disabled on the ServerIron";
        } elsif ($rspss{$hostname}{$realport} ne 'active') {
          $color = 'yellow';
          $message = "Port $realport service $portname is in state: '".$rspss{$hostname}{$realport}."'";
        }
        $message .= " according to $host";
        $message = "status $commahost.$portname $color ".scalar(localtime)."\n$message\n\n";
        $message .= "$Script Version: $VERSION\n";
        if ($is_commandline) {
          print("$ENV{'BBHOME'}/bin/bb-combo.sh add \"$message\"\n");
        } else {
          system("$ENV{'BBHOME'}/bin/bb-combo.sh add \"$message\"");
        }
      } # foreach my $realport (keys(%{$rspas{$realname}}))
      # If $ENV{'CONNTEST'} is true and 'noconn' is defined as a service,
      # then the snL4RealServerAdminStatus and snL4RealServerStatusState
      # are used to determine the 'conn' status.
      if ((exists($ENV{'CONNTEST'})) &&
          ($ENV{'CONNTEST'} eq 'TRUE') &&
          (exists($l4_bbhosts_ref->{$hostname}{'noconn'})))  {
        my $color = 'green';
        my $message = "Server $hostname is responding to layer 3 health checks";
        if ($rsas{$hostname} eq 'disabled') {
          $color = 'clear';
          $message = "Server $hostname has been disabled on the ServerIron";
        } elsif ($rsss{$hostname} eq 'serverdisabled') {
          $color = 'clear';
          $message = "Server $hostname has been disabled on the ServerIron";
        } elsif ($rsss{$hostname} ne 'serveractive') {
          $color = 'yellow';
          $message = "Server $hostname is in state: '".$rsss{$hostname}."'";
        }
        $message .= " according to $host";
        $message = "status $commahost.conn $color ".scalar(localtime)."\n$message\n\n";
        $message .= "$Script Version: $VERSION\n";
        if ($is_commandline) {
          print("$ENV{'BBHOME'}/bin/bb-combo.sh add \"$message\"\n");
        } else {
          system("$ENV{'BBHOME'}/bin/bb-combo.sh add \"$message\"");
        }
      } # If i want to determine the 'conn' status
    } # if (exists($bbhosts_ref->{$hostname}))
  } # foreach my $hostname

  # BINDS
  # Need to be able to build tree from Virtual Server ports
  # down to real server ports

  # BIND TREE
  # Shopping list:
  # 'snL4BindIndex'                       => '1.3.6.1.4.1.1991.1.1.4.6.1.1.1',
  # 'snL4BindVirtualServerName'           => '1.3.6.1.4.1.1991.1.1.4.6.1.1.2',
  # 'snL4BindVirtualPortNumber'           => '1.3.6.1.4.1.1991.1.1.4.6.1.1.3',
  # 'snL4BindRealServerName'              => '1.3.6.1.4.1.1991.1.1.4.6.1.1.4',
  # 'snL4BindRealPortNumber'              => '1.3.6.1.4.1.1991.1.1.4.6.1.1.5',
  my %binddb = ();
  while (my($oidtext,$oidnum) = each(%snmpoids)) {
    next unless $oidtext =~ /^snL4Bind/;
    if (my $result = $snmpsession->get_table(-baseoid => $oidnum)) {
      my $qoidnum = quotemeta($oidnum);
      while (my($fulloid,$value) = each(%$result)) {
        (my $shortoid = $fulloid) =~ s/^$qoidnum\.//;
        $binddb{$oidtext}{$shortoid} = $value;
      }
    } else {
      print(localtime().": ".(caller(0))[3].": Failed to get tree for $oidtext from $host: ".$snmpsession->error()."\n");
      return;
    } # if (my(%subdb) = &simpletree
  } # while (my($oidtext,$oidnum) = each(%oiddb))

  my %bindtree = ();
  while (my($indexoid,$index) = each(%{$binddb{'snL4BindIndex'}})) {
    my $virtualname = $binddb{'snL4BindVirtualServerName'}{$index};
    my $virtualport = $binddb{'snL4BindVirtualPortNumber'}{$index};
    if ($virtualport eq '65535') { $virtualport = 'default' }
    my $realname = $binddb{'snL4BindRealServerName'}{$index};
    my $realport = $binddb{'snL4BindRealPortNumber'}{$index};
    if ($realport eq '65535') { $realport = 'default' }
    $bindtree{$virtualname}{$virtualport}{$realname}{$realport} = 1;
  }


  # COMPOSE PAGE
  # Tables done in plain text for now to avoid HTML table tags for message
  # size limit considerations, but HTML might be more efficient than manually spacing
  # items.
  my $message = '';
  my $worstcolor = 'green';

  my @collen;
  @collen = (); # Maximum lengths of items in each column
  my @virtservs = sort { $a cmp $b } keys(%vn2as);
  if ($l4_show_virtservers) {
    # VIRTUAL SERVER SECTION - Green or clear
    $collen[0] = 5;
    $collen[1] = length('Name');
    $collen[2] = length('IP Address');
    $collen[3] = length('Port');
    $collen[4] = length('Admin Status');
    $message .= "<B>VIRTUAL SERVERS</B>\n";
    # Make alphebetized list of virtual servers
    foreach my $virtname (@virtservs) {
      my $adminstatus = $vn2as{$virtname};
      my $virtip = $vn2ip{$virtname};
      $collen[1] = ($collen[1] > length($virtname)) ? $collen[1] : length($virtname);
      $collen[2] = ($collen[2] > length($virtip)) ? $collen[2] : length($virtip);
      $collen[4] = ($collen[4] > length($adminstatus)) ? $collen[4] : length($adminstatus);
      while (my($port) = each(%{$vspas{$virtname}})) {
        my $adminstatus = $vspas{$virtname}{$port};
        $collen[3] = ($collen[3] > length($port)) ? $collen[3] : length($port);
        $collen[4] = ($collen[4] > length($adminstatus)) ? $collen[4] : length($adminstatus);
      }
    }
    $message .= sprintf("  %".($collen[0])."s    %".($collen[1]+1)."s %".($collen[2]+1)."s %".($collen[3]+1)."s %".($collen[4]+1)."s\n", 'Color', 'Name', 'IP Address', 'Port', 'Admin Status');
    foreach my $virtname (@virtservs) {
      my $adminstatus = $vn2as{$virtname};
      my $virtip = $vn2ip{$virtname};
      my $color = ($adminstatus eq 'enabled') ? 'green' : 'clear';
      # $message .= "  &$color $virtname / $virtip - $adminstatus\n";
      $message .= sprintf("  &%".($collen[0])."s       %".($collen[1]+1)."s %".($collen[2]+1)."s %".($collen[3]+1)."s %".($collen[4]+1)."s\n", $color, $virtname, $virtip, '',$adminstatus);
      while (my($port) = each(%{$vspas{$virtname}})) {
        my $adminstatus = $vspas{$virtname}{$port};
        my $color = ($adminstatus eq 'enabled') ? 'green' : 'clear';
        $message .= sprintf("    &%".($collen[0])."s     %".($collen[1]+1)."s %".($collen[2]+1)."s %".($collen[3]+1)."s %".($collen[4]+1)."s\n", $color,'','',$port,$adminstatus);
      }
    } # foreach my $virtname (@virtservs)
  } # if ($show_virtservers)

  if ($l4_show_realservers) {
    # REAL SERVER SECTION
    $message .= "\n<B>REAL SERVERS</B>\n";
    my @realservs = sort { $a cmp $b } keys(%rsas);
    my %realcolors = ();
    @collen = ();
    $collen[0] = length("Color");
    $collen[1] = length("Name");
    $collen[2] = length("IP Address");
    $collen[3] = length("Port");
    $collen[4] = length("Admin Status");
    $collen[5] = length("State");
    foreach my $realserv (@realservs) {
      my $servadminstatus = $rsas{$realserv};
      my $servstatusstate = $rsss{$realserv};
      my $ipaddr = $rn2ip{$realserv};
      $collen[1] = ($collen[1] > length($realserv)) ? $collen[1] : length($realserv);
      $collen[2] = ($collen[2] > length($ipaddr)) ? $collen[2] : length($ipaddr);
      $collen[4] = ($collen[4] > length($servadminstatus)) ? $collen[4] : length($servadminstatus);
      $collen[5] = ($collen[5] > length($servstatusstate)) ? $collen[5] : length($servstatusstate);
      while (my($realport) = each(%{$rspas{$realserv}})) {
        my $adminstatus = $rspas{$realserv}{$realport};
        my $statusstate = $rspss{$realserv}{$realport};
        $collen[3] = ($collen[3] > length($realport)) ? $collen[3] : length($realport);
        $collen[4] = ($collen[4] > length($adminstatus)) ? $collen[4] : length($adminstatus);
        $collen[5] = ($collen[5] > length($statusstate)) ? $collen[5] : length($statusstate);
      } # foreach my $realport (@portlist)
    } # foreach my $realserv (@realservs)
 
    $message .= sprintf("  %".($collen[0])."s  %".($collen[1] + 1)."s %".($collen[2] + 1)."s %".($collen[3] + 1)."s %".($collen[4] + 1)."s %".($collen[5] + 1)."s\n", 'Color', 'Name', 'IP Address', 'Port', 'Admin Status', 'State');
 
    foreach my $realserv (@realservs) {
      # Color is determined by admin status and statuses of
      # subsidiary ports
      my $servercolor = 'green';
      my %portcolors = ();
      my $servadminstatus = $rsas{$realserv};
      my $servstatusstate = $rsss{$realserv};
      my $ipaddr = $rn2ip{$realserv};
      while (my($port,$portstate) = each(%{$rspss{$realserv}})) {
        if ($portstate eq 'active') {
          $portcolors{$port} = 'green';
        } elsif ($portstate eq 'unbound') {
          $portcolors{$port} = 'clear';
        } elsif ($portstate eq 'disabled') {
          $portcolors{$port} = 'clear';
        } elsif ($portstate eq 'enabled') {
          $portcolors{$port} = 'clear';
        } else {
          if ($port eq 'default') {
            $portcolors{$port} = 'clear';
          } elsif ($servstatusstate eq 'serverenabled') {
            $portcolors{$port} = 'clear';
          } elsif ($servadminstatus ne 'enabled') {
            $portcolors{$port} = 'clear';
          } else {
            $portcolors{$port} = 'yellow';
            $worstcolor = 'yellow';
          }
        }
      }
      while (my($port,$portstatus) = each(%{$rspas{$realserv}})) {
        if ($portstatus eq 'enabled') {
          if ($portcolors{$port} eq 'yellow') {
            $servercolor = 'yellow';
            $worstcolor = 'yellow';
          }
        } else {
          $portcolors{$port} = 'clear';
        }
      }
      # I really do not like this logic. Should be cleaner.
      if ($servstatusstate eq 'serveractive') {
        # Do nothing
      } elsif ($servstatusstate eq 'serverenabled') {
        $servercolor = 'clear';
        foreach my $port (keys(%portcolors)) { $portcolors{$port} = 'clear' }
      } else {
        $servercolor = 'yellow';
        $worstcolor = 'yellow';
      }
      unless ($servadminstatus eq 'enabled') {
        $servercolor = 'clear';
        foreach my $port (keys(%portcolors)) { $portcolors{$port} = 'clear' }
      }
 
      $message .= sprintf("  &%".($collen[0])."s     %".($collen[1] + 1)."s %".($collen[2] + 1)."s %".($collen[3] + 1)."s %".($collen[4] + 1)."s %".($collen[5] + 1)."s\n", $servercolor, $realserv, $ipaddr, '', $servadminstatus, $servstatusstate);
 
      my @portlist = sort { $a <=> $b } keys(%portcolors);
      foreach my $realport (@portlist) {
        my $portcolor = $portcolors{$realport};
        my $adminstatus = $rspas{$realserv}{$realport};
        my $statusstate = $rspss{$realserv}{$realport};
        # $message .= "    &$portcolor $realport - $adminstatus / $statusstate\n";
        $message .= sprintf("    &%".($collen[0])."s   %".($collen[1] + 1)."s %".($collen[2] + 1)."s %".($collen[3] + 1)."s %".($collen[4] + 1)."s %".($collen[5] + 1)."s\n", $portcolor, '', '', $realport, $adminstatus, $statusstate);
      }
      $realcolors{$realserv} = $servercolor;
    } # foreach my $realserv (@realservs)
  } # if ($show_realservers)

  if ($l4_show_bindtree) {
    # BINDTREE DISPLAYED
    $message .= "\n<B>BINDS</B>\n";
    # $bindtree{$virtname}{$virtport}{$realname}{$realport} = 1;
    # First, colors.
    my %realportcolors = ();
    my %realservcolors = ();
    my %virtportcolors = ();
    my %virtservcolors = ();
    foreach my $virtname (@virtservs) {
      $virtservcolors{$virtname} = 'green';
      my @virtports = sort(keys(%{$bindtree{$virtname}}));
      if ($vn2as{$virtname} eq 'disabled') {
        $virtservcolors{$virtname} = 'clear';
        foreach my $virtport (@virtports) {
          $virtportcolors{$virtname}{$virtport} = 'clear';
          my @realnames = sort { $a cmp $b } keys(%{$bindtree{$virtname}{$virtport}});
          foreach my $realname (@realnames) {
            $realservcolors{$virtname}{$virtport}{$realname} = 'clear';
            my @realports = sort(keys(%{$bindtree{$virtname}{$virtport}{$realname}}));
            foreach my $realport (@realports) {
              $realportcolors{$virtname}{$virtport}{$realname}{$realport} = 'clear';
            }
          } # foreach my $realname (@realnames)
        } # foreach my $virtport (@virtports)
      } else {
        foreach my $virtport (@virtports) {
          # Decide the color of each virtual port
          my @realnames = sort { $a cmp $b } keys(%{$bindtree{$virtname}{$virtport}});
          if ($vspas{$virtname}{$virtport} eq 'disabled') {
            $virtportcolors{$virtname}{$virtport} = 'clear';
            foreach my $realname (@realnames) {
              $realservcolors{$virtname}{$virtport}{$realname} = 'clear';
              my @realports = sort(keys(%{$bindtree{$virtname}{$virtport}{$realname}}));
              foreach my $realport (@realports) {
                $realportcolors{$virtname}{$virtport}{$realname}{$realport} = 'clear';
              }
            }
          } else {
            foreach my $realname (@realnames) {
              $realservcolors{$virtname}{$virtport}{$realname} = 'green';
              # Decide the color of each real server
              my @realports = sort(keys(%{$bindtree{$virtname}{$virtport}{$realname}}));
              # if (($rsas_ref->{$realname} eq 'disabled') || ($rsss_ref->{$realname} eq 'serverenabled'))
              # down servers can be serverenabled
              if ($rsas{$realname} eq 'disabled') {
                $realservcolors{$virtname}{$virtport}{$realname} = 'clear';
                foreach my $realport (@realports) {
                  $realportcolors{$virtname}{$virtport}{$realname}{$realport} = 'clear';
                } # foreach my $realport (@realports)
              } else {
                foreach my $realport (@realports) {
                  $realportcolors{$virtname}{$virtport}{$realname}{$realport} = 'green';
                  # Decide the color of each port
                  # WARNING: snL4RealServerPortStatusState can be 'disabled' when the port is 'unbound'
                  if (($rspas{$realname}{$realport} eq 'disabled') ||
                      ($rspss{$realname}{$realport} eq 'disabled')) {
                    $realportcolors{$virtname}{$virtport}{$realname}{$realport} = 'clear';
                  } else {
                    if (($realport eq 'default') &&
                        ($rspss{$realname}{$realport} eq 'failed')) {
                      $realportcolors{$virtname}{$virtport}{$realname}{$realport} = 'clear';
                    } elsif ($rspss{$realname}{$realport} ne 'active') {
                      $realportcolors{$virtname}{$virtport}{$realname}{$realport} = 'yellow';
                    }
                  }
                } # foreach my $realport (@realports)
                # Look at the color of each port.
                # If any are yellow set the server yellow.
                # Don't want to turn red here.
                $realservcolors{$virtname}{$virtport}{$realname} = &color_compare(values(%{$realportcolors{$virtname}{$virtport}{$realname}}));
              } # if ($rsas_ref->{$realname} eq 'disabled')
            } # foreach my $realname (@realservs)
            # Look at the color of each real server.
            # If any are yellow set the virtual port yellow.
            # If all are yellow set the virtual port red.
            $virtportcolors{$virtname}{$virtport} = &color_compare_strict(values(%{$realservcolors{$virtname}{$virtport}}));
          } # if ($vspas_ref->{$virtname}{$virtport} eq 'disabled')
        } # foreach my $virtport (@virtports)
        # Look at the color of each virtual port.
        # If any are yellow set the virt server yellow.
        # If any are red set the virtual server red.
        $virtservcolors{$virtname} = &color_compare(values(%{$virtportcolors{$virtname}}));
      } # if ($vsas_ref->{$virtname} eq 'disabled')
    } # foreach my $virtname (@virtservs)
 
    # Then, show tree
    foreach my $virtname (@virtservs) {
      $message .= " &".$virtservcolors{$virtname}." $virtname(".$vn2ip{$virtname}.")\n";
      if ($virtservcolors{$virtname} eq 'red') {
        $worstcolor = 'red';
      } elsif (($virtservcolors{$virtname} eq 'yellow') && ($worstcolor ne 'red')) {
        $worstcolor = 'yellow';
      }
      my @virtports = sort(keys(%{$bindtree{$virtname}}));
      foreach my $virtport (@virtports) {
        $message .= "   &".$virtportcolors{$virtname}{$virtport}." $virtport\n";
        my @realnames = sort { $a cmp $b } keys(%{$bindtree{$virtname}{$virtport}});
        foreach my $realname (@realnames) {
          $message .= "     &".$realservcolors{$virtname}{$virtport}{$realname}." $realname(".$rn2ip{$realname}.")\n";
          my @realports = sort(keys(%{$bindtree{$virtname}{$virtport}{$realname}}));
          foreach my $realport (@realports) {
            $message .= "       &".$realportcolors{$virtname}{$virtport}{$realname}{$realport}." $realport\n";
          }
        } # foreach my $realserv (@realservs)
      } # foreach my $virtport (@virtports)
      $message .= "\n";
    } # foreach my $virtname (@virtservs)
  } # if ($show_bindtree)


  # HEADER GENERATED
  (my $commaname = $host) =~ s/\./\,/g;
  $message = "status $commaname.$test $worstcolor ".scalar(localtime)."\n\n$message";

  return $message;
} # sub l4_message


#----
# &l4_read_bbhosts()
#
# Read bb-hosts specifically for the benefit of comparing with L4 monitoring.
# Specific URLs don't matter as much as the fact that http or https is offered.
# Although many folks use dig with BB to check DNS, this extension creates
# the column as dns if neither dig nor dns already exist for a given host.
# If a service is mentioned with ? or ! in front, it's an indication that
# BigBrother is taking care of the checking. Same if there's a :s, :q, or :Q
# at the end.
#----

sub l4_read_bbhosts {
  my $infh = new FileHandle;
  unless ($infh->open("< $ENV{'BBHOME'}/etc/bb-hosts")) {
    print("Unable to open $ENV{'BBHOME'}/etc/bb-hosts: $!\n");
    return;
  }
  my %bbhostsdb = ();
  while (my $line = $infh->getline()) {
    $line =~ s/^\s*#.*$//g;
    next unless $line =~ /\S/;
    if ($line =~ /^(\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})\s+(\S+)/) { # If the line begins with an IP address
      my($ipaddr,$hostname,$services) = ($1,$2,$');
      $services = '' unless defined($services);
      $services =~ s/^\s+//g;
      $services =~ s/\s+$//g;
      $services =~ s/^#\s*//;
      my @svclist = split(' ',$services);
      foreach my $service (@svclist) {
        # Don't worry about http with a funny port.
        $service =~ s|^http://.*$|http|;
        $service =~ s|^https://.*$|https|;
        next if $service =~ /^BBRELAY/;
        $service =~ s/:[sqQ]$//;
        $service =~ s/^[\?\!]//;
        # Custom TCP ports by name... yes I can do this.
        # Trick is it means I do not need bb-xsnmpl4's help with this port.
        my($servname,$servport);
        if ($service =~ /^(\w+):(\d+)$/) {
          ($servname,$servport) = ($1,$2);
        } elsif ($service =~ /^(\w+)$/) {
          $servname = $1;
          $servport = scalar(getservbyname($service,'tcp'));
          $servport = 1 unless defined($servport); # Need a placeholder
        } else {
          # Quietly ignore
          # die("OOPS no patterns matched");
        }# if ($service =~ /^(\w+):(\d+)$/)
        $bbhostsdb{$hostname}{$servname} = $servport;
      } # foreach my $service (@svclist)
    } # If the line contains an IP address at the beginning
  } # while (my $line = $infh->getline())
  return \%bbhostsdb;
} # sub l4_read_bbhosts



#----
# &is_l4switch($host,$snmpsession);
#
# Check whether or not this is in fact an L4 switch
# for which I can gather data.
#----

sub is_l4switch {
  my($host,$snmpsession) = @_;

  if (my $result = $snmpsession->get_request(-varbindlist => [$snmpoids{'sysObjectID'}])) {
    my $objectoid = $result->{$snmpoids{'sysObjectID'}};
    # Is it a Foundry ServerIron?
    if (($objectoid eq $snmpoids{'snSI'}) ||
        ($objectoid eq $snmpoids{'snSIXL'}) ||
        ($objectoid eq '.1.3.6.1.4.1.1991.1.1')) { # Very old BigServerIrons
      # Does it have anything I can monitor?
      if (my $result = $snmpsession->get_table(-baseoid => $snmpoids{'snL4VirtualServerIndex'})) {
        if (scalar(keys(%$result))) {
          return 1;
        }
      }
    }
  } else {
    print(localtime().": ".(caller(0))[3].": Failed to get sysObjectID from $host: ".$snmpsession->error()."\n");
    return;
  }

  return 0;
} # sub is_l4switch



##### PORTS #####

#----
# &ports_message($host,$snmpsession,$enterprise,$version);
#
# Check port speed and duplex.
# Look for ports on auto that have chosen 100half
#----

sub ports_message {
  my($host,$snmpsession,$enterprise,$version) = @_;
  (my $commahost = $host) =~ s/\./\,/g;
  my $message = '';
  my $test = 'ports';

  my %ifidx2desc = (); # 1.3.6.1.2.1.2.2.1.2
  my %ifidx2speed = (); # 1.3.6.1.2.1.2.2.1.5
  my %ifidx2operstat = (); # 1.3.6.1.2.1.2.2.1.8
  my %ifidx2alias = (); # 1.3.6.1.2.1.31.1.1.1.18
                        # Not present in older Foundry stackables

  if (my $result = $snmpsession->get_table(-baseoid => $snmpoids{'ifOperStatus'})) {
    while (my($oid,$statuscode) = each(%$result)) {
      (my $index = $oid) =~ s/^.*\.//g;
      $ifidx2operstat{$index} = $snmpsyntaxdb{'ifOperStatus'}{$statuscode};
    }
  } else {
    print(localtime().": ".(caller(0))[3].": Unable to get ifOperStatus for $host: ".$snmpsession->error()."\n");
    return;
  }

  # Build a list of ifDescr OIDs I want to get
  # Some oddballs like ServerIrons have ifDescrs that show up in walk and get-next but not get
  if (my $result = $snmpsession->get_table(-baseoid => $snmpoids{'ifDescr'})) {
    foreach my $index (keys(%ifidx2operstat)) {
      if (exists($result->{"$snmpoids{'ifDescr'}.$index"})) {
        my $descr = $result->{"$snmpoids{'ifDescr'}.$index"};
        if ($descr =~ /^null|^vlan|^lb|^loopback|^port-channel|^v\d+/i) {
          delete($ifidx2operstat{$index});
        } else {
          $ifidx2desc{$index} = $descr;
        }
      }
    }
  } else {
    print(localtime().": ".(caller(0))[3].": Unable to get ifDescr for host $host: ".$snmpsession->error()."\n");
  }
#   my @descroidlist = keys(%ifidx2operstat);
#   foreach my $index (@descroidlist) {
#     if (my $result = $snmpsession->get_request(-varbindlist => ["$snmpoids{'ifDescr'}.$index"])) {
#       my $descr = $result->{"$snmpoids{'ifDescr'}.$index"};
#       if ($descr =~ /^null|^vlan|^lb|^loopback|^port-channel|^v\d+/i) {
#         delete($ifidx2operstat{$index});
#       } else {
#         $ifidx2desc{$index} = $descr;
#       }
#     } else {
#       print(localtime().": ".(caller(0))[3].": Unable to get ifDescr for host $host index $index: ".$snmpsession->error()."\n");
#     } # if (my $result = $snmpsession->get_request(-varbindlist => ["$snmpoids{'ifDescr'}.$index"]))
#   } # foreach my $index (@descroidlist)

  # Try to build a list of ifAlias OIDs
  # Best not to try them all in case ifAlias isn't supported
  my @indexlist = keys(%ifidx2desc);
  my $has_ifalias = 0;
  foreach my $index (@indexlist) {
    if (my $result = $snmpsession->get_request(-varbindlist => ["$snmpoids{'ifAlias'}.$index"])) {
      $ifidx2alias{$index} = $result->{"$snmpoids{'ifAlias'}.$index"};
    } else {
      $has_ifalias = 0;
      last;
    }
  }

  # Find actual speeds of interfaces that are up
  my @upindexlist = grep { $ifidx2operstat{$_} eq 'up' } keys(%ifidx2operstat);
  foreach my $index (@upindexlist) {
    if (my $result = $snmpsession->get_request(-varbindlist => ["$snmpoids{'ifSpeed'}.$index"])) {
      $ifidx2speed{$index} = $result->{"$snmpoids{'ifSpeed'}.$index"};
    } else {
      print(localtime().": ".(caller(0))[3].": Unable to get ifSpeed for $host index $index: ".$snmpsession->error()."\n");
      return;
    }
  } # foreach my $index (@upindexlist)

  # No one seems to support dot3StatsDuplexStatus, but it doesn't cover auto anyways.

  # Operational speed and status can always be determined from ifSpeed and ifOperStatus

  my %ifidx2confspeed = ();
  my %ifidx2confduplx = ();
  my %ifidx2duplx = ();

  if ($enterprise eq 'cisco') {
    # c2900PortDuplexState determines configuration for full, half, or auto
    #  1.3.6.1.4.1.9.9.87.1.4.1.1.31
    # c2900PortDuplexStatus determines actual status for full or half.
    #  1.3.6.1.4.1.9.9.87.1.4.1.1.32
    # (No point in checking for ports set to full or half.)
    # c2900PortAdminSpeed
    # Speed settings for a port: auto, 10M,100M,155M
    #  1.3.6.1.4.1.9.9.87.1.4.1.1.33

    my %port2ifindex = ();
    my %ifindex2port = ();
    if (my $result = $snmpsession->get_table(-baseoid => $snmpoids{'c2900PortIfIndex'})) {
      while (my($oidnum,$ifindex) = each(%$result)) {
        (my $portnum = $oidnum) =~ s/.*\.//g;
        $port2ifindex{$portnum} = $ifindex;
        $ifindex2port{$ifindex} = $portnum;
      }
    } else {
      print(localtime().": ".(caller(0))[3].": Unable to get c2900PortIfIndex for cisco $host: ".$snmpsession->error()."\n");
      return;
    }

    while (my($index,$status) = each(%ifidx2operstat)) {
      next unless $status eq 'up';
      my $port = $ifindex2port{$index};
      if (my $result = $snmpsession->get_request(-varbindlist => ["$snmpoids{'c2900PortAdminSpeed'}.$port"])) {
        $ifidx2confspeed{$index} = $snmpsyntaxdb{'c2900PortAdminSpeed'}{$result->{"$snmpoids{'c2900PortAdminSpeed'}.$port"}};
      } else {
        print(localtime().": ".(caller(0))[3].": Unable to get c2900PortAdminSpeed for $host index $index port $port: ".$snmpsession->error()."\n");
      }
      if (my $result = $snmpsession->get_request(-varbindlist => ["$snmpoids{'c2900PortDuplexState'}.$port"])) {
        $ifidx2confduplx{$index} = $snmpsyntaxdb{'c2900PortDuplexState'}{$result->{"$snmpoids{'c2900PortDuplexState'}.$port"}};
      } else {
        print(localtime().": ".(caller(0))[3].": Unable to get c2900PortDuplexState for $host index $index port $port: ".$snmpsession->error()."\n");
      }
      if ($ifidx2confduplx{$index} eq 'auto') {
        if (my $result = $snmpsession->get_request(-varbindlist => ["$snmpoids{'c2900PortDuplexStatus'}.$port"])) {
          $ifidx2duplx{$index} = $snmpsyntaxdb{'c2900PortDuplexStatus'}{$result->{"$snmpoids{'c2900PortDuplexStatus'}.$port"}};
        } else {
          print(localtime().": ".(caller(0))[3].": Unable to get c2900PortDuplexStatus for $host index $index port $port.\n");
        }
      } else { # if ($confduplex eq 'auto')
        $ifidx2duplx{$index} = $ifidx2confduplx{$index};
      }
    } # while (my($index,$status) = each(%ifidx2operstat))
  } elsif ($enterprise eq 'foundry') {
    # snSwPortInfoChnMode (duplex)
    #   1.3.6.1.4.1.1991.1.1.3.3.1.1.4
    # snSwPortInfoSpeed (conf speed&auto)
    #   1.3.6.1.4.1.1991.1.1.3.3.1.1.5
    # snSwPortIfIndex translates port number to interface number,
    #   1.3.6.1.4.1.1991.1.1.3.3.1.1.38
    # snSwPortName, same as ifAlias, use only if %ifidx2alias is empty
    #   1.3.6.1.4.1.1991.1.1.3.3.1.1.24
    # my $snSwPortInfoChnMode = '.1.3.6.1.4.1.1991.1.1.3.3.1.1.4';
    # my $snSwPortInfoSpeed = '.1.3.6.1.4.1.1991.1.1.3.3.1.1.5';
    # my $snSwPortIfIndex = '.1.3.6.1.4.1.1991.1.1.3.3.1.1.38';
    # my $snSwPortName = '.1.3.6.1.4.1.1991.1.1.3.3.1.1.24';
   
    my %port2ifindex = ();
    my %ifindex2port = ();
    if (my $result = $snmpsession->get_table(-baseoid => $snmpoids{'snSwPortIfIndex'})) {
      while (my($oidnum,$ifindex) = each(%$result)) {
        (my $portnum = $oidnum) =~ s/.*\.//g;
        $port2ifindex{$portnum} = $ifindex;
        $ifindex2port{$ifindex} = $portnum;
      }
    } else {
      print(localtime().": ".(caller(0))[3].": Unable to get snSwPortIfIndex for foundry $host: ".$snmpsession->error()."\n");
      return;
    }

    while (my($index,$status) = each(%ifidx2operstat)) {
      unless (exists($ifindex2port{$index})) {
        print(localtime().": ".(caller(0))[3].": No port number exists for index '$index' host '$host'\n");
        next;
      }
      unless (defined($ifindex2port{$index})) {
        print(localtime().": ".(caller(0))[3].": No port number defined for index '$index' host '$host'\n");
        next;
      }
      if (length($ifindex2port{$index}) == 0) {
        print(localtime().": ".(caller(0))[3].": Port number defined for index '$index' host '$host' has zero length.\n");
        next;
      }
      my $portnum = $ifindex2port{$index};
      unless (exists($ifidx2alias{$index})) {
        if (my $result = $snmpsession->get_request(-varbindlist => ["$snmpoids{'snSwPortName'}.$portnum"])) {
          $ifidx2alias{$index} = $result->{"$snmpoids{'snSwPortName'}.$portnum"};
        } else {
          print(localtime().": ".(caller(0))[3].": Unable to get for snSwPortName foundry $host: ".$snmpsession->error()."\n");
        }
      }
      next unless $status eq 'up';

      # Get adminspeed and duplex in one go
      if (my $result = $snmpsession->get_request(-varbindlist =>
                                              ["$snmpoids{'snSwPortInfoSpeed'}.$portnum",
                                               "$snmpoids{'snSwPortInfoChnMode'}.$portnum"])) {
        my $adminspeed = $snmpsyntaxdb{'snSwPortInfoSpeed'}{$result->{"$snmpoids{'snSwPortInfoSpeed'}.$portnum"}};
        $ifidx2confspeed{$index} = $adminspeed;
        if ($adminspeed eq 'auto') {
          $ifidx2confduplx{$index} = 'auto';
        }
        my $duplex = $snmpsyntaxdb{'snSwPortInfoChnMode'}{$result->{"$snmpoids{'snSwPortInfoChnMode'}.$portnum"}};
        $ifidx2duplx{$index} = $duplex;
        unless ($adminspeed eq 'auto') {
          $ifidx2confduplx{$index} = $duplex;
        }
      } else {
        print(localtime().": ".(caller(0))[3].": Unable to get for snSwPortInfoSpeed and snSwPortInfoChnMode foundry $host port number '$portnum': ".$snmpsession->error()."\n");
        next;
      }

    } # while (my($index,$status) = each(%ifidx2operstat))
  } else {
    print(localtime().": ".(caller(0))[3]." Unknown enterprise $enterprise");
    return;
  }

  my %ifcolordb = ();
  my $worstcolor = 'green';
  $message = "</pre>";
  $message .= "<table>";
  $message .= "<tr><td>Color</td><td>Interface</td><td>Description</td><td>Status</td><td>Conf<br>Speed</td><td>Conf<br>Duplex</td><td>Real<br>Speed</td><td>Real<br>Duplex</td></tr>\n";
  @indexlist = sort { $a <=> $b } keys(%ifidx2operstat);
  foreach my $index (@indexlist) {
    my $status = $ifidx2operstat{$index};
    if ($status eq 'up') {
      if (($ifidx2confduplx{$index} eq 'auto') &&
          ($ifidx2duplx{$index} eq 'half') &&
          ($ifidx2speed{$index} == 100000000)) {
        $ifcolordb{$index} = 'yellow';
        $worstcolor = 'yellow';
      } else {
        $ifcolordb{$index} = 'green';
      }
    } else {
      $ifcolordb{$index} = 'clear';
    }
    $message .= "<tr>";
    $message .= "<td>";
    $message .= '&'."$ifcolordb{$index}</td><td>$ifidx2desc{$index}</td><td>$ifidx2alias{$index}</td><td>$status</td>";
    if ($status eq 'up') {
      my $confspeed = $ifidx2confspeed{$index};
      unless ($confspeed eq 'auto') { $confspeed = sprintf("%dMbps",($confspeed / 1000000)) }
      my $truespeed = $ifidx2speed{$index};
      unless ($truespeed eq 'auto') { $truespeed = sprintf("%dMbps",($truespeed / 1000000)) }
      my $confduplex = $ifidx2confduplx{$index};
      my $trueduplex = $ifidx2duplx{$index};
      $message .= "<td>$confspeed</td><td>$confduplex</td><td>$truespeed</td><td>$trueduplex</td>";
    }
    $message .= "</tr>";
    $message .= "\n";
  }
  $message .= "</table>\n<pre>\n";

  $message = "status $commahost.$test $worstcolor ".scalar(localtime)."\n$message";

  return $message;  
} # sub ports_message


##### POWER #####


#----
#  &power_message($host,$snmpsession,$enterprise,$version)
#
# Gather information on power supply statuses, determine color, and create a message.
#----

sub power_message {
  my($host,$snmpsession,$enterprise,$version) = @_;
  (my $commahost = $host) =~ s/\./\,/g;
  my $message = '';
  my $test = 'power';

  if ($enterprise eq 'apc') {
    # List of OIDs to collect:
    # upsBasicIdentModel
    # upsBasicBatteryStatus
    # upsBasicOutputStatus
    # upsAdvBatteryCapacity
    # upsAdvBatteryTemperature
    # upsAdvBatteryRunTimeRemaining
    # upsAdvBatteryReplaceIndicator
    # upsAdvInputLineVoltage
    # upsAdvOutputLoad
    # upsAdvOutputVoltage
    # upsBasicBatteryLastReplaceDate
    my $result = undef;
    unless ($result = $snmpsession->get_request(-varbindlist => [
                                  $snmpoids{'upsBasicIdentModel'},
                                  $snmpoids{'upsBasicBatteryStatus'},
                                  $snmpoids{'upsBasicOutputStatus'},
                                  $snmpoids{'upsAdvBatteryCapacity'},
                                  $snmpoids{'upsAdvBatteryRunTimeRemaining'},
                                  $snmpoids{'upsAdvBatteryReplaceIndicator'},
                                  $snmpoids{'upsAdvInputLineVoltage'},
                                  $snmpoids{'upsAdvOutputLoad'},
                                  $snmpoids{'upsAdvOutputVoltage'},
                                  $snmpoids{'upsBasicBatteryLastReplaceDate'},
                                  ])) {
      print(localtime().": ".(caller(0))[3].": Unable to check UPS status for APC $host: ".$snmpsession->error()."\n");
      return;
    }

    # Digest results
    my $model = $result->{'upsBasicIdentModel'};
    my $battstatus = $snmpsyntaxdb{'upsBasicBatteryStatus'}{$result->{$snmpoids{'upsBasicBatteryStatus'}}};
    my $outstatus = $snmpsyntaxdb{'upsBasicOutputStatus'}{$result->{$snmpoids{'upsBasicOutputStatus'}}};
    my $capacity = $result->{'upsAdvBatteryCapacity'};
    my $remain = $result->{'upsAdvBatteryRunTimeRemaining'} / 100; # Seconds
    my $replace = $snmpsyntaxdb{'upsAdvBatteryReplaceIndicator'}{$result->{$snmpoids{'upsAdvBatteryReplaceIndicator'}}};
    my $inputvoltage = $result->{'upsAdvInputLineVoltage'};
    my $outputload = $result->{'upsAdvOutputLoad'};
    my $outputvoltage = $result->{'upsAdvOutputVoltage'};
    my $replacedate = $result->{'upsBasicBatteryLastReplaceDate'};

    # Determine state
    my $worstcolor = 'green';
    if ($battstatus eq 'batteryLow') {
      $worstcolor = 'red';
      $message .= "Battery time remaining low.\n";
    }
    if ($replace eq 'batteryNeedsReplacing') {
      $worstcolor = 'red';
      $message .= "Battery needs replacing.\n";
    }
    # upsAdvOutputLoad, say yellow for 90 and red for 95.
    if ($outputload > 95) {
      $worstcolor = 'red';
      $message .= "Output load above 95 percent.\n";
    } elsif ($outputload > 90) {
      $worstcolor = ($worstcolor eq 'green') ? 'yellow' : $worstcolor;
      $message .= "Output load above 90 percent.\n";
    }

    # Collect the following if currently on battery power.
    # upsAdvInputLineFailCause
    # upsBasicBatteryTimeOnBattery
    if ($outstatus eq 'onBattery') {
      $result = undef;
      unless ($result = $snmpsession->get_request(-varbindlist => [
                                      $snmpoids{'upsAdvInputLineFailCause'},
                                      $snmpoids{'upsBasicBatteryTimeOnBattery'},
                                      ])) {
        print(localtime().": ".(caller(0))[3].": Unable to check line fail cause for APC $host: ".$snmpsession->error()."\n");
        return;
      }
      my $failcause = $snmpsyntaxdb{'upsAdvInputLineFailCause'}{$result->{$snmpoids{'upsAdvInputLineFailCause'}}};
      my $timeonbatt = $result->{'upsBasicBatteryTimeOnBattery'} / 100; # Seconds
      $message .= "Reason for input line failure: $failcause\n";
      $message .= "Time elapsed since switching to battery: $timeonbatt seconds\n";
    }

    # Compose remainder of message.
    $message .= "Vendor: $enterprise\n";
    $message .= "Model: $model\n";
    $message .= "Battery Status: $battstatus\n";
    $message .= "Current State of the UPS: $outstatus\n";
    $message .= "Remaining Battery Capacity: $capacity\n";
    $message .= "Time Remaining Before Battery Exhaustion: $remain seconds\n";
    $message .= "Battery Replacement Indicator: $replace\n";
    $message .= "Input Voltage: $replace\n";
    $message .= "UPS Load: $outputload %\n";
    $message .= "Output Voltage: $outputvoltage %\n";
    $message .= "Date Batteries Last Replaced: $replacedate\n";

    $message = "status $commahost.$test $worstcolor ".scalar(localtime)."\n\n$message";

  } elsif ($enterprise eq 'cisco') {
    # walk ciscoEnvMonSupplyStatusDescr/.1.3.6.1.4.1.9.9.13.1.5.1.2 to get names
    my %descrdb = ();
    if (my $result = $snmpsession->get_table(-baseoid =>
                                             $snmpoids{'ciscoEnvMonSupplyStatusDescr'})) {
      while (my($key,$value) = each(%$result)) {
        $key = substr($key,length($snmpoids{'ciscoEnvMonSupplyStatusDescr'})+1);
        $descrdb{$key} = $value;
      }
    } else {
      print(localtime().": ".(caller(0))[3].": Unable to get power supply names for Cisco $host: ".$snmpsession->error()."\n");
      return;
    }

    # walk ciscoEnvMonSupplyState/.1.3.6.1.4.1.9.9.13.1.5.1.3 to get statuses
    my %statedb = ();
    if (my $result = $snmpsession->get_table(-baseoid => $snmpoids{'ciscoEnvMonSupplyState'})) {
      while (my($key,$value) = each(%$result)) {
        $key = substr($key,length($snmpoids{'ciscoEnvMonSupplyState'})+1);
        $statedb{$key} = $snmpsyntaxdb{'ciscoEnvMonSupplyState'}{$value};
      }
    } else {
      print(localtime().": ".(caller(0))[3].": Unable to get power supply states for Cisco $host: ".$snmpsession->error()."\n");
      return;
    }

    # walk ciscoEnvMonSupplySource/.1.3.6.1.4.1.9.9.13.1.5.1.4 to get power source
    my %sourcedb = ();
    if (my $result = $snmpsession->get_table(-baseoid => $snmpoids{'ciscoEnvMonSupplySource'})) {
      while (my($key,$value) = each(%$result)) {
        $key = substr($key,length($snmpoids{'ciscoEnvMonSupplySource'})+1);
        $sourcedb{$key} = $snmpsyntaxdb{'ciscoEnvMonSupplySource'}{$value};
      }
    } else {
      print(localtime().": ".(caller(0))[3].": Unable to get power supply sources for Cisco $host: ".$snmpsession->error()."\n");
      return;
    }

    # determine colors
    my %powercolors = ();
    while (my($index,$state) = each(%statedb)) {
      if ($state eq 'normal') {
        $powercolors{$index} = 'green';
      } elsif ($state eq 'warning') {
        $powercolors{$index} = 'yellow';
      } else {
        $powercolors{$index} = 'red';
      }
    } # while (my($index,$state) = each(%statedb))

    # find worst color
    my $worstcolor = &color_compare(values(%statedb));

    # compose message
    $message = "status $commahost.$test $worstcolor ".scalar(localtime)."\n\n";
    my @indices = sort { $a <=> $b } keys(%statedb);
    foreach my $index (@indices) {
      $message .= '&'.$powercolors{$index}.' '.$descrdb{$index}." $index";
      if (exists($sourcedb{$index})) {
        $message .= ", source: ".$sourcedb{$index}
      }
      $message .= ", state: ".$statedb{$index}."\n";
    } # foreach my $index (@indices)

  } elsif ($enterprise eq 'foundry') {
    # walk snChasPwrSupplyDescription/.1.3.6.1.4.1.1991.1.1.1.2.1.1.2 to get names
    # my $snChasPwrSupplyDescription = '.1.3.6.1.4.1.1991.1.1.1.2.1.1.2';
    my %descrdb = ();
    if (my $result = $snmpsession->get_table(-baseoid => $snmpoids{'snChasPwrSupplyDescription'})) {
      while (my($key,$value) = each(%$result)) {
        $key = substr($key,length($snmpoids{'snChasPwrSupplyDescription'})+1);
        $descrdb{$key} = $value;
      }
    } else {
      print(localtime().": ".(caller(0))[3].": Unable to get power supply names for Foundry $host: ".$snmpsession->error()."\n");
      return;
    }

    # walk snChasPwrSupplyOperStatus/.1.3.6.1.4.1.1991.1.1.1.2.1.1.3 to get statuses
    # my %statenames = ('1' => 'other', '2' => 'normal', '3' => 'failure');
    # my $snChasPwrSupplyOperStatus = '.1.3.6.1.4.1.1991.1.1.1.2.1.1.3';
    my %statedb = ();
    if (my $result = $snmpsession->get_table(-baseoid => $snmpoids{'snChasPwrSupplyOperStatus'})) {
      while (my($key,$value) = each(%$result)) {
        $key = substr($key,length($snmpoids{'snChasPwrSupplyOperStatus'})+1);
        $statedb{$key} = $snmpsyntaxdb{'snChasPwrSupplyOperStatus'}{$value};
      }
    } else {
      print(localtime().": ".(caller(0))[3].": Unable to get power supply statuses for Foundry $host: ".$snmpsession->error()."\n");
      return;
    }

    # determine colors
    my %powercolors = ();
    while (my($index,$state) = each(%statedb)) {
      if ($state eq 'normal') {
        $powercolors{$index} = 'green';
      } else {
        $powercolors{$index} = 'red';
      }
    } # while (my($index,$state) = each(%statedb))

    # find worst color
    my $worstcolor = &color_compare(values(%powercolors));

    # compose message
    $message = "status $commahost.$test $worstcolor ".scalar(localtime)."\n\n";
    my @indices = sort { $a <=> $b } keys(%statedb);
    foreach my $index (@indices) {
      $message .= '&'.$powercolors{$index}.' '.$descrdb{$index}.': '.$statedb{$index}."\n";
    }

  } elsif ($enterprise eq 'netapp') {
    # get envFailedPowerSupplyCount/.1.3.6.1.4.1.789.1.2.4.4.0 to get failed count
    # get envFailedPowerSupplyMessage/.1.3.6.1.4.1.789.1.2.4.5.0 to get failed message
    my $powercount = 0;
    my $powermessage = '';
    if (my $result = $snmpsession->get_request(-varbindlist =>
                                              [$snmpoids{'envFailedPowerSupplyCount'},
                                               $snmpoids{'envFailedPowerSupplyMessage'}])) {
      $powercount = $result->{$snmpoids{'envFailedPowerSupplyCount'}};
      $powermessage = $result->{$snmpoids{'envFailedPowerSupplyMessage'}};
    } else {
      print("Unable to get count of failed power supplies and failed power message for netapp $host: ".$snmpsession->error()."\n");
      return;
    }

    # determine color
    my $worstcolor = ($powercount > 0) ? 'red' : 'green';

    # Compose message
    $message = "status $commahost.$test $worstcolor ".scalar(localtime)."\n\n";
    $message .= "$powermessage\n";

  } else {
    print(localtime().": ".(caller(0))[3].": Unknown enterprise '$enterprise' for host '$host'\n");
    return;
  } # # if ($enterprise eq 'cisco')

  return $message;
} # sub power_message

##### TEMPERATURE #####


#----
# &temperature_message($host,$snmpsession,$enterprise,$version)
#
# Gather information on temperature statuses, determine color, and create a message.
#----

sub temperature_message {
  my($host,$snmpsession,$enterprise,$version) = @_;
  (my $commahost = $host) =~ s/\./\,/g;
  my $message = '';
  my $test = 'temperature';

  if ($enterprise eq 'apc') {
    # For now, output enough temperature data to graph it.
    # Don't worry about status for now.
    my $result = undef;
    unless ($result = $snmpsession->get_request(-varbindlist => [
                                  $snmpoids{'upsAdvBatteryTemperature'}])) {
      print(localtime().": ".(caller(0))[3].": Unable to check temperature for APC $host: ".$snmpsession->error()."\n");
      return;
    }
    my $temperature = $result->{'upsAdvBatteryTemperature'};
    my $worstcolor = 'green';

    $message = "status $commahost.$test $worstcolor ".scalar(localtime)."\n\n";
    $message .= '&'.$worstcolor." UPS Temperature: $temperature &deg C\n";

  } elsif ($enterprise eq 'cisco') {
    my $srdescr = quotemeta($snmpoids{'ciscoEnvMonTemperatureStatusDescr'});
    my $srvalue = quotemeta($snmpoids{'ciscoEnvMonTemperatureStatusValue'});
    my $srthresh = quotemeta($snmpoids{'ciscoEnvMonTemperatureThreshold'});
    my $srstate = quotemeta($snmpoids{'ciscoEnvMonTemperatureState'});

    # Walk the tree from ciscoEnvMonTemperatureStatusEntry and
    # get everything at once.
    my %descrdb = ();
    my %valuedb = ();
    my %threshdb = ();
    my %statedb = ();
    if (my $result = $snmpsession->get_table(-baseoid => $snmpoids{'ciscoEnvMonTemperatureStatusEntry'})) {
      while (my($key,$value) = each(%$result)) {
        if ($key =~ /^$srdescr\.(\d+)$/) {
          my $index = $1;
          $descrdb{$index} = $value;
        } elsif ($key =~ /^$srvalue\.(\d+)$/) {
          my $index = $1;
          $valuedb{$index} = $value;
        } elsif ($key =~ /^$srthresh\.(\d+)$/) {
          my $index = $1;
          $threshdb{$index} = $value;
        } elsif ($key =~ /^$srstate\.(\d+)$/) {
          my $index = $1;
          $statedb{$index} = $snmpsyntaxdb{'ciscoEnvMonTemperatureState'}{$value};
        }
      } # while (my($key,$value) = each(%$result))
    } else {
      print(localtime().": ".(caller(0))[3].": Unable to get temperature information for Cisco $host: ".$snmpsession->error()."\n");
      return;
    }

    # Determine colors
    my %colordb = ();
    while (my($index,$state) = each(%statedb)) {
      if ($threshdb{$index} == 0) {
        $colordb{$index} = 'clear';
      } elsif ($state eq 'normal') {
        $colordb{$index} = 'green';
      } elsif ($state eq 'warning') {
        $colordb{$index} = 'yellow';
      } else {
        $colordb{$index} = 'red';
      }
    } # while (my($index,$state) = each(%statedb))

    # find worst color
    my $worstcolor = &color_compare(values(%colordb));

    # compose message
    $message = "status $commahost.$test $worstcolor ".scalar(localtime)."\n\n";
    my @indices = sort { $a <=> $b } keys(%statedb);
    foreach my $index (@indices) {
      $message .= '&'.$colordb{$index}."  $descrdb{$index}: $valuedb{$index} &deg C, Threshold: $threshdb{$index} &deg C, State: ".$statedb{$index}."\n";
    } # foreach my $index (@indices)

  } elsif ($enterprise eq 'compaq') {
    # Use the following OIDs to grab temperature data and status
    # in a way that can be graphed by Butter and Larrd if possible
    # cpqHeThermalCondition
    # cpqHeThermalTempStatus
    # cpqHeTemperatureIndex
    # cpqHeTemperatureLocale
    # cpqHeTemperatureCelsius
    # cpqHeTemperatureThreshold
    # cpqHeTemperatureCondition
    my $thermcond = '';
    my $thermtempstat = '';
    if (my $result = $snmpsession->get_request(-varbindlist => [
                                              $snmpoids{'cpqHeThermalCondition'},
                                              $snmpoids{'cpqHeThermalTempStatus'}])) {
      $thermcond = $result->{$snmpoids{'cpqHeThermalCondition'}};
      $thermcond = $snmpsyntaxdb{'cpqHeThermalCondition'}{$thermcond};
      $thermtempstat = $result->{$snmpoids{'cpqHeThermalTempStatus'}};
      $thermtempstat = $snmpsyntaxdb{'cpqHeThermalTempStatus'}{$thermtempstat};
    } else {
      print(localtime().": ".(caller(0))[3].": Unable to get Thermal info for Compaq $host: ".$snmpsession->error()."\n");
      return;
    }
    my $thermcondcolor = ($thermcond eq 'ok') ? 'green' : 'red';
    my $thermtempstatcolor = ($thermtempstat eq 'failed') ? 'red' :
                             ($thermtempstat eq 'degraded') ? 'yellow' : 'green';

    my %locdb = ();
    my %celsdb = ();
    my %threshdb = ();
    my %conddb = ();
    my %tempcolordb = ();
    my %threshcoldb = ();
    if (my $result = $snmpsession->get_table(-baseoid => $snmpoids{'cpqHeTemperatureLocale'})) {
      while (my($key,$value) = each(%$result)) {
        (my $index = $key) =~ s/.*\.//g;
        $locdb{$index} = $snmpsyntaxdb{'cpqHeTemperatureLocale'}{$value};
      }
    } else {
      print(localtime().": ".(caller(0))[3].": Unable to get Temperature Locale for Compaq $host: ".$snmpsession->error()."\n");
      return;
    }

    if (scalar(keys(%locdb)) > 0) { # Don't bother with the others if I didn't get anything for locale
      if (my $result = $snmpsession->get_table(-baseoid => $snmpoids{'cpqHeTemperatureCelsius'})) {
        while (my($key,$value) = each(%$result)) {
          (my $index = $key) =~ s/.*\.//g;
          $celsdb{$index} = $value;
        }
      } else {
        print(localtime().": ".(caller(0))[3].": Unable to get Temperature in Celsius for Compaq $host: ".$snmpsession->error()."\n");
        return;
      }
 
      if (my $result = $snmpsession->get_table(-baseoid => $snmpoids{'cpqHeTemperatureThreshold'})) {
        while (my($key,$value) = each(%$result)) {
          (my $index = $key) =~ s/.*\.//g;
          $threshdb{$index} = $value;
          my $pct = ( $celsdb{$index} / $threshdb{$index} ) * 100;
          if ($pct > $compaq_pct_temp_panic) {
            $threshcoldb{$index} = 'red';
          } elsif ($pct > $compaq_pct_temp_warn) {
            $threshcoldb{$index} = 'yellow';
          } else {
            $threshcoldb{$index} = 'green';
          }
        }
      } else {
        print(localtime().": ".(caller(0))[3].": Unable to get cpqHeTemperatureThreshold for Compaq $host: ".$snmpsession->error()."\n");
        return;
      }
 
      if (my $result = $snmpsession->get_table(-baseoid => $snmpoids{'cpqHeTemperatureCondition'})) {
        while (my($key,$value) = each(%$result)) {
          (my $index = $key) =~ s/.*\.//g;
          $conddb{$index} = $snmpsyntaxdb{'cpqHeTemperatureCondition'}{$value};
          if ($conddb{$index} eq 'ok') {
            $tempcolordb{$index} = 'green';
          } elsif ($conddb{$index} eq 'degraded') {
            $tempcolordb{$index} = 'yellow';
          } else {
            $tempcolordb{$index} = 'red';
          }
        }
      } else {
        print(localtime().": ".(caller(0))[3].": Unable to get cpqHeTemperatureCondition for Compaq $host: ".$snmpsession->error()."\n");
        return;
      }
    } # if (scalar(keys(%locdb)) > 0)

    my $worstcolor = &color_compare($thermcondcolor,$thermtempstatcolor,values(%tempcolordb),values(%threshcoldb));

    # compose message
    $message = "status $commahost.$test $worstcolor ".scalar(localtime)."\n\n";
    # General Status: cpqHeThermalCondition and cpqHeThermalTempStatus
    $message .= '&'."$thermcondcolor Status of the system's thermal environment: $thermcond\n";
    $message .= '&'."$thermcondcolor Status of the system's temperature sensors: $thermtempstat\n";

    # Specific statuses, formatted so they can be graphed
    # Use the colors determined by threshold but also show the Condition colors
    my @indices = sort { $a <=> $b } keys(%locdb);
    foreach my $index (@indices) {
      $message .= '&'.$threshcoldb{$index}." $locdb{$index}: $celsdb{$index} &deg C, Threshold: $threshdb{$index} &deg C, Condition: &".$tempcolordb{$index}." $conddb{$index}\n";
    }

  } elsif ($enterprise eq 'foundry') {
    # Warning: some versions report in 0.5 degrees units, some in 1.0 degrees
    # Borderline seems to be 7.2: Those and above use 1.0
    # snChasActualTemperature/.1.3.6.1.4.1.1991.1.1.1.1.18.0
    # snChasWarningTemperature/.1.3.6.1.4.1.1991.1.1.1.1.19.0
    # snChasShutdownTemperature/.1.3.6.1.4.1.1991.1.1.1.1.20.0
    my $actual = 0;
    my $warning = 0;
    my $shutdown = 0;
    if (my $result = $snmpsession->get_request(-varbindlist => [$snmpoids{'snChasActualTemperature'},
                                                                $snmpoids{'snChasWarningTemperature'},
                                                                $snmpoids{'snChasShutdownTemperature'}])) {
      $actual = $result->{$snmpoids{'snChasActualTemperature'}};
      $warning = $result->{$snmpoids{'snChasWarningTemperature'}};
      $shutdown = $result->{$snmpoids{'snChasShutdownTemperature'}};
    } else {
      print(localtime().": ".(caller(0))[3].": Unable to get temperature info for Foundry $host: ".$snmpsession->error()."\n");
      return;
    }

    # Determine color
    my $worstcolor = 'green';
    if ($actual >= $shutdown) {
      $worstcolor = 'red';
    } elsif ($actual >= $warning) {
      $worstcolor = 'yellow';
    }

    # Determine whether I'm dealing with units of 0.5 or 1.0 degrees.
    if ($version < 7.2) {
    # if ($warning > 62) # another way to do it
      $actual = $actual / 2;
      $warning = $warning / 2;
      $shutdown = $shutdown / 2;
    }

    # compose message
    $message = "status $commahost.$test $worstcolor ".scalar(localtime)."\n\n";
    $message .= '&'.$worstcolor." Actual Temperature: $actual &deg C\n";
    $message .= "&yellow Warning Temperature: $warning &deg C\n";
    $message .= "&red Shutdown Temperature: $shutdown &deg C\n";

  } elsif ($enterprise eq 'netapp') {
    # Only a simple yes/no check
    # envOverTemperature/.1.3.6.1.4.1.789.1.2.4.1.0
    # my $envOverTemperature = '.1.3.6.1.4.1.789.1.2.4.1.0';
    #  no(1)
    #  yes(2)
 
    my $amihotornot = undef;
    if (my $result = $snmpsession->get_request(-varbindlist => [$snmpoids{'envOverTemperature'}])) {
      $amihotornot = $snmpsyntaxdb{'envOverTemperature'}{$result->{$snmpoids{'envOverTemperature'}}};
    } else {
      print(localtime().": ".(caller(0))[3].": Unable to check temperature for netapp $host: ".$snmpsession->error()."\n");
      return;
    }

    my $worstcolor = ($amihotornot eq 'yes') ? 'red' : 'green';
    $message = "status $commahost.$test $worstcolor ".scalar(localtime)."\n\n";
    if ($amihotornot == 2) {
      $message .= "The Network Appliance is currently operating\n";
      $message .= "above its maximum operating temperature.\n";
    } else {
      $message .= "The Network Appliance is not currently operating\n";
      $message .= "above its maximum operating temperature.\n";
    }

  } else {
    print(localtime().": ".(caller(0))[3].": Unknown enterprise '$enterprise' for host '$host'\n");
    return;
  }

  return $message;  
} # sub temperature_message



##### UPTIME #####

#----
# &uptime_message($host,$snmpsession,$enterprise,$version,$uptime);
#
# Check uptime and compose into a message.
# Simplified version of uptime_message for machines whose CPU isn't monitored.
#----

sub uptime_message {
  my($host,$snmpsession,$enterprise,$version,$uptime) = @_;

  print(localtime().": ".(caller(0))[3].": Checking host '$host'\n") if $debug;
  if (! defined($uptime)) {
    print(localtime().": ".(caller(0))[3].": Uptme for '$host' is undefined\n");
    return;
  }
  if ($uptime == 0) {
    print(localtime().": ".(caller(0))[3].": Uptme for '$host' is zero\n");
    return;
  }

  # Check uptime, hundredths of seconds
  my $uptimecolor = 'green';
  if ((($uptime / 100) / 60) < $ENV{'WARNMINSONREBOOT'}) {
    $uptimecolor = $ENV{'WARNCOLORONREBOOT'};
  }

  # Create message
  my $message = '';
  my $test = 'uptime';

  (my $commahost = $host) =~ s/\./\,/g;
  my $upword = '';
  if (($uptime / 100) < 120) {
    $upword = sprintf("%3.2f seconds",$uptime / 100);
  } elsif ((($uptime / 100) / 60) < 120) {
    $upword = sprintf("%3.2f minutes",(($uptime / 100) / 60));
  } elsif (((($uptime / 100) / 60 ) / 60) < 48) {
    $upword = sprintf("%3.2f hours",((($uptime / 100) / 60) / 60));
  } else {
    $upword = sprintf("%3.2f days",(((($uptime / 100) / 60) / 60) / 24));
  }

  $message = "status $commahost.$test $uptimecolor ".scalar(localtime)." up: $upword";

  $message .= "\n\n";
  if ((($uptime / 100) / 60) < $ENV{'WARNMINSONREBOOT'}) {
    $message .= "Warning: Machine recently rebooted\n\n";
  } else {
    $message .= "Machine not recently rebooted\n\n";
  }

  return $message;
} # sub uptime_message



##### DISK #####

#----
# &disk_message($host,$snmpsession,$enterprise,$version);
#
# Check disk usage and create a text output table as close to the
# original bb-disk.sh as possible.
# TODO: Sneak in inode checks too where possible.
#       It would be a shame to gather disk names twice.
#----

sub disk_message {
  my($host,$snmpsession,$enterprise,$version) = @_;
  (my $commahost = $host) =~ s/\./\,/g;
  my $test = 'disk';

  print(localtime().": ".(caller(0))[3].": Checking host '$host'\n") if $debug;
  my $message = '';

  my %disknamedb = ();
  my %disktotaldb = ();
  my %diskuseddb = ();
  my %diskfreedb = ();
  my %diskpctdb = ();
  my %diskmntdb = ();

  # Sneak in inode checks where possible
  my $is_inodecapable = 0;
  my %inodetotaldb = ();
  my %inodeuseddb = ();
  my %inodefreedb = ();
  my %inodepctdb = ();

  # Regardless of the system, I need:
  # Filesystem  1k-blocks(Total)  Used   Available   Use%   Mounted on
  # Mounted On can be same as FileSystem if Mounted On is not available
  if ($enterprise eq 'netapp') {
    $is_inodecapable = 1;

    # Get disk names, skip over "snapshots"
    # Skip trailing slashes to make it more like Unix
    if (my $result = $snmpsession->get_table(-baseoid => $snmpoids{'dfFileSys'})) {
      while (my($key,$value) = each(%$result)) {
        next if $value =~ /\.snapshot$/;
        (my $index = $key) =~ s/^.*\.//;
        $value =~ s/\/$//;
        $disknamedb{$index} = $value;
      } # while (($key,$value) = each(%$result))
    } else {
      print(localtime().": ".(caller(0))[3].": Unable to get dfFileSys for Netapp $host: ".$snmpsession->error()."\n");
      return;
    }

    # Get totals
    my @varbindlist = map { $_ = $snmpoids{'dfKBytesTotal'}.'.'.$_ } keys(%disknamedb);
    if (my $result = $snmpsession->get_request(-varbindlist => \@varbindlist)) {
      while (my($key,$value) = each(%$result)) {
        (my $index = $key) =~ s/^.*\.//;
        $disktotaldb{$index} = $value;
      } # while (($key,$value) = each(%$result))
    } else {
      print(localtime().": ".(caller(0))[3].": Unable to get dfKBytesTotal for Netapp $host: ".$snmpsession->error()."\n");
      return;
    }

    # Get used
    @varbindlist = map { $_ = $snmpoids{'dfKBytesUsed'}.'.'.$_ } keys(%disknamedb);
    if (my $result = $snmpsession->get_request(-varbindlist => \@varbindlist)) {
      while (my($key,$value) = each(%$result)) {
        (my $index = $key) =~ s/^.*\.//;
        $diskuseddb{$index} = $value;
      } # while (($key,$value) = each(%$result))
    } else {
      print(localtime().": ".(caller(0))[3].": Unable to get dfKBytesUsed for Netapp $host: ".$snmpsession->error()."\n");
      return;
    }

    # Get free
    @varbindlist = map { $_ = $snmpoids{'dfKBytesAvail'}.'.'.$_ } keys(%disknamedb);
    if (my $result = $snmpsession->get_request(-varbindlist => \@varbindlist)) {
      while (my($key,$value) = each(%$result)) {
        (my $index = $key) =~ s/^.*\.//;
        $diskfreedb{$index} = $value;
      } # while (($key,$value) = each(%$result))
    } else {
      print(localtime().": ".(caller(0))[3].": Unable to get dfKBytesAvail for Netapp $host: ".$snmpsession->error()."\n");
      return;
    }

    # Get MountedOn, same as the Name on the systems we have but you never know
    @varbindlist = map { $_ = $snmpoids{'dfMountedOn'}.'.'.$_ } keys(%disknamedb);
    if (my $result = $snmpsession->get_request(-varbindlist => \@varbindlist)) {
      while (my($key,$value) = each(%$result)) {
        (my $index = $key) =~ s/^.*\.//;
        $value =~ s/\/$//;
        $diskmntdb{$index} = $value;
      } # while (($key,$value) = each(%$result))
    } else {
      print(localtime().": ".(caller(0))[3].": Unable to get dfMountedOn for Netapp $host: ".$snmpsession->error()."\n");
      return;
    }

    # Get pct
    @varbindlist = map { $_ = $snmpoids{'dfPerCentKBytesCapacity'}.'.'.$_ } keys(%disknamedb);
    if (my $result = $snmpsession->get_request(-varbindlist => \@varbindlist)) {
      while (my($key,$value) = each(%$result)) {
        (my $index = $key) =~ s/^.*\.//;
        $diskpctdb{$index} = $value;
      } # while (($key,$value) = each(%$result))
    } else {
      print(localtime().": ".(caller(0))[3].": Unable to get dfPerCentKBytesCapacity for Netapp $host: ".$snmpsession->error()."\n");
      return;
    }

    # Get Inodes used
    @varbindlist = map { $_ = $snmpoids{'dfInodesUsed'}.'.'.$_ } keys(%disknamedb);
    if (my $result = $snmpsession->get_request(-varbindlist => \@varbindlist)) {
      while (my($key,$value) = each(%$result)) {
        (my $index = $key) =~ s/^.*\.//;
        $inodeuseddb{$index} = $value;
      } # while (($key,$value) = each(%$result))
    } else {
      print(localtime().": ".(caller(0))[3].": Unable to get dfInodesUsed for Netapp $host: ".$snmpsession->error()."\n");
      return;
    }

    # Get Inodes free
    @varbindlist = map { $_ = $snmpoids{'dfInodesFree'}.'.'.$_ } keys(%disknamedb);
    if (my $result = $snmpsession->get_request(-varbindlist => \@varbindlist)) {
      while (my($key,$value) = each(%$result)) {
        (my $index = $key) =~ s/^.*\.//;
        $inodefreedb{$index} = $value;
      } # while (($key,$value) = each(%$result))
    } else {
      print(localtime().": ".(caller(0))[3].": Unable to get dfInodesFree for Netapp $host: ".$snmpsession->error()."\n");
      return;
    }

    # Calculate Inode totals
    while (my($index,$used) = each(%inodeuseddb)) {
      my $free = $inodefreedb{$index};
      my $total = $used + $free;
      $inodetotaldb{$index} = $total;
    } # while

    # Get Inodes percentage
    @varbindlist = map { $_ = $snmpoids{'dfPerCentInodeCapacity'}.'.'.$_ } keys(%disknamedb);
    if (my $result = $snmpsession->get_request(-varbindlist => \@varbindlist)) {
      while (my($key,$value) = each(%$result)) {
        (my $index = $key) =~ s/^.*\.//;
        $inodepctdb{$index} = $value;
      } # while (($key,$value) = each(%$result))
    } else {
      print(localtime().": ".(caller(0))[3].": Unable to get dfPerCentInodeCapacity for Netapp $host: ".$snmpsession->error()."\n");
      return;
    }

  } else {
    print(localtime().": ".(caller(0))[3].": Unknown enterprise '$enterprise' for host '$host'\n");
    return;
  }

  my $worstcolor = 'green';
  # This is where I insert any messages about specific partitions going over their limits
  my @indices = sort { $a <=> $b } keys(%disknamedb);
  foreach my $index (@indices) {
    my $yellowlimit = $ENV{'DFWARN'};
    my $redlimit = $ENV{'DFPANIC'};
    if ((exists($diskthreshdb_ref->{$host})) &&
        (defined($diskthreshdb_ref->{$host})) &&
        (exists($diskthreshdb_ref->{$host}{$diskmntdb{$index}})) &&
        (defined($diskthreshdb_ref->{$host}{$diskmntdb{$index}}))) {
      $redlimit = $diskthreshdb_ref->{$host}{$diskmntdb{$index}}{'red'};
      $yellowlimit = $diskthreshdb_ref->{$host}{$diskmntdb{$index}}{'yellow'};
    }
    if ($diskpctdb{$index} >= $redlimit) {
      $message .= "&red $diskmntdb{$index} ($diskpctdb{$index}%) has reached the defined PANIC level ($redlimit%)\n\n";
      $worstcolor = 'red';
    } elsif ($diskpctdb{$index} >= $yellowlimit) {
      $message .= "&yellow $diskmntdb{$index} ($diskpctdb{$index}) has reached the defined WARNING level ($yellowlimit%)\n\n";
      $worstcolor = 'yellow' unless $worstcolor eq 'red';
    }
  }

  my @colheaders = ('Filesystem','1k-blocks','Used','Available','Use%','Mounted On');
  # How to get the length of the longest value in a hash?
  # length((sort { $b <=> $a } values())[0]);
  # my %disknamedb = ();
  # my %disktotaldb = ();
  # my %diskuseddb = ();
  # my %diskfreedb = ();
  # my %diskpctdb = ();
  # my %diskmntdb = ();
  my @colwidths = (
    length((sort { length($b) <=> length($a) } (values(%disknamedb),$colheaders[0]))[0]),
    length((sort { length($b) <=> length($a) } (values(%disktotaldb),$colheaders[1]))[0]),
    length((sort { length($b) <=> length($a) } (values(%diskuseddb),$colheaders[2]))[0]),
    length((sort { length($b) <=> length($a) } (values(%diskfreedb),$colheaders[3]))[0]),
    length((sort { length($b) <=> length($a) } (values(%diskpctdb),$colheaders[4]))[0]),
  );

  my $pad = '';
  for (my $n = 0; $n <= 4; $n++) {
    $pad = ' ' x (($colwidths[$n] - length($colheaders[$n])) + 1);
    if ($n == 0) {
      $message .= $colheaders[$n].$pad;
    } else {
      $message .= $pad.$colheaders[$n];
    }
  }

  $message .= ' '.$colheaders[5]."\n";
  foreach my $index (keys(%disknamedb)) {
    $pad = ' ' x (($colwidths[0] - length($disknamedb{$index})) + 1);
    $message .= $disknamedb{$index}.$pad;

    $pad = ' ' x (($colwidths[1] - length($disktotaldb{$index})) + 1);
    $message .= $pad.$disktotaldb{$index};

    $pad = ' ' x (($colwidths[2] - length($diskuseddb{$index})) + 1);
    $message .= $pad.$diskuseddb{$index};
    $pad = ' ' x (($colwidths[3] - length($diskfreedb{$index})) + 1);
    $message .= $pad.$diskfreedb{$index};

    $pad = ' ' x ($colwidths[4] - length($diskpctdb{$index}));
    $message .= $pad.$diskpctdb{$index}.'%';

    $message .= ' '.$diskmntdb{$index};
    $message .= "\n";
  }

  $message = "status $commahost.$test $worstcolor ".scalar(localtime)."\n$message";

  if ($is_inodecapable) {
    # Sneak in an inode message.
    # And why aren't I just adding the message here instead of passing it
    # back up anyways?
    my $message = ''; # Different namespace
    my $test = 'inode';
    my $worstcolor = 'green';

    # This is where I insert any messages about specific partitions going over their limits
    my @indices = sort { $a <=> $b } keys(%disknamedb);
    foreach my $index (@indices) {
      my $yellowlimit = $ENV{'DFWARN'};
      my $redlimit = $ENV{'DFPANIC'};
      if ((exists($diskthreshdb_ref->{$host})) &&
          (defined($diskthreshdb_ref->{$host})) &&
          (exists($diskthreshdb_ref->{$host}{$diskmntdb{$index}})) &&
          (defined($diskthreshdb_ref->{$host}{$diskmntdb{$index}}))) {
        $redlimit = $diskthreshdb_ref->{$host}{$diskmntdb{$index}}{'red'};
        $yellowlimit = $diskthreshdb_ref->{$host}{$diskmntdb{$index}}{'yellow'};
      }
      if ($inodepctdb{$index} >= $redlimit) {
        $message .= "&red $diskmntdb{$index} ($inodepctdb{$index}%) has reached the defined PANIC level ($redlimit%)\n\n";
        $worstcolor = 'red';
      } elsif ($inodepctdb{$index} >= $yellowlimit) {
        $message .= "&yellow $diskmntdb{$index} ($inodepctdb{$index}) has reached the defined WARNING level ($yellowlimit%)\n\n";
        $worstcolor = 'yellow' unless $worstcolor eq 'red';
      }
    } # foreach my $index (@indices)

    my @colheaders = ('Filesystem','Inodes','IUsed','IFree','IUse%','Mounted On');

    my @colwidths = (
      length((sort { length($b) <=> length($a) } (values(%disknamedb),$colheaders[0]))[0]),
      length((sort { length($b) <=> length($a) } (values(%inodetotaldb),$colheaders[1]))[0]),
      length((sort { length($b) <=> length($a) } (values(%inodeuseddb),$colheaders[2]))[0]),
      length((sort { length($b) <=> length($a) } (values(%inodefreedb),$colheaders[3]))[0]),
      length((sort { length($b) <=> length($a) } (values(%inodepctdb),$colheaders[4]))[0]),
    );

    my $pad = '';
    for (my $n = 0; $n <= 4; $n++) {
      $pad = ' ' x (($colwidths[$n] - length($colheaders[$n])) + 1);
      if ($n == 0) {
        $message .= $colheaders[$n].$pad;
      } else {
        $message .= $pad.$colheaders[$n];
      }
    }

    $message .= ' '.$colheaders[5]."\n";
    foreach my $index (keys(%disknamedb)) {
      $pad = ' ' x (($colwidths[0] - length($disknamedb{$index})) + 1);
      $message .= $disknamedb{$index}.$pad;
 
      $pad = ' ' x (($colwidths[1] - length($inodetotaldb{$index})) + 1);
      $message .= $pad.$inodetotaldb{$index};
 
      $pad = ' ' x (($colwidths[2] - length($inodeuseddb{$index})) + 1);
      $message .= $pad.$inodeuseddb{$index};
      $pad = ' ' x (($colwidths[3] - length($inodefreedb{$index})) + 1);
      $message .= $pad.$inodefreedb{$index};
 
      $pad = ' ' x ($colwidths[4] - length($inodepctdb{$index}));
      $message .= $pad.$inodepctdb{$index}.'%';
 
      $message .= ' '.$diskmntdb{$index};
      $message .= "\n";
    }
 
    $message = "status $commahost.$test $worstcolor ".scalar(localtime)."\n$message";

    $message .= "\n<!-- Enterprise: $enterprise , Version: $version -->\n";
    # Add version credit
    $message .= (caller(0))[3].", $Script Version: $VERSION\n";

    # Add to combo message
    if ($is_commandline) {
      print("$ENV{'BBHOME'}/bin/bb-combo.sh add \"$message\"\n");
    } else {
      system("$ENV{'BBHOME'}/bin/bb-combo.sh add \"$message\"");
    }
  } # if ($is_inodecapable)

  return $message;
} # sub disk_message


##### MEMORY #####

#----
# &memory_message($host,$snmpsession,$enterprise,$version);
#
# Check memory usage and create a text output table as close to the
# original bb-memory.sh as possible.
# Need to be more versatile with names of different types of memory
# and with graphing, make it more like disk checks in its logs and graphing.
#----

sub memory_message {
  my($host,$snmpsession,$enterprise,$version) = @_;
  (my $commahost = $host) =~ s/\./\,/g;
  my $test = 'memory';

  print(localtime().": ".(caller(0))[3].": Checking host '$host'\n") if $debug;
  my $message = '';

  # Some of these will be calculated as necessary
  # Keys are ostensibly index numbers but you can get away with names.
  # Numbers given in KB.
  my %memorynamedb = ();
  my %memorytotaldb = ();
  my %memoryuseddb = ();
  my %memoryfreedb = ();
  my %memorypctdb = ();
  my %memorycolordb = (); # All green for now

  if ($enterprise eq 'cisco') {
    # Walk the names
    if (my $result = $snmpsession->get_table(-baseoid => $snmpoids{'ciscoMemoryPoolName'})) {
      while (my($key,$name) = each(%$result)) {
        (my $index = $key) =~ s/^.*\.//; # Remove all but the last number
        $memorynamedb{$index} = $name;
      }
    } else {
      print(localtime().": ".(caller(0))[3].": Unable to get ciscoMemoryPoolName for $enterprise $host: ".$snmpsession->error()."\n");
      return;
    }

    # Walk the used numbers
    if (my $result = $snmpsession->get_table(-baseoid => $snmpoids{'ciscoMemoryPoolUsed'})) {
      while (my($key,$used) = each(%$result)) {
        (my $index = $key) =~ s/^.*\.//; # Remove all but the last number
          # $memoryuseddb{$index} = int($used/1024); # Cisco returns bytes
          $memoryuseddb{$index} = $used; # Cisco returns bytes, but so does Foundry
      }
    } else {
      print(localtime().": ".(caller(0))[3].": Unable to get ciscoMemoryPoolUsed for $enterprise $host: ".$snmpsession->error()."\n");
      return;
    }

    # Walk the free numbers
    if (my $result = $snmpsession->get_table(-baseoid => $snmpoids{'ciscoMemoryPoolFree'})) {
      while (my($key,$free) = each(%$result)) {
        (my $index = $key) =~ s/^.*\.//; # Remove all but the last number
          # $memoryfreedb{$index} = int($free/1024); # Cisco returns bytes
          $memoryfreedb{$index} = $free; # Cisco returns bytes, but so does Foundry
      }
    } else {
      print(localtime().": ".(caller(0))[3].": Unable to get ciscoMemoryPoolFree for $enterprise $host: ".$snmpsession->error()."\n");
      return;
    }

    # Calculate totals and percentages
    while (my($index,$used) = each(%memoryuseddb)) {
      my $free = $memoryfreedb{$index};
      my $total = $used + $free;
      my $percent = sprintf("%.2f",100*($used/$total)); # Should be using sprintf for tables
      $memorytotaldb{$index} = $total;
      $memorypctdb{$index} = $percent;
      $memorycolordb{$index} = 'green';
    }

  } elsif ($enterprise eq 'foundry') {
    # All memory named as 'System'
    # Get snAgGblDynMemUtil
    # Get snAgGblDynMemTotal
    # Get snAgGblDynMemFree
    if (my $result = $snmpsession->get_request(-varbindlist => [$snmpoids{'snAgGblDynMemUtil'},
                                                                $snmpoids{'snAgGblDynMemTotal'},
                                                                $snmpoids{'snAgGblDynMemFree'}])) {
      my $free = $result->{$snmpoids{'snAgGblDynMemFree'}};
      my $total = $result->{$snmpoids{'snAgGblDynMemTotal'}};
      # my $percent = $result->{$snmpoids{'snAgGblDynMemUtil'}};
      my $percent = sprintf("%.2f",100*(($total - $free)/$total));
      my $used = $total - $free;
      $memorynamedb{1} = 'System';
      $memorytotaldb{1} = $total;
      $memoryuseddb{1} = $used;
      $memoryfreedb{1} = $free;
      $memorypctdb{1} = $percent;
      $memorycolordb{1} = 'green';
    } else {
      print(localtime().": ".(caller(0))[3].": Unable to get memory for $enterprise $host: ".$snmpsession->error()."\n");
      return;
    }

  } else {
    print(localtime().": ".(caller(0))[3].": Unknown enterprise '$enterprise' for host '$host'\n");
    return;
  } # if ($enterprise eq ...)
 
  my @colheaders = ('Memory','Used','Total','Percentage');
  my $worstcolor = 'green';
  # And will stay green until I figure out how to configure thresholds

  my @colwidths = (
    2 + length((sort { length($b) <=> length($a) } (values(%memorynamedb),$colheaders[0]))[0]),
    2 + length((sort { length($b) <=> length($a) } (values(%memoryuseddb),$colheaders[1]))[0]),
    2 + length((sort { length($b) <=> length($a) } (values(%memorytotaldb),$colheaders[2]))[0]),
    2 + length((sort { length($b) <=> length($a) } (values(%memorypctdb),$colheaders[3]))[0]),
  );

  $message .= sprintf("   %$colwidths[0]s %$colwidths[1]s %$colwidths[2]s %$colwidths[3]s\n",@colheaders);
  my @indlist = sort(keys(%memorynamedb));
  foreach my $index (@indlist) {
    $message .= sprintf("&%s %$colwidths[0]s %$colwidths[1]s %$colwidths[2]s %$colwidths[3]s\n",$memorycolordb{$index},$memorynamedb{$index},$memoryuseddb{$index},$memorytotaldb{$index},$memorypctdb{$index}.'%');
  }

  # Add a header
  $message = "status $commahost.$test $worstcolor ".scalar(localtime)."\n$message\n\nStatus always green until I figure out a way to store thresholds.";

  return $message;
} # memory_message

##### COMMON UTILITIES #####

#----
# &env_check()
#
# Check the environment
#----

sub env_check {
  unless ((exists($ENV{'BBTMP'})) && (defined($ENV{'BBTMP'}))) {
    $is_commandline = 1;
    $ENV{'BBTMP'} = $defbbtmp;
  }
  unless (-d $ENV{'BBTMP'}) {
    print(localtime().": ".(caller(0))[3].": BBTMP directory $ENV{'BBTMP'} doesn't exist\n");
    return;
  }
  unless (-w $ENV{'BBTMP'}) {
    print(localtime().": ".(caller(0))[3].": BBTMP directory $ENV{'BBTMP'} isn't writable\n");
    return;
  }
  unless ((exists($ENV{'BBHOME'})) && (defined($ENV{'BBHOME'}))) {
    $is_commandline = 1;
    $ENV{'BBHOME'} = $defbbhome;
  }
  unless (-d $ENV{'BBHOME'}) {
    print(localtime().": ".(caller(0))[3].": BBHOME directory $ENV{'BBHOME'} doesn't exist\n");
    return;
  }
  unless ((exists($ENV{'BB'})) && (defined($ENV{'BB'}))) {
    $is_commandline = 1;
    $ENV{'BB'} = $defbb;
  }
  unless (-x $ENV{'BB'}) {
    print(localtime().": ".(caller(0))[3].": BB command $ENV{'BB'} isn't executable\n");
    return;
  }
  unless ((exists($ENV{'BBDISP'})) && (defined($ENV{'BBDISP'}))) {
    $is_commandline = 1;
    $ENV{'BBDISP'} = $defbbdisp;
    $ENV{'MACHIP'} = $defbbdisp;
    $ENV{'MACHINE'} = $defmachine;
    $ENV{'MACHINEDOTS'} = $defmachine;
  }
  if ($is_commandline) {
    $ENV{'BBDISPLAY'} = 'TRUE';
    $ENV{'BBNET'} = 'TRUE';
  }
  return 1;
}



#----
# &color_compare(@colors);
#
# Look at a list of colors and compile them into one:
# If any are red, return red.
# If all are clear, return clear.
# If any are yellow, return yellow.
# Else, return green.
#----

sub color_compare {
  my(@colors) = @_;

# If any are red, return red.
  foreach my $color (@colors) {
    if ($color eq 'red') { return 'red' }
  }

# If all are clear, return clear.
  my $all_clear = 1;
  foreach my $color (@colors) {
    if ($color ne 'clear') {
      $all_clear = 0;
      last;
    }
  }
  if ($all_clear) {
    return 'clear';
  }
 
# If any are yellow, return yellow.
  foreach my $color (@colors) {
    if ($color eq 'yellow') {
      return 'yellow';
    }
  }

# Else, return green.
  return 'green';

} # sub color_compare_nice


#----
# &color_compare_strict(@colors);
#
# Look at a list of colors and compile them into one:
# If any are red, return red.
# If all are clear, return clear.
# If all are yellow, return red.
# If any are yellow, return yellow.
# Else, return green.
#----

sub color_compare_strict {
  my(@colors) = @_;

# If any are red, return red.
  foreach my $color (@colors) {
    if ($color eq 'red') { return 'red' }
  }

# If all are clear, return clear.
  my $all_clear = 1;
  foreach my $color (@colors) {
    if ($color ne 'clear') {
      $all_clear = 0;
      last;
    }
  }
  if ($all_clear) {
    return 'clear';
  }

# If all are yellow, return red.
  my $all_yellow = 1;
  foreach my $color (@colors) {
    if ($color ne 'yellow') {
      $all_yellow = 0;
      last;
    }
  }
  if ($all_yellow) {
    return 'red';
  }

# If any are yellow, return yellow.
  foreach my $color (@colors) {
    if ($color eq 'yellow') {
      return 'yellow';
    }
  }

# Else, return green.
  return 'green';

} # sub color_compare_strict



#----
# &read_xsnmptab()
#
# Read the appropriate parts of bb-xsnmptab.
# Return a reference to a hash of hosts to tests and communities.
#----

sub read_xsnmptab {
  print("Reading $ENV{'BBHOME'}/etc/bb-xsnmptab for machine '".$ENV{'MACHINEDOTS'}."' machip '$ENV{'MACHIP'}' bbdisp '$ENV{'BBDISP'}'\n") if $debug;
  my $infh = new FileHandle;
  if ($infh->open("< $ENV{'BBHOME'}/etc/bb-xsnmptab")) {
    my %commhash = ();
    my %testhash = ();
    while (my $line = $infh->getline()) {
      chomp($line);
      next if $line =~ /^\s*\#/;
      next if $line !~ /\S/;
      if ($line =~ /^\s*(\S+)\s*\:\s*(\S+)\s*\:\s*([^\:]+)\s*\:\s*([^\:]+)/) {
        my($host,$community,$tests,$testhosts) = ($1,$2,$3,$4);
        my @testhostlist = split(/[\s\,\;]+/,$testhosts);
        my %testhostdb = map { $_ => 1 } @testhostlist;
        next unless exists($testhostdb{$ENV{'MACHINEDOTS'}});
        $commhash{$host} = $community;
        my @testlist = split(/[\s\,\;]+/,$tests);
        foreach my $test (@testlist) {
          $testhash{$host}{$test} = 1;
        }
        print("Found host '$host'\n") if $debug;
      } elsif ($line =~ /^\s*(\S+)\s*\:\s*(\S+)\s*\:\s*([^\:]+)/) {
        my($host,$community,$tests) = ($1,$2,$3);
        next unless $ENV{'BBNET'}; # If I am a BBNET machine
        $commhash{$host} = $community;
        my @testlist = split(/[\s\,\;]+/,$tests);
        foreach my $test (@testlist) {
          $testhash{$host}{$test} = 1;
        }
        print("Found host '$host'\n") if $debug;
      } # if ($line =~ /^\s*(\S+)\s*\:\s*(\S+)\s*\:\s*([^\:]+)\s*\:\s*([^\:]+)/)
    } # while (my $line = $infh->getline())
    $infh->close;
    return(\%commhash,\%testhash);
  } else {
    print("Unable to open $ENV{'BBHOME'}/etc/bb-xsnmptab: $!\n");
    return;
  } # if ($infh->open("< $ENV{'BBHOME'}/etc/bb-xsnmptab"))
} # sub read_xsnmptab



#----
# &detect_version($host,$snmpsession,$enterprise);
#
# Detect OS version.
# Return as a number with only one dot that can be used for comparisons
#----

sub detect_version {
  my($host,$snmpsession,$enterprise) = @_;

  my $version = 0;
  if ($enterprise eq 'netscreen') {
    if (my $result = $snmpsession->get_request(-varbindlist => [$snmpoids{'nsSetGenSwVer'}])) {
      my $descr = $result->{$snmpoids{'nsSetGenSwVer'}};
      if ($descr =~ /^(\d)\.(\S+) /) {
        my($major,$minor) = ($1,$2);
        $minor =~ s/\D//g;
        $version = "$major.$minor";
      }
    } else {
      print("Unable to get VersionTag for ucdavis host '$host'\n");
      return;
    }
  } elsif ($enterprise eq 'ucdavis') {
    if (my $result = $snmpsession->get_request(-varbindlist => [$snmpoids{'ucdVersionTag'}])) {
      my $descr = $result->{$snmpoids{'ucdVersionTag'}};
      if ($descr =~ /^(\d)\.(\S+)$/) {
        my($major,$minor) = ($1,$2);
        $minor =~ s/\D//g;
        $version = "$major.$minor";
      }
    } else {
      print("Unable to get VersionTag for ucdavis host '$host'\n");
      return;
    }
  } else {
    if (my $result = $snmpsession->get_request(-varbindlist => [$snmpoids{'sysDescr'}])) {
      my $descr = $result->{$snmpoids{'sysDescr'}};
      if ($enterprise eq 'cisco') {
        if ($descr =~ /Version (\d+)\.([^\,]+)\,/) {
          my($major,$minor) = ($1,$2);
          $minor =~ s/\D//g;
          $version = "$major.$minor";
        } else {
          print("Unable to find version for cisco '$host'\n");
          return;
        }
      } elsif ($enterprise eq 'foundry') {
        if ($descr =~ /are Version 0(\d+)\.(\S+) Compiled/) {
          my($major,$minor) = ($1,$2);
          $minor =~ s/\D//g;
          $version = "$major.$minor";
        } else {
          print("Failed to find version for foundry '$host' in description '$descr'\n");
          return;
        }
      } elsif ($enterprise eq 'netapp') {
        if ($descr =~ /Release (\d+)\.(\S+)\:/) {
          my($major,$minor) = ($1,$2);
          $minor =~ s/\D//g;
          $version = "$major.$minor";
        }
      } else {
        # print(localtime().": ".(caller(0))[3].": Unknown enterprise '$enterprise'\n");
        # return;
        $version = $descr;
      }
    } else {
      print("Unable to get sysDescr for host '$host'\n");
      return;
    }
  } # if ($enterprise eq 'netscreen')

  return $version;
} # sub detect_version



0
 
shanepresleyAuthor Commented:
Here's the results when I run this script, with one bad power supply.  Note the condition is GREEN (rtr.domain.com.power green) but should be RED, since three devices are RED

"
/usr/local/bb/bb/bin/bb-combo.sh add "status rtr.domain.com.power green Wed Apr 20 12:46:51 2005

&green Power Supply  1 1, source: ac, state: normal
&red Power Supply  2 2, source: ac, state: critical
&red Power Supply  3 3, source: unknown, state: notPresent
&red Power Supply  4 4, source: unknown, state: notPresent

<!-- Enterprise: cisco , Version: 12.3113 -->
0
 
jeopboyCommented:
his line seems to be the problem:

    my $worstcolor = ($fancount > 0) ? 'red' : 'green';

Is it supposed to sit inside the netapp enterprise if statement?

As it sits now, it is executed for all enterprises, but is $fancount isn't set in the cisco or foundry sections so $fancount <0 which sets $worstcolor to green.
0
Concerto's Cloud Advisory Services

Want to avoid the missteps to gaining all the benefits of the cloud? Learn more about the different assessment options from our Cloud Advisory team.

 
shanepresleyAuthor Commented:
Hmm, that sounds like it might be on the right track.  But the problem of the incorrect $worstcolor affects more than just fans?  But I tried commenting that line
out, and it failed with compile errors.

bb-xsnmp.pl Global symbol "$worstcolor" requires explicit package name at /usr/local/bb/bb/ext/bb-xsnmp.pl

So any suggestions what to change that line to?  I thought the problem might be in &color_compare, but my Perl knowledge is very limited :)

Thanks
0
 
jeopboyCommented:
I was wrong.  That worstcolor line *is* within the netapp function only so it's not the problem.

I'll look again at the color_compare routine when I get a chance...
0
 
shanepresleyAuthor Commented:
I think the problem is somewhere here...

    # determine colors
    my %powercolors = ();
    while (my($index,$state) = each(%statedb)) {
      if ($state eq 'normal') {
        $powercolors{$index} = 'green';
      } elsif ($state eq 'notPresent') {
        $powercolors{$index} = 'clear';
      } elsif ($state eq 'warning') {
        $powercolors{$index} = 'yellow';
      } else {
        $powercolors{$index} = 'red';
      }
    } # while (my($index,$state) = each(%statedb))

    # find worst color
    my $worstcolor = &color_compare(values(%statedb));


When I do some debug inside the  &color_compare routine, the values getting passed into it are "normal, critical, notPresent".  The values should
be "red" or "green" etc.
0
 
shanepresleyAuthor Commented:
Yea...got it.  This line was wrong...

my $worstcolor = &color_compare(values(%statedb));

It should have been passing in powercolors.  Correct line is
my $worstcolor = &color_compare(values(%powercolors));
0
 
jeopboyCommented:
So all of the calls are passing bad data to the color_compare routine?
0
 
shanepresleyAuthor Commented:
Not all, just power and one other.  I just threw in some debug print statements, and saw that inside color_compare,
it was seeing "critical" or "normal" instead of what it was looking for...red, green, etc.  

Anyway, thanks for the help.  I'll hand over the points since you spent some time.  THanks again!
Shane
0
 
jeopboyCommented:
Thanks!  Good luck with this...
0

Featured Post

Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

  • 6
  • 4
Tackle projects and never again get stuck behind a technical roadblock.
Join Now