librenms-librenms/doc/Developing/Support-New-OS.md

source: Developing/Support-New-OS.md This document will explain how to add basic and full support for a new OS. Some knowledge in PHP is needed for the full support.

BASIC SUPPORT FOR A NEW OS

MIB

If we have the MIB, we can copy the file into the default directory:

/opt/librenms/mibs

New OS definition

Let's begin to declare the new OS in LibreNMS. At first we create a new definition file located here:

includes/definitions/$os.yaml

This is a Yaml file. Please be careful of the formatting of this file.

os: pulse
text: 'Pulse Secure'
type: firewall
icon: pulse
over:
    - { graph: device_bits, text: 'Device Traffic' }
    - { graph: device_processor, text: 'CPU Usage' }
    - { graph: device_mempool, text: 'Memory Usage' }
discovery:
    - sysDescr:
        - Pulse Connect Secure
        - Pulse Secure
        - Juniper Networks,Inc,VA-DTE
        - VA-SPE

Icon and Logo

Create an SVG image of the icon and logo. Legacy PNG bitmaps are also supported but look bad on HiDPI.

• A vector image should not contain padding.
• The file should not be larger than 20 Kb. Simplify paths to reduce large files.
• Use plain SVG without gzip compression.

Icon

• Save the icon SVG to html/images/os/$os.svg.
• Icons should look good when viewed at 32x32 px.
• Square icons are preferred to full logos with text.
• Remove small ornaments that are almost not visible when displayed with 32px width (e.g. ® or ™).

Logo

• Save the logo SVG to html/images/logos/$os.svg.
• Logos can be any dimension, but often are wide and contain the company name.
• If a logo is not present, the icon will be used.

Hints

Hints for Inkscape:

• You can open a PDF to extract the logo.
• Ungroup elements to isolate the logo.
• Use Path -> Simplify to simplify paths of large files.
• Use File -> Document Properties… -> Resize page to content… to remove padding.
• Use File -> Clean up document to remove unused gradients, patterns, or markers.
• Use File -> Save As -> Plain SVG to save the final image.

By optimizing the SVG you can shrink the file size in some cases to less than 20 %. SVG Optimizer does a great job. There is also an online version.

OS Discovery

The discovery section of the OS yaml file contains information needed to detect this OS.

Discovery Operators

• sysObjectId The preferred operator. Checks if the sysObjectID starts with one of the strings under this item
• sysDescr Use this in addition to sysObjectId if required. Check that the sysDescr contains one of the strings under this item
• sysDescr_regex Please avoid use of this. Checks if the sysDescr matches one of the regex statements under this item

Discoery Logic

YAML is converted to an array in PHP. Consider the following YAML:

discovery: 
  - sysObjectId: foo
  - 
    sysDescr: [ snafu, exodar ]
    sysObjectId: bar

This is how the discovery array would look in PHP:

[
     [
       "sysObjectId" => "foo",
     ],
     [
       "sysDescr" => [
         "snafu",
         "exodar",
       ],
       "sysObjectId" => "bar",
     ]
]

The logic for the discovery is as follows:

1. One of the first level items must match
2. ALL of the second level items must match (sysObjectId, sysDescr)
3. One of the third level items (foo, [snafu,exodar], bar) must match

So, considering the example:

• sysObjectId: foo, sysDescr: ANYTHING matches
• sysObjectId: bar, sysDescr: ANYTHING does not match
• sysObjectId: bar, sysDescr: exodar matches
• sysObjectId: bar, sysDescr: snafu matches

Basic OS information polling

Here is the file location for polling the new OS within a vendor MIB or a standard one:

includes/polling/os/pulse.inc.php

This file will usually set the variables for $version, $hardware and $hostname retrieved from an snmp lookup.

<?php

$version = preg_replace('/[\r\n\"]+/', ' ', snmp_get($device, "productVersion.0", "-OQv", "PULSESECURE-PSG-MIB"));
$hardware = "Juniper " . preg_replace('/[\r\n\"]+/', ' ', snmp_get($device, "productName.0", "-OQv", "PULSESECURE-PSG-MIB"));
$hostname = trim($poll_device['sysName'], '"');

Quick explanation and examples :

snmpwalk -v2c -c public -m SNMPv2-MIB -M mibs
//will give the overall OIDs that can be retrieve with this standard MIB. OID on the left side and the result on the right side
//Then we have just to pick the wanted OID and do a check

snmpget -v2c -c public -OUsb -m SNMPv2-MIB -M /opt/librenms/mibs -t 30 HOSTNAME SNMPv2-SMI::mib-2.1.1.0
//sysDescr.0 = STRING: Juniper Networks,Inc,Pulse Connect Secure,VA-DTE,8.1R1 (build 33493)

snmpget -v2c -c public -OUsb -m SNMPv2-MIB -M /opt/librenms/mibs -t 30 HOSTNAME SNMPv2-SMI::mib-2.1.5.0
//sysName.0 = STRING: pulse-secure

//Here the same with the vendor MIB and the specific OID
snmpget -v2c -c public -OUsb -m PULSESECURE-PSG-MIB -M /opt/librenms_old/mibs -t 30 HOSTNAME productName.0
//productName.0 = STRING: "Pulse Connect Secure,VA-DTE"

snmpget -v2c -c public -OUsb -m PULSESECURE-PSG-MIB -M /opt/librenms/mibs -t 30 HOSTNAME productVersion.0
//productVersion.0 = STRING: "8.1R1 (build 33493)"

The final check

Discovery

./discovery.php -h HOSTNAME

Polling

./poller.php -h HOSTNAME

At this step we should see all the values retrieved in LibreNMS.

Full support for a new OS

MIB

At first we copy the MIB file into the default directory:

/opt/librenms/mibs

We are now ready to look at inside the file and find the OID we want to use. For this documentation we'll use Pulse Secure devices.

Then we can test it with the snmpget command (hostname must be reachable):

//for example the OID iveCpuUtil.0:
snmpget -v2c -c public -OUsb -m PULSESECURE-PSG-MIB -M /opt/librenms/mibs -t 30 HOSTNAME iveCpuUtil.0
//quick explanation : snmpget -v2c -c COMMUNITY -OUsb -m MIBFILE -M MIB DIRECTORY HOSTNAME OID

//Result here:
iveCpuUtil.0 = Gauge32: 28

New OS definition

Let's begin to declare the new OS in LibreNMS. At first we create a new definition file located here:

includes/definitions/$os.yaml

This is a Yaml file. Please be careful of the formatting of this file.

os: pulse
text: 'Pulse Secure'
type: firewall
icon: pulse
over:
    - { graph: device_bits, text: 'Device Traffic' }
    - { graph: device_processor, text: 'CPU Usage' }
    - { graph: device_mempool, text: 'Memory Usage' }

If you are adding custom graphs, please add the following to includes/definitions.inc.php:

//Don't forget to declare the specific graphs if needed. It will be located near the end of the file.

//Pulse Secure Graphs
$config['graph_types']['device']['pulse_users']['section']         = 'firewall';
$config['graph_types']['device']['pulse_users']['order']           = '0';
$config['graph_types']['device']['pulse_users']['descr']           = 'Active Users';
$config['graph_types']['device']['pulse_sessions']['section']      = 'firewall';
$config['graph_types']['device']['pulse_sessions']['order']        = '0';
$config['graph_types']['device']['pulse_sessions']['descr']        = 'Active Sessions';

Discovery OS

We create a new file named as our OS definition and in this directory:

includes/discovery/os/pulse.inc.php

Look at other files to get help in the code structure. For this example, it can be like this :

// Pulse Secure OS definition
if (str_contains($sysDescr, array('Pulse Connect Secure', 'Pulse Secure', 'Juniper Networks,Inc,VA-DTE', 'VA-SPE'))) {
    $os = 'pulse';
}

As we declared Memory and CPU graphs before, we declare the OID in a PHP file :

Memory

includes/discovery/mempools/pulse.inc.php

<?php
//
// Hardcoded discovery of Memory usage on Pulse Secure devices.
//
if ($device['os'] == 'pulse') {
    echo 'PULSE-MEMORY-POOL: ';

    $usage = str_replace('"', "", snmp_get($device, 'PULSESECURE-PSG-MIB::iveMemoryUtil.0', '-OvQ'));

    if (is_numeric($usage)) {
        discover_mempool($valid_mempool, $device, 0, 'pulse-mem', 'Main Memory', '100', null, null);
    }
}

CPU

includes/discovery/processors/pulse.inc.php

<?php
//
// Hardcoded discovery of CPU usage on Pulse Secure devices.
//
if ($device['os'] == 'pulse') {
    echo 'Pulse Secure : ';

    $descr = 'Processor';
    $usage = str_replace('"', "", snmp_get($device, 'PULSESECURE-PSG-MIB::iveCpuUtil.0', '-OvQ'));

    if (is_numeric($usage)) {
        discover_processor($valid['processor'], $device, 'PULSESECURE-PSG-MIB::iveCpuUtil.0', '0', 'pulse-cpu', $descr,
 '100', $usage, null, null);
    }
}

Please keep in mind that the PHP code is often different for the needs of the devices and the information we retrieve.

Polling OS

We will now do the same for the polling process:

Memory

includes/polling/mempools/pulse-mem.inc.php

<?php

// Simple hard-coded poller for Pulse Secure
echo 'Pulse Secure MemPool'.'\n';

if ($device['os'] == 'pulse') {
  $perc     = str_replace('"', "", snmp_get($device, "PULSESECURE-PSG-MIB::iveMemoryUtil.0", '-OvQ'));
  $memory_available = str_replace('"', "", snmp_get($device, "UCD-SNMP-MIB::memTotalReal.0", '-OvQ'));
  $mempool['total'] = $memory_available;

  if (is_numeric($perc)) {
    $mempool['used'] = ($memory_available / 100 * $perc);
    $mempool['free'] = ($memory_available - $mempool['used']);
  }

  echo "PERC " .$perc."%\n";
  echo "Avail " .$mempool['total']."\n";

}

CPU

includes/polling/processors/pulse-cpu.inc.php

<?php
// Simple hard-coded poller for Pulse Secure
echo 'Pulse Secure CPU Usage';

if ($device['os'] == 'pulse') {
    $usage = str_replace('"', "", snmp_get($device, 'PULSESECURE-PSG-MIB::iveCpuUtil.0', '-OvQ'));

    if (is_numeric($usage)) {
        $proc = ($usage * 100);
    }
}

Here is the file location for the specific graphs based on the OID in the vendor MIB:

includes/polling/os/pulse.inc.php

We declare two specific graphs for users and sessions numbers. Theses two graphs will be displayed on the firewall section of the graphs tab as it was written in the definition include file.

<?php

$version = preg_replace('/[\r\n\"]+/', ' ', snmp_get($device, "productVersion.0", "-OQv", "PULSESECURE-PSG-MIB"));
$hardware = "Juniper " . preg_replace('/[\r\n\"]+/', ' ', snmp_get($device, "productName.0", "-OQv", "PULSESECURE-PSG-MIB"));
$hostname = trim($poll_device['sysName'], '"');

$users = snmp_get($device, 'iveConcurrentUsers.0', '-OQv', 'PULSESECURE-PSG-MIB');

if (is_numeric($users)) {
    $rrd_def = 'DS:users:GAUGE:600:0:U';

    $fields = array(
        'users' => $users,
    );

    $tags = compact('rrd_def');
    data_update($device, 'pulse_users', $tags, $fields);
    $graphs['pulse_users'] = true;
}

$sessions = snmp_get($device, 'iveConcurrentUsers.0', '-OQv', 'PULSESECURE-PSG-MIB');

if (is_numeric($sessions)) {
    $rrd_def = 'DS:sessions:GAUGE:600:0:U';

    $fields = array(
        'sessions' => $sessions,
    );

    $tags = compact('rrd_def');
    data_update($device, 'pulse_sessions', $tags, $fields);
    $graphs['pulse_sessions'] = true;
}

We finish in the declaration of the two graph types in the database:

We can do that within a file to share our work and contribute in the development of LibreNMS. :-)

sql-schema/xxx.sql
//check the file number in GitHub

php includes/sql-schema/update.php

Or put the SQL commands directly in Mysql or PhpMyadmin for our tests:

INSERT INTO `graph_types`(`graph_type`, `graph_subtype`, `graph_section`, `graph_descr`, `graph_order`) VALUES ('device',  'pulse_users',  'firewall',  'Active Users',  '');
INSERT INTO `graph_types`(`graph_type`, `graph_subtype`, `graph_section`, `graph_descr`, `graph_order`) VALUES ('device',  'pulse_sessions',  'firewall',  'Active Sessions',  '');

Displaying

The specific graphs are not displayed automatically so we need to write the following PHP code:

Pulse Sessions

html/includes/graphs/device/pulse_sessions.inc.php

<?php

$rrd_filename = rrd_name($device['hostname'], 'pulse_sessions');

require 'includes/graphs/common.inc.php';

$ds = 'sessions';

$colour_area = '9999cc';
$colour_line = '0000cc';

$colour_area_max = '9999cc';

$graph_max = 1;
$graph_min = 0;

$unit_text = 'Sessions';

require 'includes/graphs/generic_simplex.inc.php';

Pulse Users

html/includes/graphs/device/pulse_users.inc.php

<?php

$rrd_filename = rrd_name($device['hostname'], 'pulse_users');

require 'includes/graphs/common.inc.php';

$ds = 'users';

$colour_area = '9999cc';
$colour_line = '0000cc';

$colour_area_max = '9999cc';

$graph_max = 1;

$unit_text = 'Users';

require 'includes/graphs/generic_simplex.inc.php';

The final check

Discovery

./discovery.php -h HOSTNAME

Polling

./poller.php -h HOSTNAME

At this step we should see all the values retrieved in LibreNMS.

OS Test units

We have a testing unit for new OS', please ensure you add a test for any new OS' or updates to existing OS discovery.

The OS test unit file is located tests/OSDiscoveryTest.php. An example of this is as follows:

    public function testNios()
    {
        $this->checkOS('nios');
        $this->checkOS('nios', 'nios-ipam');
    }

We utilise snmpsim to do unit testing for OS discovery. For this to work you need to supply an snmprec file. This is pretty simple and using nios as the example again this would look like:

1.3.6.1.2.1.1.1.0|4|Linux 3.14.25 #1 SMP Thu Jun 16 18:19:37 EDT 2016 x86_64
1.3.6.1.2.1.1.2.0|6|1.3.6.1.4.1.7779.1.1402

During testing LibreNMS will use any info in the snmprec file for snmp calls. This one provides sysDescr (.1.3.6.1.2.1.1.1.0, 4 = Octet String) and sysObjectID (.1.3.6.1.2.1.1.2.0, 6 = Object Identifier), which is the minimum that should be provided for new snmprec files.

To look up the numeric OID and type of an string OID with snmptranslate:

snmptranslate -On -Td SNMPv2-MIB::sysDescr.0

Common OIDs used in discovery:

String OID	Numeric OID
SNMPv2-MIB::sysDescr.0	1.3.6.1.2.1.1.1.0
SNMPv2-MIB::sysObjectID.0	1.3.6.1.2.1.1.2.0
ENTITY-MIB::entPhysicalDescr.1	1.3.6.1.2.1.47.1.1.1.1.2.1
ENTITY-MIB::entPhysicalMfgName.1	1.3.6.1.2.1.47.1.1.1.1.12.1
SML-MIB::product-Name.0	1.3.6.1.4.1.2.6.182.3.3.1.0

List of SNMP data types:

Type	Value
OCTET STRING	4
HEX STRING	4x
Integer32	2
NULL	5
OBJECT IDENTIFIER	6
IpAddress	64
Counter32	65
Gauge32	66
TimeTicks	67
Opaque	68
Counter64	70

Hex encoded strings (4x) should be used for any strings that contain line returns

You can run ./scripts/pre-commit.php -u to run the unit tests to check your code. If you would like to run tests locally against a full snmpsim instance, run ./scripts/pre-commit.php -u --snmpsim.