mirror/librenms-librenms
1
0
Fork 0
mirror of https://github.com/librenms/librenms.git synced 2024-10-07 16:52:45 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

nfdump support for with NFSen (#10376)

Browse Source 
* rework the nfsen stuff

* add the new netflow stats page

* update the docs for nfsen

* add the functions back in

* readd the defaults

* rework lowest_five_minutes into lowest_time

* nfsen_channel_rrds removed and will rework it in later post rewriting

* rework something that accidentally got nuked in merging

* rework formatting a bit

* remove a accidentaly added chunk of text =^.^=

* rework some tag capitalization

* remove a accidentally added [

* '='=>' = '
This commit is contained in:
Zane C. Bowers-Hadley
2019-08-06 19:16:34 -05:00
committed by Tony Murray
parent c18ba96f08
commit 0a6ded9ba3
7 changed files with 475 additions and 56 deletions
Download Patch File Download Diff File Expand all files Collapse all files

168
doc/Extensions/NFSen.md
View File

@@ -1,10 +1,11 @@
source: Extensions/NFSen.md
path: blob/master/doc/
# NFSen
> The installation of NFSen is out of scope for this document / LibreNMS
#### Configuration
## Configuration
The following is the configuration that can be used:
@@ -13,43 +14,184 @@ $config['nfsen_enable'] = 1;
$config['nfsen_split_char'] = '_';
$config['nfsen_rrds'][] = '/var/nfsen/profiles-stat/live/';
$config['nfsen_rrds'][] = '/var/nfsen/profiles-stat';
$config['nfsen_base'][] = '/var/nfsen/';
$config['nfsen_suffix'] = "_yourdomain_com";
```
Set `$config['nfsen_enable'] = 1;` to enable NFSen support.
`$config['nfsen_rrds']` This value tells us where your NFSen rrd files live. This can also be an array to
specify more directories like:
`$config['nfsen_rrds']` This value tells us where your NFSen rrd files
live. This can also be an array to specify more directories like:
```php
$config['nfsen_rrds'][] = '/var/nfsen/profiles-stat/sitea/';
$config['nfsen_rrds'][] = '/var/nfsen/profiles-stat/siteb/';
```
Although for most setups, it will look like below, with the profiles-stat/live directory being where it stores the general RRDs for data sources.
Although for most setups, it will look like below, with the
profiles-stat/live directory being where it stores the general RRDs
for data sources.
```php
$config['nfsen_rrds'][] = '/var/nfsen/profiles-stat/live';
```
`$config['nfsen_split_char']` This value tells us what to replace the full stops `.` in the devices hostname with.
`$config['nfsen_suffix']` This value will be removed from the domain name and can be useful if your rrd files are
something like `host1.rrd` but your device hostname is `domain.host1.rrd`. You can then set $config['nfsen_suffix'] = 'domain';
If you wish to render info for configure channels for a device, you need add the various profile-stat directories your system uses, which for most systems will be as below.
If you wish to render info for configure channels for a device, you
need add the various profile-stat directories your system uses, which
for most systems will be as below.
```php
$config['nfsen_rrds'][] = '/var/nfsen/profiles-stat';
```
When adding sources to nfsen.conf, it is important to use the hostname that matches what is configured in LibreNMS, because the rrd files NfSen creates is named after the source name (ident), and it doesn't allow you to use an IP address instead. However, in LibreNMS, if your device is added by an IP address, add your source with any name of your choice, and create a symbolic link to the rrd file.
When adding sources to nfsen.conf, it is important to use the hostname
that matches what is configured in LibreNMS, because the rrd files
NfSen creates is named after the source name (ident), and it doesn't
allow you to use an IP address instead. However, in LibreNMS, if your
device is added by an IP address, add your source with any name of
your choice, and create a symbolic link to the rrd file.
```sh
cd /var/nfsen/profiles-stat/sitea/
ln -s mychannel.rrd librenmsdeviceIP.rrd
```
When creating profiles under nfsen, be sure to use the hostname so it matches the name in LibreNMS. That is where channel data will be pulled from.
```php
$config['nfsen_split_char'] = '_';
```
You should see a new tab for the device called Nfsen.
This value tells us what to replace the full stops `.` in the devices
hostname with.
```php
$config['nfsen_suffix'] = "_yourdomain_com";
```
The above is a very important bit as device names in NfSen are limited
to 21 characters. This means full domain names for devices can be very
problematic to squeeze in, so there for this chunk is usually removed.
On a similar note, NfSen profiles for channels should be created with
the same name.
## Stats Defaults and Settings
Below are the default settings used with nfdump for stats.
For more defailted information on that, please see nfdump(1).
```php
$config['nfsen_last_max'] = 153600;
$config['nfsen_top_max'] = 500;
$config['nfsen_top_N']=array( 10, 20, 50, 100, 200, 500 );
$config['nfsen_top_default']=20;
$config['nfsen_stat_default']='srcip';
$config['nfsen_order_default']='packets';
$config['nfsen_last_default']=900;
$config['nfsen_lasts']=array(
'300'=>'5 minutes',
'600'=>'10 minutes',
'900'=>'15 minutes',
'1800'=>'30 minutes',
'3600'=>'1 hour',
'9600'=>'3 hours',
'38400'=>'12 hours',
'76800'=>'24 hours',
'115200'=>'36 hours',
'153600'=>'48 hours',
);
```
```
$config['nfsen_last_max'] = 153600;
```
The above is the max value in seconds one may pull stats for. The
higher this is, the more CPU and disk intensive the search will be.
Numbers larger than this will be set to this.
```
$config['nfsen_top_max'] = 500;
```
The above is max number of items to be displayed.
Numbers larger than this will be set to this.
```
$config['nfsen_top_N']=array( 10, 20, 50, 100, 200, 500 );
```
The above is a array containing a list for the drop down menu how many
top items should be returned.
```php
$config['nfsen_top_default']=20;
```
The above sets default top number to use from the drop down.
```php
$config['nfsen_stat_default']='srcip';
```
The above sets default stat type to use from the drop down.
```
record Flow Records
ip Any IP Address
srcip SRC IP Address
dstip DST IP Address
port Any Port
srcport SRC Port
dstport DST Port
srctos SRC TOS
dsttos DST TOS
tos TOS
as AS
srcas SRC AS
dstas DST AS
```
```php
$config['nfsen_order_default']='packets';
```
The above sets default order type to use from the drop down. Any of
the following below are currently supported.
```
flows Number of total flows for the time period.
packet Number of total packets for the time period.
bytes Number of total bytes for the time period.
pps Packets Per Second
bps Bytes Per Second
bpp Bytes Per Packet
```
```
$config['nfsen_last_default']=900;
```
The above is the last default to use from the drop down.
```
$config['nfsen_lasts']=array(
'300'=>'5 minutes',
'600'=>'10 minutes',
'900'=>'15 minutes',
'1800'=>'30 minutes',
'3600'=>'1 hour',
'9600'=>'3 hours',
'38400'=>'12 hours',
'76800'=>'24 hours',
'115200'=>'36 hours',
'153600'=>'48 hours',
);
```
The above associative array contains time intervals for how
far back to go. The keys are the length in seconds and the
value is just a description to display.

21
includes/defaults.inc.php
View File

@@ -62,6 +62,7 @@ $config['ipmitool'] = '/usr/bin/ipmitool';
$config['virsh'] = '/usr/bin/virsh';
$config['dot'] = '/usr/bin/dot';
$config['sfdp'] = '/usr/bin/sfdp';
$config['nfdump'] = '/usr/bin/nfdump';
$config['slow_statistics'] = true;
// THIS WILL CHANGE TO FALSE IN FUTURE
@@ -541,6 +542,26 @@ $config['nfsen_enable'] = 0;
// $config['nfsen_split_char'] = "_";
// $config['nfsen_rrds'] = "/var/nfsen/profiles-stat/live/";
// $config['nfsen_suffix'] = "_yourdomain_com";
$config['nfsen_subdirlayout'] = 1;
$config['nfsen_last_max'] = 153600; // 48 hours ago in seconds
$config['nfsen_top_max'] = 500; // max topN value for stats
$config['nfsen_top_N']=array( 10, 20, 50, 100, 200, 500 );
$config['nfsen_top_default']=20;
$config['nfsen_stat_default']='srcip';
$config['nfsen_order_default']='packets';
$config['nfsen_last_default']=900;
$config['nfsen_lasts']=array(
'300'=>'5 minutes',
'600'=>'10 minutes',
'900'=>'15 minutes',
'1800'=>'30 minutes',
'3600'=>'1 hour',
'9600'=>'3 hours',
'38400'=>'12 hours',
'76800'=>'24 hours',
'115200'=>'36 hours',
'153600'=>'48 hours',
);
// Location Mapping
// Use this feature to map ugly locations to pretty locations
// config['location_map']['Under the Sink'] = "Under The Sink, The Office, London, UK";

102
includes/html/functions.inc.php
View File

@@ -1640,3 +1640,105 @@ function get_sensor_label_color($sensor, $type = 'sensors')
$unit = __("$type.{$sensor['sensor_class']}.unit");
return "<span class='label $label_style'>".trim(format_si($sensor['sensor_current']).$unit)."</span>";
}
/**
* @params int unix time
* @params int seconds
* @return int
*
* Rounds down to the nearest interval.
*
* The first argument is required and it is the unix time being
* rounded down.
*
* The second value is the time interval. If not specified, it
* defaults to 300, or 5 minutes.
*/
function lowest_time($time, $seconds = 300)
{
return $time - ($time % $seconds);
}
/**
* @params int
* @return string
*
* This returns the subpath for working with nfdump.
*
* 1 value is taken and that is a unix time stamp. It will be then be rounded
* off to the lowest five minutes earlier.
*
* The return string will be a path partial you can use with nfdump to tell it what
* file or range of files to use.
*
* Below ie a explanation of the layouts as taken from the NfSen config file.
* 0 no hierachy levels - flat layout - compatible with pre NfSen version
* 1 %Y/%m/%d year/month/day
* 2 %Y/%m/%d/%H year/month/day/hour
* 3 %Y/%W/%u year/week_of_year/day_of_week
* 4 %Y/%W/%u/%H year/week_of_year/day_of_week/hour
* 5 %Y/%j year/day-of-year
* 6 %Y/%j/%H year/day-of-year/hour
* 7 %Y-%m-%d year-month-day
* 8 %Y-%m-%d/%H year-month-day/hour
*/
function time_to_nfsen_subpath($time)
{
$time=lowest_time($time);
$layout=Config::get('nfsen_subdirlayout');
if ($layout == 0) {
return 'nfcapd.'.date('YmdHi', $time);
} elseif ($layout == 1) {
return date('Y\/m\/d\/\n\f\c\a\p\d\.YmdHi', $time);
} elseif ($layout == 2) {
return date('Y\/m\/d\/H\/\n\f\c\a\p\d\.YmdHi', $time);
} elseif ($layout == 3) {
return date('Y\/W\/w\/\n\f\c\a\p\d\.YmdHi', $time);
} elseif ($layout == 4) {
return date('Y\/W\/w\/H\/\n\f\c\a\p\d\.YmdHi', $time);
} elseif ($layout == 5) {
return date('Y\/z\/\n\f\c\a\p\d\.YmdHi', $time);
} elseif ($layout == 6) {
return date('Y\/z\/H\/\n\f\c\a\p\d\.YmdHi', $time);
} elseif ($layout == 7) {
return date('Y\-m\-d\/\n\f\c\a\p\d\.YmdHi', $time);
} elseif ($layout == 8) {
return date('Y\-m\-d\/H\/\n\f\c\a\p\d\.YmdHi', $time);
}
}
/**
* @params string hostname
* @return string
*
* Takes a hostname and transforms it to the name
* used by nfsen.
*/
function nfsen_hostname($hostname)
{
$nfsen_hostname=str_replace('.', Config::get('nfsen_split_char'), $hostname);
if (!is_null(Config::get('nfsen_suffix'))) {
$nfsen_hostname=str_replace(Config::get('nfsen_suffix'), '', $nfsen_hostname);
}
return $nfsen_hostname;
}
/**
* @params string hostname
* @return string
*
* Takes a hostname and returns the path to the nfsen
* live dir.
*/
function nfsen_live_dir($hostname)
{
$hostname=nfsen_hostname($hostname);
foreach (Config::get('nfsen_base') as $base_dir) {
if (file_exists($base_dir) && is_dir($base_dir)) {
return $base_dir.'/profiles-data/live/'.$hostname;
}
}
}

48
includes/html/graphs/device/nfsen_channel_common.inc.php
View File

@@ -3,37 +3,41 @@
$simple_rrd = true;
foreach ((array)\LibreNMS\Config::get('nfsen_rrds', []) as $nfsenrrds) {
$nfsenrrds = rtrim($nfsenrrds, '/') . '/';
if (\LibreNMS\Config::get('nfsen_split_char')) {
$nfsenHostname = str_replace('.', \LibreNMS\Config::get('nfsen_split_char'), $device['hostname']);
} else {
$nfsenHostname=$device['hostname'];
if ($nfsenrrds[(strlen($nfsenrrds) - 1)] != '/') {
$nfsenrrds .= '/';
}
$rrd_filename=$nfsenrrds.$nfsenHostname.'/'.$vars['channel'].'.rrd';
if (is_file($rrd_filename)) {
$colours = 'blues';
$nototal = 0;
$units = '';
$scale_min = '0';
$unit_text = $dsdescr;
$nfsen_filename=nfsen_hostname($device['hostname']);
if (is_file($nfsenrrds.$nfsen_filename.'/'.$vars['channel'].'.rrd')) {
$rrd_filename = $nfsenrrds.$nfsen_filename.'/'.$vars['channel'].'.rrd';
$flowtypes = array('tcp', 'udp', 'icmp', 'other');
$rrd_list = array();
$nfsen_iter = 1;
foreach ($flowtypes as $flowtype) {
$rrd_list[$nfsen_iter]['filename'] = $rrd_filename;
$rrd_list[$nfsen_iter]['descr'] = $flowtype;
$rrd_list[$nfsen_iter]['ds'] = $dsprefix.$flowtype;
// set a multiplier which in turn will create a CDEF if this var is set
if ($dsprefix == 'traffic_') {
$multiplier = '8';
}
$flowtypes = array('tcp', 'udp', 'icmp', 'other');
$rrd_list = array();
foreach ($flowtypes as $flowtype) {
$rrd_list[] = array(
'filename' => $rrd_filename,
'descr' => $flowtype,
'ds' => $dsprefix.$flowtype
);
$colours = 'blues';
$nototal = 0;
$units = '';
$unit_text = $dsdescr;
$scale_min = '0';
if ($_GET['debug']) {
print_r($rrd_list);
}
$nfsen_iter++;
}
d_echo($rrd_list);
}
}

10
includes/html/graphs/device/nfsen_common.inc.php
View File

@@ -7,15 +7,7 @@ foreach ((array)\LibreNMS\Config::get('nfsen_rrds', []) as $nfsenrrds) {
$nfsenrrds .= '/';
}
// convert dots in filename to underscores
$nfsensuffix = \LibreNMS\Config::get('nfsen_suffix', '');
if (!empty(\LibreNMS\Config::get('nfsen_split_char'))) {
$basefilename_underscored = preg_replace('/\./', \LibreNMS\Config::get('nfsen_split_char'), $device['hostname']);
} else {
$basefilename_underscored = $device['hostname'];
}
$nfsen_filename = preg_replace('/'.$nfsensuffix.'/', '', $basefilename_underscored);
$nfsen_filename=nfsen_hostname($device['hostname']);
if (is_file($nfsenrrds.$nfsen_filename.'.rrd')) {
$rrd_filename = $nfsenrrds.$nfsen_filename.'.rrd';

25
includes/html/pages/device/nfsen.inc.php
View File

@@ -9,25 +9,27 @@ $link_array = array(
);
echo generate_link('General', $link_array, array('nfsen' => 'general'));
echo '|';
echo generate_link('Stats', $link_array, array('nfsen' => 'stats'));
$printedChannel=false;
$nfsenHostname = str_replace('.', \LibreNMS\Config::get('nfsen_split_char'), $device['hostname']);
foreach ((array)\LibreNMS\Config::get('nfsen_rrds', []) as $nfsenDir) {
$hostDir=$nfsenDir.'/'.$nfsenHostname.'/';
$printedChannel = false;
$nfsen_hostname = nfsen_hostname($device['hostname']);
foreach (\LibreNMS\Config::get('nfsen_rrds') as $nfsenDir) {
$hostDir = $nfsenDir.'/'.$nfsen_hostname.'/';
if (is_dir($hostDir)) {
$nfsenRRDchannelGlob=$hostDir.'*.rrd';
$nfsenRRDchannelGlob = $hostDir.'*.rrd';
foreach (glob($nfsenRRDchannelGlob) as $nfsenRRD) {
$channel = str_replace(array($hostDir, '.rrd'), '', $nfsenRRD);
if (!$printedChannel) {
echo '|Channels:';
$printedChannel=true;
$printedChannel = true;
} else {
echo ',';
}
if ($vars['channel'] == $channel) {
$channelFilter=$hostDir.$channel.'-filter.txt';
$channelFilter = $hostDir.$channel.'-filter.txt';
}
echo generate_link($channel, $link_array, array('nfsen' => 'channel', 'channel' => $channel));
@@ -37,9 +39,12 @@ foreach ((array)\LibreNMS\Config::get('nfsen_rrds', []) as $nfsenDir) {
print_optionbar_end();
$nfsen_type = basename($vars['nfsen'] ?? 'general');
if (is_file("includes/html/pages/device/nfsen/$nfsen_type.inc.php")) {
include "includes/html/pages/device/nfsen/$nfsen_type.inc.php";
if (!$vars['nfsen']) {
$vars['nfsen'] = 'general';
}
if (is_file('includes/html/pages/device/nfsen/'.mres($vars['nfsen']).'.inc.php')) {
include 'includes/html/pages/device/nfsen/'.mres($vars['nfsen']).'.inc.php';
} else {
include 'includes/html/pages/device/nfsen/general.inc.php';
}

153
includes/html/pages/device/nfsen/stats.inc.php Normal file
View File

@@ -0,0 +1,153 @@
<?php
print_optionbar_start();
echo '<form action = "'.generate_url($link_array, array('nfsen' => 'stats')).'" iOCd = "FlowStats" method = "SUBMIT">';
echo 'Top N:
<select name = "topN" id = "topN" size = 1>
';
$option_default = $vars['topN'] ?? \LibreNMS\Config::get('nfsen_top_default');
$option_int = 0;
foreach (\LibreNMS\Config::get('nfsen_top_N') as $option) {
if (strcmp($option_default, $option) == 0) {
echo '<option value = "'.$option.'" selected>'.$option.'</option>';
} else {
echo '<option value = "'.$option.'">'.$option.'</option>';
}
}
echo '
</select>
During the last:
<select name = "lastN" id = "lastN" size = 1>
';
$option_keys = array_keys(\LibreNMS\Config::get('nfsen_lasts'));
$options = \LibreNMS\Config::get('nfsen_lasts');
foreach ($option_keys as $option) {
if (strcmp($option_default, $option) == 0) {
echo '<option value = "'.$option.'" selected>'.$options[$option].'</option>';
} else {
echo '<option value = "'.$option.'">'.$options[$option].'</option>';
}
}
echo '
</select>
, Stat Type:
<select name = "stattype" id = "StatTypeSelector" size = 1>
';
$option_default = $vars['stattype'] ?? \LibreNMS\Config::get('nfsen_stats_default');
$stat_types = array(
'record'=>'Flow Records',
'ip'=>'Any IP Address',
'srcip'=>'SRC IP Address',
'dstip'=>'DST IP Address',
'port'=>'Any Port',
'srcport'=>'SRC Port',
'dstport'=>'DST Port',
'srctos'=>'SRC TOS',
'dsttos'=>'DST TOS',
'tos'=>'TOS',
'as'=>'AS',
'srcas'=>'SRC AS',
'dstas'=>'DST AS',
);
// puts together the drop down options
foreach ($stat_types as $option => $descr) {
if (strcmp($option_default, $option) == 0) {
echo '<option value = "'.$option.'" selected>'.$descr."</option>\n";
} else {
echo '<option value = "'.$option.'">'.$descr."</option>\n";
}
}
echo '
</select>
, Order By:
<select name = "statorder" id = "statorder" sizeOC = 1>
';
$option_default = \LibreNMS\Config::get('nfsen_order_default');
if (isset($vars['statorder'])) {
$option_default = $vars['statorder'];
}
// WARNING: order is relevant as it has to match the
// check later in the process part of this page.
$order_types = array(
'flows'=>1,
'packets'=>1,
'bytes'=>1,
'pps'=>1,
'bps'=>1,
'bpp'=>1,
);
// puts together the drop down options
foreach ($order_types as $option => $descr) {
if (strcmp($option_default, $option) == 0) {
echo '<option value = "'.$option.'" selected>'.$option."</option>\n";
} else {
echo '<option value = "'.$option.'">'.$option."</option>\n";
}
}
echo '
</select>
<input type = "submit" name = "process" value = "process" size = "1">
';
echo '</form>';
print_optionbar_end();
// process stuff now if we the button was clicked on
if (isset($vars['process'])) {
// Make sure we have a sane value for lastN
$lastN = 900;
if (isset($vars['lastN']) &&
is_numeric($vars['lastN']) &&
($vars['lastN'] <= \LibreNMS\Config::get('nfsen_last_max'))
) {
$lastN = $vars['lastN'];
}
// Make sure we have a sane value for lastN
$topN = 20; // The default if not set or something invalid is set
if (isset($vars['topN']) &&
is_numeric($vars['topN']) &&
($vars['topN'] <= \LibreNMS\Config::get('nfsen_top_max'))
) {
$topN = $vars['topN'];
}
// Handle the stat order.
$stat_order = 'pps'; // The default if not set or something invalid is set
if (isset($vars['statorder']) && isset($order_types[$vars['statorder']])) {
$stat_order = $vars['statorder'];
}
// Handle the stat type.
$stat_type = 'srcip'; // The default if not set or something invalid is set
if (isset($vars['stattype']) && isset($stat_types[$vars['stattype']])) {
$stat_type = $vars['stattype'];
}
$current_time = lowest_time(time() - 300);
$last_time = lowest_time($current_time - $lastN - 300);
$command = \LibreNMS\Config::get('nfdump').' -M '.nfsen_live_dir($device['hostname']).' -T -R '.
time_to_nfsen_subpath($last_time).':'.time_to_nfsen_subpath($current_time).
' -n '.$topN.' -s '.$stat_type.'/'.$stat_order;
echo '<pre>';
system($command);
echo '</pre>';
}