Files
librenms-librenms/LibreNMS/Modules/Mempools.php

193 lines
6.6 KiB
PHP
Raw Normal View History

Modernize mempools (#12282) * mempools to modern module quick hacky hrstorage port * port ucd-snmp-mib to Mempools * Populate DB for ucd Prep for yaml * initial yaml attempt * more complex conversions fixes to YamlDiscovery, make leading $ optional and allow mib::oid format * walk full tables and skip values normalize percentages above 100 * handle precent only ones (specify total as 100) * Move default polling out of YamlMempoolsDiscovery * fixes * Update test data hrstorage should be correct. * perc_warn for hrstorage * Host Resources, record buffer, cached, and shared * Host Resources is always better, don't do both HR and UCD * fix unix, include warning levels * variable size * consolidate skip_values * define mempools schema * number instead of integer * more schema refactor * one more skip_values reference * throw error for invalid oid translation aos6 * a* and Cisco * updated test data * update almost all hrstorage data files * b* * c* with test data use standard cache for hrStorage * use cache for storage module too * hand bsnmp properly * bdcom * exclude total oid from yaml so it is not polled May add a way to ignore this behavior and poll it, but I don't know if that is needed. * automatically handle percent only values * ciscowlc * only poll used or free if we have used, free, and total. * fix skipping * the dlinkoning fix find value when value "name" is numeric * support numeric oids * dnos/ftos attempt * I guess we can't filter on total > 0 * edgecos * e* * f WIP * f* * gwd (aka g*) * h* + procurve * i* * j* * m* * support 0% used memory (however unlikely) * n* * CISCO-PROCESS-MIB memory, share cache with processors module * ignore mempools with invalid total * p* * quanta * r* fix raisecom mibs terribly broken * s-z * style fixes * Move VRP back to PHP and make it actually work * fix zynos * update schema * Update Cisco processor data for description bug fixes * fix comware processors * comware mempools with memory size default precision to 1 * sophos-xg updated data * hrstorage use ram size for buffers, cache, and shared * Show memory available instead of free in device overview * UCD, use same rrd format, store available instead of free in the db. * Calculate availability for HOST-RESOURCES-MIB * Convert UCD to standard polling * rename old rrd files * initial graph work * graph WIP * Graph looking decent * Graph looking decent for hr * remove old ucd_graph code * handle availability mempool more graph cleanup * color adjustments * remove accidental free calculation * Update test data and fix corner cases * fis pfsense * update schema * add default value for mempool_class * fix whitespace * update schema * update schema correctly * one more time * fortigate_1500d-sensors missing oids * Update docs. * fix indent * add implements MempoolsDiscovery explicitly to OS * remove ucd_memory graph references remove unused device_memory graph * remove unused functions * set devices with mempools to rediscover to prevent/minimize gaps * use a subquery * add overview graph * port health mempools table * Update device mempool * only show overview if multiple * Don't override user set warn percentages in discovery * fix missed usage * fix style * Safety check to not rename rrd files incorrectly if migration has not been run. * Fix overview percent bar and represent available and used on the bar * missed an item to convert to mempool_class * percent on the wrong side
2020-11-23 15:35:35 -06:00
<?php
/*
* Mempools.php
*
* -Description-
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*
* @package LibreNMS
* @link http://librenms.org
* @copyright 2020 Tony Murray
* @author Tony Murray <murraytony@gmail.com>
*/
namespace LibreNMS\Modules;
use App\Models\Mempool;
use App\Observers\MempoolObserver;
Modernize mempools (#12282) * mempools to modern module quick hacky hrstorage port * port ucd-snmp-mib to Mempools * Populate DB for ucd Prep for yaml * initial yaml attempt * more complex conversions fixes to YamlDiscovery, make leading $ optional and allow mib::oid format * walk full tables and skip values normalize percentages above 100 * handle precent only ones (specify total as 100) * Move default polling out of YamlMempoolsDiscovery * fixes * Update test data hrstorage should be correct. * perc_warn for hrstorage * Host Resources, record buffer, cached, and shared * Host Resources is always better, don't do both HR and UCD * fix unix, include warning levels * variable size * consolidate skip_values * define mempools schema * number instead of integer * more schema refactor * one more skip_values reference * throw error for invalid oid translation aos6 * a* and Cisco * updated test data * update almost all hrstorage data files * b* * c* with test data use standard cache for hrStorage * use cache for storage module too * hand bsnmp properly * bdcom * exclude total oid from yaml so it is not polled May add a way to ignore this behavior and poll it, but I don't know if that is needed. * automatically handle percent only values * ciscowlc * only poll used or free if we have used, free, and total. * fix skipping * the dlinkoning fix find value when value "name" is numeric * support numeric oids * dnos/ftos attempt * I guess we can't filter on total > 0 * edgecos * e* * f WIP * f* * gwd (aka g*) * h* + procurve * i* * j* * m* * support 0% used memory (however unlikely) * n* * CISCO-PROCESS-MIB memory, share cache with processors module * ignore mempools with invalid total * p* * quanta * r* fix raisecom mibs terribly broken * s-z * style fixes * Move VRP back to PHP and make it actually work * fix zynos * update schema * Update Cisco processor data for description bug fixes * fix comware processors * comware mempools with memory size default precision to 1 * sophos-xg updated data * hrstorage use ram size for buffers, cache, and shared * Show memory available instead of free in device overview * UCD, use same rrd format, store available instead of free in the db. * Calculate availability for HOST-RESOURCES-MIB * Convert UCD to standard polling * rename old rrd files * initial graph work * graph WIP * Graph looking decent * Graph looking decent for hr * remove old ucd_graph code * handle availability mempool more graph cleanup * color adjustments * remove accidental free calculation * Update test data and fix corner cases * fis pfsense * update schema * add default value for mempool_class * fix whitespace * update schema * update schema correctly * one more time * fortigate_1500d-sensors missing oids * Update docs. * fix indent * add implements MempoolsDiscovery explicitly to OS * remove ucd_memory graph references remove unused device_memory graph * remove unused functions * set devices with mempools to rediscover to prevent/minimize gaps * use a subquery * add overview graph * port health mempools table * Update device mempool * only show overview if multiple * Don't override user set warn percentages in discovery * fix missed usage * fix style * Safety check to not rename rrd files incorrectly if migration has not been run. * Fix overview percent bar and represent available and used on the bar * missed an item to convert to mempool_class * percent on the wrong side
2020-11-23 15:35:35 -06:00
use Illuminate\Support\Collection;
use LibreNMS\DB\SyncsModels;
use LibreNMS\Interfaces\Discovery\MempoolsDiscovery;
use LibreNMS\Interfaces\Module;
use LibreNMS\Interfaces\Polling\MempoolsPolling;
use LibreNMS\OS;
use LibreNMS\RRD\RrdDefinition;
use Log;
class Mempools implements Module
{
use SyncsModels;
public function discover(OS $os)
{
if ($os instanceof MempoolsDiscovery) {
$mempools = $os->discoverMempools()->filter(function (Mempool $mempool) {
if ($mempool->isValid()) {
return true;
}
Log::debug("Rejecting Mempool $mempool->mempool_index $mempool->mempool_descr: Invalid total value");
return false;
});
$this->calculateAvailable($mempools);
MempoolObserver::observe('\App\Models\Mempool');
Modernize mempools (#12282) * mempools to modern module quick hacky hrstorage port * port ucd-snmp-mib to Mempools * Populate DB for ucd Prep for yaml * initial yaml attempt * more complex conversions fixes to YamlDiscovery, make leading $ optional and allow mib::oid format * walk full tables and skip values normalize percentages above 100 * handle precent only ones (specify total as 100) * Move default polling out of YamlMempoolsDiscovery * fixes * Update test data hrstorage should be correct. * perc_warn for hrstorage * Host Resources, record buffer, cached, and shared * Host Resources is always better, don't do both HR and UCD * fix unix, include warning levels * variable size * consolidate skip_values * define mempools schema * number instead of integer * more schema refactor * one more skip_values reference * throw error for invalid oid translation aos6 * a* and Cisco * updated test data * update almost all hrstorage data files * b* * c* with test data use standard cache for hrStorage * use cache for storage module too * hand bsnmp properly * bdcom * exclude total oid from yaml so it is not polled May add a way to ignore this behavior and poll it, but I don't know if that is needed. * automatically handle percent only values * ciscowlc * only poll used or free if we have used, free, and total. * fix skipping * the dlinkoning fix find value when value "name" is numeric * support numeric oids * dnos/ftos attempt * I guess we can't filter on total > 0 * edgecos * e* * f WIP * f* * gwd (aka g*) * h* + procurve * i* * j* * m* * support 0% used memory (however unlikely) * n* * CISCO-PROCESS-MIB memory, share cache with processors module * ignore mempools with invalid total * p* * quanta * r* fix raisecom mibs terribly broken * s-z * style fixes * Move VRP back to PHP and make it actually work * fix zynos * update schema * Update Cisco processor data for description bug fixes * fix comware processors * comware mempools with memory size default precision to 1 * sophos-xg updated data * hrstorage use ram size for buffers, cache, and shared * Show memory available instead of free in device overview * UCD, use same rrd format, store available instead of free in the db. * Calculate availability for HOST-RESOURCES-MIB * Convert UCD to standard polling * rename old rrd files * initial graph work * graph WIP * Graph looking decent * Graph looking decent for hr * remove old ucd_graph code * handle availability mempool more graph cleanup * color adjustments * remove accidental free calculation * Update test data and fix corner cases * fis pfsense * update schema * add default value for mempool_class * fix whitespace * update schema * update schema correctly * one more time * fortigate_1500d-sensors missing oids * Update docs. * fix indent * add implements MempoolsDiscovery explicitly to OS * remove ucd_memory graph references remove unused device_memory graph * remove unused functions * set devices with mempools to rediscover to prevent/minimize gaps * use a subquery * add overview graph * port health mempools table * Update device mempool * only show overview if multiple * Don't override user set warn percentages in discovery * fix missed usage * fix style * Safety check to not rename rrd files incorrectly if migration has not been run. * Fix overview percent bar and represent available and used on the bar * missed an item to convert to mempool_class * percent on the wrong side
2020-11-23 15:35:35 -06:00
$this->syncModels($os->getDevice(), 'mempools', $mempools);
echo PHP_EOL;
$mempools->each(function ($mempool) {
$this->printMempool($mempool);
});
}
}
public function poll(OS $os)
{
$mempools = $os->getDevice()->mempools;
if ($mempools->isEmpty()) {
return;
}
$os instanceof MempoolsPolling
? $os->pollMempools($mempools)
: $this->defaultPolling($os, $mempools);
$this->calculateAvailable($mempools)->each(function (Mempool $mempool) use ($os) {
$this->printMempool($mempool);
if (empty($mempool->mempool_class)) {
Log::debug('Mempool skipped. Does not include class.');
return;
}
Modernize mempools (#12282) * mempools to modern module quick hacky hrstorage port * port ucd-snmp-mib to Mempools * Populate DB for ucd Prep for yaml * initial yaml attempt * more complex conversions fixes to YamlDiscovery, make leading $ optional and allow mib::oid format * walk full tables and skip values normalize percentages above 100 * handle precent only ones (specify total as 100) * Move default polling out of YamlMempoolsDiscovery * fixes * Update test data hrstorage should be correct. * perc_warn for hrstorage * Host Resources, record buffer, cached, and shared * Host Resources is always better, don't do both HR and UCD * fix unix, include warning levels * variable size * consolidate skip_values * define mempools schema * number instead of integer * more schema refactor * one more skip_values reference * throw error for invalid oid translation aos6 * a* and Cisco * updated test data * update almost all hrstorage data files * b* * c* with test data use standard cache for hrStorage * use cache for storage module too * hand bsnmp properly * bdcom * exclude total oid from yaml so it is not polled May add a way to ignore this behavior and poll it, but I don't know if that is needed. * automatically handle percent only values * ciscowlc * only poll used or free if we have used, free, and total. * fix skipping * the dlinkoning fix find value when value "name" is numeric * support numeric oids * dnos/ftos attempt * I guess we can't filter on total > 0 * edgecos * e* * f WIP * f* * gwd (aka g*) * h* + procurve * i* * j* * m* * support 0% used memory (however unlikely) * n* * CISCO-PROCESS-MIB memory, share cache with processors module * ignore mempools with invalid total * p* * quanta * r* fix raisecom mibs terribly broken * s-z * style fixes * Move VRP back to PHP and make it actually work * fix zynos * update schema * Update Cisco processor data for description bug fixes * fix comware processors * comware mempools with memory size default precision to 1 * sophos-xg updated data * hrstorage use ram size for buffers, cache, and shared * Show memory available instead of free in device overview * UCD, use same rrd format, store available instead of free in the db. * Calculate availability for HOST-RESOURCES-MIB * Convert UCD to standard polling * rename old rrd files * initial graph work * graph WIP * Graph looking decent * Graph looking decent for hr * remove old ucd_graph code * handle availability mempool more graph cleanup * color adjustments * remove accidental free calculation * Update test data and fix corner cases * fis pfsense * update schema * add default value for mempool_class * fix whitespace * update schema * update schema correctly * one more time * fortigate_1500d-sensors missing oids * Update docs. * fix indent * add implements MempoolsDiscovery explicitly to OS * remove ucd_memory graph references remove unused device_memory graph * remove unused functions * set devices with mempools to rediscover to prevent/minimize gaps * use a subquery * add overview graph * port health mempools table * Update device mempool * only show overview if multiple * Don't override user set warn percentages in discovery * fix missed usage * fix style * Safety check to not rename rrd files incorrectly if migration has not been run. * Fix overview percent bar and represent available and used on the bar * missed an item to convert to mempool_class * percent on the wrong side
2020-11-23 15:35:35 -06:00
$mempool->save();
$rrd_name = ['mempool', $mempool->mempool_type, $mempool->mempool_class, $mempool->mempool_index];
$rrd_oldname = ['mempool', $mempool->mempool_type, $mempool->mempool_index];
$rrd_def = RrdDefinition::make()
->addDataset('used', 'GAUGE', 0)
->addDataset('free', 'GAUGE', 0);
$fields = [
'used' => $mempool->mempool_used,
'free' => $mempool->mempool_free,
];
$tags = compact('mempool_type', 'mempool_index', 'rrd_name', 'rrd_def', 'rrd_oldname');
data_update($os->getDeviceArray(), 'mempool', $tags, $fields);
});
}
/**
* @param OS $os
* @param \Illuminate\Support\Collection $mempools
* @return \Illuminate\Support\Collection
*/
private function defaultPolling($os, $mempools)
{
// fetch all data
$oids = $mempools->map->only(['mempool_perc_oid', 'mempool_used_oid', 'mempool_free_oid', 'mempool_total_oid'])
->flatten()->filter()->unique()->values()->all();
$data = snmp_get_multi_oid($os->getDeviceArray(), $oids);
$mempools->each(function (Mempool $mempool) use ($data) {
$mempool->fillUsage(
$data[$mempool->mempool_used_oid] ?? null,
$data[$mempool->mempool_total_oid] ?? null,
$data[$mempool->mempool_free_oid] ?? null,
$data[$mempool->mempool_perc_oid] ?? null
);
});
return $mempools;
}
public function cleanup(OS $os)
{
$os->getDevice()->mempools()->delete();
}
/**
* Calculate available memory. This is free + buffers + cached.
*
* @param \Illuminate\Support\Collection $mempools
* @return \Illuminate\Support\Collection|void
*/
private function calculateAvailable(Collection $mempools)
{
if ($mempools->count() > 2) { // optimization
$system = null;
$buffers = null;
$cached = null;
foreach ($mempools as $mempool) {
/** @var Mempool $mempool */
if ($mempool->mempool_class == 'system') {
if ($system !== null) {
Log::debug('Aborted available calculation, too many system class mempools');
return $mempools; // more than one system, abort
}
$system = $mempool;
} elseif ($mempool->mempool_class == 'buffers') {
if ($buffers !== null) {
Log::debug('Aborted available calculation, too many buffers class mempools');
return $mempools; // more than one buffer, abort
}
$buffers = $mempool->mempool_used;
} elseif ($mempool->mempool_class == 'cached') {
if ($cached !== null) {
Log::debug('Aborted available calculation, too many cached class mempools');
return $mempools; // more than one cache, abort
}
$cached = $mempool->mempool_used;
}
}
if ($system !== null) {
$old = format_bi($system->mempool_free);
$system->fillUsage(($system->mempool_used - $buffers - $cached) / $system->mempool_precision, $system->mempool_total / $system->mempool_precision);
$new = format_bi($system->mempool_free);
Log::debug("Free memory adjusted by availability calculation: {$old}iB -> {$new}iB\n");
}
}
return $mempools;
}
private function printMempool(Mempool $mempool)
{
echo "$mempool->mempool_type [$mempool->mempool_class]: $mempool->mempool_descr: $mempool->mempool_perc%";
if ($mempool->mempool_total != 100) {
$used = format_bi($mempool->mempool_used);
$total = format_bi($mempool->mempool_total);
echo " {$used}iB / {$total}iB";
}
echo PHP_EOL;
}
}