Modernize mempools (#12282)

* mempools to modern module
quick hacky hrstorage port

* port ucd-snmp-mib to Mempools

* Populate DB for ucd
Prep for yaml

* initial yaml attempt

* more complex conversions
fixes to YamlDiscovery, make leading $ optional and allow mib::oid format

* walk full tables and skip values
normalize percentages above 100

* handle precent only ones (specify total as 100)

* Move default polling out of YamlMempoolsDiscovery

* fixes

* Update test data hrstorage should be correct.

* perc_warn for hrstorage

* Host Resources, record buffer, cached, and shared

* Host Resources is always better, don't do both HR and UCD

* fix unix, include warning levels

* variable size

* consolidate skip_values

* define mempools schema

* number instead of integer

* more schema refactor

* one more skip_values reference

* throw error for invalid oid translation
aos6

* a*  and Cisco

* updated test data

* update almost all hrstorage data files

* b*

* c* with test data
use standard cache for hrStorage

* use cache for storage module too

* hand bsnmp properly

* bdcom

* exclude total oid from yaml so it is not polled
May add a way to ignore this behavior and poll it, but I don't know if that is needed.

* automatically handle percent only values

* ciscowlc

* only poll used or free if we have used, free, and total.

* fix skipping

* the dlinkoning
fix find value when value "name" is numeric

* support numeric oids

* dnos/ftos attempt

* I guess we can't filter on total > 0

* edgecos

* e*

* f WIP

* f*

* gwd (aka g*)

* h* + procurve

* i*

* j*

* m*

* support 0% used memory (however unlikely)

* n*

* CISCO-PROCESS-MIB memory, share cache with processors module

* ignore mempools with invalid total

* p*

* quanta

* r*
fix raisecom mibs terribly broken

* s-z

* style fixes

* Move VRP back to PHP and make it actually work

* fix zynos

* update schema

* Update Cisco processor data for description bug fixes

* fix comware processors

* comware mempools with memory size
default precision to 1

* sophos-xg updated data

* hrstorage use ram size for buffers, cache, and shared

* Show memory available instead of free in device overview

* UCD, use same rrd format, store available instead of free in the db.

* Calculate availability for HOST-RESOURCES-MIB

* Convert UCD to standard polling

* rename old rrd files

* initial graph work

* graph WIP

* Graph looking decent

* Graph looking decent for hr

* remove old ucd_graph code

* handle availability mempool
more graph cleanup

* color adjustments

* remove accidental free calculation

* Update test data and fix corner cases

* fis pfsense

* update schema

* add default value for mempool_class

* fix whitespace

* update schema

* update schema correctly

* one more time

* fortigate_1500d-sensors missing oids

* Update docs.

* fix indent

* add implements MempoolsDiscovery explicitly to OS

* remove ucd_memory graph references
remove unused device_memory graph

* remove unused functions

* set devices with mempools to rediscover to prevent/minimize gaps

* use a subquery

* add overview graph

* port health mempools table

* Update device mempool

* only show overview if multiple

* Don't override user set warn percentages in discovery

* fix missed usage

* fix style

* Safety check to not rename rrd files incorrectly if migration has not been run.

* Fix overview percent bar and represent available and used on the bar

* missed an item to convert to mempool_class

* percent on the wrong side

doc/Developing/os/Mem-CPU-Information.md

@@ -6,31 +6,87 @@ Processor for your new device.
 
 #### Memory
 
-Detection for memory is done via two php scripts, one for discovery
-and the other for polling:
+LibreNMS will attempt to detect memory statistics using the standard HOST-RESOURCES-MIB and UCD-SNMP-MIB MIBs.
+To detect non-standard MIBs, they can be defined via Yaml.
 
-`includes/discovery/mempools/pulse.inc.php`
+##### YAML
 
-```php
-<?php
-echo 'PULSE-MEMORY-POOL: ';
-$usage = str_replace('"', "", snmp_get($device, 'iveMemoryUtil.0', '-OvQ', 'PULSESECURE-PSG-MIB'));
-if (is_numeric($usage)) {
-    discover_mempool($valid_mempool, $device, 0, 'pulse-mem', 'Main Memory', '100', null, null);
-}
+In order to successfully detect memory amount and usage, two of the for keys below are required.  Some OS only
+provide a usage percentage, which will work, but a total RAM amount will not be displayed. 
+
+- total
+- used
+- free
+- percent_used
+
+`includes/definitions/discovery/mempools/arubaos.yaml`
+
+```yaml
+mempools:
+    data:
+        -
+            total: WLSX-SWITCH-MIB::sysXMemorySize
+            used: WLSX-SWITCH-MIB::sysXMemoryUsed
+            precision: 1024
 ```
 
-`includes/polling/mempools/pulse.inc.php`
+The code can also interpret table based OIDs and supports many of the same features as Health Sensors
+including {{ }} parsing, skip_values, and precache.
+
+Valid data entry keys:
+
+- `oid` oid to walk to collect processor data
+- `total` oid or integer total memory size in bytes (or precision), 
+- `used` oid memory used in bytes (or precision)
+- `free` oid memory free in bytes (or precision)
+- `percent_used` oid of percentage of used memory
+- `descr` A visible description of the memory measurement defaults to "Memory"
+- `warn_percent` Usage percentage to used for alert purposes
+- `precision` precision for all byte values, typically a power of 2 (1024 for example)
+- `class`used to generate rrd filename, defaults to system.  If system, buffers, and cached exist they
+will be combined to calculate available memory.
+- `type` used to generate rrd filename, defaults to the os name
+- `index` used to generate rrd filename, defaults to the oid index
+- `skip_values` skip values see [Health Sensors](../Developing/os/Health-Information.md) for specification
+- `snmp_flags` additional net-snmp flags
+
+##### Custom Processor Discovery and Polling
+
+If you need to implement custom discovery or polling you can implement
+the MempoolsDiscovery interface and the MempoolsPolling interface in the OS class.
+MempoolsPolling is optional, standard polling will be used based on OIDs stored in the database.
+
+OS Class files reside under `LibreNMS\OS`
 
 ```php
 <?php
-echo 'Pulse Secure MemPool\n';
-$perc = str_replace('"', "", snmp_get($device, "iveMemoryUtil.0", '-OvQ', 'PULSESECURE-PSG-MIB'));
-if (is_numeric($perc)) {
-    $memory_available = str_replace('"', "", snmp_get($device, "memTotalReal.0", '-OvQ', 'UCD-SNMP-MIB'));
-    $mempool['total'] = $memory_available;
-    $mempool['used'] = ($memory_available / 100 * $perc);
-    $mempool['free'] = ($memory_available - $mempool['used']);
+
+namespace LibreNMS\OS;
+
+use LibreNMS\Interfaces\Discovery\MempoolsDiscovery;
+use LibreNMS\Interfaces\Polling\MempoolsPolling;
+
+class Example extends \LibreNMS\OS implements MempoolsDiscovery, MempoolsPolling
+{
+    /**
+     * Discover a Collection of Mempool models.
+     * Will be keyed by mempool_type and mempool_index
+     *
+     * @return \Illuminate\Support\Collection \App\Models\Mempool
+     */
+    public function discoverMempools()
+    {
+        // TODO: Implement discoverMempools() method.
+    }
+
+    /**
+     * @param \Illuminate\Support\Collection $mempools \App\Models\Mempool
+     * @return \Illuminate\Support\Collection \App\Models\Mempool
+     */
+    public function pollMempools($mempools)
+    {
+        // TODO: Implement pollMempools() method.
+    }
 }
 ```
 
@@ -75,7 +131,7 @@ Accessing values within yaml:
 | {{ $count }} | The count of entries (starting with 1) |
 | {{ $`oid` }} | Any oid in the table or pre-fetched |
 
-##### Custom Discovery and Polling
+##### Custom Processor Discovery and Polling
 
 If you need to implement custom discovery or polling you can implement
 the ProcessorDiscovery interface and the ProcessorPolling interface in the OS class.