Commit Graph

110 Commits

Author SHA1 Message Date
Tony Murray
f2f169ae78 Modernize mempools (#12282)
* mempools to modern module
quick hacky hrstorage port

* port ucd-snmp-mib to Mempools

* Populate DB for ucd
Prep for yaml

* initial yaml attempt

* more complex conversions
fixes to YamlDiscovery, make leading $ optional and allow mib::oid format

* walk full tables and skip values
normalize percentages above 100

* handle precent only ones (specify total as 100)

* Move default polling out of YamlMempoolsDiscovery

* fixes

* Update test data hrstorage should be correct.

* perc_warn for hrstorage

* Host Resources, record buffer, cached, and shared

* Host Resources is always better, don't do both HR and UCD

* fix unix, include warning levels

* variable size

* consolidate skip_values

* define mempools schema

* number instead of integer

* more schema refactor

* one more skip_values reference

* throw error for invalid oid translation
aos6

* a*  and Cisco

* updated test data

* update almost all hrstorage data files

* b*

* c* with test data
use standard cache for hrStorage

* use cache for storage module too

* hand bsnmp properly

* bdcom

* exclude total oid from yaml so it is not polled
May add a way to ignore this behavior and poll it, but I don't know if that is needed.

* automatically handle percent only values

* ciscowlc

* only poll used or free if we have used, free, and total.

* fix skipping

* the dlinkoning
fix find value when value "name" is numeric

* support numeric oids

* dnos/ftos attempt

* I guess we can't filter on total > 0

* edgecos

* e*

* f WIP

* f*

* gwd (aka g*)

* h* + procurve

* i*

* j*

* m*

* support 0% used memory (however unlikely)

* n*

* CISCO-PROCESS-MIB memory, share cache with processors module

* ignore mempools with invalid total

* p*

* quanta

* r*
fix raisecom mibs terribly broken

* s-z

* style fixes

* Move VRP back to PHP and make it actually work

* fix zynos

* update schema

* Update Cisco processor data for description bug fixes

* fix comware processors

* comware mempools with memory size
default precision to 1

* sophos-xg updated data

* hrstorage use ram size for buffers, cache, and shared

* Show memory available instead of free in device overview

* UCD, use same rrd format, store available instead of free in the db.

* Calculate availability for HOST-RESOURCES-MIB

* Convert UCD to standard polling

* rename old rrd files

* initial graph work

* graph WIP

* Graph looking decent

* Graph looking decent for hr

* remove old ucd_graph code

* handle availability mempool
more graph cleanup

* color adjustments

* remove accidental free calculation

* Update test data and fix corner cases

* fis pfsense

* update schema

* add default value for mempool_class

* fix whitespace

* update schema

* update schema correctly

* one more time

* fortigate_1500d-sensors missing oids

* Update docs.

* fix indent

* add implements MempoolsDiscovery explicitly to OS

* remove ucd_memory graph references
remove unused device_memory graph

* remove unused functions

* set devices with mempools to rediscover to prevent/minimize gaps

* use a subquery

* add overview graph

* port health mempools table

* Update device mempool

* only show overview if multiple

* Don't override user set warn percentages in discovery

* fix missed usage

* fix style

* Safety check to not rename rrd files incorrectly if migration has not been run.

* Fix overview percent bar and represent available and used on the bar

* missed an item to convert to mempool_class

* percent on the wrong side
2020-11-23 15:35:35 -06:00
Jellyfrog
cfd9dce620 Convert Virtual Machine pages to Laravel (#12287)
* Convert Virtual Machine pages to Laravel

* wip

* wip

* wip

* wip

* wip

* wip

* delete

* wip

* wip

* move powerStateLabel
2020-11-11 01:15:20 +01:00
Jellyfrog
8977bb50ad Remove some unused files (#12307) 2020-11-07 22:00:47 -06:00
Hayden
e34b6877fd Add OSPF cost (TOS) (#11929)
* add OspfTos model to include ospfIfMetricEntry data (ospf interface cost)

* remove WIP comment

* add new model and db migration

* update OspfTos relationship to OspfPort and correct cleanup call

* add "Cost" to device\routing\ospf web UI interface

* updated db-schema test unit

* updated db_schema to resolve conflicts

* removed tos metrics from rrd ospf stats due to rrdcached errors

* reorder db_schema.yaml to match upstream

* readd ospf_tos to db_schema

* styleCI changes

* Capture OSPF test data

* add ospf unit tests for iosxr

* ospfAuthType

* update db_schema

* remove extra migrations

* add ospf tests for iosxe

* add ospf test to ios

* merge ospf_tos into ospf_ports

* update db_schema

* update ospf module unit tests for ios, iosxe, iosxr

* fix ospf_ports ospfIfMetricStatus column type

* update db_schema

* more efficient dropColumn in down() migration

Co-authored-by: Tony Murray <murraytony@gmail.com>
2020-11-07 09:27:25 -06:00
Tony Murray
25722bbbd5 Faster initial database creation (#12297)
* Faster initial database creation
Add db dumps to improve initial db migration
On my test system 13s -> 7s, a 46% reduction.
Should help out a lot on systems like RPi

* lnms schema:dump command

--snapshots saves db snapshots, otherwise it saves the yaml

* fix style

* fix timezones...

* not working with in-memory db yet
remove build.sql
2020-11-06 01:19:15 +01:00
Jellyfrog
50c8033099 Laravel 8.x Shift (#12235)
* Shift HTTP kernel and middleware

* Shift service providers

* Shift console routes

* Shift to class based factories

* Namespace seeders

* Shift PSR-4 autoloading

* Default config files

In an effort to make upgrading the constantly changing config files
easier, Shift defaulted them. This allows you to review the commit
diff for once for customizations when you are done Shifting.

Moving forward, consider using ENV variables or create a separate
config file to allow the core config files to remain as default
as possible.

* Shift Laravel dependencies

* Shift return type of base TestCase methods

From the [PHPUnit 8 release notes][1], the `TestCase` methods below now declare a `void` return type:

- `setUpBeforeClass()`
- `setUp()`
- `assertPreConditions()`
- `assertPostConditions()`
- `tearDown()`
- `tearDownAfterClass()`
- `onNotSuccessfulTest()`

[1]: https://phpunit.de/announcements/phpunit-8.html

* Shift cleanup

* console routes

* composer update

* factories

* phpunit

* bootstrap pagination

* model factory

* wip

* Apply fixes from StyleCI (#12236)

* wip

* Apply fixes from StyleCI (#12238)

* wip

* wip

* wip

* wip

* Apply fixes from StyleCI (#12240)

* wip

* Apply fixes from StyleCI (#12242)

* composer update

* Bump to PHP 7.3 minimum

Co-authored-by: Laravel Shift <shift@laravelshift.com>
2020-11-03 10:18:31 -06:00
Hans Erasmus
6c4596d1b9 Added new authalgo support for SNMPv3 (#11966)
* Added new authalgo support for SNMPv3

* Update 2020_07_27_00522_alter_authalgo_column.php

Fixed case

* Update addhost.inc.php

Fixed case

* Update snmp.inc.php

Fixed case

* Update db_schema.yaml

Fixed case

* Fixed SHA-* order to ascending

* Update 2020_07_27_00522_alter_authalgo_column.php

Fixed space

* Update db_schema.yaml

Fixed line to be of type string

* Update config_definitions.json

Fixed sorting

* Fixed test file case sensitivity

* Fixed rollback function

* SourceDoctor code for checks

* Fixed indentation issues on Travis test

* Update to re-trigger testing stuck on CLA step

* Added crypto-algo updates

* Incorporated suggestions from Jellyfrog

* Added documentation entry

* Fixed travis spacing error

* Fixed validations

* Delete snmpv3_version_compare.patch

Sorry, this one was not supposed to be there.

* Added poller evaluation

* Added poller comparison

* Fixed merge-conflicts

* Fixed typo in test unit

* code refinements
only can check on webui that it supports the algorithms, hopefully, the eventlog is enough to notify users of distributed pollers.

* style fixes and remove openssl setting

* fix device access

* fix missing migration column

Co-authored-by: Tony Murray <murraytony@gmail.com>
2020-10-29 13:02:26 -05:00
Hayden
774d95c118 allow nullable ospf auth (#12249)
* ospfAuthType

* update db_schema

* remove extra migrations
2020-10-21 21:07:12 -05:00
Richard Kojedzinszky
2ca6f2f8d4 schema: add missing primary keys (#12106)
* schema: add missing primary keys

* Exclude ipv4_mac.id

* Made primary key migrations SQLite compatible

Co-authored-by: Tony Murray <murraytony@gmail.com>
2020-10-14 20:37:40 -05:00
Tony Murray
1e4702fa4f Support multiple daily process locking backends with distributed polling (#11896)
* Implement locks in the file cache

* Replace custom locks

* implement restore lock
Used when re-hydrating

* remove legacy use statements

* Add class descriptions

* Fix style

* Default to database cache driver

* missed cache_locks table
prevent chicken-egg issue

* style fixes

* Remove custom file lock implementation

* missed items from file cache

* Update schema definition
hmm, other schema noise must be from manual modification as this is generated from a freshly migrated db.

* require predis, it is pure python, so no harm in adding

* and set predis as the default client
2020-10-07 07:36:35 -05:00
Tony Murray
863fb96205 Scheduled maintenance test (#12171)
* alert schedule test

* Fix style

* Skip broken tests for now
2020-10-01 03:00:42 -05:00
Negatifff
444b9544a0 new index in alert_log table (#12143)
* new index in alert_log table

* codefix1

* codefix2

* added second index

* codefix

* Apply suggestions from code review

Co-authored-by: Jellyfrog <Jellyfrog@users.noreply.github.com>
2020-09-24 17:05:21 +02:00
Jellyfrog
29f45ca352 Apply fixes from StyleCI (#12124) 2020-09-21 15:59:34 +02:00
Jellyfrog
0d56bbd946 Apply fixes from StyleCI (#12123) 2020-09-21 15:51:53 +02:00
Jellyfrog
77c531527c Apply fixes from StyleCI (#12117)
* Apply fixes from StyleCI

* Disable style check
2020-09-21 14:54:51 +02:00
Adam Bishop
bbdb1a2a5d Rewrite smokeping script to be an lnms command (#11585)
* Rewrite smokeping script to be an lnms command

* Add a default number of probes for smokeping

* Formatting fixes

* Refactor to simplify a couple of methods

* Fix a name collision when using more than 60 threads

* Simplify

* First pass at documentation

A few climate fixes too.

I'm a little dubious about the + LibreNMS fix - it could be included
in /etc/smokeping/config if it is really needed, but it sounds like
(looking at git blame) it's needed for if the script generates a broken
configuration file.

I'm thinking we should just not generate broken config.

* Improve testability and add some tests

* Load laravel to make translations available

* Second pass at documentation

* Fix brace

* Extend the device factory to include the type

Also adds a device group fake I created before I realised I didn't need it

* Mimic gen_smokeping.php a little more closely

* Update tests to properly verify old and new behaviour against each other

* Replace gen_smokeping with a wrapper

* Don't double whitespace

Render does this automatically.
Explicitly order by hostname too.

* Make faker less likely to generate duplicates

I tried adding a unique constraint here, but it didn't have the
intended effect.

Extending the hostname like this seems to work fine - I tried
generating 10 million hosts and got no duplicates, compared to
duplicates appearing with as few as 100 hosts without this
 change.

A true fix would be to add an 'fqdn' fake upstream.

* Make the tests more robust

* Assorted bug fixes

* Style corrections

* Handle 'generic' devices

* Fix an issue that came up during rebase

Co-authored-by: Tony Murray <murraytony@gmail.com>
2020-09-18 00:04:54 -05:00
Tony Murray
b22d27b988 Allow index rename migration to recover partially completed previous run (#12084) 2020-09-12 23:20:15 +02:00
SourceDoctor
427878dcaa Availability Calculation for all Devices (#12004)
* Availability Calculation for all Devices

* remove not more need uptime column
2020-08-28 13:01:54 +02:00
Jellyfrog
cf13e92f05 Add missing index for device_perf (#11974)
Makes the latency page load faster in case of a large device_perf table
2020-07-31 16:07:00 +02:00
Tony Murray
020c5fd7e1 Remove legacy code and fix missing device graphs (#11950)
* removing $graphs global

* remove unused things

* fix some additional graphs

* Fix graphs persisting too soon

* correct name for poller module performance graph

* only one type of graph is used here
2020-07-23 09:57:22 -05:00
SourceDoctor
a61b119d5f . 2020-06-23 19:31:43 +02:00
SourceDoctor
70c00d1b4a . 2020-06-23 19:12:58 +02:00
SourceDoctor
a454189a57 double to float in migration 2020-06-23 17:38:17 +02:00
SourceDoctor
c5c99e13d1 partial double - other side 2020-06-23 16:57:30 +02:00
SourceDoctor
bfcec4dc16 change column type to float 2020-06-23 16:45:08 +02:00
SourceDoctor
a8b188e9ab . 2020-06-23 10:06:15 +02:00
SourceDoctor
3ce4860860 . 2020-06-23 02:19:40 +02:00
SourceDoctor
daea015c88 . 2020-06-23 02:10:07 +02:00
SourceDoctor
14c3690b88 . 2020-06-23 02:09:17 +02:00
SourceDoctor
3275c372f4 . 2020-06-23 01:34:36 +02:00
SourceDoctor
cf7f45fe53 . 2020-06-23 01:30:09 +02:00
SourceDoctor
3416bf8fb2 . 2020-06-23 01:16:32 +02:00
SourceDoctor
9be112530a . 2020-06-23 01:14:05 +02:00
SourceDoctor
afa26d66b5 Fix datatype for Availability_perc Column 2020-06-23 01:06:49 +02:00
SourceDoctor
4e6a7291a2 Device Availability Calculation (#11784)
* Device Availability Calculation

* Travis fix

* .

* schema corrections

* flexible duration

* travis

* .

* .

* .

* .

* remove not needed code

* line Text to RRD

* update humantime

* .

* only set up again if device was marked as down

* set RRD area transparency

* save uptime also, to keep last availability as good as possible

* file description correction

* look for outages even if uptime > duration
2020-06-22 22:57:30 +02:00
Tony Murray
867aa132c1 Merge pull request #11661 from SourceDoctor/app_dhcp_enhancement
Application DHCP Upgrade
2020-06-16 17:03:45 -05:00
Tony Murray
240b84aa3f Merge pull request #11776 from TheGreatDoc/ups-traps
Added UPS-MIB Trap On Battery (upsTraps.0.1)
2020-06-14 16:41:42 -05:00
Tony Murray
d00e18a32e support sqlite 2020-06-12 09:27:25 -05:00
Tony Murray
13ade9ab6c Merge branch 'master' into alerts-carbon 2020-06-12 07:48:51 -05:00
SourceDoctor
1093315700 Merge branch 'master' into app_dhcp_enhancement 2020-06-08 20:51:41 +02:00
Tony Murray
300645388f Dispatcher Service settings (#11760)
* Poller settings WIP

* Poller settings WIP2

* working on SettingMultiple

* setting multiple working

* settings sent with all required info

* fix translation

* Fix keys

* fix groups setting

* Apply settings to service
fixes and validations for setting

* don't error when no poller_cluster entry exists

* hid tab when no poller cluster entries

* Authorization

* make prod

* daily maintenance toggle should be advanced

* Update schema def
2020-06-08 08:27:03 -05:00
TheGreatDoc
2f245c0e82 Removed sensor_id from factory()->Sensor:class 2020-06-08 06:30:13 +02:00
TheGreatDoc
e1d5b8ca5f Added sensor_oid to factory->sensor 2020-06-07 12:49:30 +02:00
TheGreatDoc
95ffe2a8b4 Fix correct field 2020-06-07 11:55:54 +02:00
TheGreatDoc
6f91e9e287 Create factory->Sensor model. Fix some format issues in UpsTrapsOnBattery.php and more UpsTrapsOnBatteryTests 2020-06-07 11:21:14 +02:00
SourceDoctor
97e2ab4c04 Merge branch 'master' into app_dhcp_enhancement 2020-05-23 00:12:14 +02:00
Tony Murray
5fc3fcb9e8 Make migrations work in SQLite (#11643)
* WIP Sqlite

Down methods don't work either, avoid them with refresh for now.

WIP persistent support

WIP db_schema

WIP db_schema 2

Update new migrations...

revert dump_db_schema changes for now, too much difference.

fix migrations on mysql

fix up some more items, this should be our target schema
lots of index renames and a two misc changes

index rename WIP

index rename WIP

another round

fix up new schema changes

try case insensitive fix

Trying tests WIP

fix down methods

DBSetupTest working (uses mysql)

Test sqlite migrations work

* Properly validate sqlite output

* revert glue changes, should be separate PR

* remove dusk workaround

* remove unused variables

* import

* sqlite capitalization

* Revert some refresh tests

* testing_mysql back to testing
2020-05-22 16:49:21 -05:00
SourceDoctor
afd2be0292 . 2020-05-22 23:41:15 +02:00
SourceDoctor
4ef5bb1ea6 class rename 2020-05-22 23:34:51 +02:00
SourceDoctor
cbe572f959 Application DHCP Upgrade 2020-05-22 23:02:19 +02:00