Commit Graph

544 Commits

Author SHA1 Message Date
Tony Murray
fff66d3c00 Feature: Generic discovery and poller tests (#7873)
* Processor Tests!

* Capture data from live devices easily.

* fix up some stuff, remove powerconnect things as they seem to be just broken.

* formatting, fix missing assignment
add netonix processor data

* fix multi-line, always add sysDescr and sysObjectID
ios cpm test file

* revert composer change and fix whitespace issues

* add help text

* missed help text

* tighter debug output

* handle empty strings properly and mibs with numbers

* use keys for sorting as intended

* fix type with empty data

* oops :)

* whitespace fix

* try installing fping

* Fix TestCase collision + cleanup

* mark TestCase as abstract
don't run two instances of snmpsim

* better database dumps, improved capture

* style fixes

* fix quotes add a few more tables

* add --prefer-new, properly merge data

* Support separate discovery and poller data. But don't waste space if they aren't needed.

* refactor to use class
collects all code in one place for reusability

* reorganize

* Print out when not saving.

* Support for running multiple (or all) modules at once.

* tidy

* Change unit test to a generic name and test all modules we have data for.

* Add documentation and a few more tidies

* whitespace fixes

* Fix tests, add a couple more modules, more docs

* More docs updates

* Fix scrutinizer issues

* add bgp-peers
2017-12-20 08:36:49 -06:00
Neil Lathwood
2c919a48ff fix: Re-added peeringdb back into daily.sh (#7884) 2017-12-09 17:19:56 +00:00
Tony Murray
f1fa61758b feature: Print out operations stats per poller/discovery module (#7839)
* Print out operations stats per poller/discovery module

* Add line return so we can see polled time better.
2017-12-02 22:55:53 +00:00
Tony Murray
2e73b75297 fix: Use memcached to lock daily processes on Distributed Pollers (#7735)
* fix: use memcached to lock daily processes on Distributed Pollers

* All the locks!
2017-11-24 09:37:52 +00:00
Tony Murray
d59695f08e fix: revert to two gets for sysDescr and sysObjectID
Some devices don't like asking for them in one get for some reason.
2017-11-15 11:26:25 -06:00
Tony Murray
29fc07f018 mib_dir consistency (#7700)
* mib_dir consistency
also, all the snmpget mib_dir lines were unneeded.

* re-add mib_dir for now, can remove the redundancy later if I come up with something.

* fix tests
2017-11-13 20:45:00 +00:00
Tony Murray
5ad28c9202 feature: Track rrdtool time for poller (#7706)
* feature: track rrdtool time for poller

* Removed var_dump()
2017-11-11 19:44:25 +00:00
Zmegolaz
f453875c0f Merge pull request #7617 from Zmegolaz/fping-options
Fix: Better sanity checks of fping options.
2017-11-09 08:42:06 +01:00
Tony Murray
acb1de5772 feature: Implement snmp_getnext() (#7678)
find correct path to snmpgetnext (will do the same for others later)
fix undefined index in get_mib_dir()
2017-11-08 15:15:49 +00:00
Zmegolaz
ddb60f4266 Interval can't be lower than timeout. Set options to the limit if configured value was over/under the limit. 2017-11-03 23:07:42 +01:00
Tony Murray
54b476c769 fix: devices detected as ibmtl or generic (#7618)
* fix: devices detected as ibmtl
Guess at the sysObjectID, please report if you have devices that should be ibmtl and are no longer detected.
Add output of os in discovery.

* Always load all os for discovery.  Should be cached most of the time.
2017-11-03 16:10:24 -05:00
Zmegolaz
5126fd06fa Better sanity checks of fping options. 2017-11-03 11:00:27 +01:00
Zmegolaz
d9df643927 fix: Use correct sensor ID when removing device. (#7611) 2017-11-02 20:40:51 +00:00
Tony Murray
0d07f85257 feature: Allow snmpget in os discovery yaml (#7587)
* feature: Allow snmpget in os discovery yaml

Re-submit after release

* Remove extra trim in discovery and in snmp_get.
also trim() always returns a string, so is_string() check is a waste of cpu cycles.
2017-11-02 12:41:18 +00:00
Tony Murray
fb45f00340 fix: ping fails on servers that don't set PATH in cron (#7603)
* fix: ping fails on servers that don't set PATH in cron
Find the full path to fping and persist it in the database.
Adds the ability to persist settings with Config::set()

* Add ability to set webui settings.
No display of for paths, because it would be inconsistent.
2017-11-01 21:56:47 +00:00
Zmegolaz
297e5ae874 fix: Fail isPingable() if fping fails, take 2 (#7585)
* Fail isPingable() if fping fails, take 2.

* Fixed scrutinizer issues.

* Filter out the values we want instead of multi level array.
2017-10-31 08:31:29 -05:00
Tony Murray
426916dbe4 Revert "feature: Allow snmpget in os discovery yaml (#7558)" (#7576)
This reverts commits:
193db02475
b73c4e9d6e
a93bb6635d
2017-10-29 14:04:26 -05:00
Tony Murray
a93bb6635d refactor: Use one snmpget during os discovery (#7566)
Use only one snmpget to fetch sysObjectId and sysDescr
fix multiline data in snmp_get_multi_oid()
add snmp_get_multi_oid() to mock.snmp.inc.php
2017-10-28 11:53:05 +01:00
Zmegolaz
f8d7ccfe0d feature: Support for up/down detection of ping only devices.
* Added support for ping only devices.

* Renamed sql-schema/206 to 207.

* Discovery tried to detect the OS when it shouldn't.

* Updated db_schema.yaml

* Added ping icon.

* Don't show unused graphs and tabs when SNMP is disabled.

* Allow the user to specify OS of ping only devices.

* Removing custom OS now changes it to 'ping'.

* Removed unnecessary use of mres().

* UI select box for SNMP enable/disable.

* Hide device_ping_perf if ping is disable on device.

* Fixed SNMP settings update status messages.

* Added functionality to add ping only devices via the web ui.

* Added ping only option to addhost.php

* Added ping only support to snmp-scan.py

* Moved sql 208 to 211

* Fixed scrutinizer issues.

* Fixed broken ossearch.

* Added ping only support to the API.

* Updated API doc.

* Added (optional) to OS and hardware description. Hid Port Settings, Applications, Modules, Storage, Processors, Memory and Components from the edit menu

* Style fix.

* Updated ping icon.

* clean() instead of mres(). More escaping. Better help in snmp-scan.py and addhost.php

* Fixed scrutinizer issue.

* Always try SNMP in snmp-scan.py, new option for it in addhost.php. Slice instead of chunk in ajax_ossuggest.php. Other minor style changes.

* Updated sql modifications to insert the new column in the same place as in db_schema.yaml.
2017-10-27 22:59:25 -05:00
Tony Murray
b73c4e9d6e refactor: Allow os discovery deferral (#7561)
Defer some os till last, generic or slow ones. (linux, freebsd, ibmtl)
last file standing is ubnt.inc.php
2017-10-27 21:55:14 +01:00
Tony Murray
193db02475 feature: Allow snmpget in os discovery yaml (#7558)
* feature: Allow snmpget in os discovery yaml
Convert all remaining os except airos.

Affected OS: asuswrt-merlin, ddnons, dsm, extrahop, huaweiups, ibmtl, pcoweb, pktj, qnap, remoteeye4, sentry3, sentry4, tomato

There should be not change in detection.

* Fix, asuswrt-merlin and tomato with snmpsim.  May not have been an issue with actual devices because of -Oa leading " was a bit odd.

* missed file
2017-10-26 20:48:33 +01:00
Tony Murray
51ba934e11 feature: Added web validation (#7474)
* refactor: convert validations to modules to prep for gui integration

* accidentally dropped, schema update

* fix accidental output to webui

* mention discovery-wrapper.py and re-arrange.

* refine some fix statements

* rename the Config validation group to Configuration.

* fix some scrutinizer issues
remove as many local functions from validator.php as possible
move extensions from pre-check
remove duplicate timezone check
looks like there is some db schema differences between mariadb 10.1 and 10.2, investigating

* mariadb version diff take2

* Check schema version first for database.
Remove stop to go back to command line for install docs.
Add helpful link when there is no devices added to /addhost

* fix incorrect validation for empty string defaults

* Fix style

* Add additional file permissions checks

* catch exception and fail for invalid timezone
Change visuals a bit.

* add php version warning

* fix space
2017-10-26 07:56:09 +01:00
Chris A. Evans
04bb65b762 newdevice: Added Brocade IronWare interface dBm sensor support (#7434)
* Added IronWare iface dBm sensor and related OID pr~e-cache

* fixed syntax violations

* added return_num() function and modified ironware dbm sensor to use it

* Update functions.php

* Update ironware.inc.php
2017-10-17 17:36:49 +01:00
Neil Lathwood
cdf31c47e1 Revert "bug: fail isPingable if fping errors" (#7496)
* Revert "fix: Fixed clickatell alert transport (#7446)"

This reverts commit 9774acd055.

* Revert "newdevice: Added support for Eaton ATS devices (#7448)"

This reverts commit 1318e5871c.

* Revert "newdevice: Updated Zhone MXK detection (#7488)"

This reverts commit 81667b9e20.

* Revert "docs: Added HipChat V2 WebUI Config Example (#7486)"

This reverts commit a31eaa9839.

* Revert "fix: Fail isPingable if fping errors (#7426)"

This reverts commit 1998845f06.
2017-10-16 11:56:47 +01:00
Daniel Preussker
1998845f06 fix: Fail isPingable if fping errors (#7426) 2017-10-15 20:29:08 +01:00
Tony Murray
c0b267b49c refactor: Discovery protocols re-write (#7380)
* refactor: discovery protocols re-write
extract more code to make it easier to read
use snmpwalk_group
walk lldpRemTable instead of lldpRemoteSystemsData
Optimize discover_device to reduce dns lookups of bad dns names
Make discovery_by_ip behavior consistent between cdp and lldp
move can_skip_discovery to discovery functions

Tested lldp, cdp, and ospf.

* always discover links even if the remote device isn't in system
Fix discover_link always updating

* fix style

* fix update, not fetching id

* increase platform length.  Most devices use sysDescr here, which is max length 255

* remove orphaned links

* update db_schema.yaml
2017-10-04 08:16:23 +01:00
Tony Murray
6ea6218f47 refactor: rewrite is_valid_port() (#7360)
* refactor: speed up is_valid_port()
Convert is_valid_port to Config
Several Config optimizations
Update documentation

* fix getCombined() when os key is not set, but global is.
Add more tests
2017-09-21 18:02:01 +01:00
Neil Lathwood
b19b9e90d2 Revert "refactor: rewrite is_valid_port()" (#7355)
* Revert "refactor: Added detection of vlan name changes (#7348)"

This reverts commit 4ad8faecdc.

* Revert "refactor: Rewrite is_valid_port() (#7337)"

This reverts commit 31607418e1.
2017-09-17 08:21:28 +01:00
Tony Murray
31607418e1 refactor: Rewrite is_valid_port() (#7337)
Convert is_valid_port to Config
Several Config optimizations
Update documentation
2017-09-16 10:13:03 +01:00
Tony Murray
7b262a6851 refactor: Improve yaml state discovery (#7221)
* feature: Improve yaml state discovery
Handle state values that are returned as strings instead of int
Synchronize state values for existing state translations so we can change them without creating a new translation and losing historical data
More extensive/verbose yaml discovery phpunit tests
dbBulkInsert, use the first entry instead of requiring the first entry to be at index 0

* Update sensor state documentation
re-order values for better readability
remove os check
Use snmpwalk_group since it is more flexible

* Add some more debug output in dynamic discovery
2017-09-03 19:58:39 +01:00
Zmegolaz
5441bafc81 fix: Fixed IPv6 host renaming (#7275)
* Fixed IPv6 host renaming

* Added get_rrd_dir() and modified other files to use it where appropriate.

* rrd_name() now uses the new function get_rrd_dir(), to make it simpler to modify the escaping in the future.

* Wrong function name in rrdstep.php
2017-09-02 13:45:31 -05:00
Neil Lathwood
932b05c38c fix: Updated dump_db_schema() to use default 0 if available (#7225)
* fix: Updated dump_db_schema() to use default 0 if available

* added removed column for testing
2017-08-26 20:23:30 +01:00
Neil Lathwood
ad6e82b3c7 fix: Fixed dump_db_schema / validate to work with newer versions of MariaDB (#7162)
* fix: Fixed dump_db_schema / validate to work with newer versions of MariaDB

* removed debug
2017-08-17 08:25:45 +01:00
Tony Murray
3ca6f84013 fix: bug in ipv62snmp function (#7135)
typo compressed instead of uncompressed
Tidied up the function and added unit tests.
2017-08-09 08:47:35 -05:00
Tony Murray
1d777693e1 feature: New IP parsing classes. Removes usage of Pear Net_IPv4 and Net_IPv6. (#7106)
* Refactor remove use of pear Net_IPv4 and Net_IPv6

Caution, I added tests to test the functionality of the new functions, but I'm not 100% certain the replaced usages match 100%.  Please test.

Left usage in generate-iplist.php because I'm not sure the use of that script, so I didn't want to implement additional functionality to support it.

Add support for ipv6 fo ipInNetwork() function

Add phpdocs to functions
Remove accidental inclusion

Use binary operations instead of string for networKFromIp()
Tidy up binary operations in ipInNetwork()

tidy parsing cidr network

Change to a class based implementation.

Update phpdocs.
Fully catch exceptions for hex parsing.

Fix mistake.

support older phpunit versions

Fix php shifting to negative with php 5

* Fix graph authorization. Uncaught exception and checking an ipv4 against an ipv6 network == fail.

* Also remove colons from hex strings.

* Fix typo in bgp polling
Two passes of removing invalid characters from hex strings. (That way we don't mangle valid IPs)
2017-08-08 14:14:58 -05:00
Neil Lathwood
c3627ec4e7 Merge pull request #7085 from murrant/device-status-event
fix: Add device status reason to up messages.
2017-08-01 21:49:15 +01:00
Tony Murray
f1fd01f276 Add device status reason to up messages. 2017-07-27 08:27:52 -05:00
Tony Murray
181b0fb7d5 refactor: Use the new locks for schema updates (#6931)
* Use the new locks for schema updates

* removed schema lock from discovery

* Add the ability to wait for a lock.
Add unit tests

* Add MysqlLock
Use that for the schema updates
Wait up to 30s for other schema updates to complete.

* Switch the schema lock back to a file lock for now.
Make FileLock support indefinite locking without polling.
Add a warning to MysqlLock for scenarios where it won't work.

* Delete MysqlLock.php

* Removed MySQL lock tests
2017-07-17 19:35:08 +01:00
Neil Lathwood
20ce209b27 Return more descriptive error when adding duplicate devices on sysName (#7019)
* Return more descriptive error when adding duplicate devices on sysName

* clean up functions to make more clear and reduce duplication

* moved return to correct place
2017-07-16 22:01:07 +01:00
Tony Murray
90aa485149 refactor: discovery code cleanups (#6856)
* Use foreach $key => value

* use get_port_by_ifIndex
get rid of the rest of the array_keys()

* improve hex_to_ip() function, add tests, use in more places

* Simplify can_skip_discovery logic
2017-06-22 21:08:20 +01:00
Tony Murray
7a87f12380 fix: custom config.php os settings (#6850)
fixes: #6848
2017-06-16 14:08:38 +01:00
Ivan
db47a43ce9 Added host to reachability failure message (#6680) 2017-05-18 21:03:55 +01:00
Neil Lathwood
6734e84382 feature: Added ability to whitelist ifDescr values from being ignored with good_if (#6584)
* feature: Added ability to whitelist ifDescr values from being ignored with good_if

* added ras to enterasys yaml
2017-05-05 15:18:33 -05:00
Tony Murray
4dbab407bb refactor: OS discovery tests are now dynamic (#6555)
* refactor: OS discovery tests are now dynamic
All that needs to be done now is supply snmprec files with the proper name. os.snmprec or os_description.snmprec
I also tweaked the loading of yaml files to speed up the tests substantially.
Verifies all os have at least one test and all snmprec files are tested.

* fix typo
2017-05-03 15:52:53 +01:00
Neil Lathwood
41e7689c77 refactor: DB Updates will now file level lock to stop duplicate updates (#6469)
* refactor: DB Updates will now file level lock to stop duplicate updates

* scrut fixes

* renamed function from clear_lock() to release_lock()
2017-05-03 14:48:23 +01:00
Tony Murray
4c0412b14d feature: Wireless Sensors Overhaul (#6471)
* feature: Wireless Sensors
Includes client counts for ios and unifi
Graphing could use some improvement.
Alerting and threshold ui not implemented

WIP: starting OO based wireless sensors.

Class based functionality working
remove old functional files
add schema file

discovery needs to be enabled, not polling

fix up schema

fix Unifi discovery not returning an array

Add some debug when discovering a sensor.
Fix style.

Add missing semicolin

Add a null object (Generic) for OS.
Fill out some phpdocs

Re-organized code
Each sensor type now has it's own discovery and polling interface
Custom polling tested with Unifi CCQ

Left to do:
Implement UI (Graphs and Custom thresholds)
Alerting
Testing

Fix event message text

Remove runDiscovery and runPolling from OS, they are unused and don't belong there.

Cleanups/docs

Missed this file.

Remove the requirement to fetch the current value to check validity.
Do that automatically if current is not specified
A few cleanups here and there

First pass at graphing.
device_ and wireless_ graphs added.

Add RouterOS support

Singleton OS instance isn't required right now.
Remove that to allow some memory to be freed.

Add wireless to the device list metrics.
Make all metrics clickable

Tweak graphs a bit

Implement limit configuration page.
Use sensors page as common code instead of duplicating.
Clean up some javascript interactions:  Allow enter on values to save. Cancel if update is not needed. Enable the clear custom button after setting a custom value.
Add some wireless alert rules to the library.

Add documentation.

Add unifi client counts by ssid in addition to radio.
Optimize Sensor polling a bit.

Add HP MSM clients support (for full controller)
Fix function accessibility

Formalize the discovery and poller interfaces.

Add Xirrus clients and noise floor
move module interfaces to a more appropriate place.
push caching code up to os, unsure about this do to the limitations

No point in selectively enabling wireless discovery.  We only discover if the device supports something.

Add RSSI, Power, and Rate.
Add these sensors for Ubnt Airos.
Clean up some copyrights.

Reduce the amount of files need to add new types.
Leave graph files for consistency and to allow customization.

Remove the old wifi clients graph completely.
ciscowlc should have improved counts (total and per-ssid)

Schema didn't get added.

Impelement the rest of the AirOS sensors
Reformat and re-organize the Airos.php class.

Add several UBNT AirFiber sensors

A few fixes add links to the section headers

Add HP MSM mibs.

* Schema file got dropped in rebase.

* Add wireless menu to view sensors across all devices.
Icons in the menu need help :/

* Add HeliOS, Mimosa, and Siklu support
Sensors added SNR + Noise

* Add power and utilization to Unifi

* Update polling to prefetch all sensor data in a few snmp requests as possible

* Add Extendair: tx+rx power, aggregate rate, frequency

* Add a check for duplicate sensors in discovery.  Just print an error for now.

* Add Bit Error Ratio (named error-ratio to allow for bit error rate to be added if needed)
Fix an incorrect link in the wireless sensors table

* Add error rate and change all bps and Hz to use si units

* Fixes to limits and frequency display

* Fix overview graph frequency display
A few decimal place tweaks

* Don't allow switching sensor and wireless-sensor graphs, it doesn't work.
Change individual distance graphs to use si units

* Go through the OS and make sure I got all the sensors I can (probably missed some still)
Because pollWirelessChannelAsFrequency() is generic and a little complex, so pull it up to OS.
Message to help developers adding supports that don't return an array from discover functions.

* Fix some issues

* Remove noise and signal for now at least
A couple more fixes
Add a notification

* Oopsie

* Bonus AirFiber sensors
2017-05-01 23:49:11 -05:00
Neil Lathwood
e7e5d51b30 newdevice: Added support for Exalt ExtendAir (#6524) 2017-04-29 15:58:06 +00:00
Neil Lathwood
d67d62eacb fix: Added more safety checking into create_state_index() so we do not end up with stale state names but no translation (#6516) 2017-04-27 22:33:56 -05:00
Neil Lathwood
153f0de12f fix: Actually reload oxidized when we should not when we think we should (#6515) 2017-04-27 22:31:48 -05:00
Tony Murray
1b1907a09f Update database tests to prepare for more (#6500)
* Update database tests to prepare for more
Add test_db_name, test_db_user, test_db_pass to allow developers to have a dedicated testing database.
Extract DBTestCase to include common functionality for database based tests. Use transactions to isolate tests.
Enable STRICT_TRANS_TABLE, test for it's existence (only when we have a user that can set it)
Move the database cleanup to register_shutdown_function, this makes it happen every time at the end of tests.
If the was not empty, only truncate the tables (that aren't prepopulated) instead of drop the database.
Use our schema functions for schema tests.
Fix some missing array indexes so it doesn't clutter test output.

* Fix style
2017-04-26 07:56:00 -05:00