Commit Graph

302 Commits

Author SHA1 Message Date
Tony Murray
010ff08e7f Fix for snmp queries that take longer than 60s (#9574) 2018-12-20 19:51:06 -06:00
PipoCanaja
2d84606abf BUG - correcting size and non significant zeroes (#9575) 2018-12-20 19:50:36 -06:00
Tony Murray
74882e3950 Dashboard widget update (#9515)
Implemented in Laravel
Doesn't use legacy PHP session
Several widgets have new features and settings, for example:

- Multiple ports in one graph
- Maps settings are configurable and override system settings but default to system settings
- Graylog stream and/or device selection
- Much improved graph widget selection
- Many more

DO NOT DELETE THIS TEXT

#### Please note

> Please read this information carefully. You can run `./scripts/pre-commit.php` to check your code before submitting.

- [x] Have you followed our [code guidelines?](http://docs.librenms.org/Developing/Code-Guidelines/)

#### Testers

If you would like to test this pull request then please run: `./scripts/github-apply <pr_id>`, i.e `./scripts/github-apply 5926`
After you are done testing, you can remove the changes with `./scripts/github-remove`.  If there are schema changes, you can ask on discord how to revert.
2018-12-16 21:18:17 +00:00
Tony Murray
fad0bffc43 Escape SNMP & IPMI shell commands (#9537)
hopefully doesn't break anything
Mostly issues with snmp oids and options containing spaces.  Try to remove all of those.

DO NOT DELETE THIS TEXT

#### Please note

> Please read this information carefully. You can run `./scripts/pre-commit.php` to check your code before submitting.

- [x] Have you followed our [code guidelines?](http://docs.librenms.org/Developing/Code-Guidelines/)

#### Testers

If you would like to test this pull request then please run: `./scripts/github-apply <pr_id>`, i.e `./scripts/github-apply 5926`
After you are done testing, you can remove the changes with `./scripts/github-remove`.  If there are schema changes, you can ask on discord how to revert.
2018-12-16 13:42:50 +00:00
Tony Murray
8bddfe7225 Fixed plugins using d_echo (#9498)
Move d_echo to helpers.php and include in autoload
Don't remove from common.php yet to be extra safe.
2018-12-08 13:16:49 +00:00
Tony Murray
c7f70b6e47 Added detection if this is a git based install or not. (#9379)
Not sure about the warning or text.

DO NOT DELETE THIS TEXT

#### Please note

> Please read this information carefully. You can run `./scripts/pre-commit.php` to check your code before submitting.

- [x] Have you followed our [code guidelines?](http://docs.librenms.org/Developing/Code-Guidelines/)

#### Testers

If you would like to test this pull request then please run: `./scripts/github-apply <pr_id>`, i.e `./scripts/github-apply 5926`
After you are done testing, you can remove the changes with `./scripts/github-remove`.  If there are schema changes, you can ask on discord how to revert.
2018-12-07 22:29:37 +00:00
Tony Murray
93f44d67f8 Fixed customers page (#9521)
move customers table backend to Laravel

DO NOT DELETE THIS TEXT

#### Please note

> Please read this information carefully. You can run `./scripts/pre-commit.php` to check your code before submitting.

- [x] Have you followed our [code guidelines?](http://docs.librenms.org/Developing/Code-Guidelines/)

#### Testers

If you would like to test this pull request then please run: `./scripts/github-apply <pr_id>`, i.e `./scripts/github-apply 5926`
After you are done testing, you can remove the changes with `./scripts/github-remove`.  If there are schema changes, you can ask on discord how to revert.
2018-12-07 22:24:53 +00:00
Tony Murray
3e35ee0e7d Refactored and update Location Geocoding (#9359)
- Fix location so it is a regular database relation (this allows multiple devices to be accurately linked to one location and saves api calls)
- Parse coordinates from the location more consistently
- Add settings to webui
- ~~Used [PHP Geocoder](http://geocoder-php.org/), which has lots of backends and is well tested. (also includes reverse and geoip)~~
- Google Maps, Bing, Mapquest, and OpenStreetMap supported initially.
- Default to OpenStreetMap, which doesn't require a key.  They will liberally hand out bans if you exceed 1 query per second though.
- All other Geocoding APIs require an API key. (Google requires a credit card on file, but seems to be the most accurate)
- Update all (I think) sql queries to handle the new structure
- Remove final vestiges of override_sysLocation as a device attribute
- Update existing device groups and rules in DB
- Tested all APIs with good/bad location, no/bad/good key, and no connection.
- Cannot fix advanced queries that use location

This blocks #8868

DO NOT DELETE THIS TEXT

#### Please note

> Please read this information carefully. You can run `./scripts/pre-commit.php` to check your code before submitting.

- [x] Have you followed our [code guidelines?](http://docs.librenms.org/Developing/Code-Guidelines/)

#### Testers

If you would like to test this pull request then please run: `./scripts/github-apply <pr_id>`, i.e `./scripts/github-apply 5926`
After you are done testing, you can remove the changes with `./scripts/github-remove`.  If there are schema changes, you can ask on discord how to revert.
2018-11-28 22:49:18 +00:00
PipoCanaja
41cc344106 VRP: Discovery improvement for optical power + Mib upgrade (#9225)
* Huawei Mib file update

* VRP: Improve sensor discovery for optical equipment

* VRP: Improve sensor discovery for optical equipment

* VRP: Adding Test data after sensor dbm update
2018-09-19 10:00:11 -05:00
Tony Murray
32a7c50189 Use Laravel authentication (#8702)
* Use Laravel for authentication
Support legacy auth methods
Always create DB entry for users (segregate by auth method)

Port api auth to Laravel

restrict poller errors to devices the user has access to

Run checks on every page load.  But set a 5 minute (configurable) timer.
Only run some checks if the user is an admin

Move toastr down a few pixels so it isn't as annoying.

Fix menu not loaded on laravel pages when twofactor is enabled for the system, but disabled for the user.
Add two missing menu entries in the laravel menu

Rewrite 2FA code
Simplify some and verify code before applying

Get http-auth working
Handle legacy $_SESSION differently.  Allows Auth::once(), etc to work.

* Fix tests and mysqli extension check

* remove duplicate Toastr messages

* Fix new items

* Rename 266.sql to 267.sql
2018-09-11 07:51:35 -05:00
Tony Murray
a2e33efba8 Use PDO for MySQL queries (#8935)
Gut dbFacile
mres() now does nothing... could have some vulnerable queries, but most are ok

TODO
- [x] IRC client broke
- [x] Install broke

DO NOT DELETE THIS TEXT

#### Please note

> Please read this information carefully. You can run `./scripts/pre-commit.php` to check your code before submitting.

- [x] Have you followed our [code guidelines?](http://docs.librenms.org/Developing/Code-Guidelines/)

#### Testers

If you would like to test this pull request then please run: `./scripts/github-apply <pr_id>`, i.e `./scripts/github-apply 5926`
2018-08-17 21:29:20 +01:00
Tony Murray
50bb72f530 Remove port association mode map from the database (#8941)
* Remove port association mode map from the database

* Renamed sql file

* Renamed sql file
2018-08-05 09:50:13 +01:00
Tony Murray
25dc44a574 fixed can_ping_device() logic (#8906)
DO NOT DELETE THIS TEXT

#### Please note

> Please read this information carefully. You can run `./scripts/pre-commit.php` to check your code before submitting.

- [x] Have you followed our [code guidelines?](http://docs.librenms.org/Developing/Code-Guidelines/)

#### Testers

If you would like to test this pull request then please run: `./scripts/github-apply <pr_id>`, i.e `./scripts/github-apply 5926`
2018-07-17 08:16:13 +01:00
Neil Lathwood
ad6ec7f72a Refactor alert templates to use Laravel Blade templating engine (#8803)
* Initial start on new templating support

* further updates

* more updates

* More working version

* Last fixes

* Small updates

* modified test to look for dbname.

* Schema update

* Added fix for not being able to disable Philips Hue transport

* Updated sql file

* Updated db_schema

* Set myclabs/deep-copy to be 1.7.x version in composer

* Fixes from murrant

* Forced nikic/php-parser to version 3.1.x in composer

* Updated composer to use custom fork of string-blade-compiler + fixed transport use

* Updated to always use correct template

* Merged legacy and blade templating engines

* Removed template type + fixed test-alert.php

* Added more template placeholders

* Added ability to reference data in templates as $alert->X

* Updated docs for templates

* Updated db_schema.yaml

* Added $alert->builder

* Clarify the use of $alert->builder

* Fixed the use of $alert->transport

* renamed schema file

* Added template validation

* Small update to fix travis issue

* Add Docs text to title bar

* Updated amqp to new of version

* Consistency in alert rule and template title bars
2018-07-14 22:15:43 +01:00
Tony Murray
eeb3d58f5b Improved Logging and Debugging (#8870)
Use Log facility when Laravel is booted.
Update init.php so we can easily boot Laravel for CLI scripts. (and just Eloquent, but that may go away)
Move all debug setup into set_debug() function and use that across all scripts.
Log Laravel database queries.
Send debug output to librenms log file when enabling debug in the webui.
Allow for colorized Log CLI output. (currently will leave % tags in log file output)

** Needs testing and perhaps tweaking still.

DO NOT DELETE THIS TEXT

#### Please note

> Please read this information carefully. You can run `./scripts/pre-commit.php` to check your code before submitting.

- [x] Have you followed our [code guidelines?](http://docs.librenms.org/Developing/Code-Guidelines/)

#### Testers

If you would like to test this pull request then please run: `./scripts/github-apply <pr_id>`, i.e `./scripts/github-apply 5926`
2018-07-13 23:08:00 +01:00
Richard Mayhew
528b91a056 corrected display of minigraph when using sysName as hostname (#8842)
* corrected display of minigraph when using sysName as hostname

* Check to see if its an IP or hostname. Make sure all 3 scenarios work.

* removed test rrd symlink

* removed test rrd symlink

* reverted old change

* Improve and add tests.
2018-07-02 08:23:09 -05:00
Richard Mayhew
ab5d17d715 Allow the hostname to resolve to the sysName, ie Dynamic DNS (#8810) 2018-06-23 21:59:09 +01:00
Tony Murray
9152becec7 refactor: Moved AD user and group filter functions into ActiveDirectoryAuthorizer (#8545) 2018-04-11 10:06:46 +01:00
Tony Murray
7250376104 refactor: Don't access $_SESSION directly for Auth (#8513)
* Don't access $_SESSION directly for Auth

* fix style

* add property annotations
2018-04-07 21:55:28 +01:00
sorribes22
c22b31506e api: Added lattitude and longitude on devices/device fetch (#8515)
* 'lat' and 'lng' columns added on list_devices()

* 'lat' and 'lng' columns added on device fetch

Helped by https://github.com/laf

* Commit user.name changed
2018-04-07 13:29:46 +01:00
Neil Lathwood
03076c4025 feature: Added new alert rule builder UI and rule mapping UI (#8293)
* feature: Added new alert rule builder UI

* Updated to export sql queries

* More updates

* more changes

* removed debug

* fix scrut

* Updated to include import options + various other fixes

* fix rule

* Populate name from collection rules.

* Fix default rule import
Allow new and old style rules in the collection.
Don't add new yet as I'm not sure GenSQL() is working.

* Fix GenSQL call

* Extract filter building to class so it is nicely contained in one place

* moved schema

* some fixes and tweaks

* travis fixes

* Some more features / updates

* Fix up my mistakes when adding default rules

* Use a modal for new alert (Incomplete)
Larger dialog!!
Remove page loading stuff.

Working:
Loading rules, resetting dialog, importing from collection.

Not working yet:
select width
device limited rule access? don't know what this is...

Lots of unused stuff to delete...

* reload "table" after save

* fixed editing rule

* Auto select2 width

* Reload window on save

* Restore per-device alert. Remove debug.

* Small cleanups. Rule Name first.

* Restore button to button type. Rename schema.

* Fixes: wrong command to reload window, remove extra attributes, rule is never passed

* Fixed old rule editing

* some small updates for old imports

* travis update to use trusty

* maybe travis fix

* Ability to set alert rule mappings on the rule edit screen

* pip installs one line, no quiet for deploy

* update schema def

* Fix style and some copyright headers

* fix docs missing file

* Allow new versions of snmpsim and libraries

* Parser WIP

* Fix default rules insert

* reorganize

* Legacy import first draft done

* Implement saving
Skip translation to sql for now

* Working on glues

* small rule collection fix

* Working on glues

* Working on glues

* Docs updates + small UI changes

* Parser WIP

* reorganize

* Legacy import first draft done

* Implement saving
Skip translation to sql for now

* Working on glues

* Working on glues

* Working on glues

* Add table mapping, should move to it's own class

* WIP

* Glue working!!

* Extract Schema class

* Some final touches.
revert alerts_rules.json for now.

* Finish up initial implementation
Needs more tests

* Fix a few places

* small doc updates

* Fix finding tables in grouped rules.

* remove unused code

* code format fixes

* Some quick tests for Schema
Simplified output for findRelationshipPath. Always includes start and target in the result.
This simplifies a lot of code in QueryBuilderParser.php
This also always loads the target table data now (which we want)

* Make bill_id the PRIMARY index for the bills table

* Load macros from a json file in misc instead of the database.

* Fix whitespace and wrong key for collection.

* Handle IN properly when generating SQL

* Fix glue (devices.device_id = ports.port_id) is incorrect :D
Show ALL tables we can resolve relationships for in the query builder filter.

* Remove all macros from the database
Remove insert statements, leave updates to update user's existing rules.
2018-03-14 20:25:19 +00:00
James Andrewartha
70714fe811 security: Hide more confidential data (SNMPv3, IPMI) from the debug output (#8303) 2018-02-27 15:41:44 +00:00
Tony Murray
b2762d9fa7 refactor: Prevent function collisions with Laravel (#8166)
Make function call compatible
drop insensitive starts_with and ends_with for now as they aren't needed.
2018-01-29 21:58:21 +00:00
crcro
7e03905e63 webui: Small graylog cleanup (#8057) 2018-01-09 20:47:33 +00:00
Zmegolaz
2c9468979f Load device attributes before load_os() in device_by_id_cache(). (#8044) 2018-01-08 13:37:34 -06:00
Neil Lathwood
7d0ad5c4ef tests: Added testing for Arista EOS BGP (#7984)
* tests: Added testing fro Arista EOS BGP

* fixing tests

* small updates

* last one

* more updates

* Account for no space after = in snmp data
always load_os() when a new device is freshly loaded by device_by_id_cache()

* don't be fatal in load_os()

* Update json to correct data
2017-12-30 10:31:06 +00:00
Neil Lathwood
27368a2d98 webui: Stop stripping html tags from snmp creds (#7951) 2017-12-25 09:51:31 -06:00
crcro
520fc4a268 webui: Graylog table refresh (#7832)
* initial release

* style fixes
2017-12-24 19:59:16 +00:00
crcro
71711326fb webui: Syslog table refresh (#7796)
* initial changes

* final changes

* added missing copyright, removed commented old code
2017-12-24 19:57:54 +00:00
Tony Murray
fff66d3c00 Feature: Generic discovery and poller tests (#7873)
* Processor Tests!

* Capture data from live devices easily.

* fix up some stuff, remove powerconnect things as they seem to be just broken.

* formatting, fix missing assignment
add netonix processor data

* fix multi-line, always add sysDescr and sysObjectID
ios cpm test file

* revert composer change and fix whitespace issues

* add help text

* missed help text

* tighter debug output

* handle empty strings properly and mibs with numbers

* use keys for sorting as intended

* fix type with empty data

* oops :)

* whitespace fix

* try installing fping

* Fix TestCase collision + cleanup

* mark TestCase as abstract
don't run two instances of snmpsim

* better database dumps, improved capture

* style fixes

* fix quotes add a few more tables

* add --prefer-new, properly merge data

* Support separate discovery and poller data. But don't waste space if they aren't needed.

* refactor to use class
collects all code in one place for reusability

* reorganize

* Print out when not saving.

* Support for running multiple (or all) modules at once.

* tidy

* Change unit test to a generic name and test all modules we have data for.

* Add documentation and a few more tidies

* whitespace fixes

* Fix tests, add a couple more modules, more docs

* More docs updates

* Fix scrutinizer issues

* add bgp-peers
2017-12-20 08:36:49 -06:00
bergroth
84fdfdf542 Junos dwdm interface sensor support (#7714)
* Junos dwdm interface sensor support  …
  four new sensors
    Chromatic Dispersion ps/nm
    Delay s (seconds)
    Q_factor (dB)
    preFEC_BER (ratio)

  reuse of
    SNR, dbm, temperature, current

* new dwdm sensors more generic, ber and quality_factor

* new sensors updated

* renamed html graphs new sensors

*  fixed ber polling

* Change to base SI Hz and s.

* update sensor divisors to bigint

* rename sql-schema/216.sql to 218.sql

* YAML junos dwmd sensor definitions, except BER

* bigint divisor fixed

* deafult 1 divisior

* small fixes including schema change for another PR

* schema fix
2017-12-12 20:27:10 -06:00
Tony Murray
c22c879983 fix: Remove faulty memcached code (not related to distributed polling) (#7881) 2017-12-10 20:40:45 +00:00
Tony Murray
b620692426 refactor: Update all applications to store metrics (#7853)
* Update all applications to store metrics

* fix db schema

* Add glue test
2017-12-06 22:13:10 +00:00
Tony Murray
5405498ed6 alerts: Adde generic alerts for state sensors (#7812)
* feature: Generic alerts for state sensors
A bit of a hack, but gets the job done. Perhaps someone can refactor the ResolveGlues() function sometime to make it better.

* Fix mistakes

* Add docs and collection rules, critical enabled by default

* moved schema file to 221.sql
2017-12-02 23:03:36 +00:00
Tony Murray
4a03e7838e feature: Save application metrics to db for alerting (#7828)
* feature: save application metrics to db for alerting
However, alerting will not work because ResolveGlues() is broken.
Can add workaround after state_translations alerting is merged

Does not update all applications yet, not sure if that should be done here or in another PR.

Introduces two handy functions dbDeleteOrphans() and array_by_column().  Will replace those in other locations after this is merged or separate them out if this is not merged.

* remove accidental inclusions

* Add db schema
2017-12-01 07:53:26 +00:00
Tony Murray
257a314fca feature: Validate session directory is writable (#7784) 2017-11-23 08:29:10 +00:00
Tony Murray
0dee809607 fix: Discovery os changes weren't properly reflected if os changed (#7779) 2017-11-22 21:48:28 +00:00
Jonathan Lassoff
fe8d011dce refactor: Allow setting the target shorthost length. (#7453)
* Create a target shorthost length. Addresses #7452.

* Update defaults.inc.php
2017-11-09 20:47:52 +00:00
Tony Murray
cbe6cb532d Fix: Discovery validation with ping-only devices (#7665)
* fix: validation check cli php extensions
Fix fatal errors when mysql is not loaded / can't connect

* fix: discovery validation incorrectly detects on ping only devices.
2017-11-07 20:09:49 +00:00
Tony Murray
54b476c769 fix: devices detected as ibmtl or generic (#7618)
* fix: devices detected as ibmtl
Guess at the sysObjectID, please report if you have devices that should be ibmtl and are no longer detected.
Add output of os in discovery.

* Always load all os for discovery.  Should be cached most of the time.
2017-11-03 16:10:24 -05:00
Neil Lathwood
3e94077a71 Revert "Update RFC1123 Hostname Check" (#7577)
* Revert "Revert "feature: Allow snmpget in os discovery yaml (#7558)" (#7576)"

This reverts commit 426916dbe4.

* Revert "fix: lldp find_port_id finds not the most correct port (#7564)"

This reverts commit 7e05fff615.

* Revert "fix: Update RFC1123 Hostname Check (#7572)"

This reverts commit ebcfa45a93.
2017-10-29 20:05:59 +00:00
Parth Laxmikant Kolekar
ebcfa45a93 fix: Update RFC1123 Hostname Check (#7572)
* Update RFC1123 Hostname Check

The regex was wrong. It failed to match devices like `cisco-3750x` even though it is a valid hostname. 
I picked this regex from the RFC1123 documentation, and updated it.

* Update CommonFunctionsTest.php

Update tests

* Updated to allow domains ending with a dot
2017-10-29 11:12:32 +01:00
Zmegolaz
f8d7ccfe0d feature: Support for up/down detection of ping only devices.
* Added support for ping only devices.

* Renamed sql-schema/206 to 207.

* Discovery tried to detect the OS when it shouldn't.

* Updated db_schema.yaml

* Added ping icon.

* Don't show unused graphs and tabs when SNMP is disabled.

* Allow the user to specify OS of ping only devices.

* Removing custom OS now changes it to 'ping'.

* Removed unnecessary use of mres().

* UI select box for SNMP enable/disable.

* Hide device_ping_perf if ping is disable on device.

* Fixed SNMP settings update status messages.

* Added functionality to add ping only devices via the web ui.

* Added ping only option to addhost.php

* Added ping only support to snmp-scan.py

* Moved sql 208 to 211

* Fixed scrutinizer issues.

* Fixed broken ossearch.

* Added ping only support to the API.

* Updated API doc.

* Added (optional) to OS and hardware description. Hid Port Settings, Applications, Modules, Storage, Processors, Memory and Components from the edit menu

* Style fix.

* Updated ping icon.

* clean() instead of mres(). More escaping. Better help in snmp-scan.py and addhost.php

* Fixed scrutinizer issue.

* Always try SNMP in snmp-scan.py, new option for it in addhost.php. Slice instead of chunk in ajax_ossuggest.php. Other minor style changes.

* Updated sql modifications to insert the new column in the same place as in db_schema.yaml.
2017-10-27 22:59:25 -05:00
Tony Murray
8a051aa295 fix: Fix a couple of OS cache invalidation issues (#7567)
* fix: fix a couple of OS cache invalidation issues
Fixes issues where we might be using outdated os definitions.

* use Config and fix style issue

* only pull in settings from config.php (otherwise will possibly contain populated os config)
2017-10-27 21:03:58 +01:00
Tony Murray
51ba934e11 feature: Added web validation (#7474)
* refactor: convert validations to modules to prep for gui integration

* accidentally dropped, schema update

* fix accidental output to webui

* mention discovery-wrapper.py and re-arrange.

* refine some fix statements

* rename the Config validation group to Configuration.

* fix some scrutinizer issues
remove as many local functions from validator.php as possible
move extensions from pre-check
remove duplicate timezone check
looks like there is some db schema differences between mariadb 10.1 and 10.2, investigating

* mariadb version diff take2

* Check schema version first for database.
Remove stop to go back to command line for install docs.
Add helpful link when there is no devices added to /addhost

* fix incorrect validation for empty string defaults

* Fix style

* Add additional file permissions checks

* catch exception and fail for invalid timezone
Change visuals a bit.

* add php version warning

* fix space
2017-10-26 07:56:09 +01:00
Tony Murray
a2b9342f05 fix: improve accuracy of is_valid_hostname() (#7435)
* fix: improve accuracy of is_valid_hostname()
fixes discovery code attempting to add discover invalid dns names
the old code allowed some invalid hostnames, this is more thorough
add tests

* fix merge error
2017-10-05 16:04:17 -05:00
Tony Murray
c0b267b49c refactor: Discovery protocols re-write (#7380)
* refactor: discovery protocols re-write
extract more code to make it easier to read
use snmpwalk_group
walk lldpRemTable instead of lldpRemoteSystemsData
Optimize discover_device to reduce dns lookups of bad dns names
Make discovery_by_ip behavior consistent between cdp and lldp
move can_skip_discovery to discovery functions

Tested lldp, cdp, and ospf.

* always discover links even if the remote device isn't in system
Fix discover_link always updating

* fix style

* fix update, not fetching id

* increase platform length.  Most devices use sysDescr here, which is max length 255

* remove orphaned links

* update db_schema.yaml
2017-10-04 08:16:23 +01:00
Neil Lathwood
f7e14df90e newdevice: Updated Arista entity-physical support to use high/low values from device (#7156)
* fix: Fixed the high/low thresholds for Arista devices

* final updates

* push threshold changes

* more updates

* another update

* updated entity-sensor for current value calc

* hoping final update
2017-09-07 08:49:32 -05:00
Tony Murray
c8436d2a4f fix: issues with Geist Watchdog
miss-named variable in sensor pre-caching
internal humidity and temperature was discovered twice
humidity was mis-spelled in yaml discovery
temperature and current had incorrect divisor in yaml

remove redundant fahrenheit to celsius function

fixes #7236
2017-08-28 13:00:29 -05:00
Neil Lathwood
25d79ce9d7 newdevice: Added further sensor support for Geist Watchdog (#7143)
* newdevice: Added further sensor support for Geist Watchdog

* relaxed if check
2017-08-19 20:22:23 +01:00