245 Commits

Author SHA1 Message Date
0ba76e6d62 New python service for poller, discovery + more (#8455)
Currently has a file handle leak (and will eventually run out of handles) related to the self update process.

Either need to fix that or rip out self-update and leave that up to cron or something.


DO NOT DELETE THIS TEXT

#### Please note

> Please read this information carefully. You can run `./scripts/pre-commit.php` to check your code before submitting.

- [x] Have you followed our [code guidelines?](http://docs.librenms.org/Developing/Code-Guidelines/)

#### Testers

If you would like to test this pull request then please run: `./scripts/github-apply <pr_id>`, i.e `./scripts/github-apply 5926`
2018-06-30 12:19:49 +01:00
e151aa94e4 github-remove the app directory too (#8849)
DO NOT DELETE THIS TEXT

#### Please note

> Please read this information carefully. You can run `./scripts/pre-commit.php` to check your code before submitting.

- [x] Have you followed our [code guidelines?](http://docs.librenms.org/Developing/Code-Guidelines/)

#### Testers

If you would like to test this pull request then please run: `./scripts/github-apply <pr_id>`, i.e `./scripts/github-apply 5926`
2018-06-29 22:57:01 +01:00
02348be861 Clear the OS cache when running collect and save test data scripts (#8767) 2018-05-26 10:33:56 -05:00
fc77900266 fix pre-commit.php --quiet (#8771)
DO NOT DELETE THIS TEXT

#### Please note

> Please read this information carefully. You can run `./scripts/pre-commit.php` to check your code before submitting.

- [x] Have you followed our [code guidelines?](http://docs.librenms.org/Developing/Code-Guidelines/)

#### Testers

If you would like to test this pull request then please run: `./scripts/github-apply <pr_id>`, i.e `./scripts/github-apply 5926`
2018-05-26 09:11:38 +01:00
8afafe7eb0 refactor: pre-commit.php updates (#8679)
Disable the log for unit tests to speed things up hopefully
2018-05-09 12:53:45 +01:00
350b3a1ef8 fix: Fixed alerting with dynamic contacts for rules made with the new builder (#8677)
Update test-alert.php too

Thanks to @dsgagi on Discord for identifying the issue
2018-05-08 21:35:21 +01:00
d799600cab refactor: Updated the oxidized script (#8572)
* refactor: Updated the oxidized script

* Updated to use rmccue::Requests + Config::get()
2018-04-26 10:14:28 -05:00
b133aaa718 feature: Added support for syslog Oxidized notification on Huawei VRP switches… (#8562) 2018-04-17 10:32:41 +01:00
d841625f12 refactor: Refactor database and config init (#8527)
* Refactor database and config init
Connect to the database without loading full config
Load config completely so post-processing is always done consistently.
Erase existing $config when loading, fixes issues in case we load the config twice.
If the database is not connected, don't try to load database settings. (Fixes some db errors on install)
Attempt to remove $config access/modification before init.php
Remove usage of db_name, that might not match the connected database.
Centralize db config loading, so we consistently apply db_test database settings.
Many of these changes are influenced by Laravel port.

* Some safety so we don't assign strings to numeric port field
Smooth out phpunit bootstrap

* Fix a couple of scrutinizer warnings.
2018-04-11 16:15:13 +01:00
ca09cf61ac auth_test.php -v override config.php (#8508) 2018-04-05 22:43:54 +01:00
5c83aa1e28 refactor: LDAP debug output (#8434)
* LDAP debug
Updated LDAP and AD docs
ldap protocol default to v3 (so we don't have to set it all the time).  If this fails it should revert to v2.
ad was using auth_ad_timeout incorrectly (1 I think)

* Add option to list all users.
2018-03-29 11:40:27 +01:00
665229be40 Follow on to PR 8168 - Fixed merge of manage_bills.php (#8467)
* Adding 1st rev of manage_bills.php

* Stlye fixes

* Freshinging PR 7633. Changing -r to -f, to be less confusing. Adding code style changes as per @laf's suggesting

* Adding in option for sysname or hostname variable. Fixing hostname lookup bug

* style fixes

* -renaming add_ports to add_ports_to_bill
- changing -i to -b for clarity sake
- Will print help if -s and -h are not set

* Adding check to ensure -i is set via cli

* I'm stupid

* I'm stupid

* Fixing help messages, fixing argument checking, moving help to it's own function

* Adding create_bill functionality, update help and opts as needed

* Adding 1st rev of manage_bills.php

* Stlye fixes

* Freshinging PR 7633. Changing -r to -f, to be less confusing. Adding code style changes as per @laf's suggesting

* Adding check to ensure -i is set via cli

* I'm stupid
2018-03-28 22:44:32 -05:00
8c94ccf4d6 refactor: Use custom phpcs ruleset for LibreNMS (#8418)
* Use custom phpcs ruleset for LibreNMS
PSR2 with exclusions

* Don't check line length

* Restore build_excludes()
Drop influxdb lib exclusion

* Fix whitespace
2018-03-20 11:32:52 +00:00
8cc8eb8dd2 refactor: Use elseif instead of else if (#8417) 2018-03-19 21:40:17 +00:00
69b066ab3d Fix mib renamer script when in a different directory (#8379)
* Fix rename mibs script when in a different directory
Add count

* fix spaces
2018-03-14 23:57:18 -05:00
03076c4025 feature: Added new alert rule builder UI and rule mapping UI (#8293)
* feature: Added new alert rule builder UI

* Updated to export sql queries

* More updates

* more changes

* removed debug

* fix scrut

* Updated to include import options + various other fixes

* fix rule

* Populate name from collection rules.

* Fix default rule import
Allow new and old style rules in the collection.
Don't add new yet as I'm not sure GenSQL() is working.

* Fix GenSQL call

* Extract filter building to class so it is nicely contained in one place

* moved schema

* some fixes and tweaks

* travis fixes

* Some more features / updates

* Fix up my mistakes when adding default rules

* Use a modal for new alert (Incomplete)
Larger dialog!!
Remove page loading stuff.

Working:
Loading rules, resetting dialog, importing from collection.

Not working yet:
select width
device limited rule access? don't know what this is...

Lots of unused stuff to delete...

* reload "table" after save

* fixed editing rule

* Auto select2 width

* Reload window on save

* Restore per-device alert. Remove debug.

* Small cleanups. Rule Name first.

* Restore button to button type. Rename schema.

* Fixes: wrong command to reload window, remove extra attributes, rule is never passed

* Fixed old rule editing

* some small updates for old imports

* travis update to use trusty

* maybe travis fix

* Ability to set alert rule mappings on the rule edit screen

* pip installs one line, no quiet for deploy

* update schema def

* Fix style and some copyright headers

* fix docs missing file

* Allow new versions of snmpsim and libraries

* Parser WIP

* Fix default rules insert

* reorganize

* Legacy import first draft done

* Implement saving
Skip translation to sql for now

* Working on glues

* small rule collection fix

* Working on glues

* Working on glues

* Docs updates + small UI changes

* Parser WIP

* reorganize

* Legacy import first draft done

* Implement saving
Skip translation to sql for now

* Working on glues

* Working on glues

* Working on glues

* Add table mapping, should move to it's own class

* WIP

* Glue working!!

* Extract Schema class

* Some final touches.
revert alerts_rules.json for now.

* Finish up initial implementation
Needs more tests

* Fix a few places

* small doc updates

* Fix finding tables in grouped rules.

* remove unused code

* code format fixes

* Some quick tests for Schema
Simplified output for findRelationshipPath. Always includes start and target in the result.
This simplifies a lot of code in QueryBuilderParser.php
This also always loads the target table data now (which we want)

* Make bill_id the PRIMARY index for the bills table

* Load macros from a json file in misc instead of the database.

* Fix whitespace and wrong key for collection.

* Handle IN properly when generating SQL

* Fix glue (devices.device_id = ports.port_id) is incorrect :D
Show ALL tables we can resolve relationships for in the query builder filter.

* Remove all macros from the database
Remove insert statements, leave updates to update user's existing rules.
2018-03-14 20:25:19 +00:00
bacab61af2 refactor: Use composer.phar first and self-update it. (#8339)
* Use composer.phar first and self-update it.

* Ansi output for interactive shells.
If HOME isn't writable, set COMPOSER_HOME.

Simulated cron with env -i

* More verbose failures when attempting to download composer.phar.

* Change the regex again
Matches all versions here https://getcomposer.org/download/
2018-03-13 20:08:40 +00:00
9459eeb55f refacotr: Use HTTPS to fetch Composer (#8309)
* use https for composer

* Fixe alternate --version output for composer
2018-02-28 20:04:44 +00:00
b2ce9b173b save-test-data.php: Print a message when we encounter a bad module name (#8274)
* Print a message when we encounter a bad module name (likely a typo)

* Raise exception for invalid module name and handle it.
2018-02-25 20:03:18 -06:00
0e0c8d1cbe feature: Added ironware syslog notify code (#8268)
* Added ironware syslog notify code

* Update syslog-notify-oxidized.php
2018-02-21 10:03:43 +00:00
1188b53192 security: Use more secure password hashes (#8213)
* More secure password hashes
Use PHP 5.5 password_hash(), currently uses bcrypt
increase password field length as per php documentation

* Use password_hash()/password_verify() for cookies too

* forgot to update db_schema.yaml
2018-02-08 23:08:21 +00:00
11147d3bbf Major Processors rewrite (#8066)
* Extract DiscoveryItem and move some things to better places.
Extract model class
Fix up model construction.  I have problem with construction...
Makeshift model working.  Switch constructor to factory.  discover() and create()
Support legacy discovery.
Remove uneeded custom pollers
Remove netonix custom detection as we try ucd on all os now.
Add a few yaml procs.  Fix a couple things.
More processor discovery conversions
Move Calix e7 to standard hrProcessorLoad, but it doesn't fully implement the HR-MIB, move things around to make it work.
Add a few yaml procs.  Fix a couple things. Correct some stupid mib stuff.
Move more, drop php 5.3
Add netscaler which uses string indexes.  Port fiberhome to yaml and use skip_values
More conversions.  BroadcomProcessorUsage Trait
Serveriron and Ironware share some mibs.  Create a common abstract os for them.
Add yaml support for mib specification in each data entry
Make legacy discover_processor() set 0 for hrDeviceIndex

Untangle Dell switch OS processors

Use use shared OS for groups if they don't have a specific group.
fix silly mib mistake

Make index optional

Move HR and UCD to Traits and out of Processor.

* forgot to update the fortiswitch index

* Make sgos and avaya-ers match the old index.

* fix comware test data

* fix merge errors

* fix dsm and remove pointless empty modules

* file not found exception is in the wrong place.

* Updated processor development docs
2018-02-05 07:39:13 -06:00
dc2883f59d Use composer wrapper in pre-commit (#8192) 2018-02-03 22:40:15 +00:00
b2762d9fa7 refactor: Prevent function collisions with Laravel (#8166)
Make function call compatible
drop insensitive starts_with and ends_with for now as they aren't needed.
2018-01-29 21:58:21 +00:00
20335bce5d Fix collect-snmp-data.php when the os is generic. (#8164) 2018-01-29 07:50:56 -06:00
6878df8c12 Prep for composer changes (#8075)
* Prep for composer changes

* Update composer.json, composer.lock and a small update to phpmailer and Yaml

* Prevent accidental `composer update` runs

* Add  --no-interaction to daily.sh
2018-01-22 08:16:37 -06:00
494c29aefa Split save-test-data.php into two scripts, allow mass update (#8115)
* Add mass update to save-test-data.
Fail when snmpsim doesn't start and try to give a hint.

* Split save-test-data.php into two scripts

One for updating snmprec files.
One for updating db dump files.
2018-01-20 20:22:47 -06:00
d4bd566d36 newdevice: Added Support for cambium PTP 300, 500, 600 and 800 (#7998)
* Add Support for cambium-ptp500

Added mibs for ptp500, ptp300

* ptp500 - Switch modulation to a state

* Update label on 500 to 500/300

300 and 500 use the same mibs

* Request changes

Added overhead graph's to all current ptp devices

* wrap text in quotes

* Fix discovery bug

* Ptp600 support

* PTP800 Support

* Fix ptp600 issues

* Remove meh copy and pasta

* Remove modulation from OS file

* Update value modifiers / typos

* +x rename-mibs.php

* Modify cambium mibs
2018-01-16 22:52:11 +00:00
cabc0eb71d feature: Quick script to rename mibs to their proper names (#8068)
So I can stop re-writing it ;)
2018-01-11 22:19:13 +00:00
d515f5043e refactor: Move install to php53 branch if running an unsupported version of PHP (#8042)
* Move install to php53 branch if running an unsupport version of PHP

* Update validate

* Move PHP version check after update check.  This will prevent us from moving no update users to the php53 branch.

Hopefully if daily.php -f update fails it returns a non-zero and we can move them to the php53 branch still.

* Wiggle things around more.

* Also, send release installs to the php53 branch.

* Set notification error for old PHP version.
2018-01-10 14:40:58 +00:00
b2a54dae3b Testing updates: module testing and collect only (#8063)
* Testing updates module testing and collect only

* cache clear no-longer needed.
2018-01-09 11:40:26 -06:00
42e5819130 refactor: Updated discovery/poller to use numeric sysObjectID (#7922)
* refactor: Updated discovery to use a core module for sysDescr/sysObjectID use

* final update hopefully

* revert changes

* more changes + docs

* migrated poller to use numerical sysObjectID

* more updates for sysObjectID

* update any alert rules which might have enterprises. in

* moved schema file

* small updates

* updated getHostOS()

* scrut fixes

* updated sysObjectId -> sysObjectID

* updated sysObjectId -> sysObjectID

* updated remainder of sysObjectId -> sysObjectID

* another sysObjectId -> sysObjectID

* fixed secureplatform test data

* Fix tests: $device is not pulled from the database before polling
Also, update the db in the core discovery module.
2018-01-06 23:00:47 -06:00
5c5de59f8e Test docs before deploying (#7992)
* Test docs before deploying
fix issue with docs

* Actually check the result of the doc build
add error for testing

* Fix up docs

* use original logic

* only deploy docs if build succeeds.
Make pip quiet
2018-01-02 08:20:19 -06:00
20865b0ca4 OS Testing update (#7971)
* OS Testing update
Option on pre-commit.php to run tests for just one OS. Reorganize tests a bit to support this.
Improved bgp-peers capture and os module capture (initial)
Better output when phpunit tests fail

* fix a couple issues.  Better output for another

* Use the correct setup function
2017-12-28 16:12:08 -06:00
fff66d3c00 Feature: Generic discovery and poller tests (#7873)
* Processor Tests!

* Capture data from live devices easily.

* fix up some stuff, remove powerconnect things as they seem to be just broken.

* formatting, fix missing assignment
add netonix processor data

* fix multi-line, always add sysDescr and sysObjectID
ios cpm test file

* revert composer change and fix whitespace issues

* add help text

* missed help text

* tighter debug output

* handle empty strings properly and mibs with numbers

* use keys for sorting as intended

* fix type with empty data

* oops :)

* whitespace fix

* try installing fping

* Fix TestCase collision + cleanup

* mark TestCase as abstract
don't run two instances of snmpsim

* better database dumps, improved capture

* style fixes

* fix quotes add a few more tables

* add --prefer-new, properly merge data

* Support separate discovery and poller data. But don't waste space if they aren't needed.

* refactor to use class
collects all code in one place for reusability

* reorganize

* Print out when not saving.

* Support for running multiple (or all) modules at once.

* tidy

* Change unit test to a generic name and test all modules we have data for.

* Add documentation and a few more tidies

* whitespace fixes

* Fix tests, add a couple more modules, more docs

* More docs updates

* Fix scrutinizer issues

* add bgp-peers
2017-12-20 08:36:49 -06:00
731261b509 feature: Added manage_bills.php script
* Adding 1st rev of manage_bills.php

* Stlye fixes

* Freshinging PR 7633. Changing -r to -f, to be less confusing. Adding code style changes as per @laf's suggesting

* Adding in option for sysname or hostname variable. Fixing hostname lookup bug

* style fixes

* -renaming add_ports to add_ports_to_bill
- changing -i to -b for clarity sake
- Will print help if -s and -h are not set

* Adding check to ensure -i is set via cli

* I'm stupid

* I'm stupid
2017-12-02 20:26:19 +00:00
6b5dccc169 refactor: AD Auth defer connection until it is needed (#7768)
* refactor: AD Auth defer connection until it is needed
Nice error if php-ldap is missing instead of http 500.

* Add the same error when ldap is missing to other auth methods.
Not as graceful looking in the authorizers since they do not defer connection.
2017-11-28 09:19:34 -06:00
c9728a1f71 refactor: Refactored authorizers to classes (#7497)
* Refactored authorizers to classes

* Merge changes for #7335

* ! fix php 5.3 incompatibility

* Update ADAuthorizationAuthorizer.php

* Fix get_user -> getUser

* Rename AuthorizerFactory to Auth, fix interface missing functions

* Add phpdocs to all interface methods and normalize the names a bit.

* Re-work auth_test.php AD bind tests to work properly with the new class.
Reflection is not the nicest tool, but I think it is appropriate here.
Handle exceptions more nicely in auth_test.php

* Restore AD getUseList fix

Not sure how it got removed

* fix auth_test.php style
2017-11-18 10:33:03 +00:00
af27195746 feature: Script to collect port polling data and compare full walk vs selective port (#7626)
* Script to collect port polling data and compare full walk vs selective port polling.

* Select hosts. Colors. Percent. Units (s). Enable sel poll.

* Moved the colors. Disable selective if needed. Report set status and count.

* Style fix.

* CLI help.

* Docs.

* Style fix.

* Allow comma separated device_id list.
Better color formatting for output
Fix issue when using &$device by reference
2017-11-17 09:10:37 +00:00
95024af2c0 feature: Added purge-port scipt to allow deleting of ports from the CLI. (#7528)
* Add purge-port scipt to allow deleting of ports from the CLI.

  This script allows deleting single ports by their port_id or a list
  of ports read from a file or STDIN.

  LibreNMS Port purge tool
        -p <port_id>    Purge single port by it's port-id
        -f <file>       Purge a list of ports, read port-ids from <file>, one on each line.
                        A filename of - means reading from STDIN.

Signed-off-by: Maximilian Wilhelm <max@sdn.clinic>

* Update CLI-Tools.md
2017-11-09 12:57:41 +00:00
a688a77864 fix: github-apply sometimes fails on PRs with lots of commits (#7604)
.patch is all commits individually, .diff is all commits merged
2017-11-01 21:19:41 +00:00
9d73cd401a feature: LDAP auth update: alerts, api, remember me (#7335)
* feature: LDAP auth update: alerts, api, remember me
Defer ldap connection until it is needed (saves connections from pollers)
Add ability to use a bind account if the server does not allow anonymous bind.
If the server does allow anonymous bind, no config change is needed.
Use Config class

FYI, I have no way to test this.

TODO: update/validate docs

* prevent duplicate users in get_userlist()

* fix bug in Config get for auth_ldap_uid_attribute, `.` should have been `,`
Change case of uidNumber to match common configs (should be case insensitive anyway)

* revert uidnumber case changes and fix up user supplied ones as it is unintuitive that they need to be lowercase.
Add auth_ldap_binddn setting to allow more a more specific way to enter the bind user.
2017-10-30 19:28:40 +00:00
51ba934e11 feature: Added web validation (#7474)
* refactor: convert validations to modules to prep for gui integration

* accidentally dropped, schema update

* fix accidental output to webui

* mention discovery-wrapper.py and re-arrange.

* refine some fix statements

* rename the Config validation group to Configuration.

* fix some scrutinizer issues
remove as many local functions from validator.php as possible
move extensions from pre-check
remove duplicate timezone check
looks like there is some db schema differences between mariadb 10.1 and 10.2, investigating

* mariadb version diff take2

* Check schema version first for database.
Remove stop to go back to command line for install docs.
Add helpful link when there is no devices added to /addhost

* fix incorrect validation for empty string defaults

* Fix style

* Add additional file permissions checks

* catch exception and fail for invalid timezone
Change visuals a bit.

* add php version warning

* fix space
2017-10-26 07:56:09 +01:00
f1b50737bb Docs refactor (#7389)
* docs: Refactored docs for new layout

* some more changes

* more doc updates

* More doc updates

* small update

* small update
2017-10-02 21:36:22 +01:00
e0040b2831 newdevice: Added support for Toshiba RemotEye4 devices (#7312)
* newdevice: Added support for Toshiba RemotEye4 devices

* small updates thanks to kate

* Update new-os.php

* Update new-os.php
2017-09-12 22:27:06 -05:00
edae33507a feature: Added script to generate config for new OS (#7161)
* feature: Added script to generate config for new OS

* scrut fixes
2017-09-04 21:38:09 +01:00
5441bafc81 fix: Fixed IPv6 host renaming (#7275)
* Fixed IPv6 host renaming

* Added get_rrd_dir() and modified other files to use it where appropriate.

* rrd_name() now uses the new function get_rrd_dir(), to make it simpler to modify the escaping in the future.

* Wrong function name in rrdstep.php
2017-09-02 13:45:31 -05:00
809c9ea591 feature: Added Syslog hook for ASA support (#7268)
* Added limited support for ASA using emblem logging

* Updated documentation

* Updated ASA support to cover regular and emblem logging

* Gave the variable a more suitable name

* Removed debug output
2017-09-01 18:37:01 +01:00
e3fab24e1c fix: Added Extreme OS mapping to 'gen_rancid' (#7261)
This is based on git-rancid code base, seems to be mirrored in upstream.
2017-08-30 20:05:32 +01:00
30c1b79b3d fix: Update gen_rancid.php to the correct arista os name (#7214) 2017-08-23 19:29:40 +01:00