41 Commits

Author SHA1 Message Date
Zane C. Bowers-Hadley 0bbcde1227 add the ability for storing app data to prevent spamming of the event log via via component usage (#14087)
* initial work on add the ability to save/fetch app data

* update to use get_app_data for ZFS

* update the poller for the new app_data stuff

* ZFS now logs changes to pools

* add schema update for app_data stuff

* small formatting fix

* add a missing \

* now adds a column

* sql-schema is no longer used, so remove the file that was added here

* misc cleanups

* rename the method in database/migrations/2022_07_03_1947_add_app_data.php

* hopefully fix the migration bit

* add the column to misc/db_schema.yaml

* more misc small DB fixes

* update the test as the json column uses collat of utf8mb4_bin

* revert the last change and try manually setting it to what is expected

* remove a extra ;

* update suricata as well

* correct the instance -> instances in one location to prevent the old instance list from being stomped

* remove a extra ;

* update fail2ban to use it as well

* remove two unused functions as suricata and fail2ban no longer use components

* style cleanup

* postgres poller updated to use it

* update html side of the postgres bits

* chronyd now uses app data bits now as well

* portactivity now uses it as well

* style fix

* sort the returned arrays from app_data

* correct log message for port activity

* collocation change

* try re-ordering it

* add in the new data column to the tests

* remove a extra ,

* hmm... ->collate('utf8mb4_unicode_ci') is not usable as apparently collate does not exist

* change the column type from json to longtext

* mv chronyd stuff while I sort out the rest of the tests... damn thing is always buggy

* hmm... fix a missing line then likely move stuff back

* style fix

* add fillable

* add the expexcted data for fail2ban json

* escape a " I missed

* add data for portactivity

* add suricata app data

* add app data to zfs legacy test

* put the moved tests back into place and update zfs-v1 test

* add app data for chronyd test

* add app data for fail2ban legacy test

* update zfs v1 app data

* add some notes on application dev work

* add Developing/Application-Notes.md to mkdocs.yml

* add data column to it

* added various suggestions from bennet-esyoil

* convert from isset to sizeof

* type fix

* fully remove the old save app data function and move it into a helper function... the other still needs cleaned up prior to removal

* update docs

* get_app_data is fully removed now as well

* a few style fixes

* add $casts

* update chronyd test

* attempt to fix the data

* more doc cleanup and try changing the cast

* style fix

* revert the changes to the chronyd test

* apply a few of murrant's suggestions

* document working with ->data as json and non-josn

* remove two no-longer used in this PR exceptions

* ->data now operates transparently

* style fix

* update data tests

* fix json

* test fix

* update the app notes to reflect how app data now works

* app test fix

* app data fix for linux_lsi

* json fix

* minor doc cleanup

* remove duplicate querty and use json_decode instead

* style fix

* modelize the app poller

* use a anon func instead of foreach

* test update

* style cleanup

* style cleanup

* another test cleanup

* more test cleanup

* reverse the test changes and add in some more glue code

* revert one of the test changes

* another small test fix

* Make things use models
Left some array access, but those will still work just fine.

* missed chronyd and portactivity

* rename poll to avoid make it any confusion

* Remove extra save and fix timestamp

* save any changes made to app->data

* nope, that was not it

* What are magic methods and how do they work?

* fix two typos

* update linux_lsi test

* change quote type

Co-authored-by: Tony Murray <murraytony@gmail.com>
2022-07-22 16:01:55 -05:00
Mark Westerterp 8a6f158ab8 fix: Add consistent output of name and app_id to Poller for all Applications (fixes #13641) (#13648)
* Add echo of name and app_id to all Applications

* Make echo not application specifc

* Fix echo

* Do spaces make StyleCI happy?

* Not spaces, but the type of quotes makes StyleCI happy
2022-01-06 12:08:52 +01:00
rasssta 3f0f133ee9 Added metric for MySQL Slave Lag (secs) (#12765)
* Added metric for MySQL Slave Lag (secs)

Added metric for MySQL Slave Lag (in seconds) to be able to build Alert Rules when slave lag value is greater than 0 (or X). It's possible to add graphs and other metrics as well, but the priority for me is to be able to build alert rules based on the "Slave Lag" metric.

* Added metric for slave lag (SlLa)
2021-04-19 20:06:55 +02:00
Jellyfrog 2fc3a21211 Apply fixes from StyleCI (#12121) 2020-09-21 15:43:38 +02:00
SourceDoctor 4b52cab776 fix RRD in MySQL (#11740) 2020-05-30 17:43:41 -05:00
Tony Murray df16de9d2f Refactor Datastores to allow future improvements. OpenTSDB Tags. (#11283)
* Datastores to object oriented code, using the Laravel IoC container

Change instantiation
better DI
move OpenTSDB

Small re-orgs

remove unused stuff

Fix graphs and other scripts

Use DI for all except rrd

fix up connection error handling

Add tests, fix up a "few" things
Add Config::forget()

Style fixes

Don't reference legacy code

remove accidental code paste

Add datastores phpunit groups

some tests

* rebase fixes

* some test fixes

* shorter tests

* shorter tests

* Don't except when rrdtool can't be started.

* restore tests

* fix rrd tests

* fix iterable change upstream

* fix isValidDataset

* fix invalid data bug

* fix mysql incorrect ds

* fix issue with data that is too long

* use regular data_update()

* Use log facade

* OpenTSDB mis-ordered arguments fix

* Making a singleton with different options makes different singletons.  Just use the global config settings to disable datastores.

* only filter tags for datastores that won't it don't modify the tags permanently

* Update copyrights to include original authors.

* Stats for all datastores

* Fix mysql sends different rrd / other ds names

* fix snmp last stats not initialized
remove unused function

* remove unused function and move single use function closer to its use

* InfluxDB does not need to update null or U values.
Skip write if all fields are empty

* Fix smart value checks

* fix style issues

* Make sure port data is stored the same way as before for Graphite and OpenTSDB
Add ifIndex tag to all to be compatible

* Missed rrdtool_tune() call

* Test update WIP

* OpenTSDB now includes tags

* fix style
2020-03-16 09:17:58 -05:00
Tony Murray 7fe895bd08 Revert "Refactor datastores to classes (#9179)" (#11273)
This reverts commit e5dad7a64e.
2020-03-11 07:52:52 -05:00
Tony Murray e5dad7a64e Refactor datastores to classes (#9179)
* Datastores to object oriented code, using the Laravel IoC container

Change instantiation
better DI
move OpenTSDB

Small re-orgs

remove unused stuff

Fix graphs and other scripts

Use DI for all except rrd

fix up connection error handling

Add tests, fix up a "few" things
Add Config::forget()

Style fixes

Don't reference legacy code

remove accidental code paste

Add datastores phpunit groups

some tests

* rebase fixes

* some test fixes

* shorter tests

* shorter tests

* Don't except when rrdtool can't be started.

* restore tests

* fix rrd tests

* fix iterable change upstream

* fix isValidDataset

* fix invalid data bug

* fix mysql incorrect ds

* fix issue with data that is too long

* use regular data_update()

* Use log facade

* OpenTSDB mis-ordered arguments fix

* Making a singleton with different options makes different singletons.  Just use the global config settings to disable datastores.

* only filter tags for datastores that won't it don't modify the tags permanently

* Update copyrights to include original authors.

* Stats for all datastores

* Fix mysql sends different rrd / other ds names

* fix snmp last stats not initialized
remove unused function

* remove unused function and move single use function closer to its use

* InfluxDB does not need to update null or U values.
Skip write if all fields are empty

* Fix smart value checks

* fix style issues
2020-03-10 19:08:00 -05:00
Tony Murray b620692426 refactor: Update all applications to store metrics (#7853)
* Update all applications to store metrics

* fix db schema

* Add glue test
2017-12-06 22:13:10 +00:00
Tony Murray 379176ceac fix: MySQL app graphs with rrdcached (#6608)
Do not send -1 values when we set the minimum as 0.  Send the proper U for non-values.
fixes: #6223
2017-05-08 19:57:51 +01:00
Neil Lathwood a0abcbb649 fix: Fix broken mysql application polling (#6317) 2017-04-01 22:22:04 +01:00
Neil Lathwood a6391ed3e8 fix: Update applications poller to use numeric oid instead of nsExtendOutputFull (#6277)
* fix: Update applications poller to use numeric oid instead of nsExtendOutputFull

* Accidental m
2017-03-30 22:01:40 -05:00
Neil Lathwood 66d1006d91 feature: Added app_state support for applications #5068 (#6061)
* feature: Added app_state support for applications #5068

* moved schema file 173 -> 174

* Re-arrange code

* added device_id to dbUpdate call

* updated function + missing apps

* Rename 174.sql to 176.sql

* Remove $device, it is unneeded when updating.  Rename update_applications() -> update_application()

* Some apps can return simply 0
2017-03-11 08:39:32 -06:00
Neil Lathwood fad5aca1b7 feature: Allow customisation of rrd step/heartbeat when creating new rrd files (#5947)
* feature: Allow customisation of rrd step/heartbeat when creating new rrd files

* revert defaults

* added docs + webui config option

* Move RrdDefinition to an Object to make them easier to create and remove the possibility of typos.

* Fix style/lint issues and missing use statements

* 3 more missing use statements

* updated doc + moved schema file
2017-02-23 22:45:50 +00:00
Søren Rosiak 414efb3539 refactor: Various Code Cleanup (#5777)
* refactor: Various Code Cleanup

* moar
2017-02-05 22:37:08 +00:00
Tony Murray 9284bc60ff Update code in includes to be PSR-2 compliant (#4220)
refactor: Update code in /includes to be psr2 compliant #4220
2016-08-28 18:32:58 +01:00
Tony Murray 01cfb960d1 Fix sensors rrd name issue
Use last instead of first, as last won't flush rrdcached
Only stream_select if timeout > 0
Attempt to fix mysql
2016-07-08 22:58:36 -05:00
Tony Murray 57650e4ffc Remove nfs requirement for distributed poller (with rrdtool 1.5+)
Use data_update instead of rrd_update/rrd_create and influx_update
Centralize rrd file check so we can check against a remote rrdcached server too
Use rrd_name() to generate rrd file names
2016-07-07 21:10:06 -05:00
laf 0f33f6d1fa Rebased 2016-01-04 12:09:32 +00:00
Luke Gopher 62e83a2f28 Fix myql empty graphs
Fix this problem when we have another plugin enabled than Mysql on a server
2015-12-18 17:07:40 +01:00
laf 7b14aada4e Rebased 2015-10-26 11:08:19 +00:00
f0o 7d95e94954 Fix rrd creation parameters 2015-09-28 08:24:57 +00:00
laf 433562d351 More updates for influx 2015-08-19 20:58:02 +00:00
laf 987634e463 Updated rrdtool_update() to use new array format 2015-08-18 16:26:55 +00:00
Job Snijders ebd0f6fc35 Apply "Squiz" code style on old (pre-2014) files 2015-07-11 16:12:35 +02:00
Adam Amstrong b273e04241 /// -> //
git-svn-id: http://www.observium.org/svn/observer/trunk@3240 61d68cd4-352d-0410-923a-c4978735b2b8
2012-05-25 12:24:34 +00:00
Adam Amstrong cacf913a55 # -> / for phpdoc
git-svn-id: http://www.observium.org/svn/observer/trunk@3239 61d68cd4-352d-0410-923a-c4978735b2b8
2012-05-25 11:29:53 +00:00
Tom Laermans 1f9eae3699 add snmpv3 support to old snmp queries - patch by gl
git-svn-id: http://www.observium.org/svn/observer/trunk@3184 61d68cd4-352d-0410-923a-c4978735b2b8
2012-05-15 13:45:26 +00:00
Tom Laermans 613e8b1e29 syntaxer run
git-svn-id: http://www.observium.org/svn/observer/trunk@3117 61d68cd4-352d-0410-923a-c4978735b2b8
2012-05-02 22:02:30 +00:00
Adam Amstrong 850359711c add memcached application graphing support. move applications to <<<app-$name>>> for easier detection. add connection tracking to mysql application. move logout to /logout/ not
/?logout=yes.


git-svn-id: http://www.observium.org/svn/observer/trunk@3096 61d68cd4-352d-0410-923a-c4978735b2b8
2012-04-27 17:18:26 +00:00
Tom Laermans e2dc9b246d syntaxer run
git-svn-id: http://www.observium.org/svn/observer/trunk@3071 61d68cd4-352d-0410-923a-c4978735b2b8
2012-04-21 00:36:55 +00:00
Adam Amstrong 2b8178bb5b move all RRA decisions to config. switch mempool/storage to new rrds (sorry, data lost, the old rrds sucked). added some more apps pages.
git-svn-id: http://www.observium.org/svn/observer/trunk@3068 61d68cd4-352d-0410-923a-c4978735b2b8
2012-04-20 17:37:58 +00:00
Adam Amstrong c09d5af16b new agent-based applications
git-svn-id: http://www.observium.org/svn/observer/trunk@3059 61d68cd4-352d-0410-923a-c4978735b2b8
2012-04-19 15:36:46 +00:00
Tom Laermans 392d97bf73 syntaxer run
git-svn-id: http://www.observium.org/svn/observer/trunk@2966 61d68cd4-352d-0410-923a-c4978735b2b8
2012-04-05 16:37:46 +00:00
Adam Amstrong 84e9f458f4 fix mysql again (broken counter set as gauge)
git-svn-id: http://www.observium.org/svn/observer/trunk@2669 61d68cd4-352d-0410-923a-c4978735b2b8
2011-10-07 22:01:19 +00:00
Adam Amstrong fdb51b5e6e fix mysql?
git-svn-id: http://www.observium.org/svn/observer/trunk@2668 61d68cd4-352d-0410-923a-c4978735b2b8
2011-10-07 19:53:09 +00:00
Tom Laermans dc0ebc7343 more code cleanup
git-svn-id: http://www.observium.org/svn/observer/trunk@2517 61d68cd4-352d-0410-923a-c4978735b2b8
2011-09-20 14:22:34 +00:00
Tom Laermans a914dfd482 remove DEVELOPING file, moved to wiki now (see Developing); minor layout fixes; remove some sql queries upon eventlog, as log_event actually prefers to receive a device array in the first place
git-svn-id: http://www.observium.org/svn/observer/trunk@2011 61d68cd4-352d-0410-923a-c4978735b2b8
2011-04-03 19:36:32 +00:00
Adam Amstrong 28e03f65ea improvement to mysql application from pkn
git-svn-id: http://www.observium.org/svn/observer/trunk@1999 61d68cd4-352d-0410-923a-c4978735b2b8
2011-03-31 13:10:58 +00:00
Tom Laermans 30bdd952ee Minor fixes, some cleanups, and standardising on echo("x") instead of echo "x"
git-svn-id: http://www.observium.org/svn/observer/trunk@1728 61d68cd4-352d-0410-923a-c4978735b2b8
2010-11-20 14:04:07 +00:00
Adam Amstrong 349cd58779 MySQL Application poller (thanks to Ove)
git-svn-id: http://www.observium.org/svn/observer/trunk@1725 61d68cd4-352d-0410-923a-c4978735b2b8
2010-10-31 20:24:21 +00:00