mirror/librenms-librenms
1
0
Fork 0
mirror of https://github.com/librenms/librenms.git synced 2024-10-07 16:52:45 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

first attempt at lessening mysql load

Browse Source
This commit is contained in:
Clint Armstrong
2015-09-14 08:40:20 -04:00
parent a761a4be77
commit 728b92f0ff
2 changed files with 16 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
doc/Extensions/Poller-Service.md
View File

@ -4,7 +4,7 @@
The Poller service is an alternative to polling and discovery cron jobs and provides support for distributed polling without memcache. It is multi-threaded and runs continuously discovering and polling devices with the oldest data attempting to honor the polling frequency configured in `config.php`. This service replaces all the required cron jobs except for `/opt/librenms/daily.sh` and `/opt/librenms/alerts.php`.
Configure the maximum number of threads for the service in `$config['poller_service_workers']`. Configure the minimum desired polling frequency in `$config['poller_service_poll_frequency']` and the minimum desired discovery frequency in `$config['poller_service_discover_frequency']`. The service will not poll or discover devices which have data newer than this this configured age in seconds. Configure how frequently the service will attempt to poll devices which are down in `$config['poller_service_down_retry']`.
Configure the maximum number of threads for the service in `$config['poller_service_workers']`. Configure the minimum desired polling frequency in `$config['poller_service_poll_frequency']` and the minimum desired discovery frequency in `$config['poller_service_discover_frequency']`. The service will not poll or discover devices which have data newer than this this configured age in seconds. Configure how frequently the service will attempt to poll devices which are down in `$config['poller_service_down_retry']`. If you have enough pollers that the worker threads run out of work, the service will query looking for devices every `$config['poller_service_retry_query']` seconds.
The poller service is designed to gracefully degrade. If not all devices can be polled within the configured frequency, the service will continuously poll devices refreshing as frequently as possible using the configured number of threads.
@ -18,6 +18,7 @@ $config['poller_service_workers'] = 16;
$config['poller_service_poll_frequency'] = 300;
$config['poller_service_discover_frequency'] = 21600;
$config['poller_service_down_retry'] = 60;
$config['poller_service_retry_query'] = 1;
```
## Distributed Polling

15
poller-service.py
View File

@ -164,6 +164,13 @@ try:
except KeyError:
down_retry = 60
try:
retry_query = int(config['poller_service_retry_query'])
if retry_query == 0:
retry_query = 1
except KeyError:
retry_query = 1
db = DB()
@ -232,16 +239,22 @@ dev_query = ('SELECT device_id, status,
next_update = datetime.now() + timedelta(minutes=1)
devices_scanned = 0
dont_query_until = datetime.fromtimestamp(0)
def poll_worker():
global dev_query
global devices_scanned
global dont_query_until
thread_id = threading.current_thread().name
db = DB()
while True:
if datetime.now() < dont_query_until:
time.sleep(1)
continue
dev_row = db.query(dev_query)
if len(dev_row) < 1:
# Sleep 1 second after getting an empty query, don't hammer the sql server for no reason.
dont_query_until = datetime.now() + timedelta(seconds=retry_query)
time.sleep(1)
continue