mirror/librenms-librenms
1
0
Fork 0
mirror of https://github.com/librenms/librenms.git synced 2024-10-07 16:52:45 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

feature: parallel snmp-scan.py (#6889)

Browse Source 
* feature: parallel snmp-scan.py
Reduces scan time of a /24 from 5 minutes to 14 seconds
Work is done by addhost.php

Just tries to addhost.php hostname/ip right now
Might need some more complexity added there, but I wasn't sure what.

* respect autodiscovery.nets-exclude

* Improvements in ip handling and output
Add compatibility arguments so it can be used as a drop in replacement for snmp-scan.php

* tidy

* Handle errors from config_to_json.php

* Handle Ctrl-C better.  This is likely to get hit when someone scans a /16 or larger or an IPv6 network :)

* Move undefined outcome to proper location

* remove snmp-scan.php
This commit is contained in:
Tony Murray
2017-07-03 15:57:56 -05:00
committed by GitHub
parent 573a4b0e62
commit f810265cc0
4 changed files with 240 additions and 244 deletions
Download Patch File Download Diff File Expand all files Collapse all files

18
doc/Extensions/Auto-Discovery.md
View File

@@ -3,7 +3,7 @@ source: Extensions/Auto-Discovery.md
### Getting Started
LibreNMS provides the ability to automatically add devices on your network, we can do this with via
LibreNMS provides the ability to automatically add devices on your network, we can do this with via
a few methods which will be explained below and also indicate if they are enabled by default.
All discovery methods run when discovery.php runs (every 6 hours by default and within 5 minutes for new devices).
@@ -35,7 +35,7 @@ These details will be attempted when adding devices, you can specify any mixture
### Allowed Networks
#### Your Networks
To add devices, we need to know what are your subnets so we don't go blindly attempting to add devices not
To add devices, we need to know what are your subnets so we don't go blindly attempting to add devices not
under your control.
```php
@@ -45,7 +45,7 @@ $config['nets'][] = '172.2.4.0/22';
### Exclusions
If you have added a network as above but a single device exists within it that you can't auto
If you have added a network as above but a single device exists within it that you can't auto
add, then you can exclude this with the following:
```php
@@ -55,17 +55,17 @@ $config['autodiscovery']['nets-exclude'][] = '192.168.0.1/32';
## Additional Options
#### Discovering devices by IP
By default we don't add devices by IP address, we look for a reverse dns name to be found and add with that. If this fails
By default we don't add devices by IP address, we look for a reverse dns name to be found and add with that. If this fails
and you would like to still add devices automatically then you will need to set `$config['discovery_by_ip'] = true;`
#### Short hostnames
If your devices only return a short hostname such as lax-fa0-dc01 but the full name should be lax-fa0-dc01.example.com then you can
If your devices only return a short hostname such as lax-fa0-dc01 but the full name should be lax-fa0-dc01.example.com then you can
set `$config['mydomain'] = 'example.com';`
#### Allow Duplicate sysName
By default we require unique sysNames when adding devices (this is returned over snmp by your devices). If you would like to allow
By default we require unique sysNames when adding devices (this is returned over snmp by your devices). If you would like to allow
devices to be added with duplicate sysNames then please set `$config['allow_duplicate_sysName'] = true;`.
@@ -127,14 +127,14 @@ This module is invoked from bgp-peers discovery module.
## SNMP Scan
This isn't actually an auto-discovery mechanism, but manually invoked.
It's designed to scan through all of the subnets in your config or what you have manually specified
It's designed to scan through all of the subnets in your config or what you have manually specified
to automatically add devices.
SNMP Scan will scan `$config['nets']` by default and respects `$config['autodiscovery']['nets-exclude']`.
An example of it's usage is:
```bash
./snmp-scan.php -r 192.168.0.0/24
./snmp-scan.py -r 192.168.0.0/24
```

28
doc/Support/Discovery Support.md
View File

@@ -46,7 +46,7 @@ With:
`33 */6 * * * librenms /opt/librenms/discovery-wrapper.php 1 >> /dev/null 2>&1`
The default is for discovery wrapper to only use 1 thread so that it mimics the current behaviour. However if your
The default is for discovery wrapper to only use 1 thread so that it mimics the current behaviour. However if your
system is powerful enough and the devices can cope then you can increase the thread count from 1 to a value of your
choosing.
@@ -209,14 +209,26 @@ Using the SNMP-Scanner may take a long time to finish depending on the size of y
If possible, divide your network into smaller subnets and scan these subnets instead. You can use an utility like the GNU Screen or tmux to avoid aborting the scan when logging out of your Shell. You can run several instances of the SNMP-Scanner simultaneously.
To run the SNMP-Scanner you need to execute the `snmp-scan.php` from within your LibreNMS installation directory.
To run the SNMP-Scanner you need to execute the `snmp-scan.py` from within your LibreNMS installation directory.
Here the script's help-page for reference:
```text
Usage: ./snmp-scan.php -r <CIDR_Range> [-d] [-l] [-h]
-r CIDR_Range CIDR noted IP-Range to scan
Example: 192.168.0.0/24
-d Enable Debug
-l Show Legend
-h Print this text
usage: snmp-scan.py [-h] [-r NETWORK] [-t THREADS] [-l] [-v]
Scan network for snmp hosts and add them to LibreNMS.
optional arguments:
-h, --help show this help message and exit
-r NETWORK CIDR noted IP-Range to scan. Can be specified multiple times
This argument is only required if $config['nets'] is not set
Example: 192.168.0.0/24 Example: 192.168.0.0/31 will be
treated as an RFC3021 p-t-p network with two addresses,
192.168.0.0 and 192.168.0.1 Example: 192.168.0.1/32 will be
treated as a single host address
-t THREADS How many IPs to scan at a time. More will increase the scan
speed, but could overload your system. Default: 32
-l, --legend Print the legend.
-v, --verbose Show debug output. Specifying multiple times increases the
verbosity.
```

227
snmp-scan.php
View File

@@ -1,227 +0,0 @@
#!/usr/bin/env php
<?php
/*
* Copyright (C) 2015 Daniel Preussker <f0o@librenms.org>
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
/**
* SNMP Scan
* @author f0o <f0o@librenms.org>
* @copyright 2015 f0o, LibreNMS
* @license GPL
* @package LibreNMS
* @subpackage Discovery
*/
use LibreNMS\Exceptions\HostExistsException;
use LibreNMS\Exceptions\HostUnreachableException;
use LibreNMS\Exceptions\HostUnreachablePingException;
chdir(__DIR__); // cwd to the directory containing this script
$ts = microtime(true);
$init_modules = array('discovery');
require __DIR__ . '/includes/init.php';
if ($config['autodiscovery']['snmpscan'] === false) {
echo 'SNMP-Scan disabled.'.PHP_EOL;
exit(2);
}
function perform_snmp_scan($net, $force_network, $force_broadcast)
{
global $stats, $config, $debug, $vdebug, $more_info;
echo 'Range: '.$net->network.'/'.$net->bitmask.PHP_EOL;
$config['snmp']['timeout'] = 1;
$config['snmp']['retries'] = 0;
$config['fping_options']['retries'] = 0;
$start = ip2long($net->network);
$end = ip2long($net->broadcast)-1;
if ($force_network === true) { //Force-scan network address
d_echo("Forcing network address scan".PHP_EOL);
$start = $start-1;
}
if ($force_broadcast === true) { //Force-scan broadcast address
d_echo("Forcing broadcast address scan".PHP_EOL);
$end = $end+1;
}
if ($net->bitmask === "31") { //Handle RFC3021 /31 prefixes
$start = ip2long($net->network)-1;
$end = ip2long($net->broadcast);
d_echo("RFC3021 network, hosts ".long2ip($start+1)." and ".long2ip($end).PHP_EOL.PHP_EOL);
} elseif ($net->bitmask === "32") { //Handle single-host /32 prefixes
$start = ip2long($net->network)-1;
$end = $start+1;
d_echo("RFC3021 network, hosts ".long2ip($start+1)." and ".long2ip($end).PHP_EOL.PHP_EOL);
} else {
d_echo("Network: ".($net->network).PHP_EOL);
d_echo("Broadcast: ".($net->broadcast).PHP_EOL.PHP_EOL);
}
while ($start++ < $end) {
$stats['count']++;
$host = long2ip($start);
if ($vdebug || $more_info === true) {
echo "Scanning: ".$host.PHP_EOL;
}
if (match_network($config['autodiscovery']['nets-exclude'], $host)) {
if ($vdebug || $more_info === true) {
echo "Excluded by config.php".PHP_EOL.PHP_EOL;
} else {
echo '|';
}
continue;
}
$test = isPingable($host);
if ($test['result'] === false) {
if ($vdebug || $more_info === true) {
echo "Unpingable Device $host".PHP_EOL.PHP_EOL;
} else {
echo '.';
}
continue;
}
if (ip_exists($host)) {
$stats['known']++;
if ($vdebug || $more_info === true) {
echo "Known Device $host".PHP_EOL;
} else {
echo '*';
}
continue;
}
foreach (array('udp','tcp') as $transport) {
try {
addHost(gethostbyaddr($host), '', $config['snmp']['port'], $transport, $config['distributed_poller_group']);
$stats['added']++;
if ($vdebug || $more_info === true) {
echo "Added Device $host".PHP_EOL.PHP_EOL;
} else {
echo '+';
}
break;
} catch (HostExistsException $e) {
$stats['known']++;
if ($vdebug || $more_info === true) {
echo "Known Device $host".PHP_EOL.PHP_EOL;
} else {
echo '*';
}
break;
} catch (HostUnreachablePingException $e) {
if ($vdebug || $more_info === true) {
echo "Unpingable Device $host".PHP_EOL.PHP_EOL;
} else {
echo '.';
}
break;
} catch (HostUnreachableException $e) {
if ($debug || $more_info === true) {
print_error($e->getMessage() . " over $transport");
foreach ($e->getReasons() as $reason) {
echo " $reason".PHP_EOL;
}
}
if ($transport === 'tcp') {
// tried both udp and tcp without success
$stats['failed']++;
if ($vdebug || $more_info === true) {
echo "Failed to Add Device $host".PHP_EOL.PHP_EOL;
} else {
echo '-';
}
}
}
}
}
echo PHP_EOL;
}
$opts = getopt('r:d::v::i::n::b::l::h::');
$stats = array('count'=> 0, 'known'=>0, 'added'=>0, 'failed'=>0);
$start = false;
$debug = false;
$quiet = 1;
$net = false;
if (isset($opts['h']) || (empty($opts) && (!isset($config['nets']) || empty($config['nets'])))) {
echo 'Usage: '.$argv[0].' -r <CIDR_Range> [-d] [-l] [-h]'.PHP_EOL;
echo ' -r CIDR_Range CIDR noted IP-Range to scan'.PHP_EOL;
echo ' This argument is only required if $config[\'nets\'] is not set'.PHP_EOL;
echo ' Example: 192.168.0.0/24'.PHP_EOL;
echo ' Example: 192.168.0.0/31 will be treated as an RFC3021 p-t-p network'.PHP_EOL;
echo ' with two addresses, 192.168.0.0 and 192.168.0.1'.PHP_EOL;
echo ' Example: 192.168.0.1/32 will be treated as a single host address'.PHP_EOL;
echo ' -n Force scan of network address'.PHP_EOL;
echo ' -b Force scan of broadcast address'.PHP_EOL;
echo ' -d Enable Debug'.PHP_EOL;
echo ' -v Enable verbose Debug'.PHP_EOL;
echo ' -i Provide more information on actions'.PHP_EOL;
echo ' -l Show Legend'.PHP_EOL;
echo ' -h Print this text'.PHP_EOL;
exit(0);
}
if (isset($opts['d']) || isset($opts['v'])) {
if (isset($opts['v'])) {
$vdebug = true;
}
$debug = true;
}
if (isset($opts['l'])) {
echo ' * = Known Device; . = Unpingable Device; + = Added Device; - = Failed To Add Device; | = Excluded by config.php'.PHP_EOL;
}
if (isset($opts['n'])) {
$force_network = true;
}
if (isset($opts['b'])) {
$force_broadcast = true;
}
if (isset($opts['i'])) {
$more_info = true;
}
if (isset($opts['r'])) {
$net = Net_IPv4::parseAddress($opts['r']);
if (ip2long($net->network) !== false) {
perform_snmp_scan($net, $force_network, $force_broadcast);
echo 'Scanned '.$stats['count'].' IPs, Already known '.$stats['known'].' Devices, Added '.$stats['added'].' Devices, Failed to add '.$stats['failed'].' Devices.'.PHP_EOL;
echo 'Runtime: '.(microtime(true)-$ts).' secs'.PHP_EOL;
} else {
echo 'Could not interpret supplied CIDR noted IP-Range: '.$opts['r'].PHP_EOL;
exit(2);
}
} elseif (isset($config['nets']) && !empty($config['nets'])) {
if (!is_array($config['nets'])) {
$config['nets'] = array( $config['nets'] );
}
foreach ($config['nets'] as $subnet) {
$net = Net_IPv4::parseAddress($subnet);
perform_snmp_scan($net, $force_network, $force_broadcast);
}
echo 'Scanned '.$stats['count'].' IPs, Already know '.$stats['known'].' Devices, Added '.$stats['added'].' Devices, Failed to add '.$stats['failed'].' Devices.'.PHP_EOL;
echo 'Runtime: '.(microtime(true)-$ts).' secs'.PHP_EOL;
} else {
echo 'Please either add a range argument with \'-r <CIDR_RANGE>\' or define $config[\'nets\'] in your config.php'.PHP_EOL;
exit(2);
}

211
snmp-scan.py Executable file
View File

@@ -0,0 +1,211 @@
#!/usr/bin/env python3
"""
Scan networks for snmp hosts and add them to LibreNMS
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
@package LibreNMS
@link http://librenms.org
@copyright 2017 Tony Murray
@author Tony Murray <murraytony@gmail.com>
"""
import argparse
import json
import socket
from collections import namedtuple
from enum import Enum
from ipaddress import ip_network, ip_address
from multiprocessing import Pool
from os import path, chdir
from subprocess import check_output, CalledProcessError
from time import time
Result = namedtuple('Result', ['ip', 'hostname', 'outcome', 'output'])
class Outcome(Enum):
UNDEFINED = 0
ADDED = 1
UNPINGABLE = 2
KNOWN = 3
FAILED = 4
EXCLUDED = 5
TERMINATED = 6
VERBOSE_LEVEL = 0
THREADS = 32
CONFIG = {}
EXCLUDED_NETS = []
start_time = time()
stats = {'count': 0, Outcome.ADDED: 0, Outcome.UNPINGABLE: 0, Outcome.KNOWN: 0, Outcome.FAILED: 0, Outcome.EXCLUDED: 0, Outcome.TERMINATED: 0}
def debug(message, level=2):
if level <= VERBOSE_LEVEL:
print(message)
def get_outcome_symbol(outcome):
return {
Outcome.UNDEFINED: '?', # should not occur
Outcome.ADDED: '+',
Outcome.UNPINGABLE: '.',
Outcome.KNOWN: '*',
Outcome.FAILED: '-',
Outcome.TERMINATED: ''
}[outcome]
def handle_result(data):
if VERBOSE_LEVEL > 0:
print('Scanned \033[1m{}\033[0m {}'.format(("{} ({})".format(data.hostname, data.ip) if data.hostname else data.ip), data.output))
else:
print(get_outcome_symbol(data.outcome), end='', flush=True)
stats['count'] += 0 if data.outcome == Outcome.TERMINATED else 1
stats[data.outcome] += 1
def check_ip_excluded(ip):
for net in EXCLUDED_NETS:
if ip in net:
debug("\033[91m{} excluded by autodiscovery.nets-exclude\033[0m".format(ip), 1)
stats[Outcome.EXCLUDED] += 1
return True
return False
def scan_host(ip):
hostname = None
try:
try:
tmp = socket.gethostbyaddr(ip)[0]
if socket.gethostbyname(tmp) == ip: # check that forward resolves
hostname = tmp
except socket.herror:
pass
try:
add_output = check_output(['/usr/bin/env', 'php', 'addhost.php', hostname or ip])
return Result(ip, hostname, Outcome.ADDED, add_output)
except CalledProcessError as err:
output = err.output.decode().rstrip()
if err.returncode == 2:
if 'Could not ping' in output:
return Result(ip, hostname, Outcome.UNPINGABLE, output)
else:
return Result(ip, hostname, Outcome.FAILED, output)
elif err.returncode == 3:
return Result(ip, hostname, Outcome.KNOWN, output)
except KeyboardInterrupt:
return Result(ip, hostname, Outcome.TERMINATED, 'Terminated')
return Result(ip, hostname, Outcome.UNDEFINED, output)
if __name__ == '__main__':
###################
# Parse arguments #
###################
parser = argparse.ArgumentParser(description='Scan network for snmp hosts and add them to LibreNMS.')
parser.add_argument('network', nargs='*', help="""CIDR noted IP-Range to scan. Can be specified multiple times
This argument is only required if $config['nets'] is not set
Example: 192.168.0.0/24
Example: 192.168.0.0/31 will be treated as an RFC3021 p-t-p network with two addresses, 192.168.0.0 and 192.168.0.1
Example: 192.168.0.1/32 will be treated as a single host address""")
parser.add_argument('-t', dest='threads', type=int, help="How many IPs to scan at a time. More will increase the scan speed, but could overload your system. Default: {}".format(THREADS))
parser.add_argument('-l', '--legend', action='store_true', help="Print the legend.")
parser.add_argument('-v', '--verbose', action='count', help="Show debug output. Specifying multiple times increases the verbosity.")
# compatibility arguments
parser.add_argument('-r', dest='network', action='append', metavar='network', help=argparse.SUPPRESS)
parser.add_argument('-d', '-i', dest='verbose', action='count', help=argparse.SUPPRESS)
parser.add_argument('-n', action='store_true', help=argparse.SUPPRESS)
parser.add_argument('-b', action='store_true', help=argparse.SUPPRESS)
args = parser.parse_args()
VERBOSE_LEVEL = args.verbose or VERBOSE_LEVEL
THREADS = args.threads or THREADS
# Import LibreNMS config
install_dir = path.dirname(path.realpath(__file__))
chdir(install_dir)
try:
CONFIG = json.loads(check_output(['/usr/bin/env', 'php', 'config_to_json.php']).decode())
except CalledProcessError as e:
parser.error("Could not execute: {}\n{}".format(' '.join(e.cmd), e.output.decode().rstrip()))
#######################
# Build network lists #
#######################
if not CONFIG.get('nets', []) and not args.network:
parser.error('$config[\'nets\'] is not set in config.php, you must specify a network to scan with -r')
networks = []
for net in (args.network if args.network else CONFIG.get('nets', [])):
try:
networks.append(ip_network(net, True))
debug('Network parsed: {}'.format(net), 2)
except ValueError as e:
parser.error('Invalid network format {}'.format(e))
for net in CONFIG.get('autodiscovery', {}).get('nets-exclude', {}):
try:
EXCLUDED_NETS.append(ip_network(net, True))
debug('Excluded network: {}'.format(net), 2)
except ValueError as e:
parser.error('Invalid excluded network format {}, check your config.php'.format(e))
#################
# Scan networks #
#################
debug('SNMP settings from config.php: {}'.format(CONFIG.get('snmp', {})), 2)
if args.legend and not VERBOSE_LEVEL:
print('Legend:\n+ Added device\n* Known device\n- Failed to add device\n. Ping failed\n')
print('Scanning IPs:')
pool = Pool(processes=THREADS)
try:
for network in networks:
if network.num_addresses == 1:
ips = [ip_address(network.network_address)]
else:
ips = network.hosts()
for ip in ips:
if not check_ip_excluded(ip):
pool.apply_async(scan_host, (str(ip),), callback=handle_result)
pool.close()
pool.join()
except KeyboardInterrupt:
pool.terminate()
if VERBOSE_LEVEL == 0:
print("\n")
base = 'Scanned {} IPs: {} known devices, added {} devices, failed to add {} devices'
summary = base.format(stats['count'], stats[Outcome.KNOWN], stats[Outcome.ADDED], stats[Outcome.FAILED])
if stats[Outcome.EXCLUDED]:
summary += ', {} ips excluded by config'.format(stats[Outcome.EXCLUDED])
print(summary)
print('Runtime: {:.2f} seconds'.format(time() - start_time))