mirror of
https://github.com/netbox-community/netbox.git
synced 2024-05-10 07:54:54 +00:00
12510 Merge Scripts and Reports (#14976)
* 12510 move reports to use BaseScript * 12510 merge report into script view * 12510 add migration for job report to script * 12510 update templates * 12510 remove reports * 12510 cleanup * 12510 legacy jobs * 12510 legacy jobs * 12510 fixes * 12510 review changes * 12510 review changes * 12510 update docs * 12510 review changes * 12510 review changes * 12510 review changes * 12510 review changes * 12510 main log results to empty string * 12510 move migration * Introduce an internal log level for debug to simplify Script logging * Misc cleanup * Remove obsolete is_valid() method * Reformat script job data (log, output, tests) * Remove ScriptLogMessageSerializer * Fix formatting of script logs * Record a timestamp with script logs * Rename _current_method to _current_test * Clean up template * Remove obsolete runreport management command * Misc cleanup & refactoring * Clean up template * Clean up migration * Clean up docs --------- Co-authored-by: Jeremy Stretch <jstretch@netboxlabs.com>
This commit is contained in:
Arthur Hanson
GitHub
@ -5,8 +5,17 @@ Custom scripting was introduced to provide a way for users to execute custom log
|
||||
* Automatically populate new devices and cables in preparation for a new site deployment
|
||||
* Create a range of new reserved prefixes or IP addresses
|
||||
* Fetch data from an external source and import it to NetBox
|
||||
* Update objects with invalid or incomplete data
|
||||
|
||||
Custom scripts are Python code and exist outside of the official NetBox code base, so they can be updated and changed without interfering with the core NetBox installation. And because they're completely custom, there is no inherent limitation on what a script can accomplish.
|
||||
They can also be used as a mechanism for validating the integrity of data within NetBox. Script authors can define test to check object against specific rules and conditions. For example, you can write script to check that:
|
||||
|
||||
* All top-of-rack switches have a console connection
|
||||
* Every router has a loopback interface with an IP address assigned
|
||||
* Each interface description conforms to a standard format
|
||||
* Every site has a minimum set of VLANs defined
|
||||
* All IP addresses have a parent prefix
|
||||
|
||||
Custom scripts are Python code which exists outside the NetBox code base, so they can be updated and changed without interfering with the core NetBox installation. And because they're completely custom, there is no inherent limitation on what a script can accomplish.
|
||||
|
||||
## Writing Custom Scripts
|
||||
|
||||
@ -135,13 +144,73 @@ These two methods will load data in YAML or JSON format, respectively, from file
|
||||
|
||||
The Script object provides a set of convenient functions for recording messages at different severity levels:
|
||||
|
||||
* `log_debug`
|
||||
* `log_success`
|
||||
* `log_info`
|
||||
* `log_warning`
|
||||
* `log_failure`
|
||||
* `log_debug(message, object=None)`
|
||||
* `log_success(message, object=None)`
|
||||
* `log_info(message, object=None)`
|
||||
* `log_warning(message, object=None)`
|
||||
* `log_failure(message, object=None)`
|
||||
|
||||
Log messages are returned to the user upon execution of the script. Markdown rendering is supported for log messages.
|
||||
Log messages are returned to the user upon execution of the script. Markdown rendering is supported for log messages. A message may optionally be associated with a particular object by passing it as the second argument to the logging method.
|
||||
|
||||
## Test Methods
|
||||
|
||||
A script can define one or more test methods to report on certain conditions. All test methods must have a name beginning with `test_` and accept no arguments beyond `self`.
|
||||
|
||||
These methods are detected and run automatically when the script is executed, unless its `run()` method has been overridden. (When overriding `run()`, `run_tests()` can be called to run all test methods present in the script.)
|
||||
|
||||
!!! info
|
||||
This functionality was ported from [legacy reports](./reports.md) in NetBox v4.0.
|
||||
|
||||
### Example
|
||||
|
||||
```
|
||||
from dcim.choices import DeviceStatusChoices
|
||||
from dcim.models import ConsolePort, Device, PowerPort
|
||||
from extras.scripts import Script
|
||||
|
||||
|
||||
class DeviceConnectionsReport(Script):
|
||||
description = "Validate the minimum physical connections for each device"
|
||||
|
||||
def test_console_connection(self):
|
||||
|
||||
# Check that every console port for every active device has a connection defined.
|
||||
active = DeviceStatusChoices.STATUS_ACTIVE
|
||||
for console_port in ConsolePort.objects.prefetch_related('device').filter(device__status=active):
|
||||
if not console_port.connected_endpoints:
|
||||
self.log_failure(
|
||||
f"No console connection defined for {console_port.name}",
|
||||
console_port.device,
|
||||
)
|
||||
elif not console_port.connection_status:
|
||||
self.log_warning(
|
||||
f"Console connection for {console_port.name} marked as planned",
|
||||
console_port.device,
|
||||
)
|
||||
else:
|
||||
self.log_success("Passed", console_port.device)
|
||||
|
||||
def test_power_connections(self):
|
||||
|
||||
# Check that every active device has at least two connected power supplies.
|
||||
for device in Device.objects.filter(status=DeviceStatusChoices.STATUS_ACTIVE):
|
||||
connected_ports = 0
|
||||
for power_port in PowerPort.objects.filter(device=device):
|
||||
if power_port.connected_endpoints:
|
||||
connected_ports += 1
|
||||
if not power_port.path.is_active:
|
||||
self.log_warning(
|
||||
f"Power connection for {power_port.name} marked as planned",
|
||||
device,
|
||||
)
|
||||
if connected_ports < 2:
|
||||
self.log_failure(
|
||||
f"{connected_ports} connected power supplies found (2 needed)",
|
||||
device,
|
||||
)
|
||||
else:
|
||||
self.log_success("Passed", device)
|
||||
```
|
||||
|
||||
## Change Logging
|
||||
|
||||
|
||||
@ -1,167 +1,63 @@
|
||||
# NetBox Reports
|
||||
|
||||
A NetBox report is a mechanism for validating the integrity of data within NetBox. Running a report allows the user to verify that the objects defined within NetBox meet certain arbitrary conditions. For example, you can write reports to check that:
|
||||
|
||||
* All top-of-rack switches have a console connection
|
||||
* Every router has a loopback interface with an IP address assigned
|
||||
* Each interface description conforms to a standard format
|
||||
* Every site has a minimum set of VLANs defined
|
||||
* All IP addresses have a parent prefix
|
||||
|
||||
...and so on. Reports are completely customizable, so there's practically no limit to what you can test for.
|
||||
|
||||
## Writing Reports
|
||||
|
||||
Reports must be saved as files in the [`REPORTS_ROOT`](../configuration/system.md#reports_root) path (which defaults to `netbox/reports/`). Each file created within this path is considered a separate module. Each module holds one or more reports (Python classes), each of which performs a certain function. The logic of each report is broken into discrete test methods, each of which applies a small portion of the logic comprising the overall test.
|
||||
|
||||
!!! warning
|
||||
The reports path includes a file named `__init__.py`, which registers the path as a Python module. Do not delete this file.
|
||||
Reports are deprecated beginning with NetBox v4.0, and their functionality has been merged with [custom scripts](./custom-scripts.md). While backward compatibility has been maintained, users are advised to convert legacy reports into custom scripts soon, as support for legacy reports will be removed in a future release.
|
||||
|
||||
For example, we can create a module named `devices.py` to hold all of our reports which pertain to devices in NetBox. Within that module, we might define several reports. Each report is defined as a Python class inheriting from `extras.reports.Report`.
|
||||
## Converting Reports to Scripts
|
||||
|
||||
```
|
||||
### Step 1: Update Class Definition
|
||||
|
||||
Change the parent class from `Report` to `Script`:
|
||||
|
||||
```python title="Old code"
|
||||
from extras.reports import Report
|
||||
|
||||
class DeviceConnectionsReport(Report):
|
||||
description = "Validate the minimum physical connections for each device"
|
||||
|
||||
class DeviceIPsReport(Report):
|
||||
description = "Check that every device has a primary IP address assigned"
|
||||
class MyReport(Report):
|
||||
```
|
||||
|
||||
Within each report class, we'll create a number of test methods to execute our report's logic. In DeviceConnectionsReport, for instance, we want to ensure that every live device has a console connection, an out-of-band management connection, and two power connections.
|
||||
```python title="New code"
|
||||
from extras.scripts import Script
|
||||
|
||||
```
|
||||
from dcim.choices import DeviceStatusChoices
|
||||
from dcim.models import ConsolePort, Device, PowerPort
|
||||
from extras.reports import Report
|
||||
|
||||
|
||||
class DeviceConnectionsReport(Report):
|
||||
description = "Validate the minimum physical connections for each device"
|
||||
|
||||
def test_console_connection(self):
|
||||
|
||||
# Check that every console port for every active device has a connection defined.
|
||||
active = DeviceStatusChoices.STATUS_ACTIVE
|
||||
for console_port in ConsolePort.objects.prefetch_related('device').filter(device__status=active):
|
||||
if not console_port.connected_endpoints:
|
||||
self.log_failure(
|
||||
console_port.device,
|
||||
"No console connection defined for {}".format(console_port.name)
|
||||
)
|
||||
elif not console_port.connection_status:
|
||||
self.log_warning(
|
||||
console_port.device,
|
||||
"Console connection for {} marked as planned".format(console_port.name)
|
||||
)
|
||||
else:
|
||||
self.log_success(console_port.device)
|
||||
|
||||
def test_power_connections(self):
|
||||
|
||||
# Check that every active device has at least two connected power supplies.
|
||||
for device in Device.objects.filter(status=DeviceStatusChoices.STATUS_ACTIVE):
|
||||
connected_ports = 0
|
||||
for power_port in PowerPort.objects.filter(device=device):
|
||||
if power_port.connected_endpoints:
|
||||
connected_ports += 1
|
||||
if not power_port.path.is_active:
|
||||
self.log_warning(
|
||||
device,
|
||||
"Power connection for {} marked as planned".format(power_port.name)
|
||||
)
|
||||
if connected_ports < 2:
|
||||
self.log_failure(
|
||||
device,
|
||||
"{} connected power supplies found (2 needed)".format(connected_ports)
|
||||
)
|
||||
else:
|
||||
self.log_success(device)
|
||||
class MyReport(Script):
|
||||
```
|
||||
|
||||
As you can see, reports are completely customizable. Validation logic can be as simple or as complex as needed. Also note that the `description` attribute support markdown syntax. It will be rendered in the report list page.
|
||||
### Step 2: Update Logging Calls
|
||||
|
||||
!!! warning
|
||||
Reports should never alter data: If you find yourself using the `create()`, `save()`, `update()`, or `delete()` methods on objects within reports, stop and re-evaluate what you're trying to accomplish. Note that there are no safeguards against the accidental alteration or destruction of data.
|
||||
Reports and scripts both provide logging methods, however their signatures differ. All script logging methods accept a message as the first parameter, and accept an object as an optional second parameter.
|
||||
|
||||
## Report Attributes
|
||||
Additionally, the Report class' generic `log()` method is **not** available on Script. Users are advised to replace calls of this method with `log_info()`.
|
||||
|
||||
### `description`
|
||||
Use the table below as a reference when updating these methods.
|
||||
|
||||
A human-friendly description of what your report does.
|
||||
| Report (old) | Script (New) |
|
||||
|-------------------------------|-----------------------------|
|
||||
| `log(message)` | `log_info(message)` |
|
||||
| `log_debug(obj, message)`[^1] | `log_debug(message, obj)` |
|
||||
| `log_info(obj, message)` | `log_info(message, obj)` |
|
||||
| `log_success(obj, message)` | `log_success(message, obj)` |
|
||||
| `log_warning(obj, message)` | `log_warning(message, obj)` |
|
||||
| `log_failure(obj, message)` | `log_failure(message, obj)` |
|
||||
|
||||
### `scheduling_enabled`
|
||||
[^1]: `log_debug()` was added to the Report class in v4.0 to avoid confusion with the same method on Script
|
||||
|
||||
By default, a report can be scheduled for execution at a later time. Setting `scheduling_enabled` to False disables this ability: Only immediate execution will be possible. (This also disables the ability to set a recurring execution interval.)
|
||||
|
||||
### `job_timeout`
|
||||
|
||||
Set the maximum allowed runtime for the report. If not set, `RQ_DEFAULT_TIMEOUT` will be used.
|
||||
|
||||
## Logging
|
||||
|
||||
The following methods are available to log results within a report:
|
||||
|
||||
* log(message)
|
||||
* log_success(object, message=None)
|
||||
* log_info(object, message)
|
||||
* log_warning(object, message)
|
||||
* log_failure(object, message)
|
||||
|
||||
The recording of one or more failure messages will automatically flag a report as failed. It is advised to log a success for each object that is evaluated so that the results will reflect how many objects are being reported on. (The inclusion of a log message is optional for successes.) Messages recorded with `log()` will appear in a report's results but are not associated with a particular object or status. Log messages also support using markdown syntax and will be rendered on the report result page.
|
||||
|
||||
To perform additional tasks, such as sending an email or calling a webhook, before or after a report is run, extend the `pre_run()` and/or `post_run()` methods, respectively.
|
||||
|
||||
By default, reports within a module are ordered alphabetically in the reports list page. To return reports in a specific order, you can define the `report_order` variable at the end of your module. The `report_order` variable is a tuple which contains each Report class in the desired order. Any reports that are omitted from this list will be listed last.
|
||||
|
||||
```
|
||||
from extras.reports import Report
|
||||
|
||||
class DeviceConnectionsReport(Report)
|
||||
pass
|
||||
|
||||
class DeviceIPsReport(Report)
|
||||
pass
|
||||
|
||||
report_order = (DeviceIPsReport, DeviceConnectionsReport)
|
||||
```python title="Old code"
|
||||
self.log_failure(
|
||||
console_port.device,
|
||||
f"No console connection defined for {console_port.name}"
|
||||
)
|
||||
```
|
||||
|
||||
Once you have created a report, it will appear in the reports list. Initially, reports will have no results associated with them. To generate results, run the report.
|
||||
|
||||
## Running Reports
|
||||
|
||||
!!! note
|
||||
To run a report, a user must be assigned permissions for `Extras > Report`, `Extras > Report Module`, and `Core > Managed File` objects. They must also be assigned the `extras.run_report` permission. This is achieved by assigning the user (or group) a permission on the Report object and specifying the `run` action in "Permissions" as shown below.
|
||||
|
||||
|
||||
|
||||
### Via the Web UI
|
||||
|
||||
Reports can be run via the web UI by navigating to the report and clicking the "run report" button at top right. Once a report has been run, its associated results will be included in the report view. It is possible to schedule a report to be executed at specified time in the future. A scheduled report can be canceled by deleting the associated job result object.
|
||||
|
||||
### Via the API
|
||||
|
||||
To run a report via the API, simply issue a POST request to its `run` endpoint. Reports are identified by their module and class name.
|
||||
|
||||
```
|
||||
POST /api/extras/reports/<module>.<name>/run/
|
||||
```python title="New code"
|
||||
self.log_failure(
|
||||
f"No console connection defined for {console_port.name}",
|
||||
obj=console_port.device,
|
||||
)
|
||||
```
|
||||
|
||||
Our example report above would be called as:
|
||||
### Other Notes
|
||||
|
||||
```
|
||||
POST /api/extras/reports/devices.DeviceConnectionsReport/run/
|
||||
```
|
||||
Existing reports will be converted to scripts automatically upon upgrading to NetBox v4.0, and previous job history will be retained. However, users are advised to convert legacy reports into custom scripts at the earliest opportunity, as support for legacy reports will be removed in a future release.
|
||||
|
||||
Optionally `schedule_at` can be passed in the form data with a datetime string to schedule a script at the specified date and time.
|
||||
The `pre_run()` and `post_run()` Report methods have been carried over to Script. These are called automatically by Script's `run()` method. (Note that if you opt to override this method, you are responsible for calling `pre_run()` and `post_run()` where applicable.)
|
||||
|
||||
### Via the CLI
|
||||
|
||||
Reports can be run on the CLI by invoking the management command:
|
||||
|
||||
```
|
||||
python3 manage.py runreport <module>
|
||||
```
|
||||
|
||||
where ``<module>`` is the name of the python file in the ``reports`` directory without the ``.py`` extension. One or more report modules may be specified.
|
||||
The `is_valid()` method on Report is no longer needed and has been removed.
|
||||
|
||||
@ -52,6 +52,7 @@ extra_css:
|
||||
markdown_extensions:
|
||||
- admonition
|
||||
- attr_list
|
||||
- footnotes
|
||||
- pymdownx.emoji:
|
||||
emoji_index: !!python/name:material.extensions.emoji.twemoji
|
||||
emoji_generator: !!python/name:material.extensions.emoji.to_svg
|
||||
|
||||
@ -50,8 +50,6 @@ __all__ = (
|
||||
'SavedFilterSerializer',
|
||||
'ScriptDetailSerializer',
|
||||
'ScriptInputSerializer',
|
||||
'ScriptLogMessageSerializer',
|
||||
'ScriptOutputSerializer',
|
||||
'ScriptSerializer',
|
||||
'TagSerializer',
|
||||
'WebhookSerializer',
|
||||
@ -604,22 +602,6 @@ class ScriptInputSerializer(serializers.Serializer):
|
||||
return value
|
||||
|
||||
|
||||
class ScriptLogMessageSerializer(serializers.Serializer):
|
||||
status = serializers.SerializerMethodField(read_only=True)
|
||||
message = serializers.SerializerMethodField(read_only=True)
|
||||
|
||||
def get_status(self, instance):
|
||||
return instance[0]
|
||||
|
||||
def get_message(self, instance):
|
||||
return instance[1]
|
||||
|
||||
|
||||
class ScriptOutputSerializer(serializers.Serializer):
|
||||
log = ScriptLogMessageSerializer(many=True, read_only=True)
|
||||
output = serializers.CharField(read_only=True)
|
||||
|
||||
|
||||
#
|
||||
# Change logging
|
||||
#
|
||||
|
||||
@ -20,7 +20,6 @@ router.register('image-attachments', views.ImageAttachmentViewSet)
|
||||
router.register('journal-entries', views.JournalEntryViewSet)
|
||||
router.register('config-contexts', views.ConfigContextViewSet)
|
||||
router.register('config-templates', views.ConfigTemplateViewSet)
|
||||
router.register('reports', views.ReportViewSet, basename='report')
|
||||
router.register('scripts', views.ScriptViewSet, basename='script')
|
||||
router.register('object-changes', views.ObjectChangeViewSet)
|
||||
router.register('content-types', views.ContentTypeViewSet)
|
||||
|
||||
@ -16,7 +16,6 @@ from core.choices import JobStatusChoices
|
||||
from core.models import Job
|
||||
from extras import filtersets
|
||||
from extras.models import *
|
||||
from extras.reports import get_module_and_report, run_report
|
||||
from extras.scripts import get_module_and_script, run_script
|
||||
from netbox.api.authentication import IsAuthenticatedOrLoginNotRequired
|
||||
from netbox.api.features import SyncedDataMixin
|
||||
@ -211,111 +210,6 @@ class ConfigTemplateViewSet(SyncedDataMixin, ConfigTemplateRenderMixin, NetBoxMo
|
||||
return self.render_configtemplate(request, configtemplate, context)
|
||||
|
||||
|
||||
#
|
||||
# Reports
|
||||
#
|
||||
|
||||
class ReportViewSet(ViewSet):
|
||||
permission_classes = [IsAuthenticatedOrLoginNotRequired]
|
||||
_ignore_model_permissions = True
|
||||
schema = None
|
||||
lookup_value_regex = '[^/]+' # Allow dots
|
||||
|
||||
def _get_report(self, pk):
|
||||
try:
|
||||
module_name, report_name = pk.split('.', maxsplit=1)
|
||||
except ValueError:
|
||||
raise Http404
|
||||
|
||||
module, report = get_module_and_report(module_name, report_name)
|
||||
if report is None:
|
||||
raise Http404
|
||||
|
||||
return module, report
|
||||
|
||||
def list(self, request):
|
||||
"""
|
||||
Compile all reports and their related results (if any). Result data is deferred in the list view.
|
||||
"""
|
||||
results = {
|
||||
job.name: job
|
||||
for job in Job.objects.filter(
|
||||
object_type=ContentType.objects.get(app_label='extras', model='reportmodule'),
|
||||
status__in=JobStatusChoices.TERMINAL_STATE_CHOICES
|
||||
).order_by('name', '-created').distinct('name').defer('data')
|
||||
}
|
||||
|
||||
report_list = []
|
||||
for report_module in ReportModule.objects.restrict(request.user):
|
||||
report_list.extend([report() for report in report_module.reports.values()])
|
||||
|
||||
# Attach Job objects to each report (if any)
|
||||
for report in report_list:
|
||||
report.result = results.get(report.name, None)
|
||||
|
||||
serializer = serializers.ReportSerializer(report_list, many=True, context={
|
||||
'request': request,
|
||||
})
|
||||
|
||||
return Response({'count': len(report_list), 'results': serializer.data})
|
||||
|
||||
def retrieve(self, request, pk):
|
||||
"""
|
||||
Retrieve a single Report identified as "<module>.<report>".
|
||||
"""
|
||||
module, report = self._get_report(pk)
|
||||
|
||||
# Retrieve the Report and Job, if any.
|
||||
object_type = ContentType.objects.get(app_label='extras', model='reportmodule')
|
||||
report.result = Job.objects.filter(
|
||||
object_type=object_type,
|
||||
name=report.name,
|
||||
status__in=JobStatusChoices.TERMINAL_STATE_CHOICES
|
||||
).first()
|
||||
|
||||
serializer = serializers.ReportDetailSerializer(report, context={
|
||||
'request': request
|
||||
})
|
||||
|
||||
return Response(serializer.data)
|
||||
|
||||
@action(detail=True, methods=['post'])
|
||||
def run(self, request, pk):
|
||||
"""
|
||||
Run a Report identified as "<module>.<script>" and return the pending Job as the result
|
||||
"""
|
||||
# Check that the user has permission to run reports.
|
||||
if not request.user.has_perm('extras.run_report'):
|
||||
raise PermissionDenied("This user does not have permission to run reports.")
|
||||
|
||||
# Check that at least one RQ worker is running
|
||||
if not Worker.count(get_connection('default')):
|
||||
raise RQWorkerNotRunningException()
|
||||
|
||||
# Retrieve and run the Report. This will create a new Job.
|
||||
module, report_cls = self._get_report(pk)
|
||||
report = report_cls
|
||||
input_serializer = serializers.ReportInputSerializer(
|
||||
data=request.data,
|
||||
context={'report': report}
|
||||
)
|
||||
|
||||
if input_serializer.is_valid():
|
||||
report.result = Job.enqueue(
|
||||
run_report,
|
||||
instance=module,
|
||||
name=report.class_name,
|
||||
user=request.user,
|
||||
job_timeout=report.job_timeout,
|
||||
schedule_at=input_serializer.validated_data.get('schedule_at'),
|
||||
interval=input_serializer.validated_data.get('interval')
|
||||
)
|
||||
serializer = serializers.ReportDetailSerializer(report, context={'request': request})
|
||||
|
||||
return Response(serializer.data)
|
||||
return Response(input_serializer.errors, status=status.HTTP_400_BAD_REQUEST)
|
||||
|
||||
|
||||
#
|
||||
# Scripts
|
||||
#
|
||||
|
||||
@ -1,3 +1,5 @@
|
||||
import logging
|
||||
|
||||
from django.utils.translation import gettext_lazy as _
|
||||
|
||||
from utilities.choices import ButtonColorChoices, ChoiceSet
|
||||
@ -164,6 +166,7 @@ class JournalEntryKindChoices(ChoiceSet):
|
||||
|
||||
class LogLevelChoices(ChoiceSet):
|
||||
|
||||
LOG_DEBUG = 'debug'
|
||||
LOG_DEFAULT = 'default'
|
||||
LOG_SUCCESS = 'success'
|
||||
LOG_INFO = 'info'
|
||||
@ -171,6 +174,7 @@ class LogLevelChoices(ChoiceSet):
|
||||
LOG_FAILURE = 'failure'
|
||||
|
||||
CHOICES = (
|
||||
(LOG_DEBUG, _('Debug'), 'teal'),
|
||||
(LOG_DEFAULT, _('Default'), 'gray'),
|
||||
(LOG_SUCCESS, _('Success'), 'green'),
|
||||
(LOG_INFO, _('Info'), 'cyan'),
|
||||
@ -178,6 +182,15 @@ class LogLevelChoices(ChoiceSet):
|
||||
(LOG_FAILURE, _('Failure'), 'red'),
|
||||
)
|
||||
|
||||
SYSTEM_LEVELS = {
|
||||
LOG_DEBUG: logging.DEBUG,
|
||||
LOG_DEFAULT: logging.INFO,
|
||||
LOG_SUCCESS: logging.INFO,
|
||||
LOG_INFO: logging.INFO,
|
||||
LOG_WARNING: logging.WARNING,
|
||||
LOG_FAILURE: logging.ERROR,
|
||||
}
|
||||
|
||||
|
||||
class DurationChoices(ChoiceSet):
|
||||
|
||||
|
||||
@ -1,65 +0,0 @@
|
||||
import time
|
||||
|
||||
from django.core.management.base import BaseCommand
|
||||
from django.utils import timezone
|
||||
|
||||
from core.choices import JobStatusChoices
|
||||
from core.models import Job
|
||||
from extras.models import ReportModule
|
||||
from extras.reports import run_report
|
||||
|
||||
|
||||
class Command(BaseCommand):
|
||||
help = "Run a report to validate data in NetBox"
|
||||
|
||||
def add_arguments(self, parser):
|
||||
parser.add_argument('reports', nargs='+', help="Report(s) to run")
|
||||
|
||||
def handle(self, *args, **options):
|
||||
|
||||
for module in ReportModule.objects.all():
|
||||
for report in module.reports.values():
|
||||
if module.name in options['reports'] or report.full_name in options['reports']:
|
||||
|
||||
# Run the report and create a new Job
|
||||
self.stdout.write(
|
||||
"[{:%H:%M:%S}] Running {}...".format(timezone.now(), report.full_name)
|
||||
)
|
||||
|
||||
job = Job.enqueue(
|
||||
run_report,
|
||||
instance=module,
|
||||
name=report.class_name,
|
||||
job_timeout=report.job_timeout
|
||||
)
|
||||
|
||||
# Wait on the job to finish
|
||||
while job.status not in JobStatusChoices.TERMINAL_STATE_CHOICES:
|
||||
time.sleep(1)
|
||||
job = Job.objects.get(pk=job.pk)
|
||||
|
||||
# Report on success/failure
|
||||
if job.status == JobStatusChoices.STATUS_FAILED:
|
||||
status = self.style.ERROR('FAILED')
|
||||
elif job == JobStatusChoices.STATUS_ERRORED:
|
||||
status = self.style.ERROR('ERRORED')
|
||||
else:
|
||||
status = self.style.SUCCESS('SUCCESS')
|
||||
|
||||
for test_name, attrs in job.data.items():
|
||||
self.stdout.write(
|
||||
"\t{}: {} success, {} info, {} warning, {} failure".format(
|
||||
test_name, attrs['success'], attrs['info'], attrs['warning'], attrs['failure']
|
||||
)
|
||||
)
|
||||
self.stdout.write(
|
||||
"[{:%H:%M:%S}] {}: {}".format(timezone.now(), report.full_name, status)
|
||||
)
|
||||
self.stdout.write(
|
||||
"[{:%H:%M:%S}] {}: Duration {}".format(timezone.now(), report.full_name, job.duration)
|
||||
)
|
||||
|
||||
# Wrap things up
|
||||
self.stdout.write(
|
||||
"[{:%H:%M:%S}] Finished".format(timezone.now())
|
||||
)
|
||||
@ -10,7 +10,6 @@ from django.db import transaction
|
||||
|
||||
from core.choices import JobStatusChoices
|
||||
from core.models import Job
|
||||
from extras.api.serializers import ScriptOutputSerializer
|
||||
from extras.context_managers import event_tracking
|
||||
from extras.scripts import get_module_and_script
|
||||
from extras.signals import clear_events
|
||||
@ -34,6 +33,7 @@ class Command(BaseCommand):
|
||||
parser.add_argument('script', help="Script to run")
|
||||
|
||||
def handle(self, *args, **options):
|
||||
|
||||
def _run_script():
|
||||
"""
|
||||
Core script execution task. We capture this within a subfunction to allow for conditionally wrapping it with
|
||||
@ -48,7 +48,7 @@ class Command(BaseCommand):
|
||||
except AbortTransaction:
|
||||
script.log_info("Database changes have been reverted automatically.")
|
||||
clear_events.send(request)
|
||||
job.data = ScriptOutputSerializer(script).data
|
||||
job.data = script.get_job_data()
|
||||
job.terminate()
|
||||
except Exception as e:
|
||||
stacktrace = traceback.format_exc()
|
||||
@ -58,9 +58,17 @@ class Command(BaseCommand):
|
||||
script.log_info("Database changes have been reverted due to error.")
|
||||
logger.error(f"Exception raised during script execution: {e}")
|
||||
clear_events.send(request)
|
||||
job.data = ScriptOutputSerializer(script).data
|
||||
job.data = script.get_job_data()
|
||||
job.terminate(status=JobStatusChoices.STATUS_ERRORED, error=repr(e))
|
||||
|
||||
# Print any test method results
|
||||
for test_name, attrs in job.data['tests'].items():
|
||||
self.stdout.write(
|
||||
"\t{}: {} success, {} info, {} warning, {} failure".format(
|
||||
test_name, attrs['success'], attrs['info'], attrs['warning'], attrs['failure']
|
||||
)
|
||||
)
|
||||
|
||||
logger.info(f"Script completed in {job.duration}")
|
||||
|
||||
User = get_user_model()
|
||||
@ -69,6 +77,7 @@ class Command(BaseCommand):
|
||||
script = options['script']
|
||||
loglevel = options['loglevel']
|
||||
commit = options['commit']
|
||||
|
||||
try:
|
||||
data = json.loads(options['data'])
|
||||
except TypeError:
|
||||
|
||||
31
netbox/extras/migrations/0107_convert_reports_to_scripts.py
Normal file
31
netbox/extras/migrations/0107_convert_reports_to_scripts.py
Normal file
@ -0,0 +1,31 @@
|
||||
from django.db import migrations
|
||||
|
||||
|
||||
def convert_reportmodule_jobs(apps, schema_editor):
|
||||
ContentType = apps.get_model('contenttypes', 'ContentType')
|
||||
Job = apps.get_model('core', 'Job')
|
||||
|
||||
# Convert all ReportModule jobs to ScriptModule jobs
|
||||
if reportmodule_ct := ContentType.objects.filter(app_label='extras', model='reportmodule').first():
|
||||
scriptmodule_ct = ContentType.objects.get(app_label='extras', model='scriptmodule')
|
||||
Job.objects.filter(object_type_id=reportmodule_ct.id).update(object_type_id=scriptmodule_ct.id)
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
|
||||
dependencies = [
|
||||
('extras', '0106_bookmark_user_cascade_deletion'),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.RunPython(
|
||||
code=convert_reportmodule_jobs,
|
||||
reverse_code=migrations.RunPython.noop
|
||||
),
|
||||
migrations.DeleteModel(
|
||||
name='Report',
|
||||
),
|
||||
migrations.DeleteModel(
|
||||
name='ReportModule',
|
||||
),
|
||||
]
|
||||
@ -3,7 +3,6 @@ from .configs import *
|
||||
from .customfields import *
|
||||
from .dashboard import *
|
||||
from .models import *
|
||||
from .reports import *
|
||||
from .scripts import *
|
||||
from .search import *
|
||||
from .staging import *
|
||||
|
||||
@ -1,80 +0,0 @@
|
||||
import inspect
|
||||
import logging
|
||||
from functools import cached_property
|
||||
|
||||
from django.db import models
|
||||
from django.urls import reverse
|
||||
from django.utils.translation import gettext_lazy as _
|
||||
|
||||
from core.choices import ManagedFileRootPathChoices
|
||||
from core.models import ManagedFile
|
||||
from extras.utils import is_report
|
||||
from netbox.models.features import JobsMixin, EventRulesMixin
|
||||
from utilities.querysets import RestrictedQuerySet
|
||||
from .mixins import PythonModuleMixin
|
||||
|
||||
logger = logging.getLogger('netbox.reports')
|
||||
|
||||
__all__ = (
|
||||
'Report',
|
||||
'ReportModule',
|
||||
)
|
||||
|
||||
|
||||
class Report(EventRulesMixin, models.Model):
|
||||
"""
|
||||
Dummy model used to generate permissions for reports. Does not exist in the database.
|
||||
"""
|
||||
class Meta:
|
||||
managed = False
|
||||
|
||||
|
||||
class ReportModuleManager(models.Manager.from_queryset(RestrictedQuerySet)):
|
||||
|
||||
def get_queryset(self):
|
||||
return super().get_queryset().filter(file_root=ManagedFileRootPathChoices.REPORTS)
|
||||
|
||||
|
||||
class ReportModule(PythonModuleMixin, JobsMixin, ManagedFile):
|
||||
"""
|
||||
Proxy model for report module files.
|
||||
"""
|
||||
objects = ReportModuleManager()
|
||||
|
||||
class Meta:
|
||||
proxy = True
|
||||
verbose_name = _('report module')
|
||||
verbose_name_plural = _('report modules')
|
||||
|
||||
def get_absolute_url(self):
|
||||
return reverse('extras:report_list')
|
||||
|
||||
def __str__(self):
|
||||
return self.python_name
|
||||
|
||||
@cached_property
|
||||
def reports(self):
|
||||
|
||||
def _get_name(cls):
|
||||
# For child objects in submodules use the full import path w/o the root module as the name
|
||||
return cls.full_name.split(".", maxsplit=1)[1]
|
||||
|
||||
try:
|
||||
module = self.get_module()
|
||||
except (ImportError, SyntaxError) as e:
|
||||
logger.error(f"Unable to load report module {self.name}, exception: {e}")
|
||||
return {}
|
||||
reports = {}
|
||||
ordered = getattr(module, 'report_order', [])
|
||||
|
||||
for cls in ordered:
|
||||
reports[_get_name(cls)] = cls
|
||||
for name, cls in inspect.getmembers(module, is_report):
|
||||
if cls not in ordered:
|
||||
reports[_get_name(cls)] = cls
|
||||
|
||||
return reports
|
||||
|
||||
def save(self, *args, **kwargs):
|
||||
self.file_root = ManagedFileRootPathChoices.REPORTS
|
||||
return super().save(*args, **kwargs)
|
||||
@ -3,6 +3,7 @@ import logging
|
||||
from functools import cached_property
|
||||
|
||||
from django.db import models
|
||||
from django.db.models import Q
|
||||
from django.urls import reverse
|
||||
from django.utils.translation import gettext_lazy as _
|
||||
|
||||
@ -32,7 +33,8 @@ class Script(EventRulesMixin, models.Model):
|
||||
class ScriptModuleManager(models.Manager.from_queryset(RestrictedQuerySet)):
|
||||
|
||||
def get_queryset(self):
|
||||
return super().get_queryset().filter(file_root=ManagedFileRootPathChoices.SCRIPTS)
|
||||
return super().get_queryset().filter(
|
||||
Q(file_root=ManagedFileRootPathChoices.SCRIPTS) | Q(file_root=ManagedFileRootPathChoices.REPORTS))
|
||||
|
||||
|
||||
class ScriptModule(PythonModuleMixin, JobsMixin, ManagedFile):
|
||||
|
||||
@ -1,248 +1,33 @@
|
||||
import inspect
|
||||
import logging
|
||||
import traceback
|
||||
from datetime import timedelta
|
||||
|
||||
from django.utils import timezone
|
||||
from django.utils.functional import classproperty
|
||||
from django_rq import job
|
||||
|
||||
from core.choices import JobStatusChoices
|
||||
from core.models import Job
|
||||
from .choices import LogLevelChoices
|
||||
from .models import ReportModule
|
||||
from .scripts import BaseScript
|
||||
|
||||
__all__ = (
|
||||
'Report',
|
||||
'get_module_and_report',
|
||||
'run_report',
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def get_module_and_report(module_name, report_name):
|
||||
module = ReportModule.objects.get(file_path=f'{module_name}.py')
|
||||
report = module.reports.get(report_name)()
|
||||
return module, report
|
||||
|
||||
|
||||
@job('default')
|
||||
def run_report(job, *args, **kwargs):
|
||||
"""
|
||||
Helper function to call the run method on a report. This is needed to get around the inability to pickle an instance
|
||||
method for queueing into the background processor.
|
||||
"""
|
||||
job.start()
|
||||
|
||||
module = ReportModule.objects.get(pk=job.object_id)
|
||||
report = module.reports.get(job.name)()
|
||||
|
||||
try:
|
||||
report.run(job)
|
||||
except Exception as e:
|
||||
job.terminate(status=JobStatusChoices.STATUS_ERRORED, error=repr(e))
|
||||
logging.error(f"Error during execution of report {job.name}")
|
||||
finally:
|
||||
# Schedule the next job if an interval has been set
|
||||
if job.interval:
|
||||
new_scheduled_time = job.scheduled + timedelta(minutes=job.interval)
|
||||
Job.enqueue(
|
||||
run_report,
|
||||
instance=job.object,
|
||||
name=job.name,
|
||||
user=job.user,
|
||||
job_timeout=report.job_timeout,
|
||||
schedule_at=new_scheduled_time,
|
||||
interval=job.interval
|
||||
)
|
||||
|
||||
|
||||
class Report(object):
|
||||
"""
|
||||
NetBox users can extend this object to write custom reports to be used for validating data within NetBox. Each
|
||||
report must have one or more test methods named `test_*`.
|
||||
|
||||
The `_results` attribute of a completed report will take the following form:
|
||||
|
||||
{
|
||||
'test_bar': {
|
||||
'failures': 42,
|
||||
'log': [
|
||||
(<datetime>, <level>, <object>, <message>),
|
||||
...
|
||||
]
|
||||
},
|
||||
'test_foo': {
|
||||
'failures': 0,
|
||||
'log': [
|
||||
(<datetime>, <level>, <object>, <message>),
|
||||
...
|
||||
]
|
||||
}
|
||||
}
|
||||
"""
|
||||
description = None
|
||||
scheduling_enabled = True
|
||||
job_timeout = None
|
||||
|
||||
def __init__(self):
|
||||
|
||||
self._results = {}
|
||||
self.active_test = None
|
||||
self.failed = False
|
||||
|
||||
self.logger = logging.getLogger(f"netbox.reports.{self.__module__}.{self.__class__.__name__}")
|
||||
|
||||
# Compile test methods and initialize results skeleton
|
||||
test_methods = []
|
||||
for method in dir(self):
|
||||
if method.startswith('test_') and callable(getattr(self, method)):
|
||||
test_methods.append(method)
|
||||
self._results[method] = {
|
||||
'success': 0,
|
||||
'info': 0,
|
||||
'warning': 0,
|
||||
'failure': 0,
|
||||
'log': [],
|
||||
}
|
||||
self.test_methods = test_methods
|
||||
|
||||
@classproperty
|
||||
def module(self):
|
||||
return self.__module__
|
||||
|
||||
@classproperty
|
||||
def class_name(self):
|
||||
return self.__name__
|
||||
|
||||
@classproperty
|
||||
def full_name(self):
|
||||
return f'{self.module}.{self.class_name}'
|
||||
|
||||
@property
|
||||
def name(self):
|
||||
"""
|
||||
Override this attribute to set a custom display name.
|
||||
"""
|
||||
return self.class_name
|
||||
|
||||
@property
|
||||
def filename(self):
|
||||
return inspect.getfile(self.__class__)
|
||||
|
||||
@property
|
||||
def source(self):
|
||||
return inspect.getsource(self.__class__)
|
||||
|
||||
@property
|
||||
def is_valid(self):
|
||||
"""
|
||||
Indicates whether the report can be run.
|
||||
"""
|
||||
return bool(self.test_methods)
|
||||
class Report(BaseScript):
|
||||
|
||||
#
|
||||
# Logging methods
|
||||
# Legacy logging methods for Reports
|
||||
#
|
||||
|
||||
def _log(self, obj, message, level=LogLevelChoices.LOG_DEFAULT):
|
||||
"""
|
||||
Log a message from a test method. Do not call this method directly; use one of the log_* wrappers below.
|
||||
"""
|
||||
if level not in LogLevelChoices.values():
|
||||
raise Exception(f"Unknown logging level: {level}")
|
||||
self._results[self.active_test]['log'].append((
|
||||
timezone.now().isoformat(),
|
||||
level,
|
||||
str(obj) if obj else None,
|
||||
obj.get_absolute_url() if hasattr(obj, 'get_absolute_url') else None,
|
||||
message,
|
||||
))
|
||||
|
||||
# There is no generic log() equivalent on BaseScript
|
||||
def log(self, message):
|
||||
"""
|
||||
Log a message which is not associated with a particular object.
|
||||
"""
|
||||
self._log(None, message, level=LogLevelChoices.LOG_DEFAULT)
|
||||
self.logger.info(message)
|
||||
self._log(message, None, level=LogLevelChoices.LOG_DEFAULT)
|
||||
|
||||
def log_success(self, obj, message=None):
|
||||
"""
|
||||
Record a successful test against an object. Logging a message is optional.
|
||||
"""
|
||||
if message:
|
||||
self._log(obj, message, level=LogLevelChoices.LOG_SUCCESS)
|
||||
self._results[self.active_test]['success'] += 1
|
||||
self.logger.info(f"Success | {obj}: {message}")
|
||||
def log_success(self, obj=None, message=None):
|
||||
super().log_success(message, obj)
|
||||
|
||||
def log_info(self, obj, message):
|
||||
"""
|
||||
Log an informational message.
|
||||
"""
|
||||
self._log(obj, message, level=LogLevelChoices.LOG_INFO)
|
||||
self._results[self.active_test]['info'] += 1
|
||||
self.logger.info(f"Info | {obj}: {message}")
|
||||
def log_info(self, obj=None, message=None):
|
||||
super().log_info(message, obj)
|
||||
|
||||
def log_warning(self, obj, message):
|
||||
"""
|
||||
Log a warning.
|
||||
"""
|
||||
self._log(obj, message, level=LogLevelChoices.LOG_WARNING)
|
||||
self._results[self.active_test]['warning'] += 1
|
||||
self.logger.info(f"Warning | {obj}: {message}")
|
||||
def log_warning(self, obj=None, message=None):
|
||||
super().log_warning(message, obj)
|
||||
|
||||
def log_failure(self, obj, message):
|
||||
"""
|
||||
Log a failure. Calling this method will automatically mark the report as failed.
|
||||
"""
|
||||
self._log(obj, message, level=LogLevelChoices.LOG_FAILURE)
|
||||
self._results[self.active_test]['failure'] += 1
|
||||
self.logger.info(f"Failure | {obj}: {message}")
|
||||
self.failed = True
|
||||
def log_failure(self, obj=None, message=None):
|
||||
super().log_failure(message, obj)
|
||||
|
||||
#
|
||||
# Run methods
|
||||
#
|
||||
|
||||
def run(self, job):
|
||||
"""
|
||||
Run the report and save its results. Each test method will be executed in order.
|
||||
"""
|
||||
self.logger.info(f"Running report")
|
||||
|
||||
# Perform any post-run tasks
|
||||
self.pre_run()
|
||||
|
||||
try:
|
||||
for method_name in self.test_methods:
|
||||
self.active_test = method_name
|
||||
test_method = getattr(self, method_name)
|
||||
test_method()
|
||||
job.data = self._results
|
||||
if self.failed:
|
||||
self.logger.warning("Report failed")
|
||||
job.terminate(status=JobStatusChoices.STATUS_FAILED)
|
||||
else:
|
||||
self.logger.info("Report completed successfully")
|
||||
job.terminate()
|
||||
except Exception as e:
|
||||
stacktrace = traceback.format_exc()
|
||||
self.log_failure(None, f"An exception occurred: {type(e).__name__}: {e} <pre>{stacktrace}</pre>")
|
||||
logger.error(f"Exception raised during report execution: {e}")
|
||||
job.terminate(status=JobStatusChoices.STATUS_ERRORED, error=repr(e))
|
||||
|
||||
# Perform any post-run tasks
|
||||
self.post_run()
|
||||
|
||||
def pre_run(self):
|
||||
"""
|
||||
Extend this method to include any tasks which should execute *before* the report is run.
|
||||
"""
|
||||
pass
|
||||
|
||||
def post_run(self):
|
||||
"""
|
||||
Extend this method to include any tasks which should execute *after* the report is run.
|
||||
"""
|
||||
pass
|
||||
# Added in v4.0 to avoid confusion with the log_debug() method provided by BaseScript
|
||||
def log_debug(self, obj=None, message=None):
|
||||
super().log_debug(message, obj)
|
||||
|
||||
@ -10,11 +10,12 @@ from django import forms
|
||||
from django.conf import settings
|
||||
from django.core.validators import RegexValidator
|
||||
from django.db import transaction
|
||||
from django.utils import timezone
|
||||
from django.utils.functional import classproperty
|
||||
from django.utils.translation import gettext as _
|
||||
|
||||
from core.choices import JobStatusChoices
|
||||
from core.models import Job
|
||||
from extras.api.serializers import ScriptOutputSerializer
|
||||
from extras.choices import LogLevelChoices
|
||||
from extras.models import ScriptModule
|
||||
from extras.signals import clear_events
|
||||
@ -25,6 +26,8 @@ from utilities.forms import add_blank_choice
|
||||
from utilities.forms.fields import DynamicModelChoiceField, DynamicModelMultipleChoiceField
|
||||
from .context_managers import event_tracking
|
||||
from .forms import ScriptForm
|
||||
from .utils import is_report
|
||||
|
||||
|
||||
__all__ = (
|
||||
'BaseScript',
|
||||
@ -270,17 +273,28 @@ class BaseScript:
|
||||
pass
|
||||
|
||||
def __init__(self):
|
||||
self.messages = [] # Primary script log
|
||||
self.tests = {} # Mapping of logs for test methods
|
||||
self.output = ''
|
||||
self.failed = False
|
||||
self._current_test = None # Tracks the current test method being run (if any)
|
||||
|
||||
# Initiate the log
|
||||
self.logger = logging.getLogger(f"netbox.scripts.{self.__module__}.{self.__class__.__name__}")
|
||||
self.log = []
|
||||
|
||||
# Declare the placeholder for the current request
|
||||
self.request = None
|
||||
|
||||
# Grab some info about the script
|
||||
self.filename = inspect.getfile(self.__class__)
|
||||
self.source = inspect.getsource(self.__class__)
|
||||
# Compile test methods and initialize results skeleton
|
||||
for method in dir(self):
|
||||
if method.startswith('test_') and callable(getattr(self, method)):
|
||||
self.tests[method] = {
|
||||
LogLevelChoices.LOG_SUCCESS: 0,
|
||||
LogLevelChoices.LOG_INFO: 0,
|
||||
LogLevelChoices.LOG_WARNING: 0,
|
||||
LogLevelChoices.LOG_FAILURE: 0,
|
||||
'log': [],
|
||||
}
|
||||
|
||||
def __str__(self):
|
||||
return self.name
|
||||
@ -331,6 +345,14 @@ class BaseScript:
|
||||
def scheduling_enabled(self):
|
||||
return getattr(self.Meta, 'scheduling_enabled', True)
|
||||
|
||||
@property
|
||||
def filename(self):
|
||||
return inspect.getfile(self.__class__)
|
||||
|
||||
@property
|
||||
def source(self):
|
||||
return inspect.getsource(self.__class__)
|
||||
|
||||
@classmethod
|
||||
def _get_vars(cls):
|
||||
vars = {}
|
||||
@ -356,9 +378,28 @@ class BaseScript:
|
||||
return ordered_vars
|
||||
|
||||
def run(self, data, commit):
|
||||
raise NotImplementedError("The script must define a run() method.")
|
||||
"""
|
||||
Override this method with custom script logic.
|
||||
"""
|
||||
|
||||
# Backward compatibility for legacy Reports
|
||||
self.pre_run()
|
||||
self.run_tests()
|
||||
self.post_run()
|
||||
|
||||
def get_job_data(self):
|
||||
"""
|
||||
Return a dictionary of data to attach to the script's Job.
|
||||
"""
|
||||
return {
|
||||
'log': self.messages,
|
||||
'output': self.output,
|
||||
'tests': self.tests,
|
||||
}
|
||||
|
||||
#
|
||||
# Form rendering
|
||||
#
|
||||
|
||||
def get_fieldsets(self):
|
||||
fieldsets = []
|
||||
@ -397,29 +438,66 @@ class BaseScript:
|
||||
|
||||
return form
|
||||
|
||||
#
|
||||
# Logging
|
||||
#
|
||||
|
||||
def log_debug(self, message):
|
||||
self.logger.log(logging.DEBUG, message)
|
||||
self.log.append((LogLevelChoices.LOG_DEFAULT, str(message)))
|
||||
def _log(self, message, obj=None, level=LogLevelChoices.LOG_DEFAULT):
|
||||
"""
|
||||
Log a message. Do not call this method directly; use one of the log_* wrappers below.
|
||||
"""
|
||||
if level not in LogLevelChoices.values():
|
||||
raise ValueError(f"Invalid logging level: {level}")
|
||||
|
||||
def log_success(self, message):
|
||||
self.logger.log(logging.INFO, message) # No syslog equivalent for SUCCESS
|
||||
self.log.append((LogLevelChoices.LOG_SUCCESS, str(message)))
|
||||
# A test method is currently active, so log the message using legacy Report logging
|
||||
if self._current_test:
|
||||
|
||||
def log_info(self, message):
|
||||
self.logger.log(logging.INFO, message)
|
||||
self.log.append((LogLevelChoices.LOG_INFO, str(message)))
|
||||
# TODO: Use a dataclass for test method logs
|
||||
self.tests[self._current_test]['log'].append((
|
||||
timezone.now().isoformat(),
|
||||
level,
|
||||
str(obj) if obj else None,
|
||||
obj.get_absolute_url() if hasattr(obj, 'get_absolute_url') else None,
|
||||
str(message),
|
||||
))
|
||||
|
||||
def log_warning(self, message):
|
||||
self.logger.log(logging.WARNING, message)
|
||||
self.log.append((LogLevelChoices.LOG_WARNING, str(message)))
|
||||
# Increment the event counter for this level
|
||||
if level in self.tests[self._current_test]:
|
||||
self.tests[self._current_test][level] += 1
|
||||
|
||||
def log_failure(self, message):
|
||||
self.logger.log(logging.ERROR, message)
|
||||
self.log.append((LogLevelChoices.LOG_FAILURE, str(message)))
|
||||
elif message:
|
||||
|
||||
# Record to the script's log
|
||||
self.messages.append({
|
||||
'time': timezone.now().isoformat(),
|
||||
'status': level,
|
||||
'message': str(message),
|
||||
})
|
||||
|
||||
# Record to the system log
|
||||
if obj:
|
||||
message = f"{obj}: {message}"
|
||||
self.logger.log(LogLevelChoices.SYSTEM_LEVELS[level], message)
|
||||
|
||||
def log_debug(self, message, obj=None):
|
||||
self._log(message, obj, level=LogLevelChoices.LOG_DEBUG)
|
||||
|
||||
def log_success(self, message, obj=None):
|
||||
self._log(message, obj, level=LogLevelChoices.LOG_SUCCESS)
|
||||
|
||||
def log_info(self, message, obj=None):
|
||||
self._log(message, obj, level=LogLevelChoices.LOG_INFO)
|
||||
|
||||
def log_warning(self, message, obj=None):
|
||||
self._log(message, obj, level=LogLevelChoices.LOG_WARNING)
|
||||
|
||||
def log_failure(self, message, obj=None):
|
||||
self._log(message, obj, level=LogLevelChoices.LOG_FAILURE)
|
||||
self.failed = True
|
||||
|
||||
#
|
||||
# Convenience functions
|
||||
#
|
||||
|
||||
def load_yaml(self, filename):
|
||||
"""
|
||||
@ -446,6 +524,39 @@ class BaseScript:
|
||||
|
||||
return data
|
||||
|
||||
#
|
||||
# Legacy Report functionality
|
||||
#
|
||||
|
||||
def run_tests(self):
|
||||
"""
|
||||
Run the report and save its results. Each test method will be executed in order.
|
||||
"""
|
||||
self.logger.info(f"Running report")
|
||||
|
||||
try:
|
||||
for test_name in self.tests:
|
||||
self._current_test = test_name
|
||||
test_method = getattr(self, test_name)
|
||||
test_method()
|
||||
self._current_test = None
|
||||
except Exception as e:
|
||||
self._current_test = None
|
||||
self.post_run()
|
||||
raise e
|
||||
|
||||
def pre_run(self):
|
||||
"""
|
||||
Legacy method for operations performed immediately prior to running a Report.
|
||||
"""
|
||||
pass
|
||||
|
||||
def post_run(self):
|
||||
"""
|
||||
Legacy method for operations performed immediately after running a Report.
|
||||
"""
|
||||
pass
|
||||
|
||||
|
||||
class Script(BaseScript):
|
||||
"""
|
||||
@ -500,7 +611,16 @@ def run_script(data, job, request=None, commit=True, **kwargs):
|
||||
# Add the current request as a property of the script
|
||||
script.request = request
|
||||
|
||||
def _run_script():
|
||||
def set_job_data(script):
|
||||
job.data = {
|
||||
'log': script.messages,
|
||||
'output': script.output,
|
||||
'tests': script.tests,
|
||||
}
|
||||
|
||||
return job
|
||||
|
||||
def _run_script(job):
|
||||
"""
|
||||
Core script execution task. We capture this within a subfunction to allow for conditionally wrapping it with
|
||||
the event_tracking context manager (which is bypassed if commit == False).
|
||||
@ -508,25 +628,39 @@ def run_script(data, job, request=None, commit=True, **kwargs):
|
||||
try:
|
||||
try:
|
||||
with transaction.atomic():
|
||||
script.output = script.run(data=data, commit=commit)
|
||||
script.output = script.run(data, commit)
|
||||
if not commit:
|
||||
raise AbortTransaction()
|
||||
except AbortTransaction:
|
||||
script.log_info("Database changes have been reverted automatically.")
|
||||
script.log_info(message=_("Database changes have been reverted automatically."))
|
||||
if request:
|
||||
clear_events.send(request)
|
||||
job.data = ScriptOutputSerializer(script).data
|
||||
job.terminate()
|
||||
|
||||
job.data = script.get_job_data()
|
||||
if script.failed:
|
||||
logger.warning(f"Script failed")
|
||||
job.terminate(status=JobStatusChoices.STATUS_FAILED)
|
||||
else:
|
||||
job.terminate()
|
||||
|
||||
except Exception as e:
|
||||
if type(e) is AbortScript:
|
||||
script.log_failure(f"Script aborted with error: {e}")
|
||||
msg = _("Script aborted with error: ") + str(e)
|
||||
if is_report(type(script)):
|
||||
script.log_failure(message=msg)
|
||||
else:
|
||||
script.log_failure(msg)
|
||||
|
||||
logger.error(f"Script aborted with error: {e}")
|
||||
else:
|
||||
stacktrace = traceback.format_exc()
|
||||
script.log_failure(f"An exception occurred: `{type(e).__name__}: {e}`\n```\n{stacktrace}\n```")
|
||||
script.log_failure(
|
||||
message=_("An exception occurred: ") + f"`{type(e).__name__}: {e}`\n```\n{stacktrace}\n```"
|
||||
)
|
||||
logger.error(f"Exception raised during script execution: {e}")
|
||||
script.log_info("Database changes have been reverted due to error.")
|
||||
job.data = ScriptOutputSerializer(script).data
|
||||
script.log_info(message=_("Database changes have been reverted due to error."))
|
||||
|
||||
job.data = script.get_job_data()
|
||||
job.terminate(status=JobStatusChoices.STATUS_ERRORED, error=repr(e))
|
||||
if request:
|
||||
clear_events.send(request)
|
||||
@ -537,9 +671,9 @@ def run_script(data, job, request=None, commit=True, **kwargs):
|
||||
# change logging, event rules, etc.
|
||||
if commit:
|
||||
with event_tracking(request):
|
||||
_run_script()
|
||||
_run_script(job)
|
||||
else:
|
||||
_run_script()
|
||||
_run_script(job)
|
||||
|
||||
# Schedule the next job if an interval has been set
|
||||
if job.interval:
|
||||
|
||||
@ -746,37 +746,6 @@ class ConfigTemplateTest(APIViewTestCases.APIViewTestCase):
|
||||
ConfigTemplate.objects.bulk_create(config_templates)
|
||||
|
||||
|
||||
class ReportTest(APITestCase):
|
||||
|
||||
class TestReport(Report):
|
||||
|
||||
def test_foo(self):
|
||||
self.log_success(None, "Report completed")
|
||||
|
||||
@classmethod
|
||||
def setUpTestData(cls):
|
||||
ReportModule.objects.create(
|
||||
file_root=ManagedFileRootPathChoices.REPORTS,
|
||||
file_path='/var/tmp/report.py'
|
||||
)
|
||||
|
||||
def get_test_report(self, *args):
|
||||
return ReportModule.objects.first(), self.TestReport()
|
||||
|
||||
def setUp(self):
|
||||
super().setUp()
|
||||
|
||||
# Monkey-patch the API viewset's _get_report() method to return our test Report above
|
||||
from extras.api.views import ReportViewSet
|
||||
ReportViewSet._get_report = self.get_test_report
|
||||
|
||||
def test_get_report(self):
|
||||
url = reverse('extras-api:report-detail', kwargs={'pk': None})
|
||||
response = self.client.get(url, **self.header)
|
||||
|
||||
self.assertEqual(response.data['name'], self.TestReport.__name__)
|
||||
|
||||
|
||||
class ScriptTest(APITestCase):
|
||||
|
||||
class TestScript(Script):
|
||||
|
||||
@ -116,15 +116,6 @@ urlpatterns = [
|
||||
path('dashboard/widgets/<uuid:id>/configure/', views.DashboardWidgetConfigView.as_view(), name='dashboardwidget_config'),
|
||||
path('dashboard/widgets/<uuid:id>/delete/', views.DashboardWidgetDeleteView.as_view(), name='dashboardwidget_delete'),
|
||||
|
||||
# Reports
|
||||
path('reports/', views.ReportListView.as_view(), name='report_list'),
|
||||
path('reports/add/', views.ReportModuleCreateView.as_view(), name='reportmodule_add'),
|
||||
path('reports/results/<int:job_pk>/', views.ReportResultView.as_view(), name='report_result'),
|
||||
path('reports/<int:pk>/', include(get_model_urls('extras', 'reportmodule'))),
|
||||
path('reports/<str:module>/<str:name>/', views.ReportView.as_view(), name='report'),
|
||||
path('reports/<str:module>/<str:name>/source/', views.ReportSourceView.as_view(), name='report_source'),
|
||||
path('reports/<str:module>/<str:name>/jobs/', views.ReportJobsView.as_view(), name='report_jobs'),
|
||||
|
||||
# Scripts
|
||||
path('scripts/', views.ScriptListView.as_view(), name='script_list'),
|
||||
path('scripts/add/', views.ScriptModuleCreateView.as_view(), name='scriptmodule_add'),
|
||||
|
||||
@ -49,11 +49,12 @@ def register_features(model, features):
|
||||
|
||||
def is_script(obj):
|
||||
"""
|
||||
Returns True if the object is a Script.
|
||||
Returns True if the object is a Script or Report.
|
||||
"""
|
||||
from .reports import Report
|
||||
from .scripts import Script
|
||||
try:
|
||||
return issubclass(obj, Script) and obj != Script
|
||||
return (issubclass(obj, Report) and obj != Report) or (issubclass(obj, Script) and obj != Script)
|
||||
except TypeError:
|
||||
return False
|
||||
|
||||
|
||||
@ -9,7 +9,7 @@ from django.urls import reverse
|
||||
from django.utils.translation import gettext as _
|
||||
from django.views.generic import View
|
||||
|
||||
from core.choices import JobStatusChoices, ManagedFileRootPathChoices
|
||||
from core.choices import ManagedFileRootPathChoices
|
||||
from core.forms import ManagedFileForm
|
||||
from core.models import Job
|
||||
from core.tables import JobTable
|
||||
@ -24,9 +24,7 @@ from utilities.templatetags.builtins.filters import render_markdown
|
||||
from utilities.utils import copy_safe_request, count_related, get_viewname, normalize_querydict, shallow_compare_dict
|
||||
from utilities.views import ContentTypePermissionRequiredMixin, register_model_view
|
||||
from . import filtersets, forms, tables
|
||||
from .forms.reports import ReportForm
|
||||
from .models import *
|
||||
from .reports import run_report
|
||||
from .scripts import run_script
|
||||
|
||||
|
||||
@ -1006,183 +1004,6 @@ class DashboardWidgetDeleteView(LoginRequiredMixin, View):
|
||||
return redirect(reverse('home'))
|
||||
|
||||
|
||||
#
|
||||
# Reports
|
||||
#
|
||||
|
||||
@register_model_view(ReportModule, 'edit')
|
||||
class ReportModuleCreateView(generic.ObjectEditView):
|
||||
queryset = ReportModule.objects.all()
|
||||
form = ManagedFileForm
|
||||
|
||||
def alter_object(self, obj, *args, **kwargs):
|
||||
obj.file_root = ManagedFileRootPathChoices.REPORTS
|
||||
return obj
|
||||
|
||||
|
||||
@register_model_view(ReportModule, 'delete')
|
||||
class ReportModuleDeleteView(generic.ObjectDeleteView):
|
||||
queryset = ReportModule.objects.all()
|
||||
default_return_url = 'extras:report_list'
|
||||
|
||||
|
||||
class ReportListView(ContentTypePermissionRequiredMixin, View):
|
||||
"""
|
||||
Retrieve all the available reports from disk and the recorded Job (if any) for each.
|
||||
"""
|
||||
def get_required_permission(self):
|
||||
return 'extras.view_report'
|
||||
|
||||
def get(self, request):
|
||||
report_modules = ReportModule.objects.restrict(request.user)
|
||||
|
||||
return render(request, 'extras/report_list.html', {
|
||||
'model': ReportModule,
|
||||
'report_modules': report_modules,
|
||||
})
|
||||
|
||||
|
||||
def get_report_module(module, request):
|
||||
return get_object_or_404(ReportModule.objects.restrict(request.user), file_path__regex=f"^{module}\\.")
|
||||
|
||||
|
||||
class ReportView(ContentTypePermissionRequiredMixin, View):
|
||||
"""
|
||||
Display a single Report and its associated Job (if any).
|
||||
"""
|
||||
def get_required_permission(self):
|
||||
return 'extras.view_report'
|
||||
|
||||
def get(self, request, module, name):
|
||||
module = get_report_module(module, request)
|
||||
report = module.reports[name]()
|
||||
jobs = module.get_jobs(report.class_name)
|
||||
|
||||
report.result = jobs.filter(
|
||||
status__in=JobStatusChoices.TERMINAL_STATE_CHOICES
|
||||
).first()
|
||||
|
||||
return render(request, 'extras/report.html', {
|
||||
'job_count': jobs.count(),
|
||||
'module': module,
|
||||
'report': report,
|
||||
'form': ReportForm(scheduling_enabled=report.scheduling_enabled),
|
||||
})
|
||||
|
||||
def post(self, request, module, name):
|
||||
if not request.user.has_perm('extras.run_report'):
|
||||
return HttpResponseForbidden()
|
||||
|
||||
module = get_report_module(module, request)
|
||||
report = module.reports[name]()
|
||||
jobs = module.get_jobs(report.class_name)
|
||||
form = ReportForm(request.POST, scheduling_enabled=report.scheduling_enabled)
|
||||
|
||||
if form.is_valid():
|
||||
|
||||
# Allow execution only if RQ worker process is running
|
||||
if not get_workers_for_queue('default'):
|
||||
messages.error(request, "Unable to run report: RQ worker process not running.")
|
||||
return render(request, 'extras/report.html', {
|
||||
'job_count': jobs.count(),
|
||||
'report': report,
|
||||
})
|
||||
|
||||
# Run the Report. A new Job is created.
|
||||
job = Job.enqueue(
|
||||
run_report,
|
||||
instance=module,
|
||||
name=report.class_name,
|
||||
user=request.user,
|
||||
schedule_at=form.cleaned_data.get('schedule_at'),
|
||||
interval=form.cleaned_data.get('interval'),
|
||||
job_timeout=report.job_timeout
|
||||
)
|
||||
|
||||
return redirect('extras:report_result', job_pk=job.pk)
|
||||
|
||||
return render(request, 'extras/report.html', {
|
||||
'job_count': jobs.count(),
|
||||
'module': module,
|
||||
'report': report,
|
||||
'form': form,
|
||||
})
|
||||
|
||||
|
||||
class ReportSourceView(ContentTypePermissionRequiredMixin, View):
|
||||
|
||||
def get_required_permission(self):
|
||||
return 'extras.view_report'
|
||||
|
||||
def get(self, request, module, name):
|
||||
module = get_report_module(module, request)
|
||||
report = module.reports[name]()
|
||||
jobs = module.get_jobs(report.class_name)
|
||||
|
||||
return render(request, 'extras/report/source.html', {
|
||||
'job_count': jobs.count(),
|
||||
'module': module,
|
||||
'report': report,
|
||||
'tab': 'source',
|
||||
})
|
||||
|
||||
|
||||
class ReportJobsView(ContentTypePermissionRequiredMixin, View):
|
||||
|
||||
def get_required_permission(self):
|
||||
return 'extras.view_report'
|
||||
|
||||
def get(self, request, module, name):
|
||||
module = get_report_module(module, request)
|
||||
report = module.reports[name]()
|
||||
jobs = module.get_jobs(report.class_name)
|
||||
|
||||
jobs_table = JobTable(
|
||||
data=jobs,
|
||||
orderable=False,
|
||||
user=request.user
|
||||
)
|
||||
jobs_table.configure(request)
|
||||
|
||||
return render(request, 'extras/report/jobs.html', {
|
||||
'job_count': jobs.count(),
|
||||
'module': module,
|
||||
'report': report,
|
||||
'table': jobs_table,
|
||||
'tab': 'jobs',
|
||||
})
|
||||
|
||||
|
||||
class ReportResultView(ContentTypePermissionRequiredMixin, View):
|
||||
"""
|
||||
Display a Job pertaining to the execution of a Report.
|
||||
"""
|
||||
def get_required_permission(self):
|
||||
return 'extras.view_report'
|
||||
|
||||
def get(self, request, job_pk):
|
||||
object_type = ContentType.objects.get_by_natural_key(app_label='extras', model='reportmodule')
|
||||
job = get_object_or_404(Job.objects.all(), pk=job_pk, object_type=object_type)
|
||||
|
||||
module = job.object
|
||||
report = module.reports[job.name]
|
||||
|
||||
# If this is an HTMX request, return only the result HTML
|
||||
if request.htmx:
|
||||
response = render(request, 'extras/htmx/report_result.html', {
|
||||
'report': report,
|
||||
'job': job,
|
||||
})
|
||||
if job.completed or not job.started:
|
||||
response.status_code = 286
|
||||
return response
|
||||
|
||||
return render(request, 'extras/report_result.html', {
|
||||
'report': report,
|
||||
'job': job,
|
||||
})
|
||||
|
||||
|
||||
#
|
||||
# Scripts
|
||||
#
|
||||
@ -1332,20 +1153,28 @@ class ScriptResultView(ContentTypePermissionRequiredMixin, View):
|
||||
module = job.object
|
||||
script = module.scripts[job.name]()
|
||||
|
||||
context = {
|
||||
'script': script,
|
||||
'job': job,
|
||||
}
|
||||
if job.data and 'log' in job.data:
|
||||
# Script
|
||||
context['tests'] = job.data.get('tests', {})
|
||||
elif job.data:
|
||||
# Legacy Report
|
||||
context['tests'] = {
|
||||
name: data for name, data in job.data.items()
|
||||
if name.startswith('test_')
|
||||
}
|
||||
|
||||
# If this is an HTMX request, return only the result HTML
|
||||
if request.htmx:
|
||||
response = render(request, 'extras/htmx/script_result.html', {
|
||||
'script': script,
|
||||
'job': job,
|
||||
})
|
||||
response = render(request, 'extras/htmx/script_result.html', context)
|
||||
if job.completed or not job.started:
|
||||
response.status_code = 286
|
||||
return response
|
||||
|
||||
return render(request, 'extras/script_result.html', {
|
||||
'script': script,
|
||||
'job': job,
|
||||
})
|
||||
return render(request, 'extras/script_result.html', context)
|
||||
|
||||
|
||||
#
|
||||
|
||||
@ -317,14 +317,8 @@ CUSTOMIZATION_MENU = Menu(
|
||||
),
|
||||
),
|
||||
MenuGroup(
|
||||
label=_('Reports & Scripts'),
|
||||
label=_('Scripts'),
|
||||
items=(
|
||||
MenuItem(
|
||||
link='extras:report_list',
|
||||
link_text=_('Reports'),
|
||||
permissions=['extras.view_report'],
|
||||
buttons=get_model_buttons('extras', "reportmodule", actions=['add'])
|
||||
),
|
||||
MenuItem(
|
||||
link='extras:script_list',
|
||||
link_text=_('Scripts'),
|
||||
|
||||
@ -1,77 +0,0 @@
|
||||
{% load humanize %}
|
||||
{% load helpers %}
|
||||
{% load i18n %}
|
||||
|
||||
<p>
|
||||
{% if job.started %}
|
||||
{% trans "Started" %}: <strong>{{ job.started|annotated_date }}</strong>
|
||||
{% elif job.scheduled %}
|
||||
{% trans "Scheduled for" %}: <strong>{{ job.scheduled|annotated_date }}</strong> ({{ job.scheduled|naturaltime }})
|
||||
{% else %}
|
||||
{% trans "Created" %}: <strong>{{ job.created|annotated_date }}</strong>
|
||||
{% endif %}
|
||||
{% if job.completed %}
|
||||
{% trans "Duration" %}: <strong>{{ job.duration }}</strong>
|
||||
{% endif %}
|
||||
<span id="pending-result-label">{% badge job.get_status_display job.get_status_color %}</span>
|
||||
</p>
|
||||
{% if job.completed %}
|
||||
<div class="card">
|
||||
<h5 class="card-header">{% trans "Report Methods" %}</h5>
|
||||
<table class="table table-hover">
|
||||
{% for method, data in job.data.items %}
|
||||
<tr>
|
||||
<td class="font-monospace"><a href="#{{ method }}">{{ method }}</a></td>
|
||||
<td class="text-end report-stats">
|
||||
<span class="badge text-bg-success">{{ data.success }}</span>
|
||||
<span class="badge text-bg-info">{{ data.info }}</span>
|
||||
<span class="badge text-bg-warning">{{ data.warning }}</span>
|
||||
<span class="badge text-bg-danger">{{ data.failure }}</span>
|
||||
</td>
|
||||
</tr>
|
||||
{% endfor %}
|
||||
</table>
|
||||
</div>
|
||||
<div class="card">
|
||||
<h5 class="card-header">{% trans "Report Results" %}</h5>
|
||||
<table class="table table-hover report">
|
||||
<thead>
|
||||
<tr>
|
||||
<th>{% trans "Time" %}</th>
|
||||
<th>{% trans "Level" %}</th>
|
||||
<th>{% trans "Object" %}</th>
|
||||
<th>{% trans "Message" %}</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{% for method, data in job.data.items %}
|
||||
<tr>
|
||||
<th colspan="4" style="font-family: monospace">
|
||||
<a name="{{ method }}"></a>{{ method }}
|
||||
</th>
|
||||
</tr>
|
||||
{% for time, level, obj, url, message in data.log %}
|
||||
<tr class="{% if level == 'failure' %}danger{% elif level %}{{ level }}{% endif %}">
|
||||
<td>{{ time }}</td>
|
||||
<td>
|
||||
<label class="badge text-bg-{% if level == 'failure' %}danger{% else %}{{ level }}{% endif %}">{{ level|title }}</label>
|
||||
</td>
|
||||
<td>
|
||||
{% if obj and url %}
|
||||
<a href="{{ url }}">{{ obj }}</a>
|
||||
{% elif obj %}
|
||||
{{ obj }}
|
||||
{% else %}
|
||||
{{ ''|placeholder }}
|
||||
{% endif %}
|
||||
</td>
|
||||
<td class="rendered-markdown">{{ message|markdown }}</td>
|
||||
</tr>
|
||||
{% endfor %}
|
||||
{% endfor %}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
{% elif job.started %}
|
||||
{% include 'extras/inc/result_pending.html' %}
|
||||
{% endif %}
|
||||
@ -17,39 +17,109 @@
|
||||
<span id="pending-result-label">{% badge job.get_status_display job.get_status_color %}</span>
|
||||
</p>
|
||||
{% if job.completed %}
|
||||
<div class="card mb-3">
|
||||
<h5 class="card-header">{% trans "Script Log" %}</h5>
|
||||
<table class="table table-hover">
|
||||
<tr>
|
||||
<th>{% trans "Line" %}</th>
|
||||
<th>{% trans "Level" %}</th>
|
||||
<th>{% trans "Message" %}</th>
|
||||
</tr>
|
||||
{% for log in job.data.log %}
|
||||
<tr>
|
||||
<td>{{ forloop.counter }}</td>
|
||||
<td>{% log_level log.status %}</td>
|
||||
<td class="rendered-markdown">{{ log.message|markdown }}</td>
|
||||
</tr>
|
||||
{% empty %}
|
||||
<tr>
|
||||
<td colspan="3" class="text-center text-muted">
|
||||
{% trans "No log output" %}
|
||||
</td>
|
||||
</tr>
|
||||
{% endfor %}
|
||||
</table>
|
||||
{% if execution_time %}
|
||||
<div class="card-footer text-end text-muted">
|
||||
<small>{% trans "Exec Time" %}: {{ execution_time|floatformat:3 }} {% trans "seconds" context "Unit of time" %}</small>
|
||||
</div>
|
||||
{% endif %}
|
||||
</div>
|
||||
<h4>{% trans "Output" %}</h4>
|
||||
{% if job.data.output %}
|
||||
<pre class="block">{{ job.data.output }}</pre>
|
||||
{% else %}
|
||||
<p class="text-muted">{% trans "None" %}</p>
|
||||
|
||||
{# Script log. Legacy reports will not have this. #}
|
||||
{% if 'log' in job.data %}
|
||||
<div class="card mb-3">
|
||||
<h5 class="card-header">{% trans "Log" %}</h5>
|
||||
{% if job.data.log %}
|
||||
<table class="table table-hover panel-body">
|
||||
<tr>
|
||||
<th>{% trans "Line" %}</th>
|
||||
<th>{% trans "Time" %}</th>
|
||||
<th>{% trans "Level" %}</th>
|
||||
<th>{% trans "Message" %}</th>
|
||||
</tr>
|
||||
{% for log in job.data.log %}
|
||||
<tr>
|
||||
<td>{{ forloop.counter }}</td>
|
||||
<td>{{ log.time|placeholder }}</td>
|
||||
<td>{% log_level log.status %}</td>
|
||||
<td>{{ log.message|markdown }}</td>
|
||||
</tr>
|
||||
{% endfor %}
|
||||
</table>
|
||||
{% else %}
|
||||
<div class="card-body text-muted">{% trans "None" %}</div>
|
||||
{% endif %}
|
||||
</div>
|
||||
{% endif %}
|
||||
|
||||
{# Script output. Legacy reports will not have this. #}
|
||||
{% if 'output' in job.data %}
|
||||
<div class="card mb-3">
|
||||
<h5 class="card-header">{% trans "Output" %}</h5>
|
||||
{% if job.data.output %}
|
||||
<pre class="card-body font-monospace">{{ job.data.output }}</pre>
|
||||
{% else %}
|
||||
<div class="card-body text-muted">{% trans "None" %}</div>
|
||||
{% endif %}
|
||||
</div>
|
||||
{% endif %}
|
||||
|
||||
{# Test method logs (for legacy Reports) #}
|
||||
{% if tests %}
|
||||
|
||||
{# Summary of test methods #}
|
||||
<div class="card">
|
||||
<h5 class="card-header">{% trans "Test Summary" %}</h5>
|
||||
<table class="table table-hover">
|
||||
{% for test, data in tests.items %}
|
||||
<tr>
|
||||
<td class="font-monospace"><a href="#{{ test }}">{{ test }}</a></td>
|
||||
<td class="text-end report-stats">
|
||||
<span class="badge text-bg-success">{{ data.success }}</span>
|
||||
<span class="badge text-bg-info">{{ data.info }}</span>
|
||||
<span class="badge text-bg-warning">{{ data.warning }}</span>
|
||||
<span class="badge text-bg-danger">{{ data.failure }}</span>
|
||||
</td>
|
||||
</tr>
|
||||
{% endfor %}
|
||||
</table>
|
||||
</div>
|
||||
|
||||
{# Detailed results for individual tests #}
|
||||
<div class="card">
|
||||
<h5 class="card-header">{% trans "Test Details" %}</h5>
|
||||
<table class="table table-hover report">
|
||||
<thead>
|
||||
<tr class="table-headings">
|
||||
<th>{% trans "Time" %}</th>
|
||||
<th>{% trans "Level" %}</th>
|
||||
<th>{% trans "Object" %}</th>
|
||||
<th>{% trans "Message" %}</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{% for test, data in tests.items %}
|
||||
<tr>
|
||||
<th colspan="4" style="font-family: monospace">
|
||||
<a name="{{ test }}"></a>{{ test }}
|
||||
</th>
|
||||
</tr>
|
||||
{% for time, level, obj, url, message in data.log %}
|
||||
<tr class="{% if level == 'failure' %}danger{% elif level %}{{ level }}{% endif %}">
|
||||
<td>{{ time }}</td>
|
||||
<td>
|
||||
<label class="badge text-bg-{% if level == 'failure' %}danger{% else %}{{ level }}{% endif %}">{{ level|title }}</label>
|
||||
</td>
|
||||
<td>
|
||||
{% if obj and url %}
|
||||
<a href="{{ url }}">{{ obj }}</a>
|
||||
{% elif obj %}
|
||||
{{ obj }}
|
||||
{% else %}
|
||||
{{ ''|placeholder }}
|
||||
{% endif %}
|
||||
</td>
|
||||
<td class="rendered-markdown">{{ message|markdown }}</td>
|
||||
</tr>
|
||||
{% endfor %}
|
||||
{% endfor %}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
|
||||
{% endif %}
|
||||
{% elif job.started %}
|
||||
{% include 'extras/inc/result_pending.html' %}
|
||||
|
||||
@ -1,43 +0,0 @@
|
||||
{% extends 'extras/report/base.html' %}
|
||||
{% load helpers %}
|
||||
{% load form_helpers %}
|
||||
{% load i18n %}
|
||||
|
||||
{% block content %}
|
||||
<div role="tabpanel" class="tab-pane active" id="report">
|
||||
{% if perms.extras.run_report %}
|
||||
<div class="row">
|
||||
<div class="col">
|
||||
{% if not report.is_valid %}
|
||||
<div class="alert alert-warning">
|
||||
<i class="mdi mdi-alert"></i>
|
||||
{% trans "This report is invalid and cannot be run." %}
|
||||
</div>
|
||||
{% endif %}
|
||||
<form action="{% url 'extras:report' module=report.module name=report.class_name %}" method="post" class="object-edit">
|
||||
{% csrf_token %}
|
||||
{% render_form form %}
|
||||
<div class="float-end">
|
||||
<button type="submit" name="_run" class="btn btn-primary"{% if not report.is_valid %} disabled{% endif %}>
|
||||
{% if report.result %}
|
||||
<i class="mdi mdi-replay"></i> {% trans "Run Again" %}
|
||||
{% else %}
|
||||
<i class="mdi mdi-play"></i> {% trans "Run Report" %}
|
||||
{% endif %}
|
||||
</button>
|
||||
</div>
|
||||
</form>
|
||||
</div>
|
||||
</div>
|
||||
{% endif %}
|
||||
<div class="row">
|
||||
<div class="col col-md-12">
|
||||
{% if report.result %}
|
||||
{% trans "Last run" %}: <a href="{% url 'extras:report_result' job_pk=report.result.pk %}">
|
||||
<strong>{{ report.result.created|annotated_date }}</strong>
|
||||
</a>
|
||||
{% endif %}
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
{% endblock content %}
|
||||
@ -1,128 +0,0 @@
|
||||
{% extends 'generic/_base.html' %}
|
||||
{% load buttons %}
|
||||
{% load helpers %}
|
||||
{% load perms %}
|
||||
{% load i18n %}
|
||||
|
||||
{% block title %}{% trans "Reports" %}{% endblock %}
|
||||
|
||||
{% block tabs %}
|
||||
<ul class="nav nav-tabs">
|
||||
<li class="nav-item" role="presentation">
|
||||
<a class="nav-link active" role="tab">{% trans "Reports" %}</a>
|
||||
</li>
|
||||
</ul>
|
||||
{% endblock tabs %}
|
||||
|
||||
{% block controls %}
|
||||
{% add_button model %}
|
||||
{% endblock controls %}
|
||||
|
||||
{% block content %}
|
||||
{% for module in report_modules %}
|
||||
<div class="card">
|
||||
<h5 class="card-header justify-content-between" id="module{{ module.pk }}">
|
||||
<div>
|
||||
<i class="mdi mdi-file-document-outline"></i> {{ module }}
|
||||
</div>
|
||||
{% if perms.extras.delete_reportmodule %}
|
||||
<a href="{% url 'extras:reportmodule_delete' pk=module.pk %}" class="btn btn-danger btn-sm">
|
||||
<i class="mdi mdi-trash-can-outline" aria-hidden="true"></i> {% trans "Delete" %}
|
||||
</a>
|
||||
{% endif %}
|
||||
</h5>
|
||||
<div class="card-body">
|
||||
{% include 'inc/sync_warning.html' with object=module %}
|
||||
{% if module.reports %}
|
||||
<table class="table table-hover reports">
|
||||
<thead>
|
||||
<tr>
|
||||
<th width="250">{% trans "Name" %}</th>
|
||||
<th>{% trans "Description" %}</th>
|
||||
<th>{% trans "Last Run" %}</th>
|
||||
<th>{% trans "Status" %}</th>
|
||||
<th width="120"></th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{% with jobs=module.get_latest_jobs %}
|
||||
{% for report_name, report in module.reports.items %}
|
||||
{% with last_job=jobs|get_key:report.class_name %}
|
||||
<tr>
|
||||
<td>
|
||||
<a href="{% url 'extras:report' module=module.python_name name=report.class_name %}" id="{{ report.module }}.{{ report.class_name }}">{{ report.name }}</a>
|
||||
</td>
|
||||
<td>{{ report.description|markdown|placeholder }}</td>
|
||||
{% if last_job %}
|
||||
<td>
|
||||
<a href="{% url 'extras:report_result' job_pk=last_job.pk %}">{{ last_job.created|annotated_date }}</a>
|
||||
</td>
|
||||
<td>
|
||||
{% badge last_job.get_status_display last_job.get_status_color %}
|
||||
</td>
|
||||
{% else %}
|
||||
<td class="text-muted">{% trans "Never" %}</td>
|
||||
<td>
|
||||
{% if report.is_valid %}
|
||||
{{ ''|placeholder }}
|
||||
{% else %}
|
||||
<span class="badge text-bg-danger" title="{% trans "Report has no test methods" %}">
|
||||
{% trans "Invalid" %}
|
||||
</span>
|
||||
{% endif %}
|
||||
</td>
|
||||
{% endif %}
|
||||
<td>
|
||||
{% if perms.extras.run_report and report.is_valid %}
|
||||
<div class="float-end d-print-none">
|
||||
<form action="{% url 'extras:report' module=report.module name=report.class_name %}" method="post">
|
||||
{% csrf_token %}
|
||||
<button type="submit" name="_run" class="btn btn-primary" style="width: 110px">
|
||||
{% if last_job %}
|
||||
<i class="mdi mdi-replay"></i> {% trans "Run Again" %}
|
||||
{% else %}
|
||||
<i class="mdi mdi-play"></i> {% trans "Run Report" %}
|
||||
{% endif %}
|
||||
</button>
|
||||
</form>
|
||||
</div>
|
||||
{% endif %}
|
||||
</td>
|
||||
</tr>
|
||||
{% for method, stats in last_job.data.items %}
|
||||
<tr>
|
||||
<td colspan="4" class="method">
|
||||
<span class="ps-3">{{ method }}</span>
|
||||
</td>
|
||||
<td class="text-end text-nowrap report-stats">
|
||||
<span class="badge text-bg-success">{{ stats.success }}</span>
|
||||
<span class="badge text-bg-info">{{ stats.info }}</span>
|
||||
<span class="badge text-bg-warning">{{ stats.warning }}</span>
|
||||
<span class="badge text-bg-danger">{{ stats.failure }}</span>
|
||||
</td>
|
||||
</tr>
|
||||
{% endfor %}
|
||||
{% endwith %}
|
||||
{% endfor %}
|
||||
{% endwith %}
|
||||
</tbody>
|
||||
</table>
|
||||
{% else %}
|
||||
<div class="alert alert-warning" role="alert">
|
||||
<i class="mdi mdi-alert"></i> Could not load reports from {{ module.name }}
|
||||
</div>
|
||||
{% endif %}
|
||||
</div>
|
||||
</div>
|
||||
{% empty %}
|
||||
<div class="alert alert-info" role="alert">
|
||||
<h4 class="alert-heading">{% trans "No Reports Found" %}</h4>
|
||||
{% if perms.extras.add_reportmodule %}
|
||||
{% url 'extras:reportmodule_add' as create_report_url %}
|
||||
{% blocktrans trimmed %}
|
||||
Get started by <a href="{{ create_report_url }}">creating a report</a> from an uploaded file or data source.
|
||||
{% endblocktrans %}
|
||||
{% endif %}
|
||||
</div>
|
||||
{% endfor %}
|
||||
{% endblock content %}
|
||||
@ -1,17 +0,0 @@
|
||||
{% extends 'extras/report.html' %}
|
||||
{% load buttons %}
|
||||
{% load perms %}
|
||||
|
||||
{% block controls %}
|
||||
{% if request.user|can_delete:job %}
|
||||
{% delete_button job %}
|
||||
{% endif %}
|
||||
{% endblock controls %}
|
||||
|
||||
{% block content %}
|
||||
<div class="row">
|
||||
<div class="col col-md-12"{% if not job.completed %} hx-get="{% url 'extras:report_result' job_pk=job.pk %}" hx-trigger="load delay:0.5s, every 5s"{% endif %}>
|
||||
{% include 'extras/htmx/report_result.html' %}
|
||||
</div>
|
||||
</div>
|
||||
{% endblock content %}
|
||||
@ -1,14 +1,11 @@
|
||||
{% extends 'generic/_base.html' %}
|
||||
{% load buttons %}
|
||||
{% load helpers %}
|
||||
{% load perms %}
|
||||
{% load i18n %}
|
||||
|
||||
{% block title %}{% trans "Scripts" %}{% endblock %}
|
||||
|
||||
{% block controls %}
|
||||
{% add_button model %}
|
||||
{% endblock controls %}
|
||||
|
||||
{% block tabs %}
|
||||
<ul class="nav nav-tabs">
|
||||
<li class="nav-item" role="presentation">
|
||||
@ -17,73 +14,117 @@
|
||||
</ul>
|
||||
{% endblock tabs %}
|
||||
|
||||
{% block controls %}
|
||||
{% add_button model %}
|
||||
{% endblock controls %}
|
||||
|
||||
{% block content %}
|
||||
{% for module in script_modules %}
|
||||
{% include 'inc/sync_warning.html' with object=module %}
|
||||
<div class="card">
|
||||
<h5 class="card-header justify-content-between" id="module{{ module.pk }}">
|
||||
<div>
|
||||
<i class="mdi mdi-file-document-outline"></i> {{ module }}
|
||||
</div>
|
||||
{% if perms.extras.delete_scriptmodule %}
|
||||
<div class="float-end">
|
||||
<a href="{% url 'extras:scriptmodule_delete' pk=module.pk %}" class="btn btn-danger btn-sm">
|
||||
<i class="mdi mdi-trash-can-outline" aria-hidden="true"></i> {% trans "Delete" %}
|
||||
</a>
|
||||
</div>
|
||||
<a href="{% url 'extras:scriptmodule_delete' pk=module.pk %}" class="btn btn-danger btn-sm">
|
||||
<i class="mdi mdi-trash-can-outline" aria-hidden="true"></i> {% trans "Delete" %}
|
||||
</a>
|
||||
{% endif %}
|
||||
</h5>
|
||||
<div class="card-body">
|
||||
{% include 'inc/sync_warning.html' with object=module %}
|
||||
{% if not module.scripts %}
|
||||
<div class="alert alert-warning d-flex align-items-center" role="alert">
|
||||
<i class="mdi mdi-alert"></i>
|
||||
{% blocktrans trimmed with file_path=module.full_path %}
|
||||
Script file at <code class="mx-1">{{ file_path }}</code> could not be loaded.
|
||||
{% endblocktrans %}
|
||||
</div>
|
||||
{% else %}
|
||||
<table class="table table-hover reports">
|
||||
<thead>
|
||||
<tr>
|
||||
<th width="250">{% trans "Name" %}</th>
|
||||
<th>{% trans "Description" %}</th>
|
||||
<th>{% trans "Last Run" %}</th>
|
||||
<th class="text-end">{% trans "Status" %}</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{% with jobs=module.get_latest_jobs %}
|
||||
{% for script_name, script_class in module.scripts.items %}
|
||||
{% if module.scripts %}
|
||||
<table class="table table-hover scripts">
|
||||
<thead>
|
||||
<tr>
|
||||
<th>{% trans "Name" %}</th>
|
||||
<th>{% trans "Description" %}</th>
|
||||
<th>{% trans "Last Run" %}</th>
|
||||
<th>{% trans "Status" %}</th>
|
||||
<th></th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{% with jobs=module.get_latest_jobs %}
|
||||
{% for script_name, script in module.scripts.items %}
|
||||
{% with last_job=jobs|get_key:script.class_name %}
|
||||
<tr>
|
||||
<td>
|
||||
<a href="{% url 'extras:script' module=module.python_name name=script_name %}" name="script.{{ script_name }}">{{ script_class.name }}</a>
|
||||
<a href="{% url 'extras:script' module=module.python_name name=script.class_name %}" id="{{ script.module }}.{{ script.class_name }}">{{ script.name }}</a>
|
||||
</td>
|
||||
<td>{{ script.description|markdown|placeholder }}</td>
|
||||
{% if last_job %}
|
||||
<td>
|
||||
<a href="{% url 'extras:script_result' job_pk=last_job.pk %}">{{ last_job.created|annotated_date }}</a>
|
||||
</td>
|
||||
<td>
|
||||
{% badge last_job.get_status_display last_job.get_status_color %}
|
||||
</td>
|
||||
{% else %}
|
||||
<td class="text-muted">{% trans "Never" %}</td>
|
||||
<td>{{ ''|placeholder }}</td>
|
||||
{% endif %}
|
||||
<td>
|
||||
{{ script_class.Meta.description|markdown|placeholder }}
|
||||
</td>
|
||||
{% with last_result=jobs|get_key:script_class.class_name %}
|
||||
{% if last_result %}
|
||||
<td>
|
||||
<a href="{% url 'extras:script_result' job_pk=last_result.pk %}">{{ last_result.created|annotated_date }}</a>
|
||||
</td>
|
||||
<td class="text-end">
|
||||
{% badge last_result.get_status_display last_result.get_status_color %}
|
||||
</td>
|
||||
{% else %}
|
||||
<td class="text-muted">{% trans "Never" %}</td>
|
||||
<td class="text-end">{{ ''|placeholder }}</td>
|
||||
{% if perms.extras.run_script %}
|
||||
<div class="float-end d-print-none">
|
||||
<form action="{% url 'extras:script' module=script.module name=script.class_name %}" method="post">
|
||||
{% csrf_token %}
|
||||
<button type="submit" name="_run" class="btn btn-primary btn-sm">
|
||||
{% if last_job %}
|
||||
<i class="mdi mdi-replay"></i> {% trans "Run Again" %}
|
||||
{% else %}
|
||||
<i class="mdi mdi-play"></i> {% trans "Run Script" %}
|
||||
{% endif %}
|
||||
</button>
|
||||
</form>
|
||||
</div>
|
||||
{% endif %}
|
||||
{% endwith %}
|
||||
</td>
|
||||
</tr>
|
||||
{% endfor %}
|
||||
{% endwith %}
|
||||
</tbody>
|
||||
</table>
|
||||
{% endif %}
|
||||
</div>
|
||||
{% if last_job %}
|
||||
{% for test_name, data in last_job.data.tests.items %}
|
||||
<tr>
|
||||
<td colspan="4" class="method">
|
||||
<span class="ps-3">{{ test_name }}</span>
|
||||
</td>
|
||||
<td class="text-end text-nowrap script-stats">
|
||||
<span class="badge text-bg-success">{{ data.success }}</span>
|
||||
<span class="badge text-bg-info">{{ data.info }}</span>
|
||||
<span class="badge text-bg-warning">{{ data.warning }}</span>
|
||||
<span class="badge text-bg-danger">{{ data.failure }}</span>
|
||||
</td>
|
||||
</tr>
|
||||
{% endfor %}
|
||||
{% elif not last_job.data.log %}
|
||||
{# legacy #}
|
||||
{% for method, stats in last_job.data.items %}
|
||||
<tr>
|
||||
<td colspan="4" class="method">
|
||||
<span class="ps-3">{{ method }}</span>
|
||||
</td>
|
||||
<td class="text-end text-nowrap report-stats">
|
||||
<span class="badge bg-success">{{ stats.success }}</span>
|
||||
<span class="badge bg-info">{{ stats.info }}</span>
|
||||
<span class="badge bg-warning">{{ stats.warning }}</span>
|
||||
<span class="badge bg-danger">{{ stats.failure }}</span>
|
||||
</td>
|
||||
</tr>
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
{% endwith %}
|
||||
{% endfor %}
|
||||
{% endwith %}
|
||||
</tbody>
|
||||
</table>
|
||||
{% else %}
|
||||
<div class="card-body">
|
||||
<div class="alert alert-warning" role="alert">
|
||||
<i class="mdi mdi-alert"></i> Could not load scripts from {{ module.name }}
|
||||
</div>
|
||||
</div>
|
||||
{% endif %}
|
||||
</div>
|
||||
{% empty %}
|
||||
<div class="alert alert-info">
|
||||
<div class="alert alert-info" role="alert">
|
||||
<h4 class="alert-heading">{% trans "No Scripts Found" %}</h4>
|
||||
{% if perms.extras.add_scriptmodule %}
|
||||
{% url 'extras:scriptmodule_add' as create_script_url %}
|
||||
|
||||
Reference in New Issue
Block a user