1
0
mirror of https://github.com/netbox-community/netbox.git synced 2024-05-10 07:54:54 +00:00

Closes #11558: Add support for remote data sources (#11646)

* WIP

* WIP

* Add git sync

* Fix file hashing

* Add last_synced to DataSource

* Build out UI & API resources

* Add status field to DataSource

* Add UI control to sync data source

* Add API endpoint to sync data sources

* Fix display of DataSource job results

* DataSource password should be write-only

* General cleanup

* Add data file UI view

* Punt on HTTP, FTP support for now

* Add DataSource URL validation

* Add HTTP proxy support to git fetcher

* Add management command to sync data sources

* DataFile REST API endpoints should be read-only

* Refactor fetch methods into backend classes

* Replace auth & git branch fields with general-purpose parameters

* Fix last_synced time

* Render discrete form fields for backend parameters

* Enable dynamic edit form for DataSource

* Register DataBackend classes in application registry

* Add search indexers for DataSource, DataFile

* Add single & bulk delete views for DataFile

* Add model documentation

* Convert DataSource to a primary model

* Introduce pre_sync & post_sync signals

* Clean up migrations

* Rename url to source_url

* Clean up filtersets

* Add API & filterset tests

* Add view tests

* Add initSelect() to HTMX refresh handler

* Render DataSourceForm fieldsets dynamically

* Update compiled static resources
This commit is contained in:
Jeremy Stretch
2023-02-02 10:06:23 -05:00
committed by jeremystretch
parent e65b2a9fb3
commit d8784d4155
53 changed files with 1865 additions and 14 deletions

View File

@ -0,0 +1,25 @@
# Data Files
A data file object is the representation in NetBox's database of some file belonging to a remote [data source](./datasource.md). Data files are synchronized automatically, and cannot be modified locally (although they can be deleted).
## Fields
### Source
The [data source](./datasource.md) to which this file belongs.
### Path
The path to the file, relative to its source's URL. For example, a file at `/opt/config-data/routing/bgp/peer.yaml` with a source URL of `file:///opt/config-data/` would have its path set to `routing/bgp/peer.yaml`.
### Last Updated
The date and time at which the file most recently updated from its source. Note that this attribute is updated only when the file's contents have been modified. Re-synchronizing the data source will not update this timestamp if the upstream file's data has not changed.
### Size
The file's size, in bytes.
### Hash
A [SHA256 hash](https://en.wikipedia.org/wiki/SHA-2) of the file's data. This can be compared to a hash taken from the original file to determine whether any changes have been made.

View File

@ -0,0 +1,47 @@
# Data Sources
A data source represents some external repository of data which NetBox can consume, such as a git repository. Files within the data source are synchronized to NetBox by saving them in the database as [data file](./datafile.md) objects.
## Fields
### Name
The data source's human-friendly name.
### Type
The type of data source. Supported options include:
* Local directory
* git repository
### URL
The URL identifying the remote source. Some examples are included below.
| Type | Example URL |
|------|-------------|
| Local | file:///var/my/data/source/ |
| git | https://https://github.com/my-organization/my-repo |
### Status
The source's current synchronization status. Note that this cannot be set manually: It is updated automatically when the source is synchronized.
### Enabled
If false, synchronization will be disabled.
### Ignore Rules
A set of rules (one per line) identifying filenames to ignore during synchronization. Some examples are provided below. See Python's [`fnmatch()` documentation](https://docs.python.org/3/library/fnmatch.html) for a complete reference.
| Rule | Description |
|----------------|------------------------------------------|
| `README` | Ignore any files named `README` |
| `*.txt` | Ignore any files with a `.txt` extension |
| `data???.json` | Ignore e.g. `data123.json` |
### Last Synced
The date and time at which the source was most recently synchronized successfully.