mirror/github-octodns
1
0
Fork 0
mirror of https://github.com/github/octodns.git synced 2024-05-11 05:55:00 +00:00
Code Issues Releases Activity

Merge pull request #1026 from octodns/dynamic-zones

Browse Source 
Dynamic zone configs
This commit is contained in:
Ross McFarland
2023-08-10 16:38:33 -06:00
committed by GitHub
parent 0b5265e1b8 866a9f9710
commit 6e7f036b6e
14 changed files with 357 additions and 80 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
CHANGELOG.md
View File

@@ -1,5 +1,13 @@
## v1.1.0 - 2023-??-?? - ???
#### Noteworthy changes
* New dynamic zone config support that allows wildcard entries in the octoDNS
config to be expanded by the source provider(s). See
[Dynamic Zone Config](/README.md#dynamic-zone-config) for more information.
#### Stuff
* Add context to general configuration and Record validation, e.g.
Some problem at filename.yaml, line 42, column 14. Our custom Yaml Loaders
attach this context information, arbitrary string. Other providers may do so

2
CONTRIBUTING.md
View File

@@ -1,6 +1,6 @@
# Contributing
Hi there! We're thrilled that you'd like to contribute to OctoDNS. Your help is essential for keeping it great.
Hi there! We're thrilled that you'd like to contribute to octoDNS. Your help is essential for keeping it great.
Please note that this project adheres to the [Contributor Covenant Code of Conduct](/CODE_OF_CONDUCT.md). By participating in this project you agree to abide by its terms.

139
README.md
View File

@@ -1,11 +1,11 @@
<img src="https://raw.githubusercontent.com/octodns/octodns/main/docs/logos/octodns-logo.png?" alt="OctoDNS Logo" height=251 width=404>
<img src="https://raw.githubusercontent.com/octodns/octodns/main/docs/logos/octodns-logo.png?" alt="octoDNS Logo" height=251 width=404>
## DNS as code - Tools for managing DNS across multiple providers
In the vein of [infrastructure as
code](https://en.wikipedia.org/wiki/Infrastructure_as_Code) OctoDNS provides a set of tools & patterns that make it easy to manage your DNS records across multiple providers. The resulting config can live in a repository and be [deployed](https://github.com/blog/1241-deploying-at-github) just like the rest of your code, maintaining a clear history and using your existing review & workflow.
code](https://en.wikipedia.org/wiki/Infrastructure_as_Code) octoDNS provides a set of tools & patterns that make it easy to manage your DNS records across multiple providers. The resulting config can live in a repository and be [deployed](https://github.com/blog/1241-deploying-at-github) just like the rest of your code, maintaining a clear history and using your existing review & workflow.
The architecture is pluggable and the tooling is flexible to make it applicable to a wide variety of use-cases. Effort has been made to make adding new providers as easy as possible. In the simple case that involves writing of a single `class` and a couple hundred lines of code, most of which is translating between the provider's schema and OctoDNS's. More on some of the ways we use it and how to go about extending it below and in the [/docs directory](/docs).
The architecture is pluggable and the tooling is flexible to make it applicable to a wide variety of use-cases. Effort has been made to make adding new providers as easy as possible. In the simple case that involves writing of a single `class` and a couple hundred lines of code, most of which is translating between the provider's schema and octoDNS's. More on some of the ways we use it and how to go about extending it below and in the [/docs directory](/docs).
## Table of Contents
@@ -13,6 +13,10 @@ The architecture is pluggable and the tooling is flexible to make it applicable
* [Workspace](#workspace)
* [Installing a specific commit SHA](#installing-a-specific-commit-sha)
* [Config](#config)
* [Dynamic Zone Config](#dynamic-zone-config)
* [Static Zone Config](#static-zone-config)
* [General Configuration Concepts](#general-configuration-concepts)
* [Quick Example Record](#quick-example-record)
* [Noop](#noop)
* [Making changes](#making-changes)
* [Workflow](#workflow)
@@ -40,9 +44,9 @@ The architecture is pluggable and the tooling is flexible to make it applicable
### Workspace
Running through the following commands will install the latest release of OctoDNS and set up a place for your config files to live. To determine if provider specific requirements are necessary see the [providers table](#providers) below.
Running through the following commands will install the latest release of octoDNS and set up a place for your config files to live. To determine if provider specific requirements are necessary see the [providers table](#providers) below.
```shell
```console
$ mkdir dns
$ cd dns
$ python -m venv env
@@ -57,18 +61,56 @@ $ mkdir config
If you'd like to install a version that has not yet been released in a repetable/safe manner you can do the following. In general octoDNS is fairly stable in between releases thanks to the plan and apply process, but care should be taken regardless.
```shell
```console
$ pip install -e git+https://git@github.com/octodns/octodns.git@<SHA>#egg=octodns
```
### Config
We start by creating a config file to tell OctoDNS about our providers and the zone(s) we want it to manage. Below we're setting up a `YamlProvider` to source records from our config files and both a `Route53Provider` and `DynProvider` to serve as the targets for those records. You can have any number of zones set up and any number of sources of data and targets for records for each. You can also have multiple config files, that make use of separate accounts and each manage a distinct set of zones. A good example of this this might be `./config/staging.yaml` & `./config/production.yaml`. We'll focus on a `config/production.yaml`.
We start by creating a config file to tell octoDNS about our providers and the zone(s) we want it to manage. Below we're setting up a `YamlProvider` to source records from our config files and both a `Route53Provider` and `DynProvider` to serve as the targets for those records. You can have any number of zones set up and any number of sources of data and targets for records for each. You can also have multiple config files, that make use of separate accounts and each manage a distinct set of zones. A good example of this this might be `./config/staging.yaml` & `./config/production.yaml`. We'll focus on a `config/production.yaml`.
#### Dynamic Zone Config
octoDNS supports dynamically building the list of zones it will work with when source providers support it. The most common use of this would be with `YamlProvider` and a single dynamic entry to in effect use the files that exist in the provider's directory as the source of truth. Other providers may support the `list_zones` method and be available to populate zones dynamically as well. This can be especially useful when using `octodns-dump` to create an initial setup from an existing provider.
An example config would look something like:
```yaml
---
providers:
config:
class: octodns.provider.yaml.YamlProvider
directory: ./config
default_ttl: 3600
enforce_order: True
ns:
class: octodns_ns1.Ns1Provider
api_key: env/NS1_API_KEY
route53:
class: octodns_route53.Route53Provider
access_key_id: env/AWS_ACCESS_KEY_ID
secret_access_key: env/AWS_SECRET_ACCESS_KEY
zones:
# This is a dynamic zone config. The source(s), here `config`, will be
# queried for a list of zone names and each will dynamically be set up to
# match the dynamic entry.
'*':
sources:
- config
targets:
- ns1
- route53
```
#### Static Zone Config
In cases where finer grained control is desired and the configuration of individual zones varies `zones` can be an explicit list with each configured zone listed along with it's specific setup. As exemplified below `alias` zones can be useful when two zones are exact copies of each other, with the same configuration and records. YAML anchors are also helpful to avoid duplication where zones share config, but not records.
```yaml
---
manager:
include_meta: False
include_meta: True
max_workers: 2
providers:
@@ -77,39 +119,55 @@ providers:
directory: ./config
default_ttl: 3600
enforce_order: True
dyn:
class: octodns.provider.dyn.DynProvider
customer: 1234
username: 'username'
password: env/DYN_PASSWORD
ns:
class: octodns_ns1.Ns1Provider
api_key: env/NS1_API_KEY
route53:
class: octodns_route53.Route53Provider
access_key_id: env/AWS_ACCESS_KEY_ID
secret_access_key: env/AWS_SECRET_ACCESS_KEY
zones:
example.com.:
example.com.: &dual_target
sources:
- config
targets:
- dyn
- ns1
- route53
# these have the same setup as example.com., but will have their own files
# in the configuration directory for records.
third.tv.: *dual_target
fourth.tv.: *dual_target
example.net.:
# example.net. is an exact copy of example.com., there will not be an
# example.net.yaml file in the config directory as `alias` includes
# duplicating the records of the aliased zone along with its config.
alias: example.com.
other.com.:
lenient: True
sources:
- config
targets:
- ns1
```
`class` is a special key that tells OctoDNS what python class should be loaded. Any other keys will be passed as configuration values to that provider. In general any sensitive or frequently rotated values should come from environmental variables. When OctoDNS sees a value that starts with `env/` it will look for that value in the process's environment and pass the result along.
#### General Configuration Concepts
`class` is a special key that tells octoDNS what python class should be loaded. Any other keys will be passed as configuration values to that provider. In general any sensitive or frequently rotated values should come from environmental variables. When octoDNS sees a value that starts with `env/` it will look for that value in the process's environment and pass the result along.
Further information can be found in the `docstring` of each source and provider class.
The `include_meta` key in the `manager` section of the config controls the creation of a TXT record at the root of a zone that is managed by OctoDNS. If set to `True`, OctoDNS will create a TXT record for the root of the zone with the value `provider=<target-provider>`. If not specified, the default value for `include_meta` is `False`.
The `include_meta` key in the `manager` section of the config controls the creation of a TXT record at the root of a zone that is managed by octoDNS. If set to `True`, octoDNS will create a TXT record for the root of the zone with the value `provider=<target-provider>`. If not specified, the default value for `include_meta` is `False`.
The `max_workers` key in the `manager` section of the config enables threading to parallelize the planning portion of the sync.
In this example, `example.net` is an alias of zone `example.com`, which means they share the same sources and targets. They will therefore have identical records.
#### Quick Example Record
Now that we have something to tell OctoDNS about our providers & zones we need to tell it about our records. We'll keep it simple for now and just create a single `A` record at the top-level of the domain.
Now that we have something to tell octoDNS about our providers & zones we need to tell it about our records. We'll keep it simple for now and just create a single `A` record at the top-level of the domain.
`config/example.com.yaml`
@@ -129,7 +187,7 @@ Further information can be found in [Records Documentation](/docs/records.md).
We're ready to do a dry-run with our new setup to see what changes it would make. Since we're pretending here we'll act like there are no existing records for `example.com.` in our accounts on either provider.
```shell
```console
$ octodns-sync --config-file=./config/production.yaml
...
********************************************************************************
@@ -145,15 +203,15 @@ $ octodns-sync --config-file=./config/production.yaml
...
```
There will be other logging information presented on the screen, but successful runs of sync will always end with a summary like the above for any providers & zones with changes. If there are no changes a message saying so will be printed instead. Above we're creating a new zone in both providers so they show the same change, but that doesn't always have to be the case. If to start one of them had a different state you would see the changes OctoDNS intends to make to sync them up.
There will be other logging information presented on the screen, but successful runs of sync will always end with a summary like the above for any providers & zones with changes. If there are no changes a message saying so will be printed instead. Above we're creating a new zone in both providers so they show the same change, but that doesn't always have to be the case. If to start one of them had a different state you would see the changes octoDNS intends to make to sync them up.
### Making changes
**WARNING**: OctoDNS assumes ownership of any domain you point it to. When you tell it to act it will do whatever is necessary to try and match up states including deleting any unexpected records. Be careful when playing around with OctoDNS. It's best to experiment with a fake zone or one without any data that matters until you're comfortable with the system.
**WARNING**: octoDNS assumes ownership of any domain you point it to. When you tell it to act it will do whatever is necessary to try and match up states including deleting any unexpected records. Be careful when playing around with octoDNS. It's best to experiment with a fake zone or one without any data that matters until you're comfortable with the system.
Now it's time to tell OctoDNS to make things happen. We'll invoke it again with the same options and add a `--doit` on the end to tell it this time we actually want it to try and make the specified changes.
Now it's time to tell octoDNS to make things happen. We'll invoke it again with the same options and add a `--doit` on the end to tell it this time we actually want it to try and make the specified changes.
```shell
```console
$ octodns-sync --config-file=./config/production.yaml --doit
...
```
@@ -162,13 +220,13 @@ The output here would be the same as before with a few more log lines at the end
### Workflow
In the above case we manually ran OctoDNS from the command line. That works and it's better than heading into the provider GUIs and making changes by clicking around, but OctoDNS is designed to be run as part of a deploy process. The implementation details are well beyond the scope of this README, but here is an example of the workflow we use at GitHub. It follows the way [GitHub itself is branch deployed](https://githubengineering.com/deploying-branches-to-github-com/).
In the above case we manually ran octoDNS from the command line. That works and it's better than heading into the provider GUIs and making changes by clicking around, but octoDNS is designed to be run as part of a deploy process. The implementation details are well beyond the scope of this README, but here is an example of the workflow we use at GitHub. It follows the way [GitHub itself is branch deployed](https://githubengineering.com/deploying-branches-to-github-com/).
The first step is to create a PR with your changes.
![GitHub user interface of a pull request](/docs/assets/pr.png)
Assuming the code tests and config validation statuses are green the next step is to do a noop deploy and verify that the changes OctoDNS plans to make are the ones you expect.
Assuming the code tests and config validation statuses are green the next step is to do a noop deploy and verify that the changes octoDNS plans to make are the ones you expect.
![Output of a noop deployment command](/docs/assets/noop.png)
@@ -184,7 +242,7 @@ If that goes smoothly, you again see the expected changes, and verify them with
Very few situations will involve starting with a blank slate which is why there's tooling built in to pull existing data out of providers into a matching config file.
```shell
```console
$ octodns-dump --config-file=config/production.yaml --output-dir=tmp/ example.com. route53
2017-03-15T13:33:34 INFO Manager __init__: config_file=tmp/production.yaml
2017-03-15T13:33:34 INFO Manager dump: zone=example.com., sources=('route53',)
@@ -196,6 +254,15 @@ $ octodns-dump --config-file=config/production.yaml --output-dir=tmp/ example.co
The above command pulled the existing data out of Route53 and placed the results into `tmp/example.com.yaml`. That file can be inspected and moved into `config/` to become the new source. If things are working as designed a subsequent noop sync should show zero changes.
Note that a [Dynamic Zone Config](#dynamic-zone-config) and be really powerful in combination with `octodns-dump` allowing you to quickly create a set of octoDNS zone files for all the zones configured in your sources.
```console
$ octodns-dump --config-file=config/production.yaml --output-dir=tmp/ '*' route53
...
```
It is important to review any `WARNING` log lines printed out during an `octodns-dump` invocation as it will give you information about records that aren't supported fully or at all by octoDNS and thus won't be exact matches or included in the dumps. Generally records that cannot be converted are either of a type that octoDNS does not support or those that include "dynamic" functionality that doesn't match octoDNS's behaviors.
## Providers
The table below lists the providers octoDNS supports. They are maintained in their own repositories and released as independent modules.
@@ -289,9 +356,9 @@ providers:
You can check out the [source](/octodns/source/) and [provider](/octodns/provider/) directory to see what's currently supported. Sources act as a source of record information. AxfrSource and TinyDnsFileSource are currently the only OSS sources, though we have several others internally that are specific to our environment. These include something to pull host data from [gPanel](https://githubengineering.com/githubs-metal-cloud/) and a similar provider that sources information about our network gear to create both `A` & `PTR` records for their interfaces. Things that might make good OSS sources might include an `ElbSource` that pulls information about [AWS Elastic Load Balancers](https://aws.amazon.com/elasticloadbalancing/) and dynamically creates `CNAME`s for them, or `Ec2Source` that pulls instance information so that records can be created for hosts similar to how our `GPanelProvider` works.
Most of the things included in OctoDNS are providers, the obvious difference being that they can serve as both sources and targets of data. We'd really like to see this list grow over time so if you use an unsupported provider then PRs are welcome. The existing providers should serve as reasonable examples. Those that have no GeoDNS support are relatively straightforward. Unfortunately most of the APIs involved to do GeoDNS style traffic management are complex and somewhat inconsistent so adding support for that function would be nice, but is optional and best done in a separate pass.
Most of the things included in octoDNS are providers, the obvious difference being that they can serve as both sources and targets of data. We'd really like to see this list grow over time so if you use an unsupported provider then PRs are welcome. The existing providers should serve as reasonable examples. Those that have no GeoDNS support are relatively straightforward. Unfortunately most of the APIs involved to do GeoDNS style traffic management are complex and somewhat inconsistent so adding support for that function would be nice, but is optional and best done in a separate pass.
The `class` key in the providers config section can be used to point to arbitrary classes in the python path so internal or 3rd party providers can easily be included with no coordination beyond getting them into PYTHONPATH, most likely installed into the virtualenv with OctoDNS.
The `class` key in the providers config section can be used to point to arbitrary classes in the python path so internal or 3rd party providers can easily be included with no coordination beyond getting them into PYTHONPATH, most likely installed into the virtualenv with octoDNS.
For examples of building third-party sources and providers, see [Related Projects & Resources](#related-projects-and-resources).
@@ -299,7 +366,7 @@ For examples of building third-party sources and providers, see [Related Project
### Syncing between providers
While the primary use-case is to sync a set of yaml config files up to one or more DNS providers, OctoDNS has been built in such a way that you can easily source and target things arbitrarily. As a quick example the config below would sync `githubtest.net.` from Route53 to Dyn.
While the primary use-case is to sync a set of yaml config files up to one or more DNS providers, octoDNS has been built in such a way that you can easily source and target things arbitrarily. As a quick example the config below would sync `githubtest.net.` from Route53 to Dyn.
```yaml
---
@@ -358,7 +425,7 @@ If you have a problem or suggestion, please [open an issue](https://github.com/o
## Related Projects and Resources
- **GitHub Action:** [OctoDNS-Sync](https://github.com/marketplace/actions/octodns-sync)
- **GitHub Action:** [octoDNS-Sync](https://github.com/marketplace/actions/octodns-sync)
- **Sample Implementations.** See how others are using it
- [`hackclub/dns`](https://github.com/hackclub/dns)
- [`kubernetes/k8s.io:/dns`](https://github.com/kubernetes/k8s.io/tree/main/dns)
@@ -371,19 +438,19 @@ If you have a problem or suggestion, please [open an issue](https://github.com/o
- [`sukiyaki/octodns-netbox`](https://github.com/sukiyaki/octodns-netbox): [NetBox](https://github.com/netbox-community/netbox) source.
- [`jcollie/octodns-netbox-dns`](https://github.com/jcollie/octodns-netbox-dns): [NetBox-DNS Plugin](https://github.com/auroraresearchlab/netbox-dns) provider.
- [`kompetenzbolzen/octodns-custom-provider`](https://github.com/kompetenzbolzen/octodns-custom-provider): zonefile provider & phpIPAM source.
- [`Financial-Times/octodns-fastly`](https://github.com/Financial-Times/octodns-fastly): An OctoDNS source for Fastly.
- [`Financial-Times/octodns-fastly`](https://github.com/Financial-Times/octodns-fastly): An octoDNS source for Fastly.
- **Resources.**
- Article: [Visualising DNS records with Neo4j](https://medium.com/@costask/querying-and-visualising-octodns-records-with-neo4j-f4f72ab2d474) + code
- Video: [FOSDEM 2019 - DNS as code with octodns](https://archive.fosdem.org/2019/schedule/event/dns_octodns/)
- GitHub Blog: [Enabling DNS split authority with OctoDNS](https://github.blog/2017-04-27-enabling-split-authority-dns-with-octodns/)
- Tutorial: [How To Deploy and Manage Your DNS using OctoDNS on Ubuntu 18.04](https://www.digitalocean.com/community/tutorials/how-to-deploy-and-manage-your-dns-using-octodns-on-ubuntu-18-04)
- GitHub Blog: [Enabling DNS split authority with octoDNS](https://github.blog/2017-04-27-enabling-split-authority-dns-with-octodns/)
- Tutorial: [How To Deploy and Manage Your DNS using octoDNS on Ubuntu 18.04](https://www.digitalocean.com/community/tutorials/how-to-deploy-and-manage-your-dns-using-octodns-on-ubuntu-18-04)
- Cloudflare Blog: [Improving the Resiliency of Our Infrastructure DNS Zone](https://blog.cloudflare.com/improving-the-resiliency-of-our-infrastructure-dns-zone/)
If you know of any other resources, please do let us know!
## License
OctoDNS is licensed under the [MIT license](LICENSE).
octoDNS is licensed under the [MIT license](LICENSE).
The MIT license grant is not for GitHub's trademarks, which include the logo designs. GitHub reserves all trademark and copyright rights in and to all GitHub trademarks. GitHub's logos include, for instance, the stylized designs that include "logo" in the file title in the following folder: https://github.com/octodns/octodns/tree/main/docs/logos/
@@ -391,4 +458,4 @@ GitHub® and its stylized versions and the Invertocat mark are GitHub's Trademar
## Authors
OctoDNS was designed and authored by [Ross McFarland](https://github.com/ross) and [Joe Williams](https://github.com/joewilliams). See https://github.com/octodns/octodns/graphs/contributors for a complete list of people who've contributed.
octoDNS was designed and authored by [Ross McFarland](https://github.com/ross) and [Joe Williams](https://github.com/joewilliams). See https://github.com/octodns/octodns/graphs/contributors for a complete list of people who've contributed.

6
docs/geo_records.md
View File

@@ -4,9 +4,9 @@ Note: Geo DNS records are still supported for the time being, but it is still st
GeoDNS is currently supported for `A` and `AAAA` records on the Dyn (via Traffic Directors) and Route53 providers. Records with geo information pushed to providers without support for them will be managed as non-geo records using the base values.
Configuring GeoDNS is complex and the details of the functionality vary widely from provider to provider. OctoDNS has an opinionated view of how GeoDNS should be set up and does its best to map that to each provider's offering in a way that will result in similar behavior. It may not fit your needs or use cases, in which case please open an issue for discussion. We expect this functionality to grow and evolve over time as it's more widely used.
Configuring GeoDNS is complex and the details of the functionality vary widely from provider to provider. octoDNS has an opinionated view of how GeoDNS should be set up and does its best to map that to each provider's offering in a way that will result in similar behavior. It may not fit your needs or use cases, in which case please open an issue for discussion. We expect this functionality to grow and evolve over time as it's more widely used.
The following is an example of GeoDNS with three entries NA-US-CA, NA-US-NY, OC-AU. Octodns creates another one labeled 'default' with the details for the actual A record, This default record is the failover record if the monitoring check fails.
The following is an example of GeoDNS with three entries NA-US-CA, NA-US-NY, OC-AU. octoDNS creates another one labeled 'default' with the details for the actual A record, This default record is the failover record if the monitoring check fails.
```yaml
---
@@ -54,7 +54,7 @@ So the example is saying:
### Health Checks
Octodns will automatically set up monitors check for a 200 response for **https://<ip_address>/_dns**.
octoDNS will automatically set up monitors check for a 200 response for **https://<ip_address>/_dns**.
These checks can be configured by adding a `healthcheck` configuration to the record:

12
docs/records.md
View File

@@ -1,8 +1,8 @@
# OctoDNS records
# octoDNS records
## Record types
OctoDNS supports the following record types:
octoDNS supports the following record types:
* `A`
* `AAAA`
@@ -22,9 +22,9 @@ OctoDNS supports the following record types:
* `TXT`
* `URLFWD`
Underlying provider support for each of these varies and some providers have extra requirements or limitations. In cases where a record type is not supported by a provider OctoDNS will ignore it there and continue to manage the record elsewhere. For example `SSHFP` is supported by Dyn, but not Route53. If your source data includes an SSHFP record OctoDNS will keep it in sync on Dyn, but not consider it when evaluating the state of Route53. The best way to find out what types are supported by a provider is to look for its `supports` method. If that method exists the logic will drive which records are supported and which are ignored. If the provider does not implement the method it will fall back to `BaseProvider.supports` which indicates full support.
Underlying provider support for each of these varies and some providers have extra requirements or limitations. In cases where a record type is not supported by a provider octoDNS will ignore it there and continue to manage the record elsewhere. For example `SSHFP` is supported by Dyn, but not Route53. If your source data includes an SSHFP record octoDNS will keep it in sync on Dyn, but not consider it when evaluating the state of Route53. The best way to find out what types are supported by a provider is to look for its `supports` method. If that method exists the logic will drive which records are supported and which are ignored. If the provider does not implement the method it will fall back to `BaseProvider.supports` which indicates full support.
Adding new record types to OctoDNS is relatively straightforward, but will require careful evaluation of each provider to determine whether or not it will be supported and the addition of code in each to handle and test the new type.
Adding new record types to octoDNS is relatively straightforward, but will require careful evaluation of each provider to determine whether or not it will be supported and the addition of code in each to handle and test the new type.
## Advanced Record Support (GeoDNS, Weighting)
@@ -33,7 +33,7 @@ Adding new record types to OctoDNS is relatively straightforward, but will requi
## Config (`YamlProvider`)
OctoDNS records and `YamlProvider`'s schema is essentially a 1:1 match. Properties on the objects will match keys in the config.
octoDNS records and `YamlProvider`'s schema is essentially a 1:1 match. Properties on the objects will match keys in the config.
### Names
@@ -125,7 +125,7 @@ If you'd like to enable lenience for a whole zone you can do so with the followi
#### Restrict Record manipulations
OctoDNS currently provides the ability to limit the number of updates/deletes on
octoDNS currently provides the ability to limit the number of updates/deletes on
DNS records by configuring a percentage of allowed operations as a threshold.
If left unconfigured, suitable defaults take over instead. In the below example,
the Dyn provider is configured with limits of 40% on both update and

5
octodns/cmds/dump.py
View File

@@ -39,7 +39,10 @@ def main():
default=False,
help='Split the dumped zone into a YAML file per record',
)
parser.add_argument('zone', help='Zone to dump')
parser.add_argument(
'zone',
help="Zone to dump, '*' (single quoted to avoid expansion) for all configured zones",
)
parser.add_argument('source', nargs='+', help='Source(s) to pull data from')
args = parser.parse_args()

131
octodns/manager.py
View File

@@ -465,6 +465,74 @@ class Manager(object):
# Return the zone as it's the desired state
return plans, zone
def _get_sources(self, decoded_zone_name, config, eligible_sources):
try:
sources = config['sources']
except KeyError:
raise ManagerException(
f'Zone {decoded_zone_name} is missing sources'
)
if eligible_sources and not [
s for s in sources if s in eligible_sources
]:
return None
self.log.info('sync: sources=%s', sources)
try:
# rather than using a list comprehension, we break this loop
# out so that the `except` block below can reference the
# `source`
collected = []
for source in sources:
collected.append(self.providers[source])
sources = collected
except KeyError:
raise ManagerException(
f'Zone {decoded_zone_name}, unknown ' f'source: {source}'
)
return sources
def _preprocess_zones(self, zones, eligible_sources=None, sources=None):
'''
This may modify the passed in zone object, it should be ignored after
the call and the zones returned from this function should be used
instead.
'''
for name, config in list(zones.items()):
if not name.startswith('*'):
continue
# we've found a dynamic config element
# find its sources
sources = sources or self._get_sources(
name, config, eligible_sources
)
self.log.info('sync: dynamic zone=%s, sources=%s', name, sources)
for source in sources:
if not hasattr(source, 'list_zones'):
raise ManagerException(
f'dynamic zone={name} includes a source, {source.id}, that does not support `list_zones`'
)
for zone_name in source.list_zones():
if zone_name in zones:
self.log.info(
'sync: zone=%s already in config, ignoring',
zone_name,
)
continue
self.log.info(
'sync: adding dynamic zone=%s', zone_name
)
zones[zone_name] = config
# remove the dynamic config element so we don't try and populate it
del zones[name]
return zones
def sync(
self,
eligible_zones=[],
@@ -485,6 +553,9 @@ class Manager(object):
)
zones = self.config['zones']
zones = self._preprocess_zones(zones, eligible_sources)
if eligible_zones:
zones = IdnaDict({n: zones.get(n) for n in eligible_zones})
@@ -531,12 +602,10 @@ class Manager(object):
continue
lenient = config.get('lenient', False)
try:
sources = config['sources']
except KeyError:
raise ManagerException(
f'Zone {decoded_zone_name} is missing sources'
)
sources = self._get_sources(
decoded_zone_name, config, eligible_sources
)
try:
targets = config['targets']
@@ -547,9 +616,7 @@ class Manager(object):
processors = config.get('processors', [])
if eligible_sources and not [
s for s in sources if s in eligible_sources
]:
if not sources:
self.log.info('sync: no eligible sources, skipping')
continue
@@ -563,7 +630,7 @@ class Manager(object):
self.log.info('sync: no eligible targets, skipping')
continue
self.log.info('sync: sources=%s -> targets=%s', sources, targets)
self.log.info('sync: targets=%s', targets)
try:
collected = []
@@ -576,19 +643,6 @@ class Manager(object):
f'processor: {processor}'
)
try:
# rather than using a list comprehension, we break this loop
# out so that the `except` block below can reference the
# `source`
collected = []
for source in sources:
collected.append(self.providers[source])
sources = collected
except KeyError:
raise ManagerException(
f'Zone {decoded_zone_name}, unknown ' f'source: {source}'
)
try:
trgs = []
for target in targets:
@@ -798,18 +852,33 @@ class Manager(object):
clz = SplitYamlProvider
target = clz('dump', output_dir)
zone = self.get_zone(zone)
for source in sources:
source.populate(zone, lenient=lenient)
zones = self.config['zones']
zones = self._preprocess_zones(zones, sources=sources)
plan = target.plan(zone)
if plan is None:
plan = Plan(zone, zone, [], False)
target.apply(plan)
if '*' in zone:
# we want to do everything, just need the names though
zones = zones.keys()
else:
# we want to do a specific zone
zones = [zone]
for zone in zones:
zone = self.get_zone(zone)
for source in sources:
source.populate(zone, lenient=lenient)
plan = target.plan(zone)
if plan is None:
plan = Plan(zone, zone, [], False)
target.apply(plan)
def validate_configs(self, lenient=False):
# TODO: this code can probably be shared with stuff in sync
for zone_name, config in self.config['zones'].items():
zones = self.config['zones']
zones = self._preprocess_zones(zones)
for zone_name, config in zones.items():
decoded_zone_name = idna_decode(zone_name)
zone = self.get_zone(zone_name)

13
octodns/provider/yaml.py
View File

@@ -190,6 +190,12 @@ class YamlProvider(BaseProvider):
join(self.directory, f'{zone.name}yaml'),
)
def list_zones(self):
for filename in listdir(self.directory):
if not filename.endswith('.yaml') or filename.count('.') < 2:
continue
yield filename[:-4]
def populate(self, zone, target=False, lenient=False):
self.log.debug(
'populate: name=%s, target=%s, lenient=%s',
@@ -324,6 +330,13 @@ class SplitYamlProvider(YamlProvider):
filename = f'{zone.name[:-1]}{self.extension}'
return join(self.directory, filename)
def list_zones(self):
n = len(self.extension) - 1
for filename in listdir(self.directory):
if not filename.endswith(self.extension):
continue
yield filename[:-n]
def populate(self, zone, target=False, lenient=False):
self.log.debug(
'populate: name=%s, target=%s, lenient=%s',

21
tests/config/dynamic-config-no-list-zones.yaml Normal file
View File

@@ -0,0 +1,21 @@
providers:
in:
# does not support list_zones
class: helpers.SimpleProvider
dump:
class: octodns.provider.yaml.YamlProvider
directory: env/YAML_TMP_DIR
zones:
'*':
sources:
- in
targets:
- dump
subzone.unit.tests.:
sources:
- in
targets:
- dump

21
tests/config/dynamic-config.yaml Normal file
View File

@@ -0,0 +1,21 @@
providers:
in:
class: octodns.provider.yaml.YamlProvider
directory: tests/config
dump:
class: octodns.provider.yaml.YamlProvider
directory: env/YAML_TMP_DIR
zones:
'*':
sources:
- in
targets:
- dump
subzone.unit.tests.:
sources:
- in
targets:
- dump

2
tests/config/processors.yaml
View File

@@ -41,6 +41,6 @@ zones:
processors:
- doesnt-exist
sources:
- in
- config
targets:
- dump

0
tests/config/split/other.thing/.gitkeep Normal file
View File

50
tests/test_octodns_manager.py
View File

@@ -2,7 +2,7 @@
#
#
from os import environ
from os import environ, listdir
from os.path import dirname, isfile, join
from unittest import TestCase
from unittest.mock import MagicMock, patch
@@ -373,12 +373,28 @@ class TestManager(TestCase):
)
self.assertEqual('Unknown source: nope', str(ctx.exception))
# specific zone
manager.dump(
zone='unit.tests.',
output_dir=tmpdir.dirname,
split=True,
sources=['in'],
)
self.assertEqual(['unit.tests.'], listdir(tmpdir.dirname))
# all configured zones
manager.dump(
zone='*', output_dir=tmpdir.dirname, split=True, sources=['in']
)
self.assertEqual(
[
'empty.',
'sub.txt.unit.tests.',
'subzone.unit.tests.',
'unit.tests.',
],
sorted(listdir(tmpdir.dirname)),
)
# make sure this fails with an ManagerException and not a KeyError
# when trying to find sub zones
@@ -963,6 +979,38 @@ class TestManager(TestCase):
tc = manager.sync(dry_run=False)
self.assertEqual(26, tc)
def test_dynamic_config(self):
with TemporaryDirectory() as tmpdir:
environ['YAML_TMP_DIR'] = tmpdir.dirname
manager = Manager(get_config_filename('dynamic-config.yaml'))
# just unit.tests. which should have been dynamically configured via
# list_zones
self.assertEqual(
23, manager.sync(eligible_zones=['unit.tests.'], dry_run=False)
)
# just subzone.unit.tests. which was explicitly configured
self.assertEqual(
3,
manager.sync(
eligible_zones=['subzone.unit.tests.'], dry_run=False
),
)
# should sync everything across all zones, total of 32 records
self.assertEqual(32, manager.sync(dry_run=False))
def test_dynamic_config_unsupported_zone(self):
manager = Manager(
get_config_filename('dynamic-config-no-list-zones.yaml')
)
with self.assertRaises(ManagerException) as ctx:
manager.sync()
self.assertTrue('does not support `list_zones`' in str(ctx.exception))
class TestMainThreadExecutor(TestCase):
def test_success(self):

27
tests/test_octodns_provider_yaml.py
View File

@@ -295,6 +295,18 @@ xn--dj-kia8a:
self.assertTrue(source.supports(DummyType(source)))
self.assertTrue(source.supports(DummyType(self)))
def test_list_zones(self):
provider = YamlProvider('test', 'tests/config')
self.assertEqual(
[
'dynamic.tests.',
'sub.txt.unit.tests.',
'subzone.unit.tests.',
'unit.tests.',
],
sorted(provider.list_zones()),
)
class TestSplitYamlProvider(TestCase):
def test_list_all_yaml_files(self):
@@ -569,6 +581,21 @@ class TestSplitYamlProvider(TestCase):
)
self.assertEqual(source.supports_root_ns, copy.supports_root_ns)
def test_list_zones(self):
provider = SplitYamlProvider(
'test', 'tests/config/split', extension='.tst'
)
self.assertEqual(
[
'dynamic.tests.',
'empty.',
'subzone.unit.tests.',
'unit.tests.',
'unordered.',
],
sorted(provider.list_zones()),
)
class TestOverridingYamlProvider(TestCase):
def test_provider(self):