mirror/github-octodns
1
0
Fork 0
mirror of https://github.com/github/octodns.git synced 2024-05-11 05:55:00 +00:00
Code Issues Releases Activity

use PowerDNS for examples so we have full/real behavior

Browse Source
This commit is contained in:
Ross McFarland
2023-09-12 10:12:50 -07:00
parent bca5308995
commit 49c732ba95
15 changed files with 316 additions and 246 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
.gitignore vendored
View File

@ -10,6 +10,7 @@
coverage.xml coverage.xml
dist/ dist/
env/ env/
examples/migrating-to-octodns/config/
htmlcov/ htmlcov/
nosetests.xml nosetests.xml
octodns.egg-info/ octodns.egg-info/

18
examples/README.md
View File

@ -3,3 +3,21 @@ Examples:
* Getting started with a [basic octoDNS configuration](basic/) - new to octoDNS * Getting started with a [basic octoDNS configuration](basic/) - new to octoDNS
this is the place to start. It'll walk you through the main pieces of DNS IaC this is the place to start. It'll walk you through the main pieces of DNS IaC
with octoDNS including the process of planning and applying changes. with octoDNS including the process of planning and applying changes.
### Running PowerDNS
If you'd like to play around with running the examples in this directory
interactively you'll need a target for pushing data to.
[octodns-powerdns](https://github.com/octodns/octodns-powerdns) is the best
stand-alone option for this and the examples directory makes extensive use of
it. There is a [docker-compose.yml](docker-compose.yml) file that should get a
fully functional copy of PowerDNS backed my MySQL with the API enabled along
with other relivant functionality. For any of the examples that refer to the
local PowerDNS instance the following instructions below should get it up and
running.
1. If you haven't already [install docker compose](https://docs.docker.com/compose/install/)
1. If you don't already have a copy of octoDNS checked out run `git clone https://github.com/octodns/octodns.git`
1. In a seperate terminal window or tab
1. cd into the examples directory `cd octodns/examples`
1. Run docker-compose up `docker-compose up`, this will start up MySQL and PowerDNS running them in the foreground with their logs printing to the terminal

49
examples/basic/README.md
View File

@ -3,7 +3,8 @@
This is the starting point octoDNS config, it's pretty similar to what you This is the starting point octoDNS config, it's pretty similar to what you
might see for managing a set of personal domains or a small business. might see for managing a set of personal domains or a small business.
Most of the actual documentation for this example is in comments in the YAML configuration files. Most of the actual documentation for this example is found in the comments in
the YAML configuration files.
* [config/octodns.yaml](config/octodns.yaml) * [config/octodns.yaml](config/octodns.yaml)
* [config/my-domain.com.yaml](config/my-domain.com.yaml) * [config/my-domain.com.yaml](config/my-domain.com.yaml)
@ -21,10 +22,14 @@ to get a copy of the example files.
$ git clone https://github.com/octodns/octodns.git $ git clone https://github.com/octodns/octodns.git
$ cd octodns/examples/basic/ $ cd octodns/examples/basic/
$ python3 -mvenv env $ python3 -mvenv env
$ source ../env.sh
$ source env/bin/activate $ source env/bin/activate
(env) $ pip install -r requirements.txt (env) $ pip install -r requirements.txt
``` ```
Finally check out [Running PowerDNS](../README.md#running-powerdns) to get a local
instance of PowerDNS up and going before continuing.
## Running octoDNS the first time ## Running octoDNS the first time
Once you have your configuration files and octoDNS installed you're ready to Once you have your configuration files and octoDNS installed you're ready to
@ -54,26 +59,26 @@ run the sync command to get it to plan an initial set of changes.
2023-08-23T15:09:51 [4577488384] INFO Manager sync: targets=['yaml'] 2023-08-23T15:09:51 [4577488384] INFO Manager sync: targets=['yaml']
2023-08-23T15:09:51 [4577488384] INFO YamlProvider[config] populate: found 9 records, exists=False 2023-08-23T15:09:51 [4577488384] INFO YamlProvider[config] populate: found 9 records, exists=False
2023-08-23T15:09:51 [4577488384] INFO SpfSource[no-mail] populate: found 0 records, exists=False 2023-08-23T15:09:51 [4577488384] INFO SpfSource[no-mail] populate: found 0 records, exists=False
2023-08-23T15:09:51 [4577488384] INFO YamlProvider[yaml] plan: desired=my-domain.com. 2023-08-23T15:09:51 [4577488384] INFO PowerDnsProvider[powerdns] plan: desired=my-domain.com.
2023-08-23T15:09:51 [4577488384] WARNING YamlProvider[yaml] root NS record supported, but no record is configured for my-domain.com. 2023-08-23T15:09:51 [4577488384] WARNING PowerDnsProvider[powerdns] root NS record supported, but no record is configured for my-domain.com.
2023-08-23T15:09:51 [4577488384] INFO YamlProvider[yaml] plan: Creates=9, Updates=0, Deletes=0, Existing Records=0 2023-08-23T15:09:51 [4577488384] INFO PowerDnsProvider[powerdns] plan: Creates=9, Updates=0, Deletes=0, Existing Records=0
2023-08-23T15:09:51 [4577488384] INFO YamlProvider[config] populate: found 0 records, exists=False 2023-08-23T15:09:51 [4577488384] INFO YamlProvider[config] populate: found 0 records, exists=False
2023-08-23T15:09:51 [4577488384] INFO SpfSource[no-mail] populate: found 1 records, exists=False 2023-08-23T15:09:51 [4577488384] INFO SpfSource[no-mail] populate: found 1 records, exists=False
2023-08-23T15:09:51 [4577488384] INFO YamlProvider[yaml] plan: desired=unused-domain.io. 2023-08-23T15:09:51 [4577488384] INFO PowerDnsProvider[powerdns] plan: desired=unused-domain.io.
2023-08-23T15:09:51 [4577488384] WARNING YamlProvider[yaml] root NS record supported, but no record is configured for unused-domain.io. 2023-08-23T15:09:51 [4577488384] WARNING PowerDnsProvider[powerdns] root NS record supported, but no record is configured for unused-domain.io.
2023-08-23T15:09:51 [4577488384] INFO YamlProvider[yaml] plan: Creates=1, Updates=0, Deletes=0, Existing Records=0 2023-08-23T15:09:51 [4577488384] INFO PowerDnsProvider[powerdns] plan: Creates=1, Updates=0, Deletes=0, Existing Records=0
2023-08-23T15:09:51 [4577488384] INFO Plan 2023-08-23T15:09:51 [4577488384] INFO Plan
******************************************************************************** ********************************************************************************
* unused-domain.io. * unused-domain.io.
******************************************************************************** ********************************************************************************
* yaml (YamlProvider) * powerdns (PowerDnsProvider)
* Create Zone<unused-domain.io.> * Create Zone<unused-domain.io.>
* Create <TxtRecord TXT 3600, unused-domain.io., ['v=spf1 -all']> () * Create <TxtRecord TXT 3600, unused-domain.io., ['v=spf1 -all']> ()
* Summary: Creates=1, Updates=0, Deletes=0, Existing Records=0 * Summary: Creates=1, Updates=0, Deletes=0, Existing Records=0
******************************************************************************** ********************************************************************************
* my-domain.com. * my-domain.com.
******************************************************************************** ********************************************************************************
* yaml (YamlProvider) * powerdns (PowerDnsProvider)
* Create Zone<my-domain.com.> * Create Zone<my-domain.com.>
* Create <ARecord A 3600, my-domain.com., ['203.0.113.42', '203.0.113.43']> (config) * Create <ARecord A 3600, my-domain.com., ['203.0.113.42', '203.0.113.43']> (config)
* Create <AaaaRecord AAAA 3600, my-domain.com., ['2001:db8::44']> (config) * Create <AaaaRecord AAAA 3600, my-domain.com., ['2001:db8::44']> (config)
@ -102,7 +107,7 @@ support managing the root NS records, some just hard code their own name
servers. servers.
```console ```console
2023-08-23T15:09:51 [4577488384] WARNING YamlProvider[yaml] root NS record supported, but no record is configured for my-domain.com. 2023-08-23T15:09:51 [4577488384] WARNING PowerDnsProvider[powerdns] root NS record supported, but no record is configured for my-domain.com.
``` ```
### The plan output ### The plan output
@ -111,14 +116,14 @@ servers.
******************************************************************************** ********************************************************************************
* unused-domain.io. * unused-domain.io.
******************************************************************************** ********************************************************************************
* yaml (YamlProvider) * powerdns (PowerDnsProvider)
* Create Zone<unused-domain.io.> * Create Zone<unused-domain.io.>
* Create <TxtRecord TXT 3600, unused-domain.io., ['v=spf1 -all']> () * Create <TxtRecord TXT 3600, unused-domain.io., ['v=spf1 -all']> ()
* Summary: Creates=1, Updates=0, Deletes=0, Existing Records=0 * Summary: Creates=1, Updates=0, Deletes=0, Existing Records=0
******************************************************************************** ********************************************************************************
* my-domain.com. * my-domain.com.
******************************************************************************** ********************************************************************************
* yaml (YamlProvider) * powerdns (PowerDnsProvider)
* Create Zone<my-domain.com.> * Create Zone<my-domain.com.>
* Create <ARecord A 3600, my-domain.com., ['203.0.113.42', '203.0.113.43']> (config) * Create <ARecord A 3600, my-domain.com., ['203.0.113.42', '203.0.113.43']> (config)
* Create <AaaaRecord AAAA 3600, my-domain.com., ['2001:db8::44']> (config) * Create <AaaaRecord AAAA 3600, my-domain.com., ['2001:db8::44']> (config)
@ -141,14 +146,14 @@ servers.
******************************************************************************** ********************************************************************************
* unused-domain.io. * unused-domain.io.
******************************************************************************** ********************************************************************************
* yaml (YamlProvider) * powerdns (PowerDnsProvider)
* Create Zone<unused-domain.io.> * Create Zone<unused-domain.io.>
* Create <TxtRecord TXT 3600, unused-domain.io., ['v=spf1 -all']> () * Create <TxtRecord TXT 3600, unused-domain.io., ['v=spf1 -all']> ()
* Summary: Creates=1, Updates=0, Deletes=0, Existing Records=0 * Summary: Creates=1, Updates=0, Deletes=0, Existing Records=0
******************************************************************************** ********************************************************************************
* my-domain.com. * my-domain.com.
******************************************************************************** ********************************************************************************
* yaml (YamlProvider) * powerdns (PowerDnsProvider)
* Create Zone<my-domain.com.> * Create Zone<my-domain.com.>
* Create <ARecord A 3600, my-domain.com., ['203.0.113.42', '203.0.113.43']> (config) * Create <ARecord A 3600, my-domain.com., ['203.0.113.42', '203.0.113.43']> (config)
* Create <AaaaRecord AAAA 3600, my-domain.com., ['2001:db8::44']> (config) * Create <AaaaRecord AAAA 3600, my-domain.com., ['2001:db8::44']> (config)
@ -162,13 +167,13 @@ servers.
* Summary: Creates=9, Updates=0, Deletes=0, Existing Records=0 * Summary: Creates=9, Updates=0, Deletes=0, Existing Records=0
******************************************************************************** ********************************************************************************
2023-08-23T15:17:00 [4671815168] INFO YamlProvider[yaml] apply: making 1 changes to unused-domain.io. 2023-08-23T15:17:00 [4671815168] INFO PowerDnsProvider[powerdns] apply: making 1 changes to unused-domain.io.
2023-08-23T15:17:00 [4671815168] INFO YamlProvider[yaml] apply: making 9 changes to my-domain.com. 2023-08-23T15:17:00 [4671815168] INFO PowerDnsProvider[powerdns] apply: making 9 changes to my-domain.com.
2023-08-23T15:17:00 [4671815168] INFO Manager sync: 10 total changes 2023-08-23T15:17:00 [4671815168] INFO Manager sync: 10 total changes
``` ```
```console ```console
(env) coho:basic ross$ octodns-sync --config-file=config/octodns.yaml (env) $ octodns-sync --config-file=config/octodns.yaml
******************************************************************************** ********************************************************************************
No changes were planned No changes were planned
******************************************************************************** ********************************************************************************
@ -189,12 +194,12 @@ Just like during the first run section above we'll first run octoDNS to see what
We'd skim the log lines again looking for unexpected WARNINGS and then take a look at the changes. We'd skim the log lines again looking for unexpected WARNINGS and then take a look at the changes.
```console ```console
(env) coho:basic ross$ octodns-sync --config-file=config/octodns.yaml (env) $ octodns-sync --config-file=config/octodns.yaml
... ...
******************************************************************************** ********************************************************************************
* my-domain.com. * my-domain.com.
******************************************************************************** ********************************************************************************
* yaml (YamlProvider) * powerdns (PowerDnsProvider)
* Create Zone<my-domain.com.> * Create Zone<my-domain.com.>
* Update * Update
* <ARecord A 3600, my-domain.com., ['203.0.113.42', '203.0.113.43']> -> * <ARecord A 3600, my-domain.com., ['203.0.113.42', '203.0.113.43']> ->
@ -217,12 +222,12 @@ Here we only see the expected changes so we're good to move forward with
applying them. applying them.
```console ```console
(env) coho:basic ross$ octodns-sync --config-file=config/octodns.yaml --doit (env) $ octodns-sync --config-file=config/octodns.yaml --doit
... ...
******************************************************************************** ********************************************************************************
* my-domain.com. * my-domain.com.
******************************************************************************** ********************************************************************************
* yaml (YamlProvider) * powerdns (PowerDnsProvider)
* Create Zone<my-domain.com.> * Create Zone<my-domain.com.>
* Update * Update
* <ARecord A 3600, my-domain.com., ['203.0.113.42', '203.0.113.43']> -> * <ARecord A 3600, my-domain.com., ['203.0.113.42', '203.0.113.43']> ->
@ -237,7 +242,7 @@ applying them.
If we want we can run another plan to make sure there are no further pending changes. If we want we can run another plan to make sure there are no further pending changes.
```console ```console
(env) coho:basic ross$ octodns-sync --config-file=config/octodns.yaml (env) $ octodns-sync --config-file=config/octodns.yaml
******************************************************************************** ********************************************************************************
No changes were planned No changes were planned
******************************************************************************** ********************************************************************************

30
examples/basic/config/octodns.yaml
View File

@ -18,22 +18,16 @@ providers:
no-mail: no-mail:
class: octodns_spf.SpfSource class: octodns_spf.SpfSource
# This is just a placeholder for the purposes of this example, in your # PowerDNS serves as a good option for a locally runnable standalone provider
# configuration you would replace it with configuration for the DNS # to use in our examples. See the top-level example README for information on
# provider(s) you wish to use, e.g. Route53, NS1, Azure DNS, ... See # how to run it and the documentation for octodns-powerdns for documentation
# https://github.com/octodns/octodns#providers for a list of supported # about the details of the config below.
# providers and have a look in each provider's repo for details on installing powerdns:
# and configuring the provider. Convention is to name the provider with the class: octodns_powerdns.PowerDnsProvider
# unique portion of the provider class/company name in all lowercase, here host: 127.0.0.1
# YamlProvider -> yaml. Route53 would be route53, NS1 -> ns1, etc. port: 8081
yaml: api_key: env/POWERDNS_API_KEY
class: octodns.provider.yaml.YamlProvider timeout: 10
directory: ./target
# This is a rarely used option that tells the YamlProvider that it should
# load existing records from disk rather than assume nothing exists
# created every a plan is made.
# TODO:
#populate_existing: true
# If you wish to push your record data to multiple providers you'd define the # If you wish to push your record data to multiple providers you'd define the
# other here and include them in the targets list down below. See the # other here and include them in the targets list down below. See the
@ -60,5 +54,5 @@ zones:
- no-mail - no-mail
# This is the place(s) we push the record data to when applying changes # This is the place(s) we push the record data to when applying changes
targets: targets:
# Here we're pushing things to our example provider. # Here we're pushing things to our local PowerDNS instance
- yaml - powerdns

1
examples/basic/requirements.txt
View File

@ -1,2 +1,3 @@
octodns>=1.0.1 octodns>=1.0.1
octodns_powerdns>=0.0.5
octodns_spf>=0.0.2 octodns_spf>=0.0.2

29
examples/docker-compose.yml Normal file
View File

@ -0,0 +1,29 @@
version: "3.9" # optional since v1.27.0
volumes:
powerdns-db-data:
name: powerdns-db-data
services:
db:
image: mariadb:10.1
ports:
- "3306:3306"
environment:
MYSQL_ROOT_PASSWORD: l3tmein
volumes:
- powerdns-db-data:/var/lib/mysql
powerdns:
image: psitrax/powerdns
ports:
- "53:53/tcp"
- "53:53/udp"
- "8081:8081"
environment:
MYSQL_HOST: db
MYSQL_PORT: 3306
MYSQL_USER: root
MYSQL_PASS: l3tmein
depends_on:
- db
command: --api=yes --api-key=its@secret --loglevel=99 --webserver=yes --webserver-address=0.0.0.0 --webserver-allow-from=0.0.0.0/0 --webserver-password=its@secret --webserver-port=8081 --enable-lua-records=shared --edns-subnet-processing=yes

1
examples/env.sh Normal file
View File

@ -0,0 +1 @@
export POWERDNS_API_KEY="its@secret"

258
examples/migrating-to-octodns/README.md
View File

@ -3,7 +3,8 @@
Importing an existing DNS setup into octoDNS management is a very Importing an existing DNS setup into octoDNS management is a very
straightforward process and can generally be completed in minutes. straightforward process and can generally be completed in minutes.
Some relevant documentation for this example is in comments in the YAML configuration files. Some relevant documentation for this example is in comments in the YAML
configuration files.
* [config/octodns.yaml](config/octodns.yaml) * [config/octodns.yaml](config/octodns.yaml)
* [config/my-domain.com.yaml](config/my-domain.com.yaml) * [config/my-domain.com.yaml](config/my-domain.com.yaml)
@ -22,6 +23,7 @@ to get a copy of the example files.
$ git clone https://github.com/octodns/octodns.git $ git clone https://github.com/octodns/octodns.git
$ cd octodns/examples/basic/ $ cd octodns/examples/basic/
$ python3 -mvenv env $ python3 -mvenv env
$ source ../env.sh
$ source env/bin/activate $ source env/bin/activate
(env) $ pip install -r requirements.txt (env) $ pip install -r requirements.txt
``` ```
@ -44,6 +46,22 @@ $ git commit -m "Importting existing records into octoDNS"
$ git push -u origin main $ git push -u origin main
``` ```
## Populating some data that we can later dump
We need zone & record data in our provider in order to have something to dump
out into a YAML config, thus our first step is to populate PowerDNS. This is
just to have something to work with the the actual process that begins in the
next step. It's not something you'd normally do when migrating to octoDNS.
Step 0 is to get a local PowerDNS instance running.
Check out out [Running PowerDNS](../README.md#running-powerdns) for info on
starting that up. Once you've done that run the following. You can ignore the
output and move on to the next step.
```console
(env) $ octodns-sync --config-file=populate/octodns.yaml --doit
```
## Running octodns-dump ## Running octodns-dump
Once you have your configuration files and octoDNS installed you're ready to Once you have your configuration files and octoDNS installed you're ready to
@ -52,52 +70,61 @@ dump your zone configs. Here we've assumed that the provider being used supports
[dynamic-zone-config](../dynamic-zone-config) for details on how to explicitly [dynamic-zone-config](../dynamic-zone-config) for details on how to explicitly
list your zones in the config file. list your zones in the config file.
The quoted * below is what tells octodns-dump to dynamically source the list of We first tell octodns-dump where to find our config file. We then tell it that
zones and dump config files for all of them. If you would only like to dump a we want to use the output provider defined in that file named `config` rather
specific zone you can replace it with the zone name, e.g. `my-domain.com.`, than having a default one auto-created. This would allow us to customize the
making sure to include the trailing `.` details of the provider, e.g. things like the `default_ttl`.
We'll then tell dump where we would like it to write our zone files, in this
case we want it to put them into `config` provider's directory `./config`.
The quoted * tells octodns-dump to dynamically source the list of zones and
dump config files for all of them. If you would only like to dump a specific
zone you can replace it with the zone name, e.g. `my-domain.com.`, making sure
to include the trailing `.`
The final argument passed to `octodns-dump` is the source of data you'd like to
dump. Here that's our existing provider `powerdns`.
Note that if you're previously run other examples you may see entries in the
log output for those zones in addition to the ones we populated for this
exercise. They shouldn't cause any problems and can be ignored.
```console ```console
(env) coho:migrating-to-octodns ross$ octodns-dump --config-file=config/octodns.yaml --output-dir config/ '*' some-provider (env) $ octodns-dump --config-file=config/octodns.yaml --output-provider config --output-dir ./config '*' powerdns
2023-08-25T10:36:00 [4413005312] INFO Manager __init__: config_file=config/octodns.yaml, (octoDNS 1.0.0+6e7f036b) 2023-09-12T09:39:59 [4580544000] INFO Manager __init__: config_file=config/octodns.yaml, (octoDNS 1.0.0+bca53089)
2023-08-25T10:36:00 [4413005312] INFO Manager _config_executor: max_workers=1 2023-09-12T09:39:59 [4580544000] INFO Manager _config_executor: max_workers=1
2023-08-25T10:36:00 [4413005312] INFO Manager _config_include_meta: include_meta=False 2023-09-12T09:39:59 [4580544000] INFO Manager _config_include_meta: include_meta=False
2023-08-25T10:36:00 [4413005312] INFO Manager _config_auto_arpa: auto_arpa=False 2023-09-12T09:39:59 [4580544000] INFO Manager _config_auto_arpa: auto_arpa=False
2023-08-25T10:36:00 [4413005312] INFO Manager __init__: global_processors=[] 2023-09-12T09:39:59 [4580544000] INFO Manager __init__: global_processors=[]
2023-08-25T10:36:00 [4413005312] INFO Manager __init__: provider=config (octodns.provider.yaml 1.0.0+6e7f036b) 2023-09-12T09:39:59 [4580544000] INFO Manager __init__: provider=config (octodns.provider.yaml 1.0.0+bca53089)
2023-08-25T10:36:00 [4413005312] INFO Manager __init__: provider=some-provider (octodns.provider.yaml 1.0.0+6e7f036b) 2023-09-12T09:40:00 [4580544000] INFO Manager __init__: provider=powerdns (octodns_powerdns 0.0.4+3189e9c2)
2023-08-25T10:36:00 [4413005312] INFO Manager dump: zone=*, output_dir=config/, output_provider=None, lenient=False, split=False, sources=['some-provider'] 2023-09-12T09:40:00 [4580544000] INFO Manager dump: zone=*, output_dir=./config, output_provider=config, lenient=False, split=False, sources=['powerdns']
2023-08-25T10:36:00 [4413005312] INFO Manager dump: using custom YamlProvider 2023-09-12T09:40:00 [4580544000] INFO Manager dump: using specified output_provider=config
2023-08-25T10:36:00 [4413005312] INFO Manager sync: dynamic zone=*, sources=[YamlProvider] 2023-09-12T09:40:00 [4580544000] INFO Manager sync: dynamic zone=*, sources=[PowerDnsProvider]
2023-08-25T10:36:00 [4413005312] INFO Manager sync: adding dynamic zone=my-domain.com. 2023-09-12T09:40:00 [4580544000] INFO Manager sync: adding dynamic zone=my-dumpable.com.
2023-08-25T10:36:00 [4413005312] INFO Manager sync: adding dynamic zone=unused-domain.io. 2023-09-12T09:40:00 [4580544000] INFO Manager sync: adding dynamic zone=unused-domain.io.
Traceback (most recent call last): Traceback (most recent call last):
File "/Users/ross/octodns/octodns/examples/migrating-to-octodns/env/bin/octodns-dump", line 8, in <module> File "/Users/ross/octodns/octodns/examples/migrating-to-octodns/env/bin/octodns-dump", line 8, in <module>
sys.exit(main()) sys.exit(main())
^^^^^^ ^^^^^^
File "/Users/ross/octodns/octodns/octodns/cmds/dump.py", line 51, in main File "/Users/ross/octodns/octodns/examples/migrating-to-octodns/env/lib/python3.11/site-packages/octodns/cmds/dump.py", line 51, in main
manager.dump( manager.dump(
File "/Users/ross/octodns/octodns/octodns/manager.py", line 868, in dump File "/Users/ross/octodns/octodns/examples/migrating-to-octodns/env/lib/python3.11/site-packages/octodns/manager.py", line 868, in dump
source.populate(zone, lenient=lenient) source.populate(zone, lenient=lenient)
File "/Users/ross/octodns/octodns/octodns/provider/yaml.py", line 229, in populate File "/Users/ross/octodns/octodns/examples/migrating-to-octodns/env/lib/python3.11/site-packages/octodns_powerdns/__init__.py", line 433, in populate
self._populate_from_file(filename, zone, lenient)
File "/Users/ross/octodns/octodns/octodns/provider/yaml.py", line 175, in _populate_from_file
record = Record.new( record = Record.new(
^^^^^^^^^^^ ^^^^^^^^^^^
File "/Users/ross/octodns/octodns/octodns/record/base.py", line 76, in new File "/Users/ross/octodns/octodns/examples/migrating-to-octodns/env/lib/python3.11/site-packages/octodns/record/base.py", line 76, in new
raise ValidationError(fqdn, reasons, context) raise ValidationError(fqdn, reasons, context)
octodns.record.exception.ValidationError: Invalid record "my-domain.com.", ./target/my-domain.com.yaml, line 17, column 5 octodns.record.exception.ValidationError: Invalid record "sshfp.my-dumpable.com."
- NS value "ns1.some-provider.com" missing trailing . - unrecognized algorithm "42"
- NS value "ns2.some-provider.com" missing trailing . - unrecognized algorithm "43"
- NS value "ns3.some-provider.com" missing trailing .
- NS value "ns4.some-provider.com" missing trailing .
``` ```
We've intentionally configured data in the provider without following We've intentionally configured data in the provider without following
(octoDNS's) best practices as it's common to run across these sorts of things (octoDNS's) best practices as it's common to run across these sorts of things
when migrating. In this case the NS values are missing their trailing `.`, when migrating. In this case there is a SSHFP record with invalid `algorithm`s.
`ns1.some-provider.com` should be `ns1.some-provider.com.`.
Regardless of whether or not you hit errors it is important to carefully look Regardless of whether or not you hit errors it is important to carefully look
over the output of octodns-dump to ensure that it was able to make sense of and over the output of octodns-dump to ensure that it was able to make sense of and
@ -112,76 +139,86 @@ this situation see [Dynamic Records](/docs/dynamic_records.md).
## Running octodns-dump again, now with --lenient ## Running octodns-dump again, now with --lenient
Back to our example, the missing `.`s aren't a huge deal and octoDNS can be Back to our example, the invalid `algorithm`s aren't a huge deal and octoDNS
asked to ignore them with the `--lenient` argument. can be asked to ignore them with the `--lenient` argument.
```console ```console
(env) coho:migrating-to-octodns ross$ octodns-dump --config-file=config/octodns.yaml --output-dir config/ '*' some-provider --lenient (env) $ octodns-dump --config-file=config/octodns.yaml --output-provider config --output-dir ./config '*' powerdns --lenient
2023-08-25T10:43:26 [4528057856] INFO Manager __init__: config_file=config/octodns.yaml, (octoDNS 1.0.0+6e7f036b) 2023-09-12T09:47:48 [4566433280] INFO Manager __init__: config_file=config/octodns.yaml, (octoDNS 1.0.0+bca53089)
2023-08-25T10:43:26 [4528057856] INFO Manager _config_executor: max_workers=1 2023-09-12T09:47:48 [4566433280] INFO Manager _config_executor: max_workers=1
2023-08-25T10:43:26 [4528057856] INFO Manager _config_include_meta: include_meta=False 2023-09-12T09:47:48 [4566433280] INFO Manager _config_include_meta: include_meta=False
2023-08-25T10:43:26 [4528057856] INFO Manager _config_auto_arpa: auto_arpa=False 2023-09-12T09:47:48 [4566433280] INFO Manager _config_auto_arpa: auto_arpa=False
2023-08-25T10:43:26 [4528057856] INFO Manager __init__: global_processors=[] 2023-09-12T09:47:48 [4566433280] INFO Manager __init__: global_processors=[]
2023-08-25T10:43:26 [4528057856] INFO Manager __init__: provider=config (octodns.provider.yaml 1.0.0+6e7f036b) 2023-09-12T09:47:48 [4566433280] INFO Manager __init__: provider=config (octodns.provider.yaml 1.0.0+bca53089)
2023-08-25T10:43:26 [4528057856] INFO Manager __init__: provider=some-provider (octodns.provider.yaml 1.0.0+6e7f036b) 2023-09-12T09:47:48 [4566433280] INFO Manager __init__: provider=powerdns (octodns_powerdns 0.0.4+3189e9c2)
2023-08-25T10:43:26 [4528057856] INFO Manager dump: zone=*, output_dir=config/, output_provider=None, lenient=True, split=False, sources=['some-provider'] 2023-09-12T09:47:48 [4566433280] INFO Manager dump: zone=*, output_dir=./config, output_provider=config, lenient=True, split=False, sources=['powerdns']
2023-08-25T10:43:26 [4528057856] INFO Manager dump: using custom YamlProvider 2023-09-12T09:47:48 [4566433280] INFO Manager dump: using specified output_provider=config
2023-08-25T10:43:26 [4528057856] INFO Manager sync: dynamic zone=*, sources=[YamlProvider] 2023-09-12T09:47:48 [4566433280] INFO Manager sync: dynamic zone=*, sources=[PowerDnsProvider]
2023-08-25T10:43:26 [4528057856] INFO Manager sync: adding dynamic zone=my-domain.com. 2023-09-12T09:47:48 [4566433280] INFO Manager sync: adding dynamic zone=my-dumpable.com.
2023-08-25T10:43:26 [4528057856] INFO Manager sync: adding dynamic zone=unused-domain.io. 2023-09-12T09:47:48 [4566433280] INFO Manager sync: adding dynamic zone=unused-dumpable.com.
2023-08-25T10:43:26 [4528057856] WARNING Record Invalid record "my-domain.com.", ./target/my-domain.com.yaml, line 17, column 5 2023-09-12T09:47:48 [4566433280] WARNING Record Invalid record "sshfp.my-dumpable.com."
- NS value "ns1.some-provider.com" missing trailing . - unrecognized algorithm "42"
- NS value "ns2.some-provider.com" missing trailing . - unrecognized algorithm "43"
- NS value "ns3.some-provider.com" missing trailing . 2023-09-12T09:47:48 [4566433280] INFO PowerDnsProvider[powerdns] populate: found 8 records, exists=True
- NS value "ns4.some-provider.com" missing trailing . 2023-09-12T09:47:48 [4566433280] INFO YamlProvider[config] plan: desired=my-dumpable.com.
2023-08-25T10:43:26 [4528057856] INFO YamlProvider[some-provider] populate: found 10 records, exists=False 2023-09-12T09:47:48 [4566433280] INFO YamlProvider[config] plan: Creates=8, Updates=0, Deletes=0, Existing Records=0
2023-08-25T10:43:26 [4528057856] INFO YamlProvider[dump] plan: desired=my-domain.com. 2023-09-12T09:47:48 [4566433280] INFO YamlProvider[config] apply: making 8 changes to my-dumpable.com.
2023-08-25T10:43:26 [4528057856] INFO YamlProvider[dump] plan: Creates=10, Updates=0, Deletes=0, Existing Records=0 2023-09-12T09:47:48 [4566433280] INFO PowerDnsProvider[powerdns] populate: found 1 records, exists=True
2023-08-25T10:43:26 [4528057856] INFO YamlProvider[dump] apply: making 10 changes to my-domain.com. 2023-09-12T09:47:49 [4566433280] INFO YamlProvider[config] plan: desired=unused-dumpable.com.
2023-08-25T10:43:26 [4528057856] INFO YamlProvider[some-provider] populate: found 0 records, exists=False 2023-09-12T09:47:49 [4566433280] WARNING YamlProvider[config] root NS record supported, but no record is configured for unused-dumpable.com.
2023-08-25T10:43:26 [4528057856] INFO YamlProvider[dump] plan: desired=unused-domain.io. 2023-09-12T09:47:49 [4566433280] INFO YamlProvider[config] plan: Creates=1, Updates=0, Deletes=0, Existing Records=0
2023-08-25T10:43:26 [4528057856] WARNING YamlProvider[dump] root NS record supported, but no record is configured for unused-domain.io. 2023-09-12T09:47:49 [4566433280] INFO YamlProvider[config] apply: making 1 changes to unused-dumpable.com.
2023-08-25T10:43:26 [4528057856] INFO YamlProvider[dump] plan: No changes
2023-08-25T10:43:26 [4528057856] INFO YamlProvider[dump] apply: making 0 changes to unused-domain.io.
``` ```
With `--lenient` we now see a warning about the validation problems, but
octoDNS does its best to continue anyway.
## Examining the results ## Examining the results
We can now take a look in the config/ directory to see the created zone files We can now take a look in the config/ directory to see the created zone files
along side our main config. along side our main config.
```console ```console
(env) coho:migrating-to-octodns ross$ ls -1 config/ (env) ls -1 config/
my-domain.com.yaml my-dumpable.com.yaml
octodns.yaml octodns.yaml
unused-domain.io.yaml unused-dumpable.com.yaml
``` ```
If you open up the my-domain.com.yaml file you'll note that the root NS record values are missing Again note if you're run through other examples you may see more files for
their trailing `.`s. octoDNS has done its best to dump things as they currently their zones in this directory. They can be ignored for the purpose of the
are in the provider. That means that if you tried to generate a plan with example, but they will be imported and managed nonetheless.
`octodns-sync` now it would fail b/c the configuration doesn't follow best
practices. If you open up the my-dumpable.com.yaml file you'll note that the SSHFP
record's values have 42 and 43 for their algorithm field.
With those values if you tried to generate a plan with `octodns-sync` now it
would fail with validation errors.
When migrating it's recommended to make as few changes to your config as When migrating it's recommended to make as few changes to your config as
possible initially, that is you want to bring the setup as-is under octoDNS possible initially, that is you want to bring the setup as-is under octoDNS
management and avoid churn as much as possible. This is the safest approach, management. This is the safest approach, change as little as possible and
change as little as possible and incrementally work towards having a fully incrementally work towards having a fully managed and compliant/best practice
managed and compliant/best practice config. config.
So for now we'll enable `lenient` on the root NS record. For more details on So for now we'll enable `lenient` on the SSHFP record by editing
see the [lenience](../lenience/) example. my-dumpable.com.yaml adding `octodns.lenient = true` as shown below. For more
details on see [lenience](/docs/records.md#lenience).
```yaml ```yaml
... ...
- octodns: sshpf:
lenient: true # TODO: look into bringing this record into compliance
type: NS octodns:
values: lenient: true
- ns1.some-provider.com type: SSHFP
- ns2.some-provider.com values:
- ns3.some-provider.com - algorithm: 42
- ns4.some-provider.com fingerprint: abcdef1234567890
fingerprint_type: 1
- algorithm: 43
fingerprint: abcdef1234567890
fingerprint_type: 1
... ...
``` ```
@ -192,53 +229,30 @@ we've looked closely at the log output zone configuration files we have a pretty
good idea the answer is no. good idea the answer is no.
```console ```console
(env) coho:migrating-to-octodns ross$ octodns-sync --config-file=config/octodns.yaml (env) $ octodns-sync --config-file=config/octodns.yaml
2023-08-25T11:07:44 [4644843008] INFO Manager __init__: config_file=config/octodns.yaml, (octoDNS 1.0.0+6e7f036b) ...
2023-08-25T11:07:44 [4644843008] INFO Manager _config_executor: max_workers=1 2023-09-12T10:04:27 [4608876032] INFO PowerDnsProvider[powerdns] plan: desired=unused-dumpable.com.
2023-08-25T11:07:44 [4644843008] INFO Manager _config_include_meta: include_meta=False 2023-09-12T10:04:27 [4608876032] INFO PowerDnsProvider[powerdns] populate: found 1 records, exists=True
2023-08-25T11:07:44 [4644843008] INFO Manager _config_auto_arpa: auto_arpa=False 2023-09-12T10:04:27 [4608876032] WARNING PowerDnsProvider[powerdns] root NS record supported, but no record is configured for unused-dumpable.com.
2023-08-25T11:07:44 [4644843008] INFO Manager __init__: global_processors=[] 2023-09-12T10:04:27 [4608876032] INFO PowerDnsProvider[powerdns] plan: No changes
2023-08-25T11:07:44 [4644843008] INFO Manager __init__: provider=config (octodns.provider.yaml 1.0.0+6e7f036b) 2023-09-12T10:04:27 [4608876032] INFO YamlProvider[config] populate: found 5 records, exists=False
2023-08-25T11:07:44 [4644843008] INFO Manager __init__: provider=some-provider (octodns.provider.yaml 1.0.0+6e7f036b) 2023-09-12T10:04:27 [4608876032] INFO PowerDnsProvider[powerdns] plan: desired=exxampled.com.
2023-08-25T11:07:44 [4644843008] INFO Manager sync: eligible_zones=[], eligible_targets=[], dry_run=True, force=False, plan_output_fh=<stdout> 2023-09-12T10:04:27 [4608876032] INFO PowerDnsProvider[powerdns] populate: found 5 records, exists=True
2023-08-25T11:07:44 [4644843008] INFO Manager sync: sources=['config'] 2023-09-12T10:04:27 [4608876032] INFO PowerDnsProvider[powerdns] plan: No changes
2023-08-25T11:07:44 [4644843008] INFO Manager sync: dynamic zone=*, sources=[YamlProvider] 2023-09-12T10:04:27 [4608876032] INFO Plan
2023-08-25T11:07:44 [4644843008] INFO Manager sync: adding dynamic zone=my-domain.com.
2023-08-25T11:07:44 [4644843008] INFO Manager sync: adding dynamic zone=unused-domain.io.
2023-08-25T11:07:44 [4644843008] INFO Manager sync: zone=my-domain.com.
2023-08-25T11:07:44 [4644843008] INFO Manager sync: sources=['config']
2023-08-25T11:07:44 [4644843008] INFO Manager sync: targets=['some-provider']
2023-08-25T11:07:44 [4644843008] INFO Manager sync: zone=unused-domain.io.
2023-08-25T11:07:44 [4644843008] INFO Manager sync: sources=['config']
2023-08-25T11:07:44 [4644843008] INFO Manager sync: targets=['some-provider']
2023-08-25T11:07:44 [4644843008] WARNING Record Invalid record "my-domain.com.", ./config/my-domain.com.yaml, line 9, column 5
- NS value "ns1.some-provider.com" missing trailing .
- NS value "ns2.some-provider.com" missing trailing .
- NS value "ns3.some-provider.com" missing trailing .
- NS value "ns4.some-provider.com" missing trailing .
2023-08-25T11:07:44 [4644843008] INFO YamlProvider[config] populate: found 10 records, exists=False
2023-08-25T11:07:44 [4644843008] INFO YamlProvider[some-provider] plan: desired=my-domain.com.
2023-08-25T11:07:44 [4644843008] WARNING Record Invalid record "my-domain.com.", ./target/my-domain.com.yaml, line 17, column 5
- NS value "ns1.some-provider.com" missing trailing .
- NS value "ns2.some-provider.com" missing trailing .
- NS value "ns3.some-provider.com" missing trailing .
- NS value "ns4.some-provider.com" missing trailing .
2023-08-25T11:07:44 [4644843008] INFO YamlProvider[some-provider] populate: found 10 records, exists=False
2023-08-25T11:07:44 [4644843008] INFO YamlProvider[some-provider] plan: No changes
2023-08-25T11:07:44 [4644843008] INFO YamlProvider[config] populate: found 0 records, exists=False
2023-08-25T11:07:44 [4644843008] INFO YamlProvider[some-provider] plan: desired=unused-domain.io.
2023-08-25T11:07:44 [4644843008] INFO YamlProvider[some-provider] populate: found 0 records, exists=False
2023-08-25T11:07:44 [4644843008] WARNING YamlProvider[some-provider] root NS record supported, but no record is configured for unused-domain.io.
2023-08-25T11:07:44 [4644843008] INFO YamlProvider[some-provider] plan: No changes
2023-08-25T11:07:44 [4644843008] INFO Plan
******************************************************************************** ********************************************************************************
No changes were planned No changes were planned
******************************************************************************** ********************************************************************************
``` ```
Most of the output above is omitted, this is the final few lines showing that
no changes need to be made to get `powerdns` to match `config`. It does have a
warning about the lack of root NS records in `unused-dumpable.com.`, but we'll
ignore that in this example.
If you had unsupported record types or advanced features in use this may not be If you had unsupported record types or advanced features in use this may not be
the case. It's completely safe to generate a plan, octoDNS won't make any the case and you may see changes. It's completely safe to generate a plan,
changes. octoDNS won't make any of the changes listed.
If you see things in the plan output section it's time to triple check and make If you see things in the plan output section it's time to triple check and make
sure they're, hopefully minimal, modifications your OK with making. sure they're, hopefully minimal, modifications your OK with making.

36
examples/migrating-to-octodns/config/octodns.yaml
View File

@ -1,31 +1,31 @@
--- ---
providers: providers:
# See the basic example for descriptions of the bulk of this config file, here # See the basic example for descriptions of the basics of this config file,
# we'll focus on things related to octodns-dump # here we'll focus on things related to octodns-dump.
# This is our config provider, it sources YAML from disk, but to start we
# don't have any files in the ./config directory. We need to dump what's in
# our provider, in this case a local PowerDNS setup, into this config
# directory as a starting point.
config: config:
class: octodns.provider.yaml.YamlProvider class: octodns.provider.yaml.YamlProvider
directory: ./config directory: ./config
# In a real setup this would be your existing DNS provider where all of your # This is our existing provider, the one we want to bring under octoDNS
# zones are configured. E.g. Route53, Azure, Cloudflare, NS1, ... Here it's # management. This example will walk through dumping the existing data in
# using a second YamlProvider so that the example can be self contained and # this provider into the config provider's directory above. At that point we
# will work for anyone. For information on how to configure your provider see # should be ready to start managing things with octoDNS going forward.
# its README.md powerdns:
some-provider: class: octodns_powerdns.PowerDnsProvider
class: octodns.provider.yaml.YamlProvider host: 127.0.0.1
directory: ./target port: 8081
# This is a rarely used option that tells the YamlProvider that it should api_key: env/POWERDNS_API_KEY
# load existing records from disk rather than assume nothing exists timeout: 10
# created every a plan is made.
# TODO:
#populate_existing: true
zones: zones:
# We're using dynamic zone config to dynamically build the list of configured
# zones using the sources & targets below for each.
'*': '*':
sources: sources:
- config - config
targets: targets:
- some-provider - powerdns

41
examples/migrating-to-octodns/populate/my-dumpable.com.yaml Normal file
View File

@ -0,0 +1,41 @@
---
# Just some records so we'll have something to dump, see the basic example for
# documentation/descriptions of what's in here.
'':
- type: A
values:
- 203.0.113.42
- 203.0.113.43
- type: AAAA
values: &WEB_AAAA_VALUES
- 2001:DB8::44
- type: NS
values:
- ns1.some-provider.com.
- ns2.some-provider.com.
- ns3.some-provider.com.
- ns4.some-provider.com.
- type: TXT
values:
- some-verification=3becb991-932f-4433-a280-9df6f39b6194
- z-other-thing=this proves i have control over this domain
'*':
- type: A
value: 203.0.113.45
- type: AAAA
value: 2001:DB8::46
sshfp:
type: SSHFP
values:
# These values won't pass validation. They'll be used to walk through the
# process of safely bringing an existing config up to octoDNS's
# recommended best practices.
- algorithm: 42
fingerprint: abcdef1234567890
fingerprint_type: 1
- algorithm: 43
fingerprint: abcdef1234567890
fingerprint_type: 1
www:
- type: CNAME
value: my-dumpable.com.

26
examples/migrating-to-octodns/populate/octodns.yaml Normal file
View File

@ -0,0 +1,26 @@
---
providers:
# This is just a quick and easy way to load some data into PowerDNS that we
# can dump while working through the example. See the basic example for
# documentation of what's in here
config:
class: octodns.provider.yaml.YamlProvider
directory: ./populate
powerdns:
class: octodns_powerdns.PowerDnsProvider
host: 127.0.0.1
port: 8081
api_key: env/POWERDNS_API_KEY
timeout: 10
zones:
'*':
# we're disabling strict validation here so we can load up data that won't
# meet octoDNS's best practices so that we can show
lenient: true
sources:
- config
targets:
- powerdns

5
examples/migrating-to-octodns/populate/unused-dumpable.com.yaml Normal file
View File

@ -0,0 +1,5 @@
---
'':
type: TXT
values:
- v=spf1 -all

1
examples/migrating-to-octodns/requirements.txt
View File

@ -1,2 +1,3 @@
octodns>=1.0.1 octodns>=1.0.1
octodns_powerdns>=0.0.5
octodns_spf>=0.0.2 octodns_spf>=0.0.2

64
examples/migrating-to-octodns/target/my-domain.com.yaml
View File

@ -1,64 +0,0 @@
---
# These are root/APEX records, things that live at the top-level of the zone.
# There are multiple records so there's a YAML list, each element in that list
# defines its own record.
'':
# We first have an A record, mapping A my-domain.com to two IPv4 addresses, we
# also use a YAML anchor to effectively store the values for later use.
- type: A
values: &WEB_A_VALUES
- 203.0.113.42
- 203.0.113.43
# Similar to the A above, this is a AAAA for my-domain.com with a single IPv6
# addresse.
- type: AAAA
values: &WEB_AAAA_VALUES
- 2001:DB8::44
- type: NS
values:
- ns1.some-provider.com
- ns2.some-provider.com
- ns3.some-provider.com
- ns4.some-provider.com
# Finally it's common to have to prove ownership of a domain with a TXT value
# in an APEX record. The following are a couple made up examples of how that
# would be done. As you'll see when you plan things The SpfSource will merge
# its SPF value in with the values defined here
- type: TXT
values:
- some-verification=3becb991-932f-4433-a280-9df6f39b6194
- z-other-thing=this proves i have control over this domain
# This is a wildcard record, any A or AAAA query not matching a more specific
# answer defined in this file will match this name and receive the associated
# value in response.
'*':
- type: A
# Note here we have a single value rather than values and use `value`.
value: 203.0.113.45
- type: AAAA
value: 2001:DB8::46
# Note that the records in this zone are sorted, by default YamlProvider
# enforces alphabetical sorting of records and even the keys within record
# data. This is an organizational best practice, but can be disabled with the
# enforce_order parameter to YamlProvider
NADCBiQKBgQ._companyname:
type: TXT
value: a-different-proof-of-ownership
# We want www and the APEX to return the same values for A/AAAA queries. We
# stored the values earlier in YAML anchors so we can use them now on.
# This is a CNAME record
pointer:
type: CNAME
# CNAMEs can only have a single value so they require `value` to be used. As a
# best practice the target name must end with a `.` If you have a specific
# case where you need to omit the `.`, or other enforced best practices, see
# the lenience example for more information.
value: look.over-here.net.
# This is an example of a record with a `.` in it's name, it would result for a
# TXT query of NADCBiQKBgQ._companyname.my-domain.com. This specific example is
# another commonly required proof of overship record.
www:
- type: A
values: *WEB_A_VALUES
- type: AAAA
values: *WEB_AAAA_VALUES

2
examples/migrating-to-octodns/target/unused-domain.io.yaml
View File

@ -1,2 +0,0 @@
---
# Nothing in here, just an empty domain b/c it's not actually used for anything