Merge branch 'master' into configurable-geo-healthcheck

.gitignore

@@ -1,6 +1,7 @@
 *.pyc
 .coverage
 .env
+/config/
 coverage.xml
 dist/
 env/
@@ -9,5 +10,3 @@ nosetests.xml
 octodns.egg-info/
 output/
 tmp/
-build/
-config/

CHANGELOG.md

@@ -1,3 +1,15 @@
+## v0.8.8 - 2017-10-24 - Google Cloud DNS, Large TXT Record support
+
+* Added support for "chunking" TXT records where individual values were larger
+  than 255 chars. This is common with DKIM records involving multiple
+  providers.
+* Added `GoogleCloudProvider`
+* Configurable `UnsafePlan` thresholds to allow modification of how many
+  updates/deletes are allowed before a plan is declared dangerous.
+* Manager.dump bug fix around empty zones.
+* Prefer use of `.` over `source` in shell scripts
+* `DynProvider` warns when it ignores unrecognized traffic directors.
+
 ## v0.8.7 - 2017-09-29 - OVH support
 
 Adds an OVH provider.
@@ -48,7 +60,7 @@ better in the future :fingers_crossed:
 
 #### Miscellaneous
 
-* Use a 3rd party lib for nautrual sorting of keys, rather than my old
+* Use a 3rd party lib for natural sorting of keys, rather than my old
   implementation. Sorting can be disabled in the YamlProvider with
   `enforce_order: False`.
 * Semi-colon/escaping fixes and improvements.

CONTRIBUTING.md

@@ -38,6 +38,10 @@ Here are a few things you can do that will increase the likelihood of your pull
 
 - Write a [good commit message](http://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html).
 
+## Development prerequisites
+
+- setuptools >= 30.3.0
+
 ## License note
 
 We can only accept contributions that are compatible with the MIT license.

MANIFEST.in

@@ -3,6 +3,5 @@ include CONTRIBUTING.md
 include LICENSE
 include docs/*
 include octodns/*
-include requirements*.txt
 include script/*
 include tests/*

README.md

@@ -119,13 +119,13 @@ The first step is to create a PR with your changes.
 
 Assuming the code tests and config validation statuses are green the next step is to do a noop deploy and verify that the changes OctoDNS plans to make are the ones you expect.
 
-![](/docs/assets/noop.png)    
+![](/docs/assets/noop.png)
 
 After that comes a set of reviews. One from a teammate who should have full context on what you're trying to accomplish and visibility in to the changes you're making to do it. The other is from a member of the team here at GitHub that owns DNS, mostly as a sanity check and to make sure that best practices are being followed. As much of that as possible is baked into `octodns-validate`.
 
 After the reviews it's time to branch deploy the change.
 
-![](/docs/assets/deploy.png)    
+![](/docs/assets/deploy.png)
 
 If that goes smoothly, you again see the expected changes, and verify them with `dig` and/or `octodns-report` you're good to hit the merge button. If there are problems you can quickly do a `.deploy dns/master` to go back to the previous state.
 
@@ -150,13 +150,16 @@ The above command pulled the existing data out of Route53 and placed the results
 | Provider | Record Support | GeoDNS Support | Notes |
 |--|--|--|--|
 | [AzureProvider](/octodns/provider/azuredns.py) | A, AAAA, CNAME, MX, NS, PTR, SRV, TXT | No | |
-| [CloudflareProvider](/octodns/provider/cloudflare.py) | A, AAAA, CAA, CNAME, MX, NS, SPF, TXT | No | CAA tags restricted |
+| [CloudflareProvider](/octodns/provider/cloudflare.py) | A, AAAA, ALIAS, CAA, CNAME, MX, NS, SPF, SRV, TXT | No | CAA tags restricted |
+| [DigitalOceanProvider](/octodns/provider/digitalocean.py) | A, AAAA, CAA, CNAME, MX, NS, TXT, SRV | No | CAA tags restricted |
+| [DnsMadeEasyProvider](/octodns/provider/dnsmadeeasy.py) | A, AAAA, CAA, CNAME, MX, NS, PTR, SPF, SRV, TXT | No | CAA tags restricted |
 | [DnsimpleProvider](/octodns/provider/dnsimple.py) | All | No | CAA tags restricted |
 | [DynProvider](/octodns/provider/dyn.py) | All | Yes | |
 | [GoogleCloudProvider](/octodns/provider/googlecloud.py) | A, AAAA, CAA, CNAME, MX, NAPTR, NS, PTR, SPF, SRV, TXT  | No | |
-| [Ns1Provider](/octodns/provider/ns1.py) | All | No | |
-| [OVH](/octodns/provider/ovh.py) | A, AAAA, CNAME, MX, NAPTR, NS, PTR, SPF, SRV, SSHFP, TXT | No | |
+| [Ns1Provider](/octodns/provider/ns1.py) | All | Yes | No health checking for GeoDNS |
+| [OVH](/octodns/provider/ovh.py) | A, AAAA, CNAME, MX, NAPTR, NS, PTR, SPF, SRV, SSHFP, TXT, DKIM | No | |
 | [PowerDnsProvider](/octodns/provider/powerdns.py) | All | No | |
+| [Rackspace](/octodns/provider/rackspace.py) | A, AAAA, ALIAS, CNAME, MX, NS, PTR, SPF, TXT | No |  |
 | [Route53](/octodns/provider/route53.py) | A, AAAA, CAA, CNAME, MX, NAPTR, NS, PTR, SPF, SRV, TXT | Yes | |
 | [TinyDNSSource](/octodns/source/tinydns.py) | A, CNAME, MX, NS, PTR | No | read-only |
 | [YamlProvider](/octodns/provider/yaml.py) | All | Yes | config |
@@ -166,6 +169,7 @@ The above command pulled the existing data out of Route53 and placed the results
 * ALIAS support varies a lot from provider to provider care should be taken to verify that your needs are met in detail.
    * Dyn's UI doesn't allow editing or view of TTL, but the API accepts and stores the value provided, this value does not appear to be used when served
    * Dnsimple's uses the configured TTL when serving things through the ALIAS, there's also a secondary TXT record created alongside the ALIAS that octoDNS ignores
+* octoDNS itself supports non-ASCII character sets, but in testing Cloudflare is the only provider where that is currently functional end-to-end. Others have failures either in the client libraries or API calls
 
 ## Custom Sources and Providers
 

docs/records.md

@@ -26,6 +26,55 @@ GeoDNS is currently supported for `A` and `AAAA` records on the Dyn (via Traffic
 
 Configuring GeoDNS is complex and the details of the functionality vary widely from provider to provider. OctoDNS has an opinionated view of how GeoDNS should be set up and does its best to map that to each provider's offering in a way that will result in similar behavior. It may not fit your needs or use cases, in which case please open an issue for discussion. We expect this functionality to grow and evolve over time as it's more widely used.
 
+The following is an example of GeoDNS with three entries NA-US-CA, NA-US-NY, OC-AU. Octodns creates another one labeled 'default' with the details for the actual A record, This default record is the failover record if the monitoring check fails.
+
+```yaml
+---
+? ''
+: type: TXT
+  value: v=spf1 -all
+test:
+  geo:
+    NA-US-NY:
+    - 111.111.111.1
+    NA-US-CA:
+    - 111.111.111.2
+    OC-AU:
+    - 111.111.111.3
+    EU:
+    - 111.111.111.4
+  ttl: 300
+  type: A
+  value: 111.111.111.5
+```
+
+
+The geo labels breakdown based on:
+
+1.
+    - 'AF': 14,  # Continental Africa
+    - 'AN': 17,  # Continental Antarctica
+    - 'AS': 15,  # Continental Asia
+    - 'EU': 13,  # Continental Europe
+    - 'NA': 11,  # Continental North America
+    - 'OC': 16,  # Continental Australia/Oceania
+    - 'SA': 12,  # Continental South America
+
+2. ISO Country Code https://en.wikipedia.org/wiki/ISO_3166-2
+
+3. ISO Country Code Subdevision as per https://en.wikipedia.org/wiki/ISO_3166-2:US   (change the code at the end for the country you are subdividing) * these may not always be supported depending on the provider.
+
+So the example is saying:
+
+- North America - United States - New York:  gets served an "A" record of  111.111.111.1
+- North America - United States - California:   gets served an "A" record of  111.111.111.2
+- Oceania - Australia: Gets served an "A" record of 111.111.111.3
+- Europe: gets an "A" record of 111.111.111.4
+- Everyone else gets an "A" record of 111.111.111.5
+
+
+Octodns will automatically set up a monitor and check for **https://<ip_address>/_dns** and check for a 200 response.
+
 ## Config (`YamlProvider`)
 
 OctoDNS records and `YamlProvider`'s schema is essentially a 1:1 match. Properties on the objects will match keys in the config.

octodns/__init__.py

@@ -3,4 +3,4 @@
 from __future__ import absolute_import, division, print_function, \
     unicode_literals
 
-__VERSION__ = '0.8.7'
+__VERSION__ = '0.8.8'

octodns/cmds/report.py

@@ -65,7 +65,7 @@ def main():
         resolver = AsyncResolver(configure=False,
                                  num_workers=int(args.num_workers))
         if not ip_addr_re.match(server):
-            server = str(query(server, 'A')[0])
+            server = unicode(query(server, 'A')[0])
         log.info('server=%s', server)
         resolver.nameservers = [server]
         resolver.lifetime = int(args.timeout)
@@ -81,12 +81,12 @@ def main():
         stdout.write(',')
         stdout.write(record._type)
         stdout.write(',')
-        stdout.write(str(record.ttl))
+        stdout.write(unicode(record.ttl))
         compare = {}
         for future in futures:
             stdout.write(',')
             try:
-                answers = [str(r) for r in future.result()]
+                answers = [unicode(r) for r in future.result()]
             except (NoAnswer, NoNameservers):
                 answers = ['*no answer*']
             except NXDOMAIN:

octodns/cmds/sync.py

@@ -26,7 +26,7 @@ def main():
                         help='Limit sync to the specified zone(s)')
 
     # --sources isn't an option here b/c filtering sources out would be super
-    # dangerous since you could eaily end up with an empty zone and delete
+    # dangerous since you could easily end up with an empty zone and delete
     # everything, or even just part of things when there are multiple sources
 
     parser.add_argument('--target', default=[], action='append',

octodns/cmds/validate.py

@@ -15,7 +15,7 @@ from octodns.manager import Manager
 def main():
     parser = ArgumentParser(description=__doc__.split('\n')[1])
 
-    parser.add_argument('--config-file', default='./config/production.yaml',
+    parser.add_argument('--config-file', required=True,
                         help='The Manager configuration file to use')
 
     args = parser.parse_args(WARN)

octodns/manager.py

@@ -5,13 +5,13 @@
 from __future__ import absolute_import, division, print_function, \
     unicode_literals
 
-from StringIO import StringIO
 from concurrent.futures import ThreadPoolExecutor
 from importlib import import_module
 from os import environ
 import logging
 
-from .provider.base import BaseProvider, Plan
+from .provider.base import BaseProvider
+from .provider.plan import Plan
 from .provider.yaml import YamlProvider
 from .record import Record
 from .yaml import safe_load
@@ -51,7 +51,7 @@ class MakeThreadFuture(object):
 
 class MainThreadExecutor(object):
     '''
-    Dummy executor that runs things on the main thread during the involcation
+    Dummy executor that runs things on the main thread during the invocation
     of submit, but still returns a future object with the result. This allows
     code to be written to handle async, even in the case where we don't want to
     use multiple threads/workers and would prefer that things flow as if
@@ -95,23 +95,8 @@ class Manager(object):
                 self.log.exception('Invalid provider class')
                 raise Exception('Provider {} is missing class'
                                 .format(provider_name))
-            _class = self._get_provider_class(_class)
-            # Build up the arguments we need to pass to the provider
-            kwargs = {}
-            for k, v in provider_config.items():
-                try:
-                    if v.startswith('env/'):
-                        try:
-                            env_var = v[4:]
-                            v = environ[env_var]
-                        except KeyError:
-                            self.log.exception('Invalid provider config')
-                            raise Exception('Incorrect provider config, '
-                                            'missing env var {}'
-                                            .format(env_var))
-                except AttributeError:
-                    pass
-                kwargs[k] = v
+            _class = self._get_named_class('provider', _class)
+            kwargs = self._build_kwargs(provider_config)
             try:
                 self.providers[provider_name] = _class(provider_name, **kwargs)
             except TypeError:
@@ -139,20 +124,64 @@ class Manager(object):
                     where = where[piece]
         self.zone_tree = zone_tree
 
-    def _get_provider_class(self, _class):
+        self.plan_outputs = {}
+        plan_outputs = manager_config.get('plan_outputs', {
+            'logger': {
+                'class': 'octodns.provider.plan.PlanLogger',
+                'level': 'info'
+            }
+        })
+        for plan_output_name, plan_output_config in plan_outputs.items():
+            try:
+                _class = plan_output_config.pop('class')
+            except KeyError:
+                self.log.exception('Invalid plan_output class')
+                raise Exception('plan_output {} is missing class'
+                                .format(plan_output_name))
+            _class = self._get_named_class('plan_output', _class)
+            kwargs = self._build_kwargs(plan_output_config)
+            try:
+                self.plan_outputs[plan_output_name] = \
+                    _class(plan_output_name, **kwargs)
+            except TypeError:
+                self.log.exception('Invalid plan_output config')
+                raise Exception('Incorrect plan_output config for {}'
+                                .format(plan_output_name))
+
+    def _get_named_class(self, _type, _class):
         try:
             module_name, class_name = _class.rsplit('.', 1)
             module = import_module(module_name)
         except (ImportError, ValueError):
-            self.log.exception('_get_provider_class: Unable to import '
+            self.log.exception('_get_{}_class: Unable to import '
                                'module %s', _class)
-            raise Exception('Unknown provider class: {}'.format(_class))
+            raise Exception('Unknown {} class: {}'.format(_type, _class))
         try:
             return getattr(module, class_name)
         except AttributeError:
-            self.log.exception('_get_provider_class: Unable to get class %s '
+            self.log.exception('_get_{}_class: Unable to get class %s '
                                'from module %s', class_name, module)
-            raise Exception('Unknown provider class: {}'.format(_class))
+            raise Exception('Unknown {} class: {}'.format(_type, _class))
+
+    def _build_kwargs(self, source):
+        # Build up the arguments we need to pass to the provider
+        kwargs = {}
+        for k, v in source.items():
+            try:
+                if v.startswith('env/'):
+                    try:
+                        env_var = v[4:]
+                        v = environ[env_var]
+                    except KeyError:
+                        self.log.exception('Invalid provider config')
+                        raise Exception('Incorrect provider config, '
+                                        'missing env var {}'
+                                        .format(env_var))
+            except AttributeError:
+                pass
+            kwargs[k] = v
+
+        return kwargs
 
     def configured_sub_zones(self, zone_name):
         # Reversed pieces of the zone name
@@ -259,39 +288,8 @@ class Manager(object):
         # plan pairs.
         plans = [p for f in futures for p in f.result()]
 
-        hr = '*************************************************************' \
-            '*******************\n'
-        buf = StringIO()
-        buf.write('\n')
-        if plans:
-            current_zone = None
-            for target, plan in plans:
-                if plan.desired.name != current_zone:
-                    current_zone = plan.desired.name
-                    buf.write(hr)
-                    buf.write('* ')
-                    buf.write(current_zone)
-                    buf.write('\n')
-                    buf.write(hr)
-
-                buf.write('* ')
-                buf.write(target.id)
-                buf.write(' (')
-                buf.write(target)
-                buf.write(')\n*   ')
-                for change in plan.changes:
-                    buf.write(change.__repr__(leader='* '))
-                    buf.write('\n*   ')
-
-                buf.write('Summary: ')
-                buf.write(plan)
-                buf.write('\n')
-        else:
-            buf.write(hr)
-            buf.write('No changes were planned\n')
-        buf.write(hr)
-        buf.write('\n')
-        self.log.info(buf.getvalue())
+        for output in self.plan_outputs.values():
+            output.run(plans=plans, log=self.log)
 
         if not force:
             self.log.debug('sync:   checking safety')
@@ -363,7 +361,7 @@ class Manager(object):
 
         plan = target.plan(zone)
         if plan is None:
-            plan = Plan(zone, zone, [])
+            plan = Plan(zone, zone, [], False)
         target.apply(plan)
 
     def validate_configs(self):

octodns/provider/azuredns.py

@@ -39,7 +39,7 @@ class _AzureRecord(object):
     }
 
     def __init__(self, resource_group, record, delete=False):
-        '''Contructor for _AzureRecord.
+        '''Constructor for _AzureRecord.
 
             Notes on Azure records: An Azure record set has the form
             RecordSet(name=<...>, type=<...>, arecords=[...], aaaa_records, ..)
@@ -222,7 +222,7 @@ class AzureProvider(BaseProvider):
               azuredns:
                 class: octodns.provider.azuredns.AzureProvider
                 client_id: env/AZURE_APPLICATION_ID
-                key: env/AZURE_AUTHENICATION_KEY
+                key: env/AZURE_AUTHENTICATION_KEY
                 directory_id: env/AZURE_DIRECTORY_ID
                 sub_id: env/AZURE_SUBSCRIPTION_ID
                 resource_group: 'TestResource1'
@@ -322,6 +322,8 @@ class AzureProvider(BaseProvider):
             :type return: void
         '''
         self.log.debug('populate: name=%s', zone.name)
+
+        exists = False
         before = len(zone.records)
 
         zone_name = zone.name[:len(zone.name) - 1]
@@ -331,6 +333,7 @@ class AzureProvider(BaseProvider):
         _records = set()
         records = self._dns_client.record_sets.list_by_dns_zone
         if self._check_zone(zone_name):
+            exists = True
             for azrecord in records(self._resource_group, zone_name):
                 if _parse_azure_type(azrecord.type) in self.SUPPORTS:
                     _records.add(azrecord)
@@ -344,7 +347,9 @@ class AzureProvider(BaseProvider):
                 record = Record.new(zone, record_name, data, source=self)
                 zone.add_record(record)
 
-        self.log.info('populate: found %s records', len(zone.records) - before)
+        self.log.info('populate: found %s records, exists=%s',
+                      len(zone.records) - before, exists)
+        return exists
 
     def _data_for_A(self, azrecord):
         return {'values': [ar.ipv4_address for ar in azrecord.arecords]}

octodns/provider/base.py

@@ -7,78 +7,7 @@ from __future__ import absolute_import, division, print_function, \
 
 from ..source.base import BaseSource
 from ..zone import Zone
-from logging import getLogger
-
-
-class UnsafePlan(Exception):
-    pass
-
-
-class Plan(object):
-    log = getLogger('Plan')
-
-    MAX_SAFE_UPDATE_PCENT = .3
-    MAX_SAFE_DELETE_PCENT = .3
-    MIN_EXISTING_RECORDS = 10
-
-    def __init__(self, existing, desired, changes,
-                 update_pcent_threshold=MAX_SAFE_UPDATE_PCENT,
-                 delete_pcent_threshold=MAX_SAFE_DELETE_PCENT):
-        self.existing = existing
-        self.desired = desired
-        self.changes = changes
-        self.update_pcent_threshold = update_pcent_threshold
-        self.delete_pcent_threshold = delete_pcent_threshold
-
-        change_counts = {
-            'Create': 0,
-            'Delete': 0,
-            'Update': 0
-        }
-        for change in changes:
-            change_counts[change.__class__.__name__] += 1
-        self.change_counts = change_counts
-
-        try:
-            existing_n = len(self.existing.records)
-        except AttributeError:
-            existing_n = 0
-
-        self.log.debug('__init__: Creates=%d, Updates=%d, Deletes=%d'
-                       'Existing=%d',
-                       self.change_counts['Create'],
-                       self.change_counts['Update'],
-                       self.change_counts['Delete'], existing_n)
-
-    def raise_if_unsafe(self):
-        # TODO: what is safe really?
-        if self.existing and \
-           len(self.existing.records) >= self.MIN_EXISTING_RECORDS:
-
-            existing_record_count = len(self.existing.records)
-            update_pcent = self.change_counts['Update'] / existing_record_count
-            delete_pcent = self.change_counts['Delete'] / existing_record_count
-
-            if update_pcent > self.update_pcent_threshold:
-                raise UnsafePlan('Too many updates, {} is over {} percent'
-                                 '({}/{})'.format(
-                                     update_pcent,
-                                     self.MAX_SAFE_UPDATE_PCENT * 100,
-                                     self.change_counts['Update'],
-                                     existing_record_count))
-            if delete_pcent > self.delete_pcent_threshold:
-                raise UnsafePlan('Too many deletes, {} is over {} percent'
-                                 '({}/{})'.format(
-                                     delete_pcent,
-                                     self.MAX_SAFE_DELETE_PCENT * 100,
-                                     self.change_counts['Delete'],
-                                     existing_record_count))
-
-    def __repr__(self):
-        return 'Creates={}, Updates={}, Deletes={}, Existing Records={}' \
-            .format(self.change_counts['Create'], self.change_counts['Update'],
-                    self.change_counts['Delete'],
-                    len(self.existing.records))
+from .plan import Plan
 
 
 class BaseProvider(BaseSource):
@@ -88,7 +17,8 @@ class BaseProvider(BaseSource):
                  delete_pcent_threshold=Plan.MAX_SAFE_DELETE_PCENT):
         super(BaseProvider, self).__init__(id)
         self.log.debug('__init__: id=%s, apply_disabled=%s, '
-                       'update_pcent_threshold=%d, delete_pcent_threshold=%d',
+                       'update_pcent_threshold=%.2f'
+                       'delete_pcent_threshold=%.2f',
                        id,
                        apply_disabled,
                        update_pcent_threshold,
@@ -100,14 +30,14 @@ class BaseProvider(BaseSource):
     def _include_change(self, change):
         '''
         An opportunity for providers to filter out false positives due to
-        pecularities in their implementation. E.g. minimum TTLs.
+        peculiarities in their implementation. E.g. minimum TTLs.
         '''
         return True
 
     def _extra_changes(self, existing, desired, changes):
         '''
         An opportunity for providers to add extra changes to the plan that are
-        necessary to update ancilary record data or configure the zone. E.g.
+        necessary to update ancillary record data or configure the zone. E.g.
         base NS records.
         '''
         return []
@@ -116,7 +46,12 @@ class BaseProvider(BaseSource):
         self.log.info('plan: desired=%s', desired.name)
 
         existing = Zone(desired.name, desired.sub_zones)
-        self.populate(existing, target=True, lenient=True)
+        exists = self.populate(existing, target=True, lenient=True)
+        if exists is None:
+            # If your code gets this warning see Source.populate for more
+            # information
+            self.log.warn('Provider %s used in target mode did not return '
+                          'exists', self.id)
 
         # compute the changes at the zone/record level
         changes = existing.changes(desired, self)
@@ -132,11 +67,11 @@ class BaseProvider(BaseSource):
         extra = self._extra_changes(existing, desired, changes)
         if extra:
             self.log.info('plan:   extra changes\n  %s', '\n  '
-                          .join([str(c) for c in extra]))
+                          .join([unicode(c) for c in extra]))
             changes += extra
 
         if changes:
-            plan = Plan(existing, desired, changes,
+            plan = Plan(existing, desired, changes, exists,
                         self.update_pcent_threshold,
                         self.delete_pcent_threshold)
             self.log.info('plan:   %s', plan)

octodns/provider/cloudflare.py

@@ -7,20 +7,25 @@ from __future__ import absolute_import, division, print_function, \
 
 from collections import defaultdict
 from logging import getLogger
+from json import dumps
 from requests import Session
 
 from ..record import Record, Update
 from .base import BaseProvider
 
 
-class CloudflareAuthenticationError(Exception):
-
+class CloudflareError(Exception):
     def __init__(self, data):
         try:
             message = data['errors'][0]['message']
         except (IndexError, KeyError):
-            message = 'Authentication error'
-        super(CloudflareAuthenticationError, self).__init__(message)
+            message = 'Cloudflare error'
+        super(CloudflareError, self).__init__(message)
+
+
+class CloudflareAuthenticationError(CloudflareError):
+    def __init__(self, data):
+        CloudflareError.__init__(self, data)
 
 
 class CloudflareProvider(BaseProvider):
@@ -33,17 +38,24 @@ class CloudflareProvider(BaseProvider):
         email: [email protected]
         # The api key (required)
         token: foo
+        # Import CDN enabled records as CNAME to {}.cdn.cloudflare.net. Records
+        # ending at .cdn.cloudflare.net. will be ignored when this provider is
+        # not used as the source and the cdn option is enabled.
+        #
+        # See: https://support.cloudflare.com/hc/en-us/articles/115000830351
+        cdn: false
     '''
     SUPPORTS_GEO = False
-    # TODO: support SRV
-    SUPPORTS = set(('A', 'AAAA', 'CAA', 'CNAME', 'MX', 'NS', 'SPF', 'TXT'))
+    SUPPORTS = set(('ALIAS', 'A', 'AAAA', 'CAA', 'CNAME', 'MX', 'NS', 'SRV',
+                    'SPF', 'TXT'))
 
     MIN_TTL = 120
     TIMEOUT = 15
 
-    def __init__(self, id, email, token, *args, **kwargs):
+    def __init__(self, id, email, token, cdn=False, *args, **kwargs):
         self.log = getLogger('CloudflareProvider[{}]'.format(id))
-        self.log.debug('__init__: id=%s, email=%s, token=***', id, email)
+        self.log.debug('__init__: id=%s, email=%s, token=***, cdn=%s', id,
+                       email, cdn)
         super(CloudflareProvider, self).__init__(id, *args, **kwargs)
 
         sess = Session()
@@ -51,6 +63,7 @@ class CloudflareProvider(BaseProvider):
             'X-Auth-Email': email,
             'X-Auth-Key': token,
         })
+        self.cdn = cdn
         self._sess = sess
 
         self._zones = None
@@ -63,8 +76,11 @@ class CloudflareProvider(BaseProvider):
         resp = self._sess.request(method, url, params=params, json=data,
                                   timeout=self.TIMEOUT)
         self.log.debug('_request:   status=%d', resp.status_code)
+        if resp.status_code == 400:
+            raise CloudflareError(resp.json())
         if resp.status_code == 403:
             raise CloudflareAuthenticationError(resp.json())
+
         resp.raise_for_status()
         return resp.json()
 
@@ -86,6 +102,18 @@ class CloudflareProvider(BaseProvider):
 
         return self._zones
 
+    def _data_for_cdn(self, name, _type, records):
+        self.log.info('CDN rewrite for %s', records[0]['name'])
+        _type = "CNAME"
+        if name == "":
+            _type = "ALIAS"
+
+        return {
+            'ttl': records[0]['ttl'],
+            'type': _type,
+            'value': '{}.cdn.cloudflare.net.'.format(records[0]['name']),
+        }
+
     def _data_for_multiple(self, _type, records):
         return {
             'ttl': records[0]['ttl'],
@@ -123,6 +151,8 @@ class CloudflareProvider(BaseProvider):
             'value': '{}.'.format(only['content'])
         }
 
+    _data_for_ALIAS = _data_for_CNAME
+
     def _data_for_MX(self, _type, records):
         values = []
         for r in records:
@@ -143,6 +173,21 @@ class CloudflareProvider(BaseProvider):
             'values': ['{}.'.format(r['content']) for r in records],
         }
 
+    def _data_for_SRV(self, _type, records):
+        values = []
+        for r in records:
+            values.append({
+                'priority': r['data']['priority'],
+                'weight': r['data']['weight'],
+                'port': r['data']['port'],
+                'target': '{}.'.format(r['data']['target']),
+            })
+        return {
+            'type': _type,
+            'ttl': records[0]['ttl'],
+            'values': values
+        }
+
     def zone_records(self, zone):
         if zone.name not in self._zone_records:
             zone_id = self.zones.get(zone.name, False)
@@ -165,13 +210,29 @@ class CloudflareProvider(BaseProvider):
 
         return self._zone_records[zone.name]
 
+    def _record_for(self, zone, name, _type, records, lenient):
+        # rewrite Cloudflare proxied records
+        if self.cdn and records[0]['proxied']:
+            data = self._data_for_cdn(name, _type, records)
+        else:
+            # Cloudflare supports ALIAS semantics with root CNAMEs
+            if _type == 'CNAME' and name == '':
+                _type = 'ALIAS'
+
+            data_for = getattr(self, '_data_for_{}'.format(_type))
+            data = data_for(_type, records)
+
+        return Record.new(zone, name, data, source=self, lenient=lenient)
+
     def populate(self, zone, target=False, lenient=False):
         self.log.debug('populate: name=%s, target=%s, lenient=%s', zone.name,
                        target, lenient)
 
+        exists = False
         before = len(zone.records)
         records = self.zone_records(zone)
         if records:
+            exists = True
             values = defaultdict(lambda: defaultdict(list))
             for record in records:
                 name = zone.hostname_from_fqdn(record['name'])
@@ -181,22 +242,37 @@ class CloudflareProvider(BaseProvider):
 
             for name, types in values.items():
                 for _type, records in types.items():
-                    data_for = getattr(self, '_data_for_{}'.format(_type))
-                    data = data_for(_type, records)
-                    record = Record.new(zone, name, data, source=self,
-                                        lenient=lenient)
+                    record = self._record_for(zone, name, _type, records,
+                                              lenient)
+
+                    # only one rewrite is needed for names where the proxy is
+                    # enabled at multiple records with a different type but
+                    # the same name
+                    if (self.cdn and records[0]['proxied'] and
+                       record in zone._records[name]):
+                        self.log.info('CDN rewrite %s already in zone', name)
+                        continue
+
                     zone.add_record(record)
 
-        self.log.info('populate:   found %s records',
-                      len(zone.records) - before)
+        self.log.info('populate:   found %s records, exists=%s',
+                      len(zone.records) - before, exists)
+        return exists
 
     def _include_change(self, change):
         if isinstance(change, Update):
             existing = change.existing.data
             new = change.new.data
-            new['ttl'] = max(120, new['ttl'])
+            new['ttl'] = max(self.MIN_TTL, new['ttl'])
             if new == existing:
                 return False
+
+        # If this is a record to enable Cloudflare CDN don't update as
+        # we don't know the original values.
+        if (change.record._type in ('ALIAS', 'CNAME') and
+                change.record.value.endswith('.cdn.cloudflare.net.')):
+            return False
+
         return True
 
     def _contents_for_multiple(self, record):
@@ -232,25 +308,125 @@ class CloudflareProvider(BaseProvider):
                 'content': value.exchange
             }
 
+    def _contents_for_SRV(self, record):
+        service, proto = record.name.split('.', 2)
+        for value in record.values:
+            yield {
+                'data': {
+                    'service': service,
+                    'proto': proto,
+                    'name': record.zone.name,
+                    'priority': value.priority,
+                    'weight': value.weight,
+                    'port': value.port,
+                    'target': value.target[:-1],
+                }
+            }
+
+    def _gen_contents(self, record):
+        name = record.fqdn[:-1]
+        _type = record._type
+        ttl = max(self.MIN_TTL, record.ttl)
+
+        # Cloudflare supports ALIAS semantics with a root CNAME
+        if _type == 'ALIAS':
+            _type = 'CNAME'
+
+        contents_for = getattr(self, '_contents_for_{}'.format(_type))
+        for content in contents_for(record):
+            content.update({
+                'name': name,
+                'type': _type,
+                'ttl': ttl,
+            })
+            yield content
+
     def _apply_Create(self, change):
         new = change.new
         zone_id = self.zones[new.zone.name]
-        contents_for = getattr(self, '_contents_for_{}'.format(new._type))
         path = '/zones/{}/dns_records'.format(zone_id)
-        name = new.fqdn[:-1]
-        for content in contents_for(change.new):
-            content.update({
-                'name': name,
-                'type': new._type,
-                # Cloudflare has a min ttl of 120s
-                'ttl': max(self.MIN_TTL, new.ttl),
-            })
+        for content in self._gen_contents(new):
             self._request('POST', path, data=content)
 
+    def _hash_content(self, content):
+        # Some of the dicts are nested so this seems about as good as any
+        # option we have for consistently hashing them (within a single run)
+        return hash(dumps(content, sort_keys=True))
+
     def _apply_Update(self, change):
-        # Create the new and delete the old
-        self._apply_Create(change)
-        self._apply_Delete(change)
+
+        # Ugh, this is pretty complicated and ugly, mainly due to the
+        # sub-optimal API/semantics. Ideally we'd have a batch change API like
+        # Route53's to make this 100% clean and safe without all this PITA, but
+        # we don't so we'll have to work around that and manually do it as
+        # safely as possible. Note this still isn't perfect as we don't/can't
+        # practically take into account things like the different "types" of
+        # CAA records so when we "swap" there may be brief periods where things
+        # are invalid or even worse Cloudflare may update their validations to
+        # prevent dups. I see no clean way around that short of making this
+        # understand 100% of the details of each record type and develop an
+        # individual/specific ordering of changes that prevents it. That'd
+        # probably result in more code than this whole provider currently has
+        # so... :-(
+
+        existing_contents = {
+            self._hash_content(c): c
+            for c in self._gen_contents(change.existing)
+        }
+        new_contents = {
+            self._hash_content(c): c
+            for c in self._gen_contents(change.new)
+        }
+
+        # Find the things we need to add
+        adds = []
+        for k, content in new_contents.items():
+            try:
+                existing_contents.pop(k)
+                self.log.debug('_apply_Update: leaving %s', content)
+            except KeyError:
+                adds.append(content)
+
+        zone = change.new.zone
+        zone_id = self.zones[zone.name]
+
+        # Find things we need to remove
+        hostname = zone.hostname_from_fqdn(change.new.fqdn[:-1])
+        _type = change.new._type
+        # OK, work through each record from the zone
+        for record in self.zone_records(zone):
+            name = zone.hostname_from_fqdn(record['name'])
+            # Use the _record_for so that we include all of standard
+            # conversion logic
+            r = self._record_for(zone, name, record['type'], [record], True)
+            if hostname == r.name and _type == r._type:
+
+                # Round trip the single value through a record to contents flow
+                # to get a consistent _gen_contents result that matches what
+                # went in to new_contents
+                content = self._gen_contents(r).next()
+
+                # If the hash of that dict isn't in new this record isn't
+                # needed
+                if self._hash_content(content) not in new_contents:
+                    rid = record['id']
+                    path = '/zones/{}/dns_records/{}'.format(record['zone_id'],
+                                                             rid)
+                    try:
+                        add_content = adds.pop(0)
+                        self.log.debug('_apply_Update: swapping %s -> %s, %s',
+                                       content, add_content, rid)
+                        self._request('PUT', path, data=add_content)
+                    except IndexError:
+                        self.log.debug('_apply_Update: removing %s, %s',
+                                       content, rid)
+                        self._request('DELETE', path)
+
+        # Any remaining adds just need to be created
+        path = '/zones/{}/dns_records'.format(zone_id)
+        for content in adds:
+            self.log.debug('_apply_Update: adding %s', content)
+            self._request('POST', path, data=content)
 
     def _apply_Delete(self, change):
         existing = change.existing

octodns/provider/digitalocean.py

@@ -0,0 +1,343 @@
+#
+#
+#
+
+from __future__ import absolute_import, division, print_function, \
+    unicode_literals
+
+from collections import defaultdict
+from requests import Session
+import logging
+
+from ..record import Record
+from .base import BaseProvider
+
+
+class DigitalOceanClientException(Exception):
+    pass
+
+
+class DigitalOceanClientNotFound(DigitalOceanClientException):
+
+    def __init__(self):
+        super(DigitalOceanClientNotFound, self).__init__('Not Found')
+
+
+class DigitalOceanClientUnauthorized(DigitalOceanClientException):
+
+    def __init__(self):
+        super(DigitalOceanClientUnauthorized, self).__init__('Unauthorized')
+
+
+class DigitalOceanClient(object):
+    BASE = 'https://api.digitalocean.com/v2'
+
+    def __init__(self, token):
+        sess = Session()
+        sess.headers.update({'Authorization': 'Bearer {}'.format(token)})
+        self._sess = sess
+
+    def _request(self, method, path, params=None, data=None):
+        url = '{}{}'.format(self.BASE, path)
+        resp = self._sess.request(method, url, params=params, json=data)
+        if resp.status_code == 401:
+            raise DigitalOceanClientUnauthorized()
+        if resp.status_code == 404:
+            raise DigitalOceanClientNotFound()
+        resp.raise_for_status()
+        return resp
+
+    def domain(self, name):
+        path = '/domains/{}'.format(name)
+        return self._request('GET', path).json()
+
+    def domain_create(self, name):
+        # Digitalocean requires an IP on zone creation
+        self._request('POST', '/domains', data={'name': name,
+                                                'ip_address': '192.0.2.1'})
+
+        # After the zone is created, immediately delete the record
+        records = self.records(name)
+        for record in records:
+            if record['name'] == '' and record['type'] == 'A':
+                self.record_delete(name, record['id'])
+
+    def records(self, zone_name):
+        path = '/domains/{}/records'.format(zone_name)
+        ret = []
+
+        page = 1
+        while True:
+            data = self._request('GET', path, {'page': page}).json()
+
+            ret += data['domain_records']
+            links = data['links']
+
+            # https://developers.digitalocean.com/documentation/v2/#links
+            # pages exists if there is more than 1 page
+            # last doesn't exist if you're on the last page
+            try:
+                links['pages']['last']
+                page += 1
+            except KeyError:
+                break
+
+        for record in ret:
+            # change any apex record to empty string
+            if record['name'] == '@':
+                record['name'] = ''
+
+            # change any apex value to zone name
+            if record['data'] == '@':
+                record['data'] = zone_name
+
+        return ret
+
+    def record_create(self, zone_name, params):
+        path = '/domains/{}/records'.format(zone_name)
+        # change empty name string to @, DO uses @ for apex record names
+        if params['name'] == '':
+            params['name'] = '@'
+
+        self._request('POST', path, data=params)
+
+    def record_delete(self, zone_name, record_id):
+        path = '/domains/{}/records/{}'.format(zone_name, record_id)
+        self._request('DELETE', path)
+
+
+class DigitalOceanProvider(BaseProvider):
+    '''
+    DigitalOcean DNS provider using API v2
+
+    digitalocean:
+        class: octodns.provider.digitalocean.DigitalOceanProvider
+        # Your DigitalOcean API token (required)
+        token: foo
+    '''
+    SUPPORTS_GEO = False
+    SUPPORTS = set(('A', 'AAAA', 'CAA', 'CNAME', 'MX', 'NS', 'TXT', 'SRV'))
+
+    def __init__(self, id, token, *args, **kwargs):
+        self.log = logging.getLogger('DigitalOceanProvider[{}]'.format(id))
+        self.log.debug('__init__: id=%s, token=***', id)
+        super(DigitalOceanProvider, self).__init__(id, *args, **kwargs)
+        self._client = DigitalOceanClient(token)
+
+        self._zone_records = {}
+
+    def _data_for_multiple(self, _type, records):
+        return {
+            'ttl': records[0]['ttl'],
+            'type': _type,
+            'values': [r['data'] for r in records]
+        }
+
+    _data_for_A = _data_for_multiple
+    _data_for_AAAA = _data_for_multiple
+
+    def _data_for_CAA(self, _type, records):
+        values = []
+        for record in records:
+            values.append({
+                'flags': record['flags'],
+                'tag': record['tag'],
+                'value': record['data'],
+            })
+        return {
+            'ttl': records[0]['ttl'],
+            'type': _type,
+            'values': values
+        }
+
+    def _data_for_CNAME(self, _type, records):
+        record = records[0]
+        return {
+            'ttl': record['ttl'],
+            'type': _type,
+            'value': '{}.'.format(record['data'])
+        }
+
+    def _data_for_MX(self, _type, records):
+        values = []
+        for record in records:
+            values.append({
+                'preference': record['priority'],
+                'exchange': '{}.'.format(record['data'])
+            })
+        return {
+            'ttl': records[0]['ttl'],
+            'type': _type,
+            'values': values
+        }
+
+    def _data_for_NS(self, _type, records):
+        values = []
+        for record in records:
+            data = '{}.'.format(record['data'])
+            values.append(data)
+        return {
+            'ttl': records[0]['ttl'],
+            'type': _type,
+            'values': values,
+        }
+
+    def _data_for_SRV(self, _type, records):
+        values = []
+        for record in records:
+            values.append({
+                'port': record['port'],
+                'priority': record['priority'],
+                'target': '{}.'.format(record['data']),
+                'weight': record['weight']
+            })
+        return {
+            'type': _type,
+            'ttl': records[0]['ttl'],
+            'values': values
+        }
+
+    def _data_for_TXT(self, _type, records):
+        values = [value['data'].replace(';', '\;') for value in records]
+        return {
+            'ttl': records[0]['ttl'],
+            'type': _type,
+            'values': values
+        }
+
+    def zone_records(self, zone):
+        if zone.name not in self._zone_records:
+            try:
+                self._zone_records[zone.name] = \
+                    self._client.records(zone.name[:-1])
+            except DigitalOceanClientNotFound:
+                return []
+
+        return self._zone_records[zone.name]
+
+    def populate(self, zone, target=False, lenient=False):
+        self.log.debug('populate: name=%s, target=%s, lenient=%s', zone.name,
+                       target, lenient)
+
+        values = defaultdict(lambda: defaultdict(list))
+        for record in self.zone_records(zone):
+            _type = record['type']
+            values[record['name']][record['type']].append(record)
+
+        before = len(zone.records)
+        for name, types in values.items():
+            for _type, records in types.items():
+                data_for = getattr(self, '_data_for_{}'.format(_type))
+                record = Record.new(zone, name, data_for(_type, records),
+                                    source=self, lenient=lenient)
+                zone.add_record(record)
+
+        exists = zone.name in self._zone_records
+        self.log.info('populate:   found %s records, exists=%s',
+                      len(zone.records) - before, exists)
+        return exists
+
+    def _params_for_multiple(self, record):
+        for value in record.values:
+            yield {
+                'data': value,
+                'name': record.name,
+                'ttl': record.ttl,
+                'type': record._type
+            }
+
+    _params_for_A = _params_for_multiple
+    _params_for_AAAA = _params_for_multiple
+    _params_for_NS = _params_for_multiple
+
+    def _params_for_CAA(self, record):
+        for value in record.values:
+            yield {
+                'data': '{}.'.format(value.value),
+                'flags': value.flags,
+                'name': record.name,
+                'tag': value.tag,
+                'ttl': record.ttl,
+                'type': record._type
+            }
+
+    def _params_for_single(self, record):
+        yield {
+            'data': record.value,
+            'name': record.name,
+            'ttl': record.ttl,
+            'type': record._type
+        }
+
+    _params_for_CNAME = _params_for_single
+
+    def _params_for_MX(self, record):
+        for value in record.values:
+            yield {
+                'data': value.exchange,
+                'name': record.name,
+                'priority': value.preference,
+                'ttl': record.ttl,
+                'type': record._type
+            }
+
+    def _params_for_SRV(self, record):
+        for value in record.values:
+            yield {
+                'data': value.target,
+                'name': record.name,
+                'port': value.port,
+                'priority': value.priority,
+                'ttl': record.ttl,
+                'type': record._type,
+                'weight': value.weight
+            }
+
+    def _params_for_TXT(self, record):
+        # DigitalOcean doesn't want things escaped in values so we
+        # have to strip them here and add them when going the other way
+        for value in record.values:
+            yield {
+                'data': value.replace('\;', ';'),
+                'name': record.name,
+                'ttl': record.ttl,
+                'type': record._type
+            }
+
+    def _apply_Create(self, change):
+        new = change.new
+        params_for = getattr(self, '_params_for_{}'.format(new._type))
+        for params in params_for(new):
+            self._client.record_create(new.zone.name[:-1], params)
+
+    def _apply_Update(self, change):
+        self._apply_Delete(change)
+        self._apply_Create(change)
+
+    def _apply_Delete(self, change):
+        existing = change.existing
+        zone = existing.zone
+        for record in self.zone_records(zone):
+            if existing.name == record['name'] and \
+               existing._type == record['type']:
+                self._client.record_delete(zone.name[:-1], record['id'])
+
+    def _apply(self, plan):
+        desired = plan.desired
+        changes = plan.changes
+        self.log.debug('_apply: zone=%s, len(changes)=%d', desired.name,
+                       len(changes))
+
+        domain_name = desired.name[:-1]
+        try:
+            self._client.domain(domain_name)
+        except DigitalOceanClientNotFound:
+            self.log.debug('_apply:   no matching zone, creating domain')
+            self._client.domain_create(domain_name)
+
+        for change in changes:
+            class_name = change.__class__.__name__
+            getattr(self, '_apply_{}'.format(class_name))(change)
+
+        # Clear out the cache if any
+        self._zone_records.pop(desired.name, None)

octodns/provider/dnsimple.py

@@ -160,7 +160,7 @@ class DnsimpleProvider(BaseProvider):
                     record['content'].split(' ', 5)
             except ValueError:
                 # their api will let you create invalid records, this
-                # essnetially handles that by ignoring them for values
+                # essentially handles that by ignoring them for values
                 # purposes. That will cause updates to happen to delete them if
                 # they shouldn't exist or update them if they're wrong
                 continue
@@ -272,8 +272,10 @@ class DnsimpleProvider(BaseProvider):
                                     source=self, lenient=lenient)
                 zone.add_record(record)
 
-        self.log.info('populate:   found %s records',
-                      len(zone.records) - before)
+        exists = zone.name in self._zone_records
+        self.log.info('populate:   found %s records, exists=%s',
+                      len(zone.records) - before, exists)
+        return exists
 
     def _params_for_multiple(self, record):
         for value in record.values:

octodns/provider/dnsmadeeasy.py

@@ -0,0 +1,382 @@
+#
+#
+#
+
+from __future__ import absolute_import, division, print_function, \
+    unicode_literals
+
+from collections import defaultdict
+from requests import Session
+from time import strftime, gmtime, sleep
+import hashlib
+import hmac
+import logging
+
+from ..record import Record
+from .base import BaseProvider
+
+
+class DnsMadeEasyClientException(Exception):
+    pass
+
+
+class DnsMadeEasyClientBadRequest(DnsMadeEasyClientException):
+
+    def __init__(self, resp):
+        errors = resp.json()['error']
+        super(DnsMadeEasyClientBadRequest, self).__init__(
+            '\n  - {}'.format('\n  - '.join(errors)))
+
+
+class DnsMadeEasyClientUnauthorized(DnsMadeEasyClientException):
+
+    def __init__(self):
+        super(DnsMadeEasyClientUnauthorized, self).__init__('Unauthorized')
+
+
+class DnsMadeEasyClientNotFound(DnsMadeEasyClientException):
+
+    def __init__(self):
+        super(DnsMadeEasyClientNotFound, self).__init__('Not Found')
+
+
+class DnsMadeEasyClient(object):
+    PRODUCTION = 'https://api.dnsmadeeasy.com/V2.0/dns/managed'
+    SANDBOX = 'https://api.sandbox.dnsmadeeasy.com/V2.0/dns/managed'
+
+    def __init__(self, api_key, secret_key, sandbox=False,
+                 ratelimit_delay=0.0):
+        self.api_key = api_key
+        self.secret_key = secret_key
+        self._base = self.SANDBOX if sandbox else self.PRODUCTION
+        self.ratelimit_delay = ratelimit_delay
+        self._sess = Session()
+        self._sess.headers.update({'x-dnsme-apiKey': self.api_key})
+        self._domains = None
+
+    def _current_time(self):
+        return strftime("%a, %d %b %Y %H:%M:%S +0000", gmtime())
+
+    def _hmac_hash(self, now):
+        return hmac.new(self.secret_key.encode(), now.encode(),
+                        hashlib.sha1).hexdigest()
+
+    def _request(self, method, path, params=None, data=None):
+        now = self._current_time()
+        hmac_hash = self._hmac_hash(now)
+
+        headers = {
+            'x-dnsme-hmac': hmac_hash,
+            'x-dnsme-requestDate': now
+        }
+
+        url = '{}{}'.format(self._base, path)
+        resp = self._sess.request(method, url, headers=headers,
+                                  params=params, json=data)
+        if resp.status_code == 400:
+            raise DnsMadeEasyClientBadRequest(resp)
+        if resp.status_code in [401, 403]:
+            raise DnsMadeEasyClientUnauthorized()
+        if resp.status_code == 404:
+            raise DnsMadeEasyClientNotFound()
+        resp.raise_for_status()
+        sleep(self.ratelimit_delay)
+        return resp
+
+    @property
+    def domains(self):
+        if self._domains is None:
+            zones = []
+
+            # has pages in resp, do we need paging?
+            resp = self._request('GET', '/').json()
+            zones += resp['data']
+
+            self._domains = {'{}.'.format(z['name']): z['id'] for z in zones}
+
+        return self._domains
+
+    def domain(self, name):
+        path = '/id/{}'.format(name)
+        return self._request('GET', path).json()
+
+    def domain_create(self, name):
+        self._request('POST', '/', data={'name': name})
+
+    def records(self, zone_name):
+        zone_id = self.domains.get(zone_name, False)
+        path = '/{}/records'.format(zone_id)
+        ret = []
+
+        # has pages in resp, do we need paging?
+        resp = self._request('GET', path).json()
+        ret += resp['data']
+
+        # change relative values to absolute
+        for record in ret:
+            value = record['value']
+            if record['type'] in ['CNAME', 'MX', 'NS', 'SRV']:
+                if value == '':
+                    record['value'] = zone_name
+                elif not value.endswith('.'):
+                    record['value'] = '{}.{}'.format(value, zone_name)
+
+        return ret
+
+    def record_create(self, zone_name, params):
+        zone_id = self.domains.get(zone_name, False)
+        path = '/{}/records'.format(zone_id)
+
+        self._request('POST', path, data=params)
+
+    def record_delete(self, zone_name, record_id):
+        zone_id = self.domains.get(zone_name, False)
+        path = '/{}/records/{}'.format(zone_id, record_id)
+        self._request('DELETE', path)
+
+
+class DnsMadeEasyProvider(BaseProvider):
+    '''
+    DNSMadeEasy DNS provider using v2.0 API
+
+    dnsmadeeasy:
+        class: octodns.provider.dnsmadeeasy.DnsMadeEasyProvider
+        # Your DnsMadeEasy api key (required)
+        api_key: env/DNSMADEEASY_API_KEY
+        # Your DnsMadeEasy secret key (required)
+        secret_key: env/DNSMADEEASY_SECRET_KEY
+        # Whether or not to use Sandbox environment
+        # (optional, default is false)
+        sandbox: true
+    '''
+    SUPPORTS_GEO = False
+    SUPPORTS = set(('A', 'AAAA', 'CAA', 'CNAME', 'MX',
+                    'NS', 'PTR', 'SPF', 'SRV', 'TXT'))
+
+    def __init__(self, id, api_key, secret_key, sandbox=False,
+                 ratelimit_delay=0.0, *args, **kwargs):
+        self.log = logging.getLogger('DnsMadeEasyProvider[{}]'.format(id))
+        self.log.debug('__init__: id=%s, api_key=***, secret_key=***, '
+                       'sandbox=%s', id, sandbox)
+        super(DnsMadeEasyProvider, self).__init__(id, *args, **kwargs)
+        self._client = DnsMadeEasyClient(api_key, secret_key, sandbox,
+                                         ratelimit_delay)
+
+        self._zone_records = {}
+
+    def _data_for_multiple(self, _type, records):
+        return {
+            'ttl': records[0]['ttl'],
+            'type': _type,
+            'values': [r['value'] for r in records]
+        }
+
+    _data_for_A = _data_for_multiple
+    _data_for_AAAA = _data_for_multiple
+    _data_for_NS = _data_for_multiple
+
+    def _data_for_CAA(self, _type, records):
+        values = []
+        for record in records:
+            values.append({
+                'flags': record['issuerCritical'],
+                'tag': record['caaType'],
+                'value': record['value'][1:-1]
+            })
+        return {
+            'ttl': records[0]['ttl'],
+            'type': _type,
+            'values': values
+        }
+
+    def _data_for_TXT(self, _type, records):
+        values = [value['value'].replace(';', '\;') for value in records]
+        return {
+            'ttl': records[0]['ttl'],
+            'type': _type,
+            'values': values
+        }
+
+    _data_for_SPF = _data_for_TXT
+
+    def _data_for_MX(self, _type, records):
+        values = []
+        for record in records:
+            values.append({
+                'preference': record['mxLevel'],
+                'exchange': record['value']
+            })
+        return {
+            'ttl': records[0]['ttl'],
+            'type': _type,
+            'values': values
+        }
+
+    def _data_for_single(self, _type, records):
+        record = records[0]
+        return {
+            'ttl': record['ttl'],
+            'type': _type,
+            'value': record['value']
+        }
+
+    _data_for_CNAME = _data_for_single
+    _data_for_PTR = _data_for_single
+
+    def _data_for_SRV(self, _type, records):
+        values = []
+        for record in records:
+            values.append({
+                'port': record['port'],
+                'priority': record['priority'],
+                'target': record['value'],
+                'weight': record['weight']
+            })
+        return {
+            'type': _type,
+            'ttl': records[0]['ttl'],
+            'values': values
+        }
+
+    def zone_records(self, zone):
+        if zone.name not in self._zone_records:
+            try:
+                self._zone_records[zone.name] = \
+                    self._client.records(zone.name)
+            except DnsMadeEasyClientNotFound:
+                return []
+
+        return self._zone_records[zone.name]
+
+    def populate(self, zone, target=False, lenient=False):
+        self.log.debug('populate: name=%s, target=%s, lenient=%s', zone.name,
+                       target, lenient)
+
+        values = defaultdict(lambda: defaultdict(list))
+        for record in self.zone_records(zone):
+            _type = record['type']
+            values[record['name']][record['type']].append(record)
+
+        before = len(zone.records)
+        for name, types in values.items():
+            for _type, records in types.items():
+                data_for = getattr(self, '_data_for_{}'.format(_type))
+                record = Record.new(zone, name, data_for(_type, records),
+                                    source=self, lenient=lenient)
+                zone.add_record(record)
+
+        exists = zone.name in self._zone_records
+        self.log.info('populate:   found %s records, exists=%s',
+                      len(zone.records) - before, exists)
+        return exists
+
+    def _params_for_multiple(self, record):
+        for value in record.values:
+            yield {
+                'value': value,
+                'name': record.name,
+                'ttl': record.ttl,
+                'type': record._type
+            }
+
+    _params_for_A = _params_for_multiple
+    _params_for_AAAA = _params_for_multiple
+
+    # An A record with this name must exist in this domain for
+    # this NS record to be valid. Need to handle checking if
+    # there is an A record before creating NS
+    _params_for_NS = _params_for_multiple
+
+    def _params_for_single(self, record):
+        yield {
+            'value': record.value,
+            'name': record.name,
+            'ttl': record.ttl,
+            'type': record._type
+        }
+
+    _params_for_CNAME = _params_for_single
+    _params_for_PTR = _params_for_single
+
+    def _params_for_MX(self, record):
+        for value in record.values:
+            yield {
+                'value': value.exchange,
+                'name': record.name,
+                'mxLevel': value.preference,
+                'ttl': record.ttl,
+                'type': record._type
+            }
+
+    def _params_for_SRV(self, record):
+        for value in record.values:
+            yield {
+                'value': value.target,
+                'name': record.name,
+                'port': value.port,
+                'priority': value.priority,
+                'ttl': record.ttl,
+                'type': record._type,
+                'weight': value.weight
+            }
+
+    def _params_for_TXT(self, record):
+        # DNSMadeEasy does not want values escaped
+        for value in record.chunked_values:
+            yield {
+                'value': value.replace('\;', ';'),
+                'name': record.name,
+                'ttl': record.ttl,
+                'type': record._type
+            }
+
+    _params_for_SPF = _params_for_TXT
+
+    def _params_for_CAA(self, record):
+        for value in record.values:
+            yield {
+                'value': value.value,
+                'issuerCritical': value.flags,
+                'name': record.name,
+                'caaType': value.tag,
+                'ttl': record.ttl,
+                'type': record._type
+            }
+
+    def _apply_Create(self, change):
+        new = change.new
+        params_for = getattr(self, '_params_for_{}'.format(new._type))
+        for params in params_for(new):
+            self._client.record_create(new.zone.name, params)
+
+    def _apply_Update(self, change):
+        self._apply_Delete(change)
+        self._apply_Create(change)
+
+    def _apply_Delete(self, change):
+        existing = change.existing
+        zone = existing.zone
+        for record in self.zone_records(zone):
+            if existing.name == record['name'] and \
+               existing._type == record['type']:
+                    self._client.record_delete(zone.name, record['id'])
+
+    def _apply(self, plan):
+        desired = plan.desired
+        changes = plan.changes
+        self.log.debug('_apply: zone=%s, len(changes)=%d', desired.name,
+                       len(changes))
+
+        domain_name = desired.name[:-1]
+        try:
+            self._client.domain(domain_name)
+        except DnsMadeEasyClientNotFound:
+            self.log.debug('_apply:   no matching zone, creating domain')
+            self._client.domain_create(domain_name)
+
+        for change in changes:
+            class_name = change.__class__.__name__
+            getattr(self, '_apply_{}'.format(class_name))(change)
+
+        # Clear out the cache if any
+        self._zone_records.pop(desired.name, None)

octodns/provider/dyn.py

@@ -91,7 +91,7 @@ class _CachingDynZone(DynZone):
                 cls.log.debug('get:   fetched')
             except DynectGetError:
                 if not create:
-                    cls.log.debug("get:   does't exist")
+                    cls.log.debug("get:   doesn't exist")
                     return None
                 # this value shouldn't really matter, it's not tied to
                 # whois or anything
@@ -180,11 +180,11 @@ class DynProvider(BaseProvider):
     REGION_CODES = {
         'NA': 11,  # Continental North America
         'SA': 12,  # Continental South America
-        'EU': 13,  # Contentinal Europe
+        'EU': 13,  # Continental Europe
         'AF': 14,  # Continental Africa
-        'AS': 15,  # Contentinal Asia
-        'OC': 16,  # Contentinal Austrailia/Oceania
-        'AN': 17,  # Continental Antartica
+        'AS': 15,  # Continental Asia
+        'OC': 16,  # Continental Australia/Oceania
+        'AN': 17,  # Continental Antarctica
     }
 
     MONITOR_HEADER = 'User-Agent: Dyn Monitor'
@@ -221,7 +221,7 @@ class DynProvider(BaseProvider):
         if DynectSession.get_session() is None:
             # We need to create a new session for this thread and DynectSession
             # creation is not thread-safe so we have to do the locking. If we
-            # don't and multiple sessions start creattion before the the first
+            # don't and multiple sessions start creation before the the first
             # has finished (long time b/c it makes http calls) the subsequent
             # creates will blow away DynectSession._instances, potentially
             # multiple times if there are multiple creates in flight. Only the
@@ -346,7 +346,7 @@ class DynProvider(BaseProvider):
                 try:
                     fqdn, _type = td.label.split(':', 1)
                 except ValueError as e:
-                    self.log.warn("Failed to load TraficDirector '%s': %s",
+                    self.log.warn("Failed to load TrafficDirector '%s': %s",
                                   td.label, e.message)
                     continue
                 tds[fqdn][_type] = td
@@ -408,6 +408,7 @@ class DynProvider(BaseProvider):
         self.log.debug('populate: name=%s, target=%s, lenient=%s', zone.name,
                        target, lenient)
 
+        exists = False
         before = len(zone.records)
 
         self._check_dyn_sess()
@@ -415,10 +416,12 @@ class DynProvider(BaseProvider):
         td_records = set()
         if self.traffic_directors_enabled:
             td_records = self._populate_traffic_directors(zone)
+            exists = True
 
         dyn_zone = _CachingDynZone.get(zone.name[:-1])
 
         if dyn_zone:
+            exists = True
             values = defaultdict(lambda: defaultdict(list))
             for _type, records in dyn_zone.get_all_records().items():
                 if _type == 'soa_records':
@@ -437,8 +440,9 @@ class DynProvider(BaseProvider):
                     if record not in td_records:
                         zone.add_record(record)
 
-        self.log.info('populate:   found %s records',
-                      len(zone.records) - before)
+        self.log.info('populate:   found %s records, exists=%s',
+                      len(zone.records) - before, exists)
+        return exists
 
     def _extra_changes(self, _, desired, changes):
         self.log.debug('_extra_changes: desired=%s', desired.name)

octodns/provider/googlecloud.py

@@ -9,6 +9,7 @@ import shlex
 import time
 from logging import getLogger
 from uuid import uuid4
+import re
 
 from google.cloud import dns
 
@@ -127,9 +128,10 @@ class GoogleCloudProvider(BaseProvider):
             :type return: new google.cloud.dns.ManagedZone
         """
         # Zone name must begin with a letter, end with a letter or digit,
-        # and only contain lowercase letters, digits or dashes
-        zone_name = '{}-{}'.format(
-            dns_name[:-1].replace('.', '-'), uuid4().hex)
+        # and only contain lowercase letters, digits or dashes,
+        # and be 63 characters or less
+        zone_name = 'zone-{}-{}'.format(
+            dns_name.replace('.', '-'), uuid4().hex)[:63]
 
         gcloud_zone = self.gcloud_client.zone(
             name=zone_name,
@@ -202,11 +204,14 @@ class GoogleCloudProvider(BaseProvider):
 
         self.log.debug('populate: name=%s, target=%s, lenient=%s', zone.name,
                        target, lenient)
+
+        exists = False
         before = len(zone.records)
 
         gcloud_zone = self.gcloud_zones.get(zone.name)
 
         if gcloud_zone:
+            exists = True
             for gcloud_record in self._get_gcloud_records(gcloud_zone):
                 if gcloud_record.record_type.upper() not in self.SUPPORTS:
                     continue
@@ -227,7 +232,9 @@ class GoogleCloudProvider(BaseProvider):
                 record = Record.new(zone, record_name, data, source=self)
                 zone.add_record(record)
 
-        self.log.info('populate: found %s records', len(zone.records) - before)
+        self.log.info('populate: found %s records, exists=%s',
+                      len(zone.records) - before, exists)
+        return exists
 
     def _data_for_A(self, gcloud_record):
         return {
@@ -269,12 +276,15 @@ class GoogleCloudProvider(BaseProvider):
 
     _data_for_PTR = _data_for_CNAME
 
+    _fix_semicolons = re.compile(r'(?<!\\);')
+
     def _data_for_SPF(self, gcloud_record):
         if len(gcloud_record.rrdatas) > 1:
             return {
-                'values': gcloud_record.rrdatas}
+                'values': [self._fix_semicolons.sub('\;', rr)
+                           for rr in gcloud_record.rrdatas]}
         return {
-            'value': gcloud_record.rrdatas[0]}
+            'value': self._fix_semicolons.sub('\;', gcloud_record.rrdatas[0])}
 
     def _data_for_SRV(self, gcloud_record):
         return {'values': [{

octodns/provider/ns1.py

@@ -6,8 +6,11 @@ from __future__ import absolute_import, division, print_function, \
     unicode_literals
 
 from logging import getLogger
+from itertools import chain
+from collections import OrderedDict, defaultdict
 from nsone import NSONE
 from nsone.rest.errors import RateLimitException, ResourceException
+from incf.countryutils import transformations
 from time import sleep
 
 from ..record import Record
@@ -22,9 +25,9 @@ class Ns1Provider(BaseProvider):
         class: octodns.provider.ns1.Ns1Provider
         api_key: env/NS1_API_KEY
     '''
-    SUPPORTS_GEO = False
-    SUPPORTS = set(('A', 'AAAA', 'ALIAS', 'CAA', 'CNAME', 'MX', 'NAPTR', 'NS',
-                    'PTR', 'SPF', 'SRV', 'TXT'))
+    SUPPORTS_GEO = True
+    SUPPORTS = set(('A', 'AAAA', 'ALIAS', 'CAA', 'CNAME', 'MX', 'NAPTR',
+                    'NS', 'PTR', 'SPF', 'SRV', 'TXT'))
 
     ZONE_NOT_FOUND_MESSAGE = 'server error: zone not found'
 
@@ -35,11 +38,50 @@ class Ns1Provider(BaseProvider):
         self._client = NSONE(apiKey=api_key)
 
     def _data_for_A(self, _type, record):
-        return {
+        # record meta (which would include geo information is only
+        # returned when getting a record's detail, not from zone detail
+        geo = defaultdict(list)
+        data = {
             'ttl': record['ttl'],
             'type': _type,
-            'values': record['short_answers'],
         }
+        values, codes = [], []
+        if 'answers' not in record:
+            values = record['short_answers']
+        for answer in record.get('answers', []):
+            meta = answer.get('meta', {})
+            if meta:
+                # country + state and country + province are allowed
+                # in that case though, supplying a state/province would
+                # be redundant since the country would supercede in when
+                # resolving the record.  it is syntactically valid, however.
+                country = meta.get('country', [])
+                us_state = meta.get('us_state', [])
+                ca_province = meta.get('ca_province', [])
+                for cntry in country:
+                    cn = transformations.cc_to_cn(cntry)
+                    con = transformations.cn_to_ctca2(cn)
+                    key = '{}-{}'.format(con, cntry)
+                    geo[key].extend(answer['answer'])
+                for state in us_state:
+                    key = 'NA-US-{}'.format(state)
+                    geo[key].extend(answer['answer'])
+                for province in ca_province:
+                    key = 'NA-CA-{}'.format(province)
+                    geo[key].extend(answer['answer'])
+                for code in meta.get('iso_region_code', []):
+                    key = code
+                    geo[key].extend(answer['answer'])
+            else:
+                values.extend(answer['answer'])
+                codes.append([])
+        values = [unicode(x) for x in values]
+        geo = OrderedDict(
+            {unicode(k): [unicode(x) for x in v] for k, v in geo.items()}
+        )
+        data['values'] = values
+        data['geo'] = geo
+        return data
 
     _data_for_AAAA = _data_for_A
 
@@ -69,10 +111,14 @@ class Ns1Provider(BaseProvider):
         }
 
     def _data_for_CNAME(self, _type, record):
+        try:
+            value = record['short_answers'][0]
+        except IndexError:
+            value = None
         return {
             'ttl': record['ttl'],
             'type': _type,
-            'value': record['short_answers'][0],
+            'value': value,
         }
 
     _data_for_ALIAS = _data_for_CNAME
@@ -136,39 +182,81 @@ class Ns1Provider(BaseProvider):
         }
 
     def populate(self, zone, target=False, lenient=False):
-        self.log.debug('populate: name=%s, target=%s, lenient=%s', zone.name,
+        self.log.debug('populate: name=%s, target=%s, lenient=%s',
+                       zone.name,
                        target, lenient)
 
         try:
             nsone_zone = self._client.loadZone(zone.name[:-1])
             records = nsone_zone.data['records']
+            geo_records = nsone_zone.search(has_geo=True)
+            exists = True
         except ResourceException as e:
             if e.message != self.ZONE_NOT_FOUND_MESSAGE:
                 raise
             records = []
+            geo_records = []
+            exists = False
 
         before = len(zone.records)
-        for record in records:
+        # geo information isn't returned from the main endpoint, so we need
+        # to query for all records with geo information
+        zone_hash = {}
+        for record in chain(records, geo_records):
             _type = record['type']
             data_for = getattr(self, '_data_for_{}'.format(_type))
             name = zone.hostname_from_fqdn(record['domain'])
             record = Record.new(zone, name, data_for(_type, record),
                                 source=self, lenient=lenient)
-            zone.add_record(record)
-
-        self.log.info('populate:   found %s records',
-                      len(zone.records) - before)
+            zone_hash[(_type, name)] = record
+        [zone.add_record(r) for r in zone_hash.values()]
+        self.log.info('populate:   found %s records, exists=%s',
+                      len(zone.records) - before, exists)
+        return exists
 
     def _params_for_A(self, record):
-        return {'answers': record.values, 'ttl': record.ttl}
+        params = {'answers': record.values, 'ttl': record.ttl}
+        if hasattr(record, 'geo'):
+            # purposefully set non-geo answers to have an empty meta,
+            # so that we know we did this on purpose if/when troubleshooting
+            params['answers'] = [{"answer": [x], "meta": {}}
+                                 for x in record.values]
+            has_country = False
+            for iso_region, target in record.geo.items():
+                key = 'iso_region_code'
+                value = iso_region
+                if not has_country and \
+                   len(value.split('-')) > 1:  # pragma: nocover
+                    has_country = True
+                for answer in target.values:
+                    params['answers'].append(
+                        {
+                            'answer': [answer],
+                            'meta': {key: [value]},
+                        },
+                    )
+            params['filters'] = []
+            if has_country:
+                params['filters'].append(
+                    {"filter": "shuffle", "config": {}}
+                )
+                params['filters'].append(
+                    {"filter": "geotarget_country", "config": {}}
+                )
+                params['filters'].append(
+                    {"filter": "select_first_n",
+                     "config": {"N": 1}}
+                )
+        self.log.debug("params for A: %s", params)
+        return params
 
     _params_for_AAAA = _params_for_A
     _params_for_NS = _params_for_A
 
     def _params_for_SPF(self, record):
-        # NS1 seems to be the only provider that doesn't want things escaped in
-        # values so we have to strip them here and add them when going the
-        # other way
+        # NS1 seems to be the only provider that doesn't want things
+        # escaped in values so we have to strip them here and add
+        # them when going the other way
         values = [v.replace('\;', ';') for v in record.values]
         return {'answers': values, 'ttl': record.ttl}
 
@@ -210,9 +298,10 @@ class Ns1Provider(BaseProvider):
         try:
             meth(name, **params)
         except RateLimitException as e:
+            period = float(e.period)
             self.log.warn('_apply_Create: rate limit encountered, pausing '
-                          'for %ds and trying again', e.period)
-            sleep(e.period)
+                          'for %ds and trying again', period)
+            sleep(period)
             meth(name, **params)
 
     def _apply_Update(self, nsone_zone, change):
@@ -225,9 +314,10 @@ class Ns1Provider(BaseProvider):
         try:
             record.update(**params)
         except RateLimitException as e:
+            period = float(e.period)
             self.log.warn('_apply_Update: rate limit encountered, pausing '
-                          'for %ds and trying again', e.period)
-            sleep(e.period)
+                          'for %ds and trying again', period)
+            sleep(period)
             record.update(**params)
 
     def _apply_Delete(self, nsone_zone, change):
@@ -238,9 +328,10 @@ class Ns1Provider(BaseProvider):
         try:
             record.delete()
         except RateLimitException as e:
+            period = float(e.period)
             self.log.warn('_apply_Delete: rate limit encountered, pausing '
-                          'for %ds and trying again', e.period)
-            sleep(e.period)
+                          'for %ds and trying again', period)
+            sleep(period)
             record.delete()
 
     def _apply(self, plan):
@@ -260,4 +351,5 @@ class Ns1Provider(BaseProvider):
 
         for change in changes:
             class_name = change.__class__.__name__
-            getattr(self, '_apply_{}'.format(class_name))(nsone_zone, change)
+            getattr(self, '_apply_{}'.format(class_name))(nsone_zone,
+                                                          change)

octodns/provider/ovh.py

@@ -5,10 +5,13 @@
 from __future__ import absolute_import, division, print_function, \
     unicode_literals
 
+import base64
+import binascii
 import logging
 from collections import defaultdict
 
 import ovh
+from ovh import ResourceNotFoundError
 
 from octodns.record import Record
 from .base import BaseProvider
@@ -31,9 +34,12 @@ class OvhProvider(BaseProvider):
     """
 
     SUPPORTS_GEO = False
+    ZONE_NOT_FOUND_MESSAGE = 'This service does not exist'
 
-    SUPPORTS = set(('A', 'AAAA', 'CNAME', 'MX', 'NAPTR', 'NS', 'PTR', 'SPF',
-                    'SRV', 'SSHFP', 'TXT'))
+    # This variable is also used in populate method to filter which OVH record
+    # types are supported by octodns
+    SUPPORTS = set(('A', 'AAAA', 'CNAME', 'DKIM', 'MX', 'NAPTR', 'NS', 'PTR',
+                    'SPF', 'SRV', 'SSHFP', 'TXT'))
 
     def __init__(self, id, endpoint, application_key, application_secret,
                  consumer_key, *args, **kwargs):
@@ -53,7 +59,14 @@ class OvhProvider(BaseProvider):
         self.log.debug('populate: name=%s, target=%s, lenient=%s', zone.name,
                        target, lenient)
         zone_name = zone.name[:-1]
-        records = self.get_records(zone_name=zone_name)
+        try:
+            records = self.get_records(zone_name=zone_name)
+            exists = True
+        except ResourceNotFoundError as e:
+            if e.message != self.ZONE_NOT_FOUND_MESSAGE:
+                raise
+            exists = False
+            records = []
 
         values = defaultdict(lambda: defaultdict(list))
         for record in records:
@@ -62,13 +75,18 @@ class OvhProvider(BaseProvider):
         before = len(zone.records)
         for name, types in values.items():
             for _type, records in types.items():
+                if _type not in self.SUPPORTS:
+                    self.log.warning('Not managed record of type %s, skip',
+                                     _type)
+                    continue
                 data_for = getattr(self, '_data_for_{}'.format(_type))
                 record = Record.new(zone, name, data_for(_type, records),
                                     source=self, lenient=lenient)
                 zone.add_record(record)
 
-        self.log.info('populate:   found %s records',
-                      len(zone.records) - before)
+        self.log.info('populate:   found %s records, exists=%s',
+                      len(zone.records) - before, exists)
+        return exists
 
     def _apply(self, plan):
         desired = plan.desired
@@ -96,7 +114,11 @@ class OvhProvider(BaseProvider):
 
     def _apply_delete(self, zone_name, change):
         existing = change.existing
-        self.delete_records(zone_name, existing._type, existing.name)
+        record_type = existing._type
+        if record_type == "TXT":
+            if self._is_valid_dkim(existing.values[0]):
+                record_type = 'DKIM'
+        self.delete_records(zone_name, record_type, existing.name)
 
     @staticmethod
     def _data_for_multiple(_type, records):
@@ -184,6 +206,15 @@ class OvhProvider(BaseProvider):
             'values': values
         }
 
+    @staticmethod
+    def _data_for_DKIM(_type, records):
+        return {
+            'ttl': records[0]['ttl'],
+            'type': "TXT",
+            'values': [record['target'].replace(';', '\;')
+                       for record in records]
+        }
+
     _data_for_A = _data_for_multiple
     _data_for_AAAA = _data_for_multiple
     _data_for_NS = _data_for_multiple
@@ -238,10 +269,11 @@ class OvhProvider(BaseProvider):
     def _params_for_SRV(record):
         for value in record.values:
             yield {
-                'subDomain': '{} {} {} {}'.format(value.priority,
-                                                  value.weight, value.port,
-                                                  value.target),
-                'target': record.name,
+                'target': '{} {} {} {}'.format(value.priority,
+                                               value.weight,
+                                               value.port,
+                                               value.target),
+                'subDomain': record.name,
                 'ttl': record.ttl,
                 'fieldType': record._type
             }
@@ -250,23 +282,71 @@ class OvhProvider(BaseProvider):
     def _params_for_SSHFP(record):
         for value in record.values:
             yield {
-                'subDomain': '{} {} {}'.format(value.algorithm,
-                                               value.fingerprint_type,
-                                               value.fingerprint),
-                'target': record.name,
+                'target': '{} {} {}'.format(value.algorithm,
+                                            value.fingerprint_type,
+                                            value.fingerprint),
+                'subDomain': record.name,
                 'ttl': record.ttl,
                 'fieldType': record._type
             }
 
+    def _params_for_TXT(self, record):
+        for value in record.values:
+            field_type = 'TXT'
+            if self._is_valid_dkim(value):
+                field_type = 'DKIM'
+                value = value.replace("\;", ";")
+            yield {
+                'target': value,
+                'subDomain': record.name,
+                'ttl': record.ttl,
+                'fieldType': field_type
+            }
+
     _params_for_A = _params_for_multiple
     _params_for_AAAA = _params_for_multiple
     _params_for_NS = _params_for_multiple
     _params_for_SPF = _params_for_multiple
-    _params_for_TXT = _params_for_multiple
 
     _params_for_CNAME = _params_for_single
     _params_for_PTR = _params_for_single
 
+    def _is_valid_dkim(self, value):
+        """Check if value is a valid DKIM"""
+        validator_dict = {'h': lambda val: val in ['sha1', 'sha256'],
+                          's': lambda val: val in ['*', 'email'],
+                          't': lambda val: val in ['y', 's'],
+                          'v': lambda val: val == 'DKIM1',
+                          'k': lambda val: val == 'rsa',
+                          'n': lambda _: True,
+                          'g': lambda _: True}
+
+        splitted = value.split('\;')
+        found_key = False
+        for splitted_value in splitted:
+            sub_split = map(lambda x: x.strip(), splitted_value.split("=", 1))
+            if len(sub_split) < 2:
+                return False
+            key, value = sub_split[0], sub_split[1]
+            if key == "p":
+                is_valid_key = self._is_valid_dkim_key(value)
+                if not is_valid_key:
+                    return False
+                found_key = True
+            else:
+                is_valid_key = validator_dict.get(key, lambda _: False)(value)
+                if not is_valid_key:
+                    return False
+        return found_key
+
+    @staticmethod
+    def _is_valid_dkim_key(key):
+        try:
+            base64.decodestring(key)
+        except binascii.Error:
+            return False
+        return True
+
     def get_records(self, zone_name):
         """
         List all records of a DNS zone

octodns/provider/plan.py

@@ -0,0 +1,285 @@
+#
+#
+#
+
+from __future__ import absolute_import, division, print_function, \
+    unicode_literals
+
+from StringIO import StringIO
+from logging import DEBUG, ERROR, INFO, WARN, getLogger
+from sys import stdout
+
+
+class UnsafePlan(Exception):
+    pass
+
+
+class Plan(object):
+    log = getLogger('Plan')
+
+    MAX_SAFE_UPDATE_PCENT = .3
+    MAX_SAFE_DELETE_PCENT = .3
+    MIN_EXISTING_RECORDS = 10
+
+    def __init__(self, existing, desired, changes, exists,
+                 update_pcent_threshold=MAX_SAFE_UPDATE_PCENT,
+                 delete_pcent_threshold=MAX_SAFE_DELETE_PCENT):
+        self.existing = existing
+        self.desired = desired
+        self.changes = changes
+        self.exists = exists
+        self.update_pcent_threshold = update_pcent_threshold
+        self.delete_pcent_threshold = delete_pcent_threshold
+
+        change_counts = {
+            'Create': 0,
+            'Delete': 0,
+            'Update': 0
+        }
+        for change in changes:
+            change_counts[change.__class__.__name__] += 1
+        self.change_counts = change_counts
+
+        try:
+            existing_n = len(self.existing.records)
+        except AttributeError:
+            existing_n = 0
+
+        self.log.debug('__init__: Creates=%d, Updates=%d, Deletes=%d'
+                       'Existing=%d',
+                       self.change_counts['Create'],
+                       self.change_counts['Update'],
+                       self.change_counts['Delete'], existing_n)
+
+    def raise_if_unsafe(self):
+        # TODO: what is safe really?
+        if self.existing and \
+           len(self.existing.records) >= self.MIN_EXISTING_RECORDS:
+
+            existing_record_count = len(self.existing.records)
+            update_pcent = self.change_counts['Update'] / existing_record_count
+            delete_pcent = self.change_counts['Delete'] / existing_record_count
+
+            if update_pcent > self.update_pcent_threshold:
+                raise UnsafePlan('Too many updates, {:.2f} is over {:.2f} %'
+                                 '({}/{})'.format(
+                                     update_pcent * 100,
+                                     self.update_pcent_threshold * 100,
+                                     self.change_counts['Update'],
+                                     existing_record_count))
+            if delete_pcent > self.delete_pcent_threshold:
+                raise UnsafePlan('Too many deletes, {:.2f} is over {:.2f} %'
+                                 '({}/{})'.format(
+                                     delete_pcent * 100,
+                                     self.delete_pcent_threshold * 100,
+                                     self.change_counts['Delete'],
+                                     existing_record_count))
+
+    def __repr__(self):
+        return 'Creates={}, Updates={}, Deletes={}, Existing Records={}' \
+            .format(self.change_counts['Create'], self.change_counts['Update'],
+                    self.change_counts['Delete'],
+                    len(self.existing.records))
+
+
+class _PlanOutput(object):
+
+    def __init__(self, name):
+        self.name = name
+
+
+class PlanLogger(_PlanOutput):
+
+    def __init__(self, name, level='info'):
+        super(PlanLogger, self).__init__(name)
+        try:
+            self.level = {
+                'debug': DEBUG,
+                'info': INFO,
+                'warn': WARN,
+                'warning': WARN,
+                'error': ERROR
+            }[level.lower()]
+        except (AttributeError, KeyError):
+            raise Exception('Unsupported level: {}'.format(level))
+
+    def run(self, log, plans, *args, **kwargs):
+        hr = '*************************************************************' \
+            '*******************\n'
+        buf = StringIO()
+        buf.write('\n')
+        if plans:
+            current_zone = None
+            for target, plan in plans:
+                if plan.desired.name != current_zone:
+                    current_zone = plan.desired.name
+                    buf.write(hr)
+                    buf.write('* ')
+                    buf.write(current_zone)
+                    buf.write('\n')
+                    buf.write(hr)
+
+                buf.write('* ')
+                buf.write(target.id)
+                buf.write(' (')
+                buf.write(target)
+                buf.write(')\n*   ')
+
+                if plan.exists is False:
+                    buf.write('Create ')
+                    buf.write(str(plan.desired))
+                    buf.write('\n*   ')
+
+                for change in plan.changes:
+                    buf.write(change.__repr__(leader='* '))
+                    buf.write('\n*   ')
+
+                buf.write('Summary: ')
+                buf.write(plan)
+                buf.write('\n')
+        else:
+            buf.write(hr)
+            buf.write('No changes were planned\n')
+        buf.write(hr)
+        buf.write('\n')
+        log.log(self.level, buf.getvalue())
+
+
+def _value_stringifier(record, sep):
+    try:
+        values = [unicode(v) for v in record.values]
+    except AttributeError:
+        values = [record.value]
+    for code, gv in sorted(getattr(record, 'geo', {}).items()):
+        vs = ', '.join([unicode(v) for v in gv.values])
+        values.append('{}: {}'.format(code, vs))
+    return sep.join(values)
+
+
+class PlanMarkdown(_PlanOutput):
+
+    def run(self, plans, fh=stdout, *args, **kwargs):
+        if plans:
+            current_zone = None
+            for target, plan in plans:
+                if plan.desired.name != current_zone:
+                    current_zone = plan.desired.name
+                    fh.write('## ')
+                    fh.write(current_zone)
+                    fh.write('\n\n')
+
+                fh.write('### ')
+                fh.write(target.id)
+                fh.write('\n\n')
+
+                fh.write('| Operation | Name | Type | TTL | Value | Source |\n'
+                         '|--|--|--|--|--|--|\n')
+
+                if plan.exists is False:
+                    fh.write('| Create | ')
+                    fh.write(str(plan.desired))
+                    fh.write(' | | | | |\n')
+
+                for change in plan.changes:
+                    existing = change.existing
+                    new = change.new
+                    record = change.record
+                    fh.write('| ')
+                    fh.write(change.__class__.__name__)
+                    fh.write(' | ')
+                    fh.write(record.name)
+                    fh.write(' | ')
+                    fh.write(record._type)
+                    fh.write(' | ')
+                    # TTL
+                    if existing:
+                        fh.write(unicode(existing.ttl))
+                        fh.write(' | ')
+                        fh.write(_value_stringifier(existing, '; '))
+                        fh.write(' | |\n')
+                        if new:
+                            fh.write('| | | | ')
+
+                    if new:
+                        fh.write(unicode(new.ttl))
+                        fh.write(' | ')
+                        fh.write(_value_stringifier(new, '; '))
+                        fh.write(' | ')
+                        if new.source:
+                            fh.write(new.source.id)
+                        fh.write(' |\n')
+
+                fh.write('\nSummary: ')
+                fh.write(unicode(plan))
+                fh.write('\n\n')
+        else:
+            fh.write('## No changes were planned\n')
+
+
+class PlanHtml(_PlanOutput):
+
+    def run(self, plans, fh=stdout, *args, **kwargs):
+        if plans:
+            current_zone = None
+            for target, plan in plans:
+                if plan.desired.name != current_zone:
+                    current_zone = plan.desired.name
+                    fh.write('<h2>')
+                    fh.write(current_zone)
+                    fh.write('</h2>\n')
+
+                fh.write('<h3>')
+                fh.write(target.id)
+                fh.write('''</h3>
+<table>
+  <tr>
+    <th>Operation</th>
+    <th>Name</th>
+    <th>Type</th>
+    <th>TTL</th>
+    <th>Value</th>
+    <th>Source</th>
+  </tr>
+''')
+
+                if plan.exists is False:
+                    fh.write('  <tr>\n    <td>Create</td>\n    <td colspan=5>')
+                    fh.write(str(plan.desired))
+                    fh.write('</td>\n  </tr>\n')
+
+                for change in plan.changes:
+                    existing = change.existing
+                    new = change.new
+                    record = change.record
+                    fh.write('  <tr>\n    <td>')
+                    fh.write(change.__class__.__name__)
+                    fh.write('</td>\n    <td>')
+                    fh.write(record.name)
+                    fh.write('</td>\n    <td>')
+                    fh.write(record._type)
+                    fh.write('</td>\n')
+                    # TTL
+                    if existing:
+                        fh.write('    <td>')
+                        fh.write(unicode(existing.ttl))
+                        fh.write('</td>\n    <td>')
+                        fh.write(_value_stringifier(existing, '<br/>'))
+                        fh.write('</td>\n    <td></td>\n  </tr>\n')
+                        if new:
+                            fh.write('  <tr>\n    <td colspan=3></td>\n')
+
+                    if new:
+                        fh.write('    <td>')
+                        fh.write(unicode(new.ttl))
+                        fh.write('</td>\n    <td>')
+                        fh.write(_value_stringifier(new, '<br/>'))
+                        fh.write('</td>\n    <td>')
+                        if new.source:
+                            fh.write(new.source.id)
+                        fh.write('</td>\n  </tr>\n')
+
+                fh.write('  <tr>\n    <td colspan=6>Summary: ')
+                fh.write(unicode(plan))
+                fh.write('</td>\n  </tr>\n</table>\n')
+        else:
+            fh.write('<b>No changes were planned</b>')

octodns/provider/powerdns.py

@@ -18,13 +18,14 @@ class PowerDnsBaseProvider(BaseProvider):
                     'PTR', 'SPF', 'SSHFP', 'SRV', 'TXT'))
     TIMEOUT = 5
 
-    def __init__(self, id, host, api_key, port=8081, scheme="http", *args,
-                 **kwargs):
+    def __init__(self, id, host, api_key, port=8081, scheme="http",
+                 timeout=TIMEOUT, *args, **kwargs):
         super(PowerDnsBaseProvider, self).__init__(id, *args, **kwargs)
 
         self.host = host
         self.port = port
         self.scheme = scheme
+        self.timeout = timeout
 
         sess = Session()
         sess.headers.update({'X-API-Key': api_key})
@@ -35,7 +36,7 @@ class PowerDnsBaseProvider(BaseProvider):
 
         url = '{}://{}:{}/api/v1/servers/localhost/{}' \
             .format(self.scheme, self.host, self.port, path)
-        resp = self._sess.request(method, url, json=data, timeout=self.TIMEOUT)
+        resp = self._sess.request(method, url, json=data, timeout=self.timeout)
         self.log.debug('_request:   status=%d', resp.status_code)
         resp.raise_for_status()
         return resp
@@ -177,7 +178,7 @@ class PowerDnsBaseProvider(BaseProvider):
                 raise Exception('PowerDNS unauthorized host={}'
                                 .format(self.host))
             elif e.response.status_code == 422:
-                # 422 means powerdns doesn't know anything about the requsted
+                # 422 means powerdns doesn't know anything about the requested
                 # domain. We'll just ignore it here and leave the zone
                 # untouched.
                 pass
@@ -186,8 +187,10 @@ class PowerDnsBaseProvider(BaseProvider):
                 raise
 
         before = len(zone.records)
+        exists = False
 
         if resp:
+            exists = True
             for rrset in resp.json()['rrsets']:
                 _type = rrset['type']
                 if _type == 'SOA':
@@ -198,8 +201,9 @@ class PowerDnsBaseProvider(BaseProvider):
                                     source=self, lenient=lenient)
                 zone.add_record(record)
 
-        self.log.info('populate:   found %s records',
-                      len(zone.records) - before)
+        self.log.info('populate:   found %s records, exists=%s',
+                      len(zone.records) - before, exists)
+        return exists
 
     def _records_for_multiple(self, record):
         return [{'content': v, 'disabled': False}
@@ -293,8 +297,8 @@ class PowerDnsBaseProvider(BaseProvider):
             return []
 
         # sorting mostly to make things deterministic for testing, but in
-        # theory it let us find what we're after quickier (though sorting would
-        # ve more exepensive.)
+        # theory it let us find what we're after quicker (though sorting would
+        # be more expensive.)
         for record in sorted(existing.records):
             if record == ns:
                 # We've found the top-level NS record, return any changes
@@ -340,7 +344,7 @@ class PowerDnsBaseProvider(BaseProvider):
                                e.response.text)
                 raise
             self.log.info('_apply:   creating zone=%s', desired.name)
-            # 422 means powerdns doesn't know anything about the requsted
+            # 422 means powerdns doesn't know anything about the requested
             # domain. We'll try to create it with the correct records instead
             # of update. Hopefully all the mods are creates :-)
             data = {

octodns/provider/rackspace.py

@@ -0,0 +1,376 @@
+#
+#
+#
+from __future__ import absolute_import, division, print_function, \
+    unicode_literals
+
+from requests import HTTPError, Session, post
+from collections import defaultdict
+import logging
+import string
+import time
+
+from ..record import Record
+from .base import BaseProvider
+
+
+def add_trailing_dot(s):
+    assert s
+    assert s[-1] != '.'
+    return s + '.'
+
+
+def remove_trailing_dot(s):
+    assert s
+    assert s[-1] == '.'
+    return s[:-1]
+
+
+def escape_semicolon(s):
+    assert s
+    return string.replace(s, ';', '\;')
+
+
+def unescape_semicolon(s):
+    assert s
+    return string.replace(s, '\;', ';')
+
+
+class RackspaceProvider(BaseProvider):
+    SUPPORTS_GEO = False
+    SUPPORTS = set(('A', 'AAAA', 'ALIAS', 'CNAME', 'MX', 'NS', 'PTR', 'SPF',
+                    'TXT'))
+    TIMEOUT = 5
+
+    def __init__(self, id, username, api_key, ratelimit_delay=0.0, *args,
+                 **kwargs):
+        '''
+        Rackspace API v1 Provider
+
+        rackspace:
+            class: octodns.provider.rackspace.RackspaceProvider
+            # The the username to authenticate with (required)
+            username: username
+            # The api key that grants access for that user (required)
+            api_key: api-key
+        '''
+        self.log = logging.getLogger('RackspaceProvider[{}]'.format(id))
+        super(RackspaceProvider, self).__init__(id, *args, **kwargs)
+
+        auth_token, dns_endpoint = self._get_auth_token(username, api_key)
+        self.dns_endpoint = dns_endpoint
+
+        self.ratelimit_delay = float(ratelimit_delay)
+
+        sess = Session()
+        sess.headers.update({'X-Auth-Token': auth_token})
+        self._sess = sess
+
+        # Map record type, name, and data to an id when populating so that
+        # we can find the id for update and delete operations.
+        self._id_map = {}
+
+    def _get_auth_token(self, username, api_key):
+        ret = post('https://identity.api.rackspacecloud.com/v2.0/tokens',
+                   json={"auth": {
+                       "RAX-KSKEY:apiKeyCredentials": {"username": username,
+                                                       "apiKey": api_key}}},
+                   )
+        cloud_dns_endpoint = \
+            [x for x in ret.json()['access']['serviceCatalog'] if
+             x['name'] == 'cloudDNS'][0]['endpoints'][0]['publicURL']
+        return ret.json()['access']['token']['id'], cloud_dns_endpoint
+
+    def _get_zone_id_for(self, zone):
+        ret = self._request('GET', 'domains', pagination_key='domains')
+        return [x for x in ret if x['name'] == zone.name[:-1]][0]['id']
+
+    def _request(self, method, path, data=None, pagination_key=None):
+        self.log.debug('_request: method=%s, path=%s', method, path)
+        url = '{}/{}'.format(self.dns_endpoint, path)
+
+        if pagination_key:
+            resp = self._paginated_request_for_url(method, url, data,
+                                                   pagination_key)
+        else:
+            resp = self._request_for_url(method, url, data)
+        time.sleep(self.ratelimit_delay)
+        return resp
+
+    def _request_for_url(self, method, url, data):
+        resp = self._sess.request(method, url, json=data, timeout=self.TIMEOUT)
+        self.log.debug('_request:   status=%d', resp.status_code)
+        resp.raise_for_status()
+        return resp
+
+    def _paginated_request_for_url(self, method, url, data, pagination_key):
+        acc = []
+
+        resp = self._sess.request(method, url, json=data, timeout=self.TIMEOUT)
+        self.log.debug('_request:   status=%d', resp.status_code)
+        resp.raise_for_status()
+        acc.extend(resp.json()[pagination_key])
+
+        next_page = [x for x in resp.json().get('links', []) if
+                     x['rel'] == 'next']
+        if next_page:
+            url = next_page[0]['href']
+            acc.extend(self._paginated_request_for_url(method, url, data,
+                                                       pagination_key))
+            return acc
+        else:
+            return acc
+
+    def _post(self, path, data=None):
+        return self._request('POST', path, data=data)
+
+    def _put(self, path, data=None):
+        return self._request('PUT', path, data=data)
+
+    def _delete(self, path, data=None):
+        return self._request('DELETE', path, data=data)
+
+    @classmethod
+    def _key_for_record(cls, rs_record):
+        return rs_record['type'], rs_record['name'], rs_record['data']
+
+    def _data_for_multiple(self, rrset):
+        return {
+            'type': rrset[0]['type'],
+            'values': [r['data'] for r in rrset],
+            'ttl': rrset[0]['ttl']
+        }
+
+    _data_for_A = _data_for_multiple
+    _data_for_AAAA = _data_for_multiple
+
+    def _data_for_NS(self, rrset):
+        return {
+            'type': rrset[0]['type'],
+            'values': [add_trailing_dot(r['data']) for r in rrset],
+            'ttl': rrset[0]['ttl']
+        }
+
+    def _data_for_single(self, record):
+        return {
+            'type': record[0]['type'],
+            'value': add_trailing_dot(record[0]['data']),
+            'ttl': record[0]['ttl']
+        }
+
+    _data_for_ALIAS = _data_for_single
+    _data_for_CNAME = _data_for_single
+    _data_for_PTR = _data_for_single
+
+    def _data_for_textual(self, rrset):
+        return {
+            'type': rrset[0]['type'],
+            'values': [escape_semicolon(r['data']) for r in rrset],
+            'ttl': rrset[0]['ttl']
+        }
+
+    _data_for_SPF = _data_for_textual
+    _data_for_TXT = _data_for_textual
+
+    def _data_for_MX(self, rrset):
+        values = []
+        for record in rrset:
+            values.append({
+                'priority': record['priority'],
+                'value': add_trailing_dot(record['data']),
+            })
+        return {
+            'type': rrset[0]['type'],
+            'values': values,
+            'ttl': rrset[0]['ttl']
+        }
+
+    def populate(self, zone, target=False, lenient=False):
+        self.log.debug('populate: name=%s', zone.name)
+        resp_data = None
+        try:
+            domain_id = self._get_zone_id_for(zone)
+            resp_data = self._request('GET',
+                                      'domains/{}/records'.format(domain_id),
+                                      pagination_key='records')
+            self.log.debug('populate:   loaded')
+        except HTTPError as e:
+            if e.response.status_code == 401:
+                # Nicer error message for auth problems
+                raise Exception('Rackspace request unauthorized')
+            elif e.response.status_code == 404:
+                # Zone not found leaves the zone empty instead of failing.
+                return False
+            raise
+
+        before = len(zone.records)
+
+        if resp_data:
+            records = self._group_records(resp_data)
+            for record_type, records_of_type in records.items():
+                for raw_record_name, record_set in records_of_type.items():
+                    data_for = getattr(self,
+                                       '_data_for_{}'.format(record_type))
+                    record_name = zone.hostname_from_fqdn(raw_record_name)
+                    record = Record.new(zone, record_name,
+                                        data_for(record_set),
+                                        source=self)
+                    zone.add_record(record)
+
+        self.log.info('populate:   found %s records, exists=True',
+                      len(zone.records) - before)
+        return True
+
+    def _group_records(self, all_records):
+        records = defaultdict(lambda: defaultdict(list))
+        for record in all_records:
+            self._id_map[self._key_for_record(record)] = record['id']
+            records[record['type']][record['name']].append(record)
+        return records
+
+    @staticmethod
+    def _record_for_single(record, value):
+        return {
+            'name': remove_trailing_dot(record.fqdn),
+            'type': record._type,
+            'data': value,
+            'ttl': max(record.ttl, 300),
+        }
+
+    _record_for_A = _record_for_single
+    _record_for_AAAA = _record_for_single
+
+    @staticmethod
+    def _record_for_named(record, value):
+        return {
+            'name': remove_trailing_dot(record.fqdn),
+            'type': record._type,
+            'data': remove_trailing_dot(value),
+            'ttl': max(record.ttl, 300),
+        }
+
+    _record_for_NS = _record_for_named
+    _record_for_ALIAS = _record_for_named
+    _record_for_CNAME = _record_for_named
+    _record_for_PTR = _record_for_named
+
+    @staticmethod
+    def _record_for_textual(record, value):
+        return {
+            'name': remove_trailing_dot(record.fqdn),
+            'type': record._type,
+            'data': unescape_semicolon(value),
+            'ttl': max(record.ttl, 300),
+        }
+
+    _record_for_SPF = _record_for_textual
+    _record_for_TXT = _record_for_textual
+
+    @staticmethod
+    def _record_for_MX(record, value):
+        return {
+            'name': remove_trailing_dot(record.fqdn),
+            'type': record._type,
+            'data': remove_trailing_dot(value.exchange),
+            'ttl': max(record.ttl, 300),
+            'priority': value.preference
+        }
+
+    def _get_values(self, record):
+        try:
+            return record.values
+        except AttributeError:
+            return [record.value]
+
+    def _mod_Create(self, change):
+        return self._create_given_change_values(change,
+                                                self._get_values(change.new))
+
+    def _create_given_change_values(self, change, values):
+        transformer = getattr(self, "_record_for_{}".format(change.new._type))
+        return [transformer(change.new, v) for v in values]
+
+    def _mod_Update(self, change):
+        existing_values = self._get_values(change.existing)
+        new_values = self._get_values(change.new)
+
+        # A reduction in number of values in an update record needs
+        # to get upgraded into a Delete change for the removed values.
+        deleted_values = set(existing_values) - set(new_values)
+        delete_out = self._delete_given_change_values(change, deleted_values)
+
+        # An increase in number of values in an update record needs
+        # to get upgraded into a Create change for the added values.
+        create_values = set(new_values) - set(existing_values)
+        create_out = self._create_given_change_values(change, create_values)
+
+        update_out = []
+        update_values = set(new_values).intersection(set(existing_values))
+        for value in update_values:
+            transformer = getattr(self,
+                                  "_record_for_{}".format(change.new._type))
+            prior_rs_record = transformer(change.existing, value)
+            prior_key = self._key_for_record(prior_rs_record)
+            next_rs_record = transformer(change.new, value)
+            next_key = self._key_for_record(next_rs_record)
+            next_rs_record["id"] = self._id_map[prior_key]
+            del next_rs_record["type"]
+            update_out.append(next_rs_record)
+            self._id_map[next_key] = self._id_map[prior_key]
+            del self._id_map[prior_key]
+        return create_out, update_out, delete_out
+
+    def _mod_Delete(self, change):
+        return self._delete_given_change_values(change, self._get_values(
+            change.existing))
+
+    def _delete_given_change_values(self, change, values):
+        transformer = getattr(self, "_record_for_{}".format(
+            change.existing._type))
+        out = []
+        for value in values:
+            rs_record = transformer(change.existing, value)
+            key = self._key_for_record(rs_record)
+            out.append('id=' + self._id_map[key])
+            del self._id_map[key]
+        return out
+
+    def _apply(self, plan):
+        desired = plan.desired
+        changes = plan.changes
+        self.log.debug('_apply: zone=%s, len(changes)=%d', desired.name,
+                       len(changes))
+
+        # Creates, updates, and deletes are processed by different endpoints
+        # and are broken out by record-set entries; pre-process everything
+        # into these buckets in order to minimize the number of API calls.
+        domain_id = self._get_zone_id_for(desired)
+        creates = []
+        updates = []
+        deletes = []
+        for change in changes:
+            if change.__class__.__name__ == 'Create':
+                creates += self._mod_Create(change)
+            elif change.__class__.__name__ == 'Update':
+                add_creates, add_updates, add_deletes = self._mod_Update(
+                    change)
+                creates += add_creates
+                updates += add_updates
+                deletes += add_deletes
+            else:
+                assert change.__class__.__name__ == 'Delete'
+                deletes += self._mod_Delete(change)
+
+        if deletes:
+            params = "&".join(sorted(deletes))
+            self._delete('domains/{}/records?{}'.format(domain_id, params))
+
+        if updates:
+            data = {"records": sorted(updates, key=lambda v: v['name'])}
+            self._put('domains/{}/records'.format(domain_id), data=data)
+
+        if creates:
+            data = {"records": sorted(creates, key=lambda v: v['type'] +
+                                      v['name'] +
+                                      v.get('data', ''))}
+            self._post('domains/{}/records'.format(domain_id), data=data)

octodns/provider/route53.py

@@ -61,7 +61,7 @@ class _Route53Record(object):
 
     # NOTE: we're using __hash__ and __cmp__ methods that consider
     # _Route53Records equivalent if they have the same class, fqdn, and _type.
-    # Values are ignored. This is usful when computing diffs/changes.
+    # Values are ignored. This is useful when computing diffs/changes.
 
     def __hash__(self):
         'sub-classes should never use this method'
@@ -385,10 +385,10 @@ class Route53Provider(BaseProvider):
             values.append({
                 'order': order,
                 'preference': preference,
-                'flags': flags if flags else None,
-                'service': service if service else None,
-                'regexp': regexp if regexp else None,
-                'replacement': replacement if replacement else None,
+                'flags': flags,
+                'service': service,
+                'regexp': regexp,
+                'replacement': replacement,
             })
         return {
             'type': rrset['Type'],
@@ -451,15 +451,23 @@ class Route53Provider(BaseProvider):
                        target, lenient)
 
         before = len(zone.records)
+        exists = False
 
         zone_id = self._get_zone_id(zone.name)
         if zone_id:
+            exists = True
             records = defaultdict(lambda: defaultdict(list))
             for rrset in self._load_records(zone_id):
                 record_name = zone.hostname_from_fqdn(rrset['Name'])
                 record_name = _octal_replace(record_name)
                 record_type = rrset['Type']
-                if record_type == 'SOA':
+                if record_type not in self.SUPPORTS:
+                    continue
+                if 'AliasTarget' in rrset:
+                    # Alias records are Route53 specific and are not
+                    # portable, so we need to skip them
+                    self.log.warning("%s is an Alias record. Skipping..."
+                                     % rrset['Name'])
                     continue
                 data = getattr(self, '_data_for_{}'.format(record_type))(rrset)
                 records[record_name][record_type].append(data)
@@ -483,8 +491,9 @@ class Route53Provider(BaseProvider):
                                         lenient=lenient)
                     zone.add_record(record)
 
-        self.log.info('populate:   found %s records',
-                      len(zone.records) - before)
+        self.log.info('populate:   found %s records, exists=%s',
+                      len(zone.records) - before, exists)
+        return exists
 
     def _gen_mods(self, action, records):
         '''
@@ -701,7 +710,7 @@ class Route53Provider(BaseProvider):
                    .get('CountryCode', False) == '*':
                     # it's a default record
                     continue
-                # we expect a healtcheck now
+                # we expect a healthcheck now
                 try:
                     health_check_id = rrset['HealthCheckId']
                     health_check = self.health_checks[health_check_id]
@@ -755,7 +764,7 @@ class Route53Provider(BaseProvider):
                               batch_rs_count)
                 # send the batch
                 self._really_apply(batch, zone_id)
-                # start a new batch with the lefovers
+                # start a new batch with the leftovers
                 batch = mods
                 batch_rs_count = mods_rs_count
 

octodns/provider/yaml.py

@@ -31,8 +31,8 @@ class YamlProvider(BaseProvider):
         enforce_order: True
     '''
     SUPPORTS_GEO = True
-    SUPPORTS = set(('A', 'AAAA', 'ALIAS', 'CNAME', 'MX', 'NAPTR', 'NS', 'PTR',
-                   'SSHFP', 'SPF', 'SRV', 'TXT'))
+    SUPPORTS = set(('A', 'AAAA', 'ALIAS', 'CAA', 'CNAME', 'MX', 'NAPTR', 'NS',
+                    'PTR', 'SSHFP', 'SPF', 'SRV', 'TXT'))
 
     def __init__(self, id, directory, default_ttl=3600, enforce_order=True,
                  *args, **kwargs):
@@ -52,7 +52,7 @@ class YamlProvider(BaseProvider):
         if target:
             # When acting as a target we ignore any existing records so that we
             # create a completely new copy
-            return
+            return False
 
         before = len(zone.records)
         filename = join(self.directory, '{}yaml'.format(zone.name))
@@ -69,8 +69,9 @@ class YamlProvider(BaseProvider):
                                             lenient=lenient)
                         zone.add_record(record)
 
-        self.log.info('populate:   found %s records',
+        self.log.info('populate:   found %s records, exists=False',
                       len(zone.records) - before)
+        return False
 
     def _apply(self, plan):
         desired = plan.desired

octodns/record.py

@@ -95,6 +95,10 @@ class Record(object):
         except KeyError:
             raise Exception('Unknown record type: "{}"'.format(_type))
         reasons = _class.validate(name, data)
+        try:
+            lenient |= data['octodns']['lenient']
+        except KeyError:
+            pass
         if reasons:
             if lenient:
                 cls.log.warn(ValidationError.build_message(fqdn, reasons))
@@ -118,11 +122,14 @@ class Record(object):
                        self.__class__.__name__, name)
         self.zone = zone
         # force everything lower-case just to be safe
-        self.name = str(name).lower() if name else name
+        self.name = unicode(name).lower() if name else name
         self.source = source
         self.ttl = int(data['ttl'])
 
         self._octodns = data.get('octodns', {})
+        self.ignored = octodns.get('ignored', False)
+        self.excluded = octodns.get('excluded', [])
+        self.included = octodns.get('included', [])
 
     def _data(self):
         return {'ttl': self.ttl}
@@ -162,7 +169,7 @@ class Record(object):
 
     # NOTE: we're using __hash__ and __cmp__ methods that consider Records
     # equivalent if they have the same name & _type. Values are ignored. This
-    # is usful when computing diffs/changes.
+    # is useful when computing diffs/changes.
 
     def __hash__(self):
         return '{}:{}'.format(self.name, self._type).__hash__()
@@ -195,7 +202,7 @@ class GeoValue(object):
         self.continent_code = match.group('continent_code')
         self.country_code = match.group('country_code')
         self.subdivision_code = match.group('subdivision_code')
-        self.values = values
+        self.values = sorted(values)
 
     @property
     def parents(self):
@@ -224,9 +231,30 @@ class _ValuesMixin(object):
         values = []
         try:
             values = data['values']
+            if not values:
+                values = []
+                reasons.append('missing value(s)')
+            else:
+                # loop through copy of values
+                # remove invalid value from values
+                for value in list(values):
+                    if value is None:
+                        reasons.append('missing value(s)')
+                        values.remove(value)
+                    elif len(value) == 0:
+                        reasons.append('empty value')
+                        values.remove(value)
         except KeyError:
             try:
-                values = [data['value']]
+                value = data['value']
+                if value is None:
+                    reasons.append('missing value(s)')
+                    values = []
+                elif len(value) == 0:
+                    reasons.append('empty value')
+                    values = []
+                else:
+                    values = [value]
             except KeyError:
                 reasons.append('missing value(s)')
 
@@ -251,14 +279,21 @@ class _ValuesMixin(object):
     def _data(self):
         ret = super(_ValuesMixin, self)._data()
         if len(self.values) > 1:
-            ret['values'] = [getattr(v, 'data', v) for v in self.values]
-        else:
+            values = [getattr(v, 'data', v) for v in self.values if v]
+            if len(values) > 1:
+                ret['values'] = values
+            elif len(values) == 1:
+                ret['value'] = values[0]
+        elif len(self.values) == 1:
             v = self.values[0]
-            ret['value'] = getattr(v, 'data', v)
+            if v:
+                ret['value'] = getattr(v, 'data', v)
+
         return ret
 
     def __repr__(self):
-        values = "['{}']".format("', '".join([str(v) for v in self.values]))
+        values = "['{}']".format("', '".join([unicode(v)
+                                              for v in self.values]))
         return '<{} {} {}, {}, {}>'.format(self.__class__.__name__,
                                            self._type, self.ttl,
                                            self.fqdn, values)
@@ -362,6 +397,10 @@ class _ValueMixin(object):
         value = None
         try:
             value = data['value']
+            if value is None:
+                reasons.append('missing value')
+            elif value == '':
+                reasons.append('empty value')
         except KeyError:
             reasons.append('missing value')
         if value:
@@ -379,7 +418,8 @@ class _ValueMixin(object):
 
     def _data(self):
         ret = super(_ValueMixin, self)._data()
-        ret['value'] = getattr(self.value, 'data', self.value)
+        if self.value:
+            ret['value'] = getattr(self.value, 'data', self.value)
         return ret
 
     def __repr__(self):
@@ -485,7 +525,10 @@ class MxValue(object):
     def _validate_value(cls, value):
         reasons = []
         try:
-            int(value.get('preference', None) or value['priority'])
+            try:
+                int(value['preference'])
+            except KeyError:
+                int(value['priority'])
         except KeyError:
             reasons.append('missing preference')
         except ValueError:
@@ -654,8 +697,8 @@ class PtrRecord(_ValueMixin, Record):
 
 
 class SshfpValue(object):
-    VALID_ALGORITHMS = (1, 2)
-    VALID_FINGERPRINT_TYPES = (1,)
+    VALID_ALGORITHMS = (1, 2, 3)
+    VALID_FINGERPRINT_TYPES = (1, 2)
 
     @classmethod
     def _validate_value(cls, value):

octodns/source/base.py

@@ -22,7 +22,7 @@ class BaseSource(object):
 
     def populate(self, zone, target=False, lenient=False):
         '''
-        Loads all zones the provider knows about
+        Loads all records the provider knows about for the provided zone
 
         When `target` is True the populate call is being made to load the
         current state of the provider.
@@ -31,6 +31,9 @@ class BaseSource(object):
         do a "best effort" load of data. That will allow through some common,
         but not best practices stuff that we otherwise would reject. E.g. no
         trailing . or mising escapes for ;.
+
+        When target is True (loading current state) this method should return
+        True if the zone exists or False if it does not.
         '''
         raise NotImplementedError('Abstract base class, populate method '
                                   'missing')

octodns/zone.py

@@ -37,10 +37,10 @@ class Zone(object):
         if not name[-1] == '.':
             raise Exception('Invalid zone name {}, missing ending dot'
                             .format(name))
-        # Force everyting to lowercase just to be safe
-        self.name = str(name).lower() if name else name
+        # Force everything to lowercase just to be safe
+        self.name = unicode(name).lower() if name else name
         self.sub_zones = sub_zones
-        # We're grouping by node, it allows us to efficently search for
+        # We're grouping by node, it allows us to efficiently search for
         # duplicates and detect when CNAMEs co-exist with other records
         self._records = defaultdict(set)
         # optional leading . to match empty hostname
@@ -110,10 +110,29 @@ class Zone(object):
         for record in filter(_is_eligible, self.records):
             if record.ignored:
                 continue
+            elif len(record.included) > 0 and \
+                    target.id not in record.included:
+                self.log.debug('changes:  skipping record=%s %s - %s not'
+                               ' included ', record.fqdn, record._type,
+                               target.id)
+                continue
+            elif target.id in record.excluded:
+                self.log.debug('changes:  skipping record=%s %s - %s '
+                               'excluded ', record.fqdn, record._type,
+                               target.id)
+                continue
             try:
                 desired_record = desired_records[record]
                 if desired_record.ignored:
                     continue
+                elif len(desired_record.included) > 0 and \
+                        target.id not in desired_record.included:
+                    self.log.debug('changes:  skipping record=%s %s - %s'
+                                   'not included ', record.fqdn, record._type,
+                                   target.id)
+                    continue
+                elif target.id in desired_record.excluded:
+                    continue
             except KeyError:
                 if not target.supports(record):
                     self.log.debug('changes:  skipping record=%s %s - %s does '
@@ -141,6 +160,18 @@ class Zone(object):
         for record in filter(_is_eligible, desired.records - self.records):
             if record.ignored:
                 continue
+            elif len(record.included) > 0 and \
+                    target.id not in record.included:
+                self.log.debug('changes:  skipping record=%s %s - %s not'
+                               ' included ', record.fqdn, record._type,
+                               target.id)
+                continue
+            elif target.id in record.excluded:
+                self.log.debug('changes:  skipping record=%s %s - %s '
+                               'excluded ', record.fqdn, record._type,
+                               target.id)
+                continue
+
             if not target.supports(record):
                 self.log.debug('changes:  skipping record=%s %s - %s does not '
                                'support it', record.fqdn, record._type,

requirements-dev.txt

@@ -1,7 +0,0 @@
-coverage
-mock
-nose
-pep8
-pyflakes
-requests_mock
-setuptools>=36.4.0

requirements.txt

@@ -1,23 +0,0 @@
-# These are known good versions. You're free to use others and things will
-# likely work, but no promises are made, especilly if you go older.
-PyYaml==3.12
-azure-mgmt-dns==1.0.1
-azure-common==1.1.6
-boto3==1.4.6
-botocore==1.6.8
-dnspython==1.15.0
-docutils==0.14
-dyn==1.8.0
-futures==3.1.1
-google-cloud==0.27.0
-incf.countryutils==1.0
-ipaddress==1.0.18
-jmespath==0.9.3
-msrestazure==0.4.10
-natsort==5.0.3
-nsone==0.9.14
-ovh==0.4.7
-python-dateutil==2.6.1
-requests==2.13.0
-s3transfer==0.1.10
-six==1.10.0

script/bootstrap

@@ -19,10 +19,10 @@ if [ ! -d "$VENV_NAME" ]; then
 fi
 . "$VENV_NAME/bin/activate"
 
-pip install -U -r requirements.txt
+pip install -e .
 
 if [ "$ENV" != "production" ]; then
-    pip install -U -r requirements-dev.txt
+    pip install -e .[dev,test]
 fi
 
 if [ ! -L ".git/hooks/pre-commit" ]; then

script/lint

@@ -17,5 +17,5 @@ fi
 
 SOURCES="*.py octodns/*.py octodns/*/*.py tests/*.py"
 
-pep8 --ignore=E221,E241,E251 $SOURCES
+pycodestyle --ignore=E221,E241,E251,E722 $SOURCES
 pyflakes $SOURCES

script/release

@@ -11,4 +11,4 @@ git tag -s v$VERSION -m "Release $VERSION"
 git push origin v$VERSION
 echo "Tagged and pushed v$VERSION"
 python setup.py sdist upload
-echo "Updloaded $VERSION"
+echo "Uploaded $VERSION"

setup.cfg

@@ -0,0 +1,68 @@
+[metadata]
+name = octodns
+description = "DNS as code - Tools for managing DNS across multiple providers"
+long_description = file: README.md
+version = attr: octodns.__VERSION__
+author = Ross McFarland
+author_email = [email protected]
+url = https://github.com/github/octodns
+license = MIT
+keywords = dns, providers
+classifiers =
+    License :: OSI Approved :: MIT License
+    Programming Language :: Python
+    Programming Language :: Python :: 2.7
+    Programming Language :: Python :: 3
+    Programming Language :: Python :: 3.3
+    Programming Language :: Python :: 3.4
+    Programming Language :: Python :: 3.5
+    Programming Language :: Python :: 3.6
+
+[options]
+install_requires =
+    PyYaml>=3.12
+    dnspython>=1.15.0
+    futures>=3.1.1
+    incf.countryutils>=1.0
+    ipaddress>=1.0.18
+    natsort>=5.0.3
+    python-dateutil>=2.6.1
+    requests>=2.13.0
+packages = find:
+include_package_data = True
+
+[options.entry_points]
+console_scripts =
+    octodns-compare = octodns.cmds.compare:main
+    octodns-dump = octodns.cmds.dump:main
+    octodns-report = octodns.cmds.report:main
+    octodns-sync = octodns.cmds.sync:main
+    octodns-validate = octodns.cmds.validate:main
+
+[options.packages.find]
+exclude =
+    tests
+
+[options.extras_require]
+dev =
+    azure-mgmt-dns==1.0.1
+    azure-common==1.1.6
+    boto3>=1.4.6
+    botocore>=1.6.8
+    docutils>=0.14
+    dyn>=1.8.0
+    google-cloud>=0.27.0
+    jmespath>=0.9.3
+    msrestazure==0.4.10
+    nsone>=0.9.17
+    ovh>=0.4.7
+    s3transfer>=0.1.10
+    six>=1.10.0
+test =
+    coverage
+    mock
+    nose
+    pycodestyle
+    pyflakes
+    requests_mock
+    setuptools>=36.4.0

setup.py

@@ -1,47 +1,5 @@
 #!/usr/bin/env python
+from setuptools import setup
 
-from os.path import dirname, join
-import octodns
 
-try:
-    from setuptools import find_packages, setup
-except ImportError:
-    from distutils.core import find_packages, setup
-
-cmds = (
-    'compare',
-    'dump',
-    'report',
-    'sync',
-    'validate'
-)
-cmds_dir = join(dirname(__file__), 'octodns', 'cmds')
-console_scripts = {
-    'octodns-{name} = octodns.cmds.{name}:main'.format(name=name)
-    for name in cmds
-}
-
-setup(
-    author='Ross McFarland',
-    author_email='[email protected]',
-    description=octodns.__doc__,
-    entry_points={
-        'console_scripts': console_scripts,
-    },
-    install_requires=[
-        'PyYaml>=3.12',
-        'dnspython>=1.15.0',
-        'futures>=3.0.5',
-        'incf.countryutils>=1.0',
-        'ipaddress>=1.0.18',
-        'natsort>=5.0.3',
-        'python-dateutil>=2.6.0',
-        'requests>=2.13.0'
-    ],
-    license='MIT',
-    long_description=open('README.md').read(),
-    name='octodns',
-    packages=find_packages(),
-    url='https://github.com/github/octodns',
-    version=octodns.__VERSION__,
-)
+setup()

tests/config/bad-plan-output-config.yaml

@@ -0,0 +1,7 @@
+manager:
+  plan_outputs:
+    'bad': 
+      class: octodns.provider.plan.PlanLogger
+      invalid: config
+providers: {}
+zones: {}

tests/config/bad-plan-output-missing-class.yaml

@@ -0,0 +1,5 @@
+manager:
+  plan_outputs:
+    'bad': {}
+providers: {}
+zones: {}

tests/config/unit.tests.yaml

@@ -56,11 +56,23 @@ cname:
   ttl: 300
   type: CNAME
   value: unit.tests.
+excluded:
+  octodns:
+    excluded:
+    - test
+  type: CNAME
+  value: unit.tests.
 ignored:
   octodns:
     ignored: true
   type: A
   value: 9.9.9.9
+included:
+  octodns:
+    included:
+    - test
+  type: CNAME
+  value: unit.tests.
 mx:
   ttl: 300
   type: MX

tests/fixtures/cloudflare-dns_records-page-1.json

@@ -180,7 +180,7 @@
     "per_page": 10,
     "total_pages": 2,
     "count": 10,
-    "total_count": 17
+    "total_count": 19
   },
   "success": true,
   "errors": [],

tests/fixtures/cloudflare-dns_records-page-2.json

@@ -139,14 +139,81 @@
       "meta": {
         "auto_added": false
       }
+    },
+    {
+      "id": "fc12ab34cd5611334422ab3322997656",
+      "type": "CNAME",
+      "name": "included.unit.tests",
+      "content": "unit.tests",
+      "proxiable": true,
+      "proxied": false,
+      "ttl": 3600,
+      "locked": false,
+      "zone_id": "ff12ab34cd5611334422ab3322997650",
+      "zone_name": "unit.tests",
+      "modified_on": "2017-03-11T18:01:43.940682Z",
+      "created_on": "2017-03-11T18:01:43.940682Z",
+      "meta": {
+        "auto_added": false
+      }
+    },
+    {
+      "id": "fc12ab34cd5611334422ab3322997656",
+      "type": "SRV",
+      "name": "_srv._tcp.unit.tests",
+      "data": {
+        "service": "_srv",
+        "proto": "_tcp",
+        "name": "unit.tests",
+        "priority": 12,
+        "weight": 20,
+        "port": 30,
+        "target": "foo-2.unit.tests"
+      },
+      "proxiable": true,
+      "proxied": false,
+      "ttl": 600,
+      "locked": false,
+      "zone_id": "ff12ab34cd5611334422ab3322997650",
+      "zone_name": "unit.tests",
+      "modified_on": "2017-03-11T18:01:43.940682Z",
+      "created_on": "2017-03-11T18:01:43.940682Z",
+      "meta": {
+        "auto_added": false
+      }
+    },
+    {
+      "id": "fc12ab34cd5611334422ab3322997656",
+      "type": "SRV",
+      "name": "_srv._tcp.unit.tests",
+      "data": {
+        "service": "_srv",
+        "proto": "_tcp",
+        "name": "unit.tests",
+        "priority": 10,
+        "weight": 20,
+        "port": 30,
+        "target": "foo-1.unit.tests"
+      },
+      "proxiable": true,
+      "proxied": false,
+      "ttl": 600,
+      "locked": false,
+      "zone_id": "ff12ab34cd5611334422ab3322997650",
+      "zone_name": "unit.tests",
+      "modified_on": "2017-03-11T18:01:43.940682Z",
+      "created_on": "2017-03-11T18:01:43.940682Z",
+      "meta": {
+        "auto_added": false
+      }
     }
   ],
   "result_info": {
     "page": 2,
-    "per_page": 10,
+    "per_page": 11,
     "total_pages": 2,
-    "count": 8,
-    "total_count": 19
+    "count": 9,
+    "total_count": 21
   },
   "success": true,
   "errors": [],

tests/fixtures/digitalocean-page-1.json

@@ -0,0 +1,177 @@
+{
+	"domain_records": [{
+		"id": 11189874,
+		"type": "NS",
+		"name": "@",
+		"data": "ns1.digitalocean.com",
+		"priority": null,
+		"port": null,
+		"ttl": 3600,
+		"weight": null,
+		"flags": null,
+		"tag": null
+	}, {
+		"id": 11189875,
+		"type": "NS",
+		"name": "@",
+		"data": "ns2.digitalocean.com",
+		"priority": null,
+		"port": null,
+		"ttl": 3600,
+		"weight": null,
+		"flags": null,
+		"tag": null
+	}, {
+		"id": 11189876,
+		"type": "NS",
+		"name": "@",
+		"data": "ns3.digitalocean.com",
+		"priority": null,
+		"port": null,
+		"ttl": 3600,
+		"weight": null,
+		"flags": null,
+		"tag": null
+	}, {
+		"id": 11189877,
+		"type": "NS",
+		"name": "under",
+		"data": "ns1.unit.tests",
+		"priority": null,
+		"port": null,
+		"ttl": 3600,
+		"weight": null,
+		"flags": null,
+		"tag": null
+	}, {
+		"id": 11189878,
+		"type": "NS",
+		"name": "under",
+		"data": "ns2.unit.tests",
+		"priority": null,
+		"port": null,
+		"ttl": 3600,
+		"weight": null,
+		"flags": null,
+		"tag": null
+	}, {
+		"id": 11189879,
+		"type": "SRV",
+		"name": "_srv._tcp",
+		"data": "foo-1.unit.tests",
+		"priority": 10,
+		"port": 30,
+		"ttl": 600,
+		"weight": 20,
+		"flags": null,
+		"tag": null
+	}, {
+		"id": 11189880,
+		"type": "SRV",
+		"name": "_srv._tcp",
+		"data": "foo-2.unit.tests",
+		"priority": 12,
+		"port": 30,
+		"ttl": 600,
+		"weight": 20,
+		"flags": null,
+		"tag": null
+	}, {
+		"id": 11189881,
+		"type": "TXT",
+		"name": "txt",
+		"data": "Bah bah black sheep",
+		"priority": null,
+		"port": null,
+		"ttl": 600,
+		"weight": null,
+		"flags": null,
+		"tag": null
+	}, {
+		"id": 11189882,
+		"type": "TXT",
+		"name": "txt",
+		"data": "have you any wool.",
+		"priority": null,
+		"port": null,
+		"ttl": 600,
+		"weight": null,
+		"flags": null,
+		"tag": null
+	}, {
+		"id": 11189883,
+		"type": "A",
+		"name": "@",
+		"data": "1.2.3.4",
+		"priority": null,
+		"port": null,
+		"ttl": 300,
+		"weight": null,
+		"flags": null,
+		"tag": null
+	}, {
+		"id": 11189884,
+		"type": "A",
+		"name": "@",
+		"data": "1.2.3.5",
+		"priority": null,
+		"port": null,
+		"ttl": 300,
+		"weight": null,
+		"flags": null,
+		"tag": null
+	}, {
+		"id": 11189885,
+		"type": "A",
+		"name": "www",
+		"data": "2.2.3.6",
+		"priority": null,
+		"port": null,
+		"ttl": 300,
+		"weight": null,
+		"flags": null,
+		"tag": null
+	}, {
+		"id": 11189886,
+		"type": "MX",
+		"name": "mx",
+		"data": "smtp-4.unit.tests",
+		"priority": 10,
+		"port": null,
+		"ttl": 300,
+		"weight": null,
+		"flags": null,
+		"tag": null
+	}, {
+		"id": 11189887,
+		"type": "MX",
+		"name": "mx",
+		"data": "smtp-2.unit.tests",
+		"priority": 20,
+		"port": null,
+		"ttl": 300,
+		"weight": null,
+		"flags": null,
+		"tag": null
+	}, {
+		"id": 11189888,
+		"type": "MX",
+		"name": "mx",
+		"data": "smtp-3.unit.tests",
+		"priority": 30,
+		"port": null,
+		"ttl": 300,
+		"weight": null,
+		"flags": null,
+		"tag": null
+	}],
+	"links": {
+		"pages": {
+			"last": "https://api.digitalocean.com/v2/domains/unit.tests/records?page=2",
+			"next": "https://api.digitalocean.com/v2/domains/unit.tests/records?page=2"
+		}
+	},
+	"meta": {
+		"total": 21
+	}
+}

tests/fixtures/digitalocean-page-2.json

@@ -0,0 +1,89 @@
+{
+	"domain_records": [{
+		"id": 11189889,
+		"type": "MX",
+		"name": "mx",
+		"data": "smtp-1.unit.tests",
+		"priority": 40,
+		"port": null,
+		"ttl": 300,
+		"weight": null,
+		"flags": null,
+		"tag": null
+	}, {
+		"id": 11189890,
+		"type": "AAAA",
+		"name": "aaaa",
+		"data": "2601:644:500:e210:62f8:1dff:feb8:947a",
+		"priority": null,
+		"port": null,
+		"ttl": 600,
+		"weight": null,
+		"flags": null,
+		"tag": null
+	}, {
+		"id": 11189891,
+		"type": "CNAME",
+		"name": "cname",
+		"data": "@",
+		"priority": null,
+		"port": null,
+		"ttl": 300,
+		"weight": null,
+		"flags": null,
+		"tag": null
+	}, {
+		"id": 11189892,
+		"type": "A",
+		"name": "www.sub",
+		"data": "2.2.3.6",
+		"priority": null,
+		"port": null,
+		"ttl": 300,
+		"weight": null,
+		"flags": null,
+		"tag": null
+	}, {
+		"id": 11189893,
+		"type": "TXT",
+		"name": "txt",
+		"data": "v=DKIM1;k=rsa;s=email;h=sha256;p=A/kinda+of/long/string+with+numb3rs",
+		"priority": null,
+		"port": null,
+		"ttl": 600,
+		"weight": null,
+		"flags": null,
+		"tag": null
+	}, {
+		"id": 11189894,
+		"type": "CAA",
+		"name": "@",
+		"data": "ca.unit.tests",
+		"priority": null,
+		"port": null,
+		"ttl": 3600,
+		"weight": null,
+		"flags": 0,
+		"tag": "issue"
+	}, {
+		"id": 11189895,
+		"type": "CNAME",
+		"name": "included",
+		"data": "@",
+		"priority": null,
+		"port": null,
+		"ttl": 3600,
+		"weight": null,
+		"flags": null,
+		"tag": null
+	}],
+	"links": {
+		"pages": {
+			"first": "https://api.digitalocean.com/v2/domains/unit.tests/records?page=1",
+			"prev": "https://api.digitalocean.com/v2/domains/unit.tests/records?page=1"
+		}
+	},
+	"meta": {
+		"total": 21
+	}
+}

tests/fixtures/dnsimple-page-2.json

@@ -175,12 +175,28 @@
       "system_record": false,
       "created_at": "2017-03-09T15:55:09Z",
       "updated_at": "2017-03-09T15:55:09Z"
+    },
+    {
+      "id": 12188805,
+      "zone_id": "unit.tests",
+      "parent_id": null,
+      "name": "included",
+      "content": "unit.tests",
+      "ttl": 3600,
+      "priority": null,
+      "type": "CNAME",
+      "regions": [
+        "global"
+      ],
+      "system_record": false,
+      "created_at": "2017-03-09T15:55:09Z",
+      "updated_at": "2017-03-09T15:55:09Z"
     }
   ],
   "pagination": {
     "current_page": 2,
     "per_page": 20,
-    "total_entries": 30,
+    "total_entries": 32,
     "total_pages": 2
   }
 }

tests/fixtures/dnsmadeeasy-domains.json

@@ -0,0 +1,16 @@
+{
+	"totalPages": 1,
+	"totalRecords": 1,
+	"data": [{
+		"created": 1511740800000,
+		"folderId": 1990,
+		"gtdEnabled": false,
+		"pendingActionId": 0,
+		"updated": 1511766661574,
+		"processMulti": false,
+		"activeThirdParties": [],
+		"name": "unit.tests",
+		"id": 123123
+	}],
+	"page": 0
+}

tests/fixtures/dnsmadeeasy-records.json

@@ -0,0 +1,312 @@
+{
+	"totalPages": 1,
+	"totalRecords": 21,
+	"data": [{
+		"failover": false,
+		"monitor": false,
+		"sourceId": 123123,
+		"caaType": "issue",
+		"dynamicDns": false,
+		"failed": false,
+		"gtdLocation": "DEFAULT",
+		"hardLink": false,
+		"issuerCritical": 0,
+		"ttl": 3600,
+		"source": 1,
+		"name": "",
+		"value": "\"ca.unit.tests\"",
+		"id": 11189874,
+		"type": "CAA"
+	}, {
+		"failover": false,
+		"monitor": false,
+		"sourceId": 123123,
+		"dynamicDns": false,
+		"failed": false,
+		"gtdLocation": "DEFAULT",
+		"hardLink": false,
+		"ttl": 300,
+		"source": 1,
+		"name": "",
+		"value": "1.2.3.4",
+		"id": 11189875,
+		"type": "A"
+	}, {
+		"failover": false,
+		"monitor": false,
+		"sourceId": 123123,
+		"dynamicDns": false,
+		"failed": false,
+		"gtdLocation": "DEFAULT",
+		"hardLink": false,
+		"ttl": 300,
+		"source": 1,
+		"name": "",
+		"value": "1.2.3.5",
+		"id": 11189876,
+		"type": "A"
+	}, {
+		"failover": false,
+		"monitor": false,
+		"sourceId": 123123,
+		"dynamicDns": false,
+		"failed": false,
+		"gtdLocation": "DEFAULT",
+		"hardLink": false,
+		"ttl": 600,
+		"weight": 20,
+		"source": 1,
+		"name": "_srv._tcp",
+		"value": "foo-1.unit.tests.",
+		"id": 11189877,
+		"priority": 10,
+		"type": "SRV",
+		"port": 30
+	}, {
+		"failover": false,
+		"monitor": false,
+		"sourceId": 123123,
+		"dynamicDns": false,
+		"failed": false,
+		"gtdLocation": "DEFAULT",
+		"hardLink": false,
+		"ttl": 600,
+		"weight": 20,
+		"source": 1,
+		"name": "_srv._tcp",
+		"value": "foo-2.unit.tests.",
+		"id": 11189878,
+		"priority": 12,
+		"type": "SRV",
+		"port": 30
+	}, {
+		"failover": false,
+		"monitor": false,
+		"sourceId": 123123,
+		"dynamicDns": false,
+		"failed": false,
+		"gtdLocation": "DEFAULT",
+		"hardLink": false,
+		"ttl": 600,
+		"source": 1,
+		"name": "aaaa",
+		"value": "2601:644:500:e210:62f8:1dff:feb8:947a",
+		"id": 11189879,
+		"type": "AAAA"
+	}, {
+		"failover": false,
+		"monitor": false,
+		"sourceId": 123123,
+		"dynamicDns": false,
+		"failed": false,
+		"gtdLocation": "DEFAULT",
+		"hardLink": false,
+		"ttl": 300,
+		"source": 1,
+		"name": "cname",
+		"value": "",
+		"id": 11189880,
+		"type": "CNAME"
+	}, {
+		"failover": false,
+		"monitor": false,
+		"sourceId": 123123,
+		"dynamicDns": false,
+		"failed": false,
+		"gtdLocation": "DEFAULT",
+		"hardLink": false,
+		"ttl": 3600,
+		"source": 1,
+		"name": "included",
+		"value": "",
+		"id": 11189881,
+		"type": "CNAME"
+	}, {
+		"failover": false,
+		"monitor": false,
+		"sourceId": 123123,
+		"dynamicDns": false,
+		"failed": false,
+		"gtdLocation": "DEFAULT",
+		"hardLink": false,
+		"mxLevel": 30,
+		"ttl": 300,
+		"source": 1,
+		"name": "mx",
+		"value": "smtp-3.unit.tests.",
+		"id": 11189882,
+		"type": "MX"
+	}, {
+		"failover": false,
+		"monitor": false,
+		"sourceId": 123123,
+		"dynamicDns": false,
+		"failed": false,
+		"gtdLocation": "DEFAULT",
+		"hardLink": false,
+		"mxLevel": 20,
+		"ttl": 300,
+		"source": 1,
+		"name": "mx",
+		"value": "smtp-2.unit.tests.",
+		"id": 11189883,
+		"type": "MX"
+	}, {
+		"failover": false,
+		"monitor": false,
+		"sourceId": 123123,
+		"dynamicDns": false,
+		"failed": false,
+		"gtdLocation": "DEFAULT",
+		"hardLink": false,
+		"mxLevel": 10,
+		"ttl": 300,
+		"source": 1,
+		"name": "mx",
+		"value": "smtp-4.unit.tests.",
+		"id": 11189884,
+		"type": "MX"
+	}, {
+		"failover": false,
+		"monitor": false,
+		"sourceId": 123123,
+		"dynamicDns": false,
+		"failed": false,
+		"gtdLocation": "DEFAULT",
+		"hardLink": false,
+		"mxLevel": 40,
+		"ttl": 300,
+		"source": 1,
+		"name": "mx",
+		"value": "smtp-1.unit.tests.",
+		"id": 11189885,
+		"type": "MX"
+	}, {
+		"failover": false,
+		"monitor": false,
+		"sourceId": 123123,
+		"dynamicDns": false,
+		"failed": false,
+		"gtdLocation": "DEFAULT",
+		"hardLink": false,
+		"ttl": 600,
+		"source": 1,
+		"name": "spf",
+		"value": "\"v=spf1 ip4:192.168.0.1/16-all\"",
+		"id": 11189886,
+		"type": "SPF"
+	}, {
+		"failover": false,
+		"monitor": false,
+		"sourceId": 123123,
+		"dynamicDns": false,
+		"failed": false,
+		"gtdLocation": "DEFAULT",
+		"hardLink": false,
+		"ttl": 600,
+		"source": 1,
+		"name": "txt",
+		"value": "\"Bah bah black sheep\"",
+		"id": 11189887,
+		"type": "TXT"
+	}, {
+		"failover": false,
+		"monitor": false,
+		"sourceId": 123123,
+		"dynamicDns": false,
+		"failed": false,
+		"gtdLocation": "DEFAULT",
+		"hardLink": false,
+		"ttl": 600,
+		"source": 1,
+		"name": "txt",
+		"value": "\"have you any wool.\"",
+		"id": 11189888,
+		"type": "TXT"
+	}, {
+		"failover": false,
+		"monitor": false,
+		"sourceId": 123123,
+		"dynamicDns": false,
+		"failed": false,
+		"gtdLocation": "DEFAULT",
+		"hardLink": false,
+		"ttl": 600,
+		"source": 1,
+		"name": "txt",
+		"value": "\"v=DKIM1;k=rsa;s=email;h=sha256;p=A/kinda+of/long/string+with+numb3rs\"",
+		"id": 11189889,
+		"type": "TXT"
+	}, {
+		"failover": false,
+		"monitor": false,
+		"sourceId": 123123,
+		"dynamicDns": false,
+		"failed": false,
+		"gtdLocation": "DEFAULT",
+		"hardLink": false,
+		"ttl": 3600,
+		"source": 1,
+		"name": "under",
+		"value": "ns1.unit.tests.",
+		"id": 11189890,
+		"type": "NS"
+	}, {
+		"failover": false,
+		"monitor": false,
+		"sourceId": 123123,
+		"dynamicDns": false,
+		"failed": false,
+		"gtdLocation": "DEFAULT",
+		"hardLink": false,
+		"ttl": 3600,
+		"source": 1,
+		"name": "under",
+		"value": "ns2",
+		"id": 11189891,
+		"type": "NS"
+	}, {
+		"failover": false,
+		"monitor": false,
+		"sourceId": 123123,
+		"dynamicDns": false,
+		"failed": false,
+		"gtdLocation": "DEFAULT",
+		"hardLink": false,
+		"ttl": 300,
+		"source": 1,
+		"name": "www",
+		"value": "2.2.3.6",
+		"id": 11189892,
+		"type": "A"
+	}, {
+		"failover": false,
+		"monitor": false,
+		"sourceId": 123123,
+		"dynamicDns": false,
+		"failed": false,
+		"gtdLocation": "DEFAULT",
+		"hardLink": false,
+		"ttl": 300,
+		"source": 1,
+		"name": "www.sub",
+		"value": "2.2.3.6",
+		"id": 11189893,
+		"type": "A"
+	}, {
+		"failover": false,
+		"monitor": false,
+		"sourceId": 123123,
+		"dynamicDns": false,
+		"failed": false,
+		"gtdLocation": "DEFAULT",
+		"hardLink": false,
+		"ttl": 300,
+		"source": 1,
+		"name": "ptr",
+		"value": "foo.bar.com.",
+		"id": 11189894,
+		"type": "PTR"
+	}],
+	"page": 0
+}

tests/fixtures/powerdns-full-data.json

@@ -242,6 +242,18 @@
             ],
             "ttl": 3600,
             "type": "CAA"
+        },
+	{
+            "comments": [],
+            "name": "included.unit.tests.",
+            "records": [
+                {
+                    "content": "unit.tests.",
+                    "disabled": false
+                }
+            ],
+            "ttl": 3600,
+            "type": "CNAME"
         }
     ],
     "serial": 2017012803,

tests/fixtures/rackspace-auth-response.json

@@ -0,0 +1,87 @@
+{
+        "access": {
+                "token": {
+                        "id": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
+                        "expires": "2014-11-24T22:05:39.115Z",
+                        "tenant": {
+                                "id": "110011",
+                                "name": "110011"
+                        },
+                        "RAX-AUTH:authenticatedBy": [
+                                "APIKEY"
+                        ]
+                },
+                "serviceCatalog": [
+                        {
+                                "name": "cloudDatabases",
+                                "endpoints": [
+                                        {
+                                                "publicURL": "https://syd.databases.api.rackspacecloud.com/v1.0/110011",
+                                                "region": "SYD",
+                                                "tenantId": "110011"
+                                        },
+                                        {
+                                                "publicURL": "https://dfw.databases.api.rackspacecloud.com/v1.0/110011",
+                                                "region": "DFW",
+                                                "tenantId": "110011"
+                                        },
+                                        {
+                                                "publicURL": "https://ord.databases.api.rackspacecloud.com/v1.0/110011",
+                                                "region": "ORD",
+                                                "tenantId": "110011"
+                                        },
+                                        {
+                                                "publicURL": "https://iad.databases.api.rackspacecloud.com/v1.0/110011",
+                                                "region": "IAD",
+                                                "tenantId": "110011"
+                                        },
+                                        {
+                                                "publicURL": "https://hkg.databases.api.rackspacecloud.com/v1.0/110011",
+                                                "region": "HKG",
+                                                "tenantId": "110011"
+                                        }
+                                ],
+                                "type": "rax:database"
+                        },
+                        {
+                                "name": "cloudDNS",
+                                "endpoints": [
+                                        {
+                                                "publicURL": "https://dns.api.rackspacecloud.com/v1.0/110011",
+                                                "tenantId": "110011"
+                                        }
+                                ],
+                                "type": "rax:dns"
+                        },
+                        {
+                                "name": "rackCDN",
+                                "endpoints": [
+                                        {
+                                                "internalURL": "https://global.cdn.api.rackspacecloud.com/v1.0/110011",
+                                                "publicURL": "https://global.cdn.api.rackspacecloud.com/v1.0/110011",
+                                                "tenantId": "110011"
+                                        }
+                                ],
+                                "type": "rax:cdn"
+                        }
+                ],
+                "user": {
+                        "id": "123456",
+                        "roles": [
+                                {
+                                        "description": "A Role that allows a user access to keystone Service methods",
+                                        "id": "6",
+                                        "name": "compute:default",
+                                        "tenantId": "110011"
+                                },
+                                {
+                                        "description": "User Admin Role.",
+                                        "id": "3",
+                                        "name": "identity:user-admin"
+                                }
+                        ],
+                        "name": "jsmith",
+                        "RAX-AUTH:defaultRegion": "ORD"
+                }
+        }
+}

tests/fixtures/rackspace-list-domains-response.json

@@ -0,0 +1,68 @@
+{
+  "totalEntries" : 10,
+  "domains" : [ {
+    "name" : "example.com",
+    "id" : 2725233,
+    "comment" : "Optional domain comment...",
+    "updated" : "2011-06-24T01:23:15.000+0000",
+    "accountId" : 1234,
+    "emailAddress" : "[email protected]",
+    "created" : "2011-06-24T01:12:51.000+0000"
+  }, {
+    "name" : "sub1.example.com",
+    "id" : 2725257,
+    "comment" : "1st sample subdomain",
+    "updated" : "2011-06-23T03:09:34.000+0000",
+    "accountId" : 1234,
+    "emailAddress" : "[email protected]",
+    "created" : "2011-06-23T03:09:33.000+0000"
+  }, {
+    "name" : "sub2.example.com",
+    "id" : 2725258,
+    "comment" : "1st sample subdomain",
+    "updated" : "2011-06-23T03:52:55.000+0000",
+    "accountId" : 1234,
+    "emailAddress" : "[email protected]",
+    "created" : "2011-06-23T03:52:55.000+0000"
+  }, {
+    "name" : "north.example.com",
+    "id" : 2725260,
+    "updated" : "2011-06-23T03:53:10.000+0000",
+    "accountId" : 1234,
+    "emailAddress" : "[email protected]",
+    "created" : "2011-06-23T03:53:09.000+0000"
+  }, {
+    "name" : "south.example.com",
+    "id" : 2725261,
+    "comment" : "Final sample subdomain",
+    "updated" : "2011-06-23T03:53:14.000+0000",
+    "accountId" : 1234,
+    "emailAddress" : "[email protected]",
+    "created" : "2011-06-23T03:53:14.000+0000"
+  }, {
+    "name" : "region2.example.net",
+    "id" : 2725352,
+    "updated" : "2011-06-23T20:21:06.000+0000",
+    "accountId" : 1234,
+    "created" : "2011-06-23T19:24:27.000+0000"
+  }, {
+    "name" : "example.org",
+    "id" : 2718984,
+    "updated" : "2011-05-03T14:47:32.000+0000",
+    "accountId" : 1234,
+    "created" : "2011-05-03T14:47:30.000+0000"
+  }, {
+    "name" : "rackspace.example",
+    "id" : 2722346,
+    "updated" : "2011-06-21T15:54:31.000+0000",
+    "accountId" : 1234,
+    "created" : "2011-06-15T19:02:07.000+0000"
+  }, {
+    "name" : "unit.tests",
+    "id" : 2722347,
+    "comment" : "Sample comment",
+    "updated" : "2011-06-21T15:54:31.000+0000",
+    "accountId" : 1234,
+    "created" : "2011-06-15T19:02:07.000+0000"
+  } ]
+}

tests/fixtures/rackspace-sample-recordset-existing-nameservers.json

@@ -0,0 +1,29 @@
+{
+  "totalEntries" : 3,
+  "records" : [{
+    "name" : "unit.tests.",
+    "id" : "A-6822995",
+    "type" : "A",
+    "data" : "1.2.3.4",
+    "updated" : "2011-06-24T01:12:53.000+0000",
+    "ttl" : 600,
+    "created" : "2011-06-24T01:12:53.000+0000"
+  }, {
+    "name" : "unit.tests.",
+    "id" : "NS-454454",
+    "type" : "NS",
+    "data" : "ns1.example.com",
+    "updated" : "2011-06-24T01:12:51.000+0000",
+    "ttl" : 600,
+    "created" : "2011-06-24T01:12:51.000+0000"
+  }, {
+    "name" : "unit.tests.",
+    "id" : "NS-454455",
+    "type" : "NS",
+    "data" : "ns2.example.com",
+    "updated" : "2011-06-24T01:12:52.000+0000",
+    "ttl" : 600,
+    "created" : "2011-06-24T01:12:52.000+0000"
+  }],
+  "links" : []
+}

tests/fixtures/rackspace-sample-recordset-page1.json

@@ -0,0 +1,33 @@
+{
+  "totalEntries" : 6,
+  "records" : [ {
+    "name" : "ftp.example.com",
+    "id" : "A-6817754",
+    "type" : "A",
+    "data" : "192.0.2.8",
+    "updated" : "2011-05-19T13:07:08.000+0000",
+    "ttl" : 5771,
+    "created" : "2011-05-18T19:53:09.000+0000"
+  }, {
+    "name" : "example.com",
+    "id" : "A-6822994",
+    "type" : "A",
+    "data" : "192.0.2.17",
+    "updated" : "2011-06-24T01:12:52.000+0000",
+    "ttl" : 86400,
+    "created" : "2011-06-24T01:12:52.000+0000"
+  }, {
+    "name" : "example.com",
+    "id" : "NS-6251982",
+    "type" : "NS",
+    "data" : "ns.rackspace.com",
+    "updated" : "2011-06-24T01:12:51.000+0000",
+    "ttl" : 3600,
+    "created" : "2011-06-24T01:12:51.000+0000"
+  } ],
+  "links" : [ {
+    "content" : "",
+    "href" : "https://localhost/v1.0/1234/domains/domain_id/records?limit=3&offset=3",
+    "rel" : "next"
+  } ]
+}

tests/fixtures/rackspace-sample-recordset-page2.json

@@ -0,0 +1,35 @@
+{
+  "totalEntries" : 6,
+  "records" : [ {
+    "name" : "example.com",
+    "id" : "NS-6251983",
+    "type" : "NS",
+    "data" : "ns2.rackspace.com",
+    "updated" : "2011-06-24T01:12:51.000+0000",
+    "ttl" : 3600,
+    "created" : "2011-06-24T01:12:51.000+0000"
+  }, {
+    "name" : "example.com",
+    "priority" : 5,
+    "id" : "MX-3151218",
+    "type" : "MX",
+    "data" : "mail.example.com",
+    "updated" : "2011-06-24T01:12:53.000+0000",
+    "ttl" : 3600,
+    "created" : "2011-06-24T01:12:53.000+0000"
+  }, {
+    "name" : "www.example.com",
+    "id" : "CNAME-9778009",
+    "type" : "CNAME",
+    "comment" : "This is a comment on the CNAME record",
+    "data" : "example.com",
+    "updated" : "2011-06-24T01:12:54.000+0000",
+    "ttl" : 5400,
+    "created" : "2011-06-24T01:12:54.000+0000"
+  } ],
+  "links" : [ {
+    "content" : "",
+    "href" : "https://dns.api.rackspacecloud.com/v1.0/1234/domains/domain_id/records?limit=3&offset=0",
+    "rel" : "previous"
+  }]
+}

tests/helpers.py

@@ -18,6 +18,7 @@ class SimpleSource(object):
 class SimpleProvider(object):
     SUPPORTS_GEO = False
     SUPPORTS = set(('A',))
+    id = 'test'
 
     def __init__(self, id='test'):
         pass
@@ -34,6 +35,7 @@ class SimpleProvider(object):
 
 class GeoProvider(object):
     SUPPORTS_GEO = True
+    id = 'test'
 
     def __init__(self, id='test'):
         pass

tests/test_octodns_manager.py

@@ -83,6 +83,19 @@ class TestManager(TestCase):
                 .sync(['unknown.target.'])
         self.assertTrue('unknown target' in ctx.exception.message)
 
+    def test_bad_plan_output_class(self):
+        with self.assertRaises(Exception) as ctx:
+            name = 'bad-plan-output-missing-class.yaml'
+            Manager(get_config_filename(name)).sync()
+        self.assertEquals('plan_output bad is missing class',
+                          ctx.exception.message)
+
+    def test_bad_plan_output_config(self):
+        with self.assertRaises(Exception) as ctx:
+            Manager(get_config_filename('bad-plan-output-config.yaml')).sync()
+        self.assertEqual('Incorrect plan_output config for bad',
+                         ctx.exception.message)
+
     def test_source_only_as_a_target(self):
         with self.assertRaises(Exception) as ctx:
             Manager(get_config_filename('unknown-provider.yaml')) \
@@ -102,12 +115,12 @@ class TestManager(TestCase):
             environ['YAML_TMP_DIR'] = tmpdir.dirname
             tc = Manager(get_config_filename('simple.yaml')) \
                 .sync(dry_run=False)
-            self.assertEquals(19, tc)
+            self.assertEquals(21, tc)
 
             # try with just one of the zones
             tc = Manager(get_config_filename('simple.yaml')) \
                 .sync(dry_run=False, eligible_zones=['unit.tests.'])
-            self.assertEquals(13, tc)
+            self.assertEquals(15, tc)
 
             # the subzone, with 2 targets
             tc = Manager(get_config_filename('simple.yaml')) \
@@ -122,18 +135,18 @@ class TestManager(TestCase):
             # Again with force
             tc = Manager(get_config_filename('simple.yaml')) \
                 .sync(dry_run=False, force=True)
-            self.assertEquals(19, tc)
+            self.assertEquals(21, tc)
 
             # Again with max_workers = 1
             tc = Manager(get_config_filename('simple.yaml'), max_workers=1) \
                 .sync(dry_run=False, force=True)
-            self.assertEquals(19, tc)
+            self.assertEquals(21, tc)
 
             # Include meta
             tc = Manager(get_config_filename('simple.yaml'), max_workers=1,
                          include_meta=True) \
                 .sync(dry_run=False, force=True)
-            self.assertEquals(23, tc)
+            self.assertEquals(25, tc)
 
     def test_eligible_targets(self):
         with TemporaryDirectory() as tmpdir:
@@ -159,13 +172,13 @@ class TestManager(TestCase):
                 fh.write('---\n{}')
 
             changes = manager.compare(['in'], ['dump'], 'unit.tests.')
-            self.assertEquals(13, len(changes))
+            self.assertEquals(15, len(changes))
 
             # Compound sources with varying support
             changes = manager.compare(['in', 'nosshfp'],
                                       ['dump'],
                                       'unit.tests.')
-            self.assertEquals(12, len(changes))
+            self.assertEquals(14, len(changes))
 
             with self.assertRaises(Exception) as ctx:
                 manager.compare(['nope'], ['dump'], 'unit.tests.')

tests/test_octodns_plan.py

@@ -0,0 +1,113 @@
+#
+#
+#
+
+from __future__ import absolute_import, division, print_function, \
+    unicode_literals
+
+from StringIO import StringIO
+from logging import getLogger
+from unittest import TestCase
+
+from octodns.provider.plan import Plan, PlanHtml, PlanLogger, PlanMarkdown
+from octodns.record import Create, Delete, Record, Update
+from octodns.zone import Zone
+
+from helpers import SimpleProvider
+
+
+simple = SimpleProvider()
+zone = Zone('unit.tests.', [])
+existing = Record.new(zone, 'a', {
+    'ttl': 300,
+    'type': 'A',
+    # This matches the zone data above, one to swap, one to leave
+    'values': ['1.1.1.1', '2.2.2.2'],
+})
+new = Record.new(zone, 'a', {
+    'geo': {
+        'AF': ['5.5.5.5'],
+        'NA-US': ['6.6.6.6']
+    },
+    'ttl': 300,
+    'type': 'A',
+    # This leaves one, swaps ones, and adds one
+    'values': ['2.2.2.2', '3.3.3.3', '4.4.4.4'],
+}, simple)
+create = Create(Record.new(zone, 'b', {
+    'ttl': 60,
+    'type': 'CNAME',
+    'value': 'foo.unit.tests.'
+}, simple))
+create2 = Create(Record.new(zone, 'c', {
+    'ttl': 60,
+    'type': 'CNAME',
+    'value': 'foo.unit.tests.'
+}))
+update = Update(existing, new)
+delete = Delete(new)
+changes = [create, create2, delete, update]
+plans = [
+    (simple, Plan(zone, zone, changes, True)),
+    (simple, Plan(zone, zone, changes, False)),
+]
+
+
+class TestPlanLogger(TestCase):
+
+    def test_invalid_level(self):
+        with self.assertRaises(Exception) as ctx:
+            PlanLogger('invalid', 'not-a-level')
+        self.assertEquals('Unsupported level: not-a-level',
+                          ctx.exception.message)
+
+    def test_create(self):
+
+        class MockLogger(object):
+
+            def __init__(self):
+                self.out = StringIO()
+
+            def log(self, level, msg):
+                self.out.write(msg)
+
+        log = MockLogger()
+        PlanLogger('logger').run(log, plans)
+        out = log.out.getvalue()
+        self.assertTrue('Summary: Creates=2, Updates=1, '
+                        'Deletes=1, Existing Records=0' in out)
+
+
+class TestPlanHtml(TestCase):
+    log = getLogger('TestPlanHtml')
+
+    def test_empty(self):
+        out = StringIO()
+        PlanHtml('html').run([], fh=out)
+        self.assertEquals('<b>No changes were planned</b>', out.getvalue())
+
+    def test_simple(self):
+        out = StringIO()
+        PlanHtml('html').run(plans, fh=out)
+        out = out.getvalue()
+        self.assertTrue('    <td colspan=6>Summary: Creates=2, Updates=1, '
+                        'Deletes=1, Existing Records=0</td>' in out)
+
+
+class TestPlanMarkdown(TestCase):
+    log = getLogger('TestPlanMarkdown')
+
+    def test_empty(self):
+        out = StringIO()
+        PlanMarkdown('markdown').run([], fh=out)
+        self.assertEquals('## No changes were planned\n', out.getvalue())
+
+    def test_simple(self):
+        out = StringIO()
+        PlanMarkdown('markdown').run(plans, fh=out)
+        out = out.getvalue()
+        self.assertTrue('## unit.tests.' in out)
+        self.assertTrue('Create | b | CNAME | 60 | foo.unit.tests.' in out)
+        self.assertTrue('Update | a | A | 300 | 1.1.1.1;' in out)
+        self.assertTrue('NA-US: 6.6.6.6 | test' in out)
+        self.assertTrue('Delete | a | A | 300 | 2.2.2.2;' in out)

tests/test_octodns_provider_azuredns.py

@@ -302,7 +302,8 @@ class TestAzureDnsProvider(TestCase):
         record_list = provider._dns_client.record_sets.list_by_dns_zone
         record_list.return_value = rs
 
-        provider.populate(zone)
+        exists = provider.populate(zone)
+        self.assertTrue(exists)
 
         self.assertEquals(len(zone.records), 16)
 
@@ -338,8 +339,10 @@ class TestAzureDnsProvider(TestCase):
             changes.append(Create(i))
             deletes.append(Delete(i))
 
-        self.assertEquals(13, provider.apply(Plan(None, zone, changes)))
-        self.assertEquals(13, provider.apply(Plan(zone, zone, deletes)))
+        self.assertEquals(13, provider.apply(Plan(None, zone,
+                                                  changes, True)))
+        self.assertEquals(13, provider.apply(Plan(zone, zone,
+                                                  deletes, True)))
 
     def test_create_zone(self):
         provider = self._get_provider()
@@ -354,7 +357,8 @@ class TestAzureDnsProvider(TestCase):
         _get = provider._dns_client.zones.get
         _get.side_effect = CloudError(Mock(status=404), err_msg)
 
-        self.assertEquals(13, provider.apply(Plan(None, desired, changes)))
+        self.assertEquals(13, provider.apply(Plan(None, desired, changes,
+                                                  True)))
 
     def test_check_zone_no_create(self):
         provider = self._get_provider()
@@ -374,6 +378,7 @@ class TestAzureDnsProvider(TestCase):
         _get = provider._dns_client.zones.get
         _get.side_effect = CloudError(Mock(status=404), err_msg)
 
-        provider.populate(Zone('unit3.test.', []))
+        exists = provider.populate(Zone('unit3.test.', []))
+        self.assertFalse(exists)
 
         self.assertEquals(len(zone.records), 0)

tests/test_octodns_provider_base.py

@@ -9,7 +9,8 @@ from logging import getLogger
 from unittest import TestCase
 
 from octodns.record import Create, Delete, Record, Update
-from octodns.provider.base import BaseProvider, Plan, UnsafePlan
+from octodns.provider.base import BaseProvider
+from octodns.provider.plan import Plan, UnsafePlan
 from octodns.zone import Zone
 
 
@@ -17,6 +18,7 @@ class HelperProvider(BaseProvider):
     log = getLogger('HelperProvider')
 
     SUPPORTS = set(('A',))
+    id = 'test'
 
     def __init__(self, extra_changes, apply_disabled=False,
                  include_change_callback=None):
@@ -61,14 +63,14 @@ class TestBaseProvider(TestCase):
 
         zone = Zone('unit.tests.', [])
         with self.assertRaises(NotImplementedError) as ctx:
-            HasSupportsGeo('hassupportesgeo').populate(zone)
+            HasSupportsGeo('hassupportsgeo').populate(zone)
         self.assertEquals('Abstract base class, SUPPORTS property missing',
                           ctx.exception.message)
 
         class HasSupports(HasSupportsGeo):
             SUPPORTS = set(('A',))
         with self.assertRaises(NotImplementedError) as ctx:
-            HasSupports('hassupportes').populate(zone)
+            HasSupports('hassupports').populate(zone)
         self.assertEquals('Abstract base class, populate method missing',
                           ctx.exception.message)
 
@@ -92,7 +94,7 @@ class TestBaseProvider(TestCase):
             'value': '1.2.3.4'
         }))
 
-        self.assertTrue(HasSupports('hassupportesgeo')
+        self.assertTrue(HasSupports('hassupportsgeo')
                         .supports(list(zone.records)[0]))
 
         plan = HasPopulate('haspopulate').plan(zone)
@@ -151,7 +153,7 @@ class TestBaseProvider(TestCase):
 
     def test_safe_none(self):
         # No changes is safe
-        Plan(None, None, []).raise_if_unsafe()
+        Plan(None, None, [], True).raise_if_unsafe()
 
     def test_safe_creates(self):
         # Creates are safe when existing records is under MIN_EXISTING_RECORDS
@@ -162,7 +164,8 @@ class TestBaseProvider(TestCase):
             'type': 'A',
             'value': '1.2.3.4',
         })
-        Plan(zone, zone, [Create(record) for i in range(10)]).raise_if_unsafe()
+        Plan(zone, zone, [Create(record) for i in range(10)], True) \
+            .raise_if_unsafe()
 
     def test_safe_min_existing_creates(self):
         # Creates are safe when existing records is over MIN_EXISTING_RECORDS
@@ -175,13 +178,14 @@ class TestBaseProvider(TestCase):
         })
 
         for i in range(int(Plan.MIN_EXISTING_RECORDS)):
-            zone.add_record(Record.new(zone, str(i), {
+            zone.add_record(Record.new(zone, unicode(i), {
                             'ttl': 60,
                             'type': 'A',
                             'value': '2.3.4.5'
                             }))
 
-        Plan(zone, zone, [Create(record) for i in range(10)]).raise_if_unsafe()
+        Plan(zone, zone, [Create(record) for i in range(10)], True) \
+            .raise_if_unsafe()
 
     def test_safe_no_existing(self):
         # existing records fewer than MIN_EXISTING_RECORDS is safe
@@ -193,7 +197,7 @@ class TestBaseProvider(TestCase):
         })
 
         updates = [Update(record, record), Update(record, record)]
-        Plan(zone, zone, updates).raise_if_unsafe()
+        Plan(zone, zone, updates, True).raise_if_unsafe()
 
     def test_safe_updates_min_existing(self):
         # MAX_SAFE_UPDATE_PCENT+1 fails when more
@@ -206,7 +210,7 @@ class TestBaseProvider(TestCase):
         })
 
         for i in range(int(Plan.MIN_EXISTING_RECORDS)):
-            zone.add_record(Record.new(zone, str(i), {
+            zone.add_record(Record.new(zone, unicode(i), {
                             'ttl': 60,
                             'type': 'A',
                             'value': '2.3.4.5'
@@ -217,7 +221,7 @@ class TestBaseProvider(TestCase):
                                       Plan.MAX_SAFE_UPDATE_PCENT) + 1)]
 
         with self.assertRaises(UnsafePlan) as ctx:
-            Plan(zone, zone, changes).raise_if_unsafe()
+            Plan(zone, zone, changes, True).raise_if_unsafe()
 
         self.assertTrue('Too many updates' in ctx.exception.message)
 
@@ -232,7 +236,7 @@ class TestBaseProvider(TestCase):
         })
 
         for i in range(int(Plan.MIN_EXISTING_RECORDS)):
-            zone.add_record(Record.new(zone, str(i), {
+            zone.add_record(Record.new(zone, unicode(i), {
                             'ttl': 60,
                             'type': 'A',
                             'value': '2.3.4.5'
@@ -241,7 +245,7 @@ class TestBaseProvider(TestCase):
                    for i in range(int(Plan.MIN_EXISTING_RECORDS *
                                       Plan.MAX_SAFE_UPDATE_PCENT))]
 
-        Plan(zone, zone, changes).raise_if_unsafe()
+        Plan(zone, zone, changes, True).raise_if_unsafe()
 
     def test_safe_deletes_min_existing(self):
         # MAX_SAFE_DELETE_PCENT+1 fails when more
@@ -254,7 +258,7 @@ class TestBaseProvider(TestCase):
         })
 
         for i in range(int(Plan.MIN_EXISTING_RECORDS)):
-            zone.add_record(Record.new(zone, str(i), {
+            zone.add_record(Record.new(zone, unicode(i), {
                             'ttl': 60,
                             'type': 'A',
                             'value': '2.3.4.5'
@@ -265,7 +269,7 @@ class TestBaseProvider(TestCase):
                                       Plan.MAX_SAFE_DELETE_PCENT) + 1)]
 
         with self.assertRaises(UnsafePlan) as ctx:
-            Plan(zone, zone, changes).raise_if_unsafe()
+            Plan(zone, zone, changes, True).raise_if_unsafe()
 
         self.assertTrue('Too many deletes' in ctx.exception.message)
 
@@ -280,7 +284,7 @@ class TestBaseProvider(TestCase):
         })
 
         for i in range(int(Plan.MIN_EXISTING_RECORDS)):
-            zone.add_record(Record.new(zone, str(i), {
+            zone.add_record(Record.new(zone, unicode(i), {
                             'ttl': 60,
                             'type': 'A',
                             'value': '2.3.4.5'
@@ -289,7 +293,7 @@ class TestBaseProvider(TestCase):
                    for i in range(int(Plan.MIN_EXISTING_RECORDS *
                                       Plan.MAX_SAFE_DELETE_PCENT))]
 
-        Plan(zone, zone, changes).raise_if_unsafe()
+        Plan(zone, zone, changes, True).raise_if_unsafe()
 
     def test_safe_updates_min_existing_override(self):
         safe_pcent = .4
@@ -303,7 +307,7 @@ class TestBaseProvider(TestCase):
         })
 
         for i in range(int(Plan.MIN_EXISTING_RECORDS)):
-            zone.add_record(Record.new(zone, str(i), {
+            zone.add_record(Record.new(zone, unicode(i), {
                             'ttl': 60,
                             'type': 'A',
                             'value': '2.3.4.5'
@@ -314,7 +318,7 @@ class TestBaseProvider(TestCase):
                                       safe_pcent) + 1)]
 
         with self.assertRaises(UnsafePlan) as ctx:
-            Plan(zone, zone, changes,
+            Plan(zone, zone, changes, True,
                  update_pcent_threshold=safe_pcent).raise_if_unsafe()
 
         self.assertTrue('Too many updates' in ctx.exception.message)
@@ -331,7 +335,7 @@ class TestBaseProvider(TestCase):
         })
 
         for i in range(int(Plan.MIN_EXISTING_RECORDS)):
-            zone.add_record(Record.new(zone, str(i), {
+            zone.add_record(Record.new(zone, unicode(i), {
                             'ttl': 60,
                             'type': 'A',
                             'value': '2.3.4.5'
@@ -342,7 +346,7 @@ class TestBaseProvider(TestCase):
                                       safe_pcent) + 1)]
 
         with self.assertRaises(UnsafePlan) as ctx:
-            Plan(zone, zone, changes,
+            Plan(zone, zone, changes, True,
                  delete_pcent_threshold=safe_pcent).raise_if_unsafe()
 
         self.assertTrue('Too many deletes' in ctx.exception.message)

tests/test_octodns_provider_cloudflare.py

@@ -11,7 +11,8 @@ from requests import HTTPError
 from requests_mock import ANY, mock as requests_mock
 from unittest import TestCase
 
-from octodns.record import Record
+from octodns.record import Record, Update
+from octodns.provider.base import Plan
 from octodns.provider.cloudflare import CloudflareProvider
 from octodns.provider.yaml import YamlProvider
 from octodns.zone import Zone
@@ -41,6 +42,20 @@ class TestCloudflareProvider(TestCase):
     def test_populate(self):
         provider = CloudflareProvider('test', 'email', 'token')
 
+        # Bad requests
+        with requests_mock() as mock:
+            mock.get(ANY, status_code=400,
+                     text='{"success":false,"errors":[{"code":1101,'
+                     '"message":"request was invalid"}],'
+                     '"messages":[],"result":null}')
+
+            with self.assertRaises(Exception) as ctx:
+                zone = Zone('unit.tests.', [])
+                provider.populate(zone)
+
+            self.assertEquals('CloudflareError', type(ctx.exception).__name__)
+            self.assertEquals('request was invalid', ctx.exception.message)
+
         # Bad auth
         with requests_mock() as mock:
             mock.get(ANY, status_code=403,
@@ -51,6 +66,8 @@ class TestCloudflareProvider(TestCase):
             with self.assertRaises(Exception) as ctx:
                 zone = Zone('unit.tests.', [])
                 provider.populate(zone)
+            self.assertEquals('CloudflareAuthenticationError',
+                              type(ctx.exception).__name__)
             self.assertEquals('Unknown X-Auth-Key or X-Auth-Email',
                               ctx.exception.message)
 
@@ -61,7 +78,9 @@ class TestCloudflareProvider(TestCase):
             with self.assertRaises(Exception) as ctx:
                 zone = Zone('unit.tests.', [])
                 provider.populate(zone)
-            self.assertEquals('Authentication error', ctx.exception.message)
+            self.assertEquals('CloudflareAuthenticationError',
+                              type(ctx.exception).__name__)
+            self.assertEquals('Cloudflare error', ctx.exception.message)
 
         # General error
         with requests_mock() as mock:
@@ -118,15 +137,16 @@ class TestCloudflareProvider(TestCase):
 
             zone = Zone('unit.tests.', [])
             provider.populate(zone)
-            self.assertEquals(10, len(zone.records))
+            self.assertEquals(12, len(zone.records))
 
             changes = self.expected.changes(zone, provider)
+
             self.assertEquals(0, len(changes))
 
         # re-populating the same zone/records comes out of cache, no calls
         again = Zone('unit.tests.', [])
         provider.populate(again)
-        self.assertEquals(10, len(again.records))
+        self.assertEquals(12, len(again.records))
 
     def test_apply(self):
         provider = CloudflareProvider('test', 'email', 'token')
@@ -140,12 +160,13 @@ class TestCloudflareProvider(TestCase):
                     'id': 42,
                 }
             },  # zone create
-        ] + [None] * 17  # individual record creates
+        ] + [None] * 20  # individual record creates
 
         # non-existant zone, create everything
         plan = provider.plan(self.expected)
-        self.assertEquals(10, len(plan.changes))
-        self.assertEquals(10, provider.apply(plan))
+        self.assertEquals(12, len(plan.changes))
+        self.assertEquals(12, provider.apply(plan))
+        self.assertFalse(plan.exists)
 
         provider._request.assert_has_calls([
             # created the domain
@@ -170,7 +191,7 @@ class TestCloudflareProvider(TestCase):
             }),
         ], True)
         # expected number of total calls
-        self.assertEquals(19, provider._request.call_count)
+        self.assertEquals(22, provider._request.call_count)
 
         provider._request.reset_mock()
 
@@ -265,17 +286,406 @@ class TestCloudflareProvider(TestCase):
         # only see the delete & ttl update, below min-ttl is filtered out
         self.assertEquals(2, len(plan.changes))
         self.assertEquals(2, provider.apply(plan))
+        self.assertTrue(plan.exists)
         # recreate for update, and deletes for the 2 parts of the other
         provider._request.assert_has_calls([
-            call('POST', '/zones/42/dns_records', data={
-                'content': '3.2.3.4',
-                'type': 'A',
-                'name': 'ttl.unit.tests',
-                'ttl': 300}),
-            call('DELETE', '/zones/ff12ab34cd5611334422ab3322997650/'
-                 'dns_records/fc12ab34cd5611334422ab3322997655'),
+            call('PUT', '/zones/ff12ab34cd5611334422ab3322997650/dns_records/'
+                 'fc12ab34cd5611334422ab3322997655',
+                 data={'content': '3.2.3.4',
+                       'type': 'A',
+                       'name': 'ttl.unit.tests',
+                       'ttl': 300}),
             call('DELETE', '/zones/ff12ab34cd5611334422ab3322997650/'
                  'dns_records/fc12ab34cd5611334422ab3322997653'),
             call('DELETE', '/zones/ff12ab34cd5611334422ab3322997650/'
                  'dns_records/fc12ab34cd5611334422ab3322997654')
         ])
+
+    def test_update_add_swap(self):
+        provider = CloudflareProvider('test', 'email', 'token')
+
+        provider.zone_records = Mock(return_value=[
+            {
+                "id": "fc12ab34cd5611334422ab3322997653",
+                "type": "A",
+                "name": "a.unit.tests",
+                "content": "1.1.1.1",
+                "proxiable": True,
+                "proxied": False,
+                "ttl": 300,
+                "locked": False,
+                "zone_id": "ff12ab34cd5611334422ab3322997650",
+                "zone_name": "unit.tests",
+                "modified_on": "2017-03-11T18:01:43.420689Z",
+                "created_on": "2017-03-11T18:01:43.420689Z",
+                "meta": {
+                    "auto_added": False
+                }
+            },
+            {
+                "id": "fc12ab34cd5611334422ab3322997654",
+                "type": "A",
+                "name": "a.unit.tests",
+                "content": "2.2.2.2",
+                "proxiable": True,
+                "proxied": False,
+                "ttl": 300,
+                "locked": False,
+                "zone_id": "ff12ab34cd5611334422ab3322997650",
+                "zone_name": "unit.tests",
+                "modified_on": "2017-03-11T18:01:43.420689Z",
+                "created_on": "2017-03-11T18:01:43.420689Z",
+                "meta": {
+                    "auto_added": False
+                }
+            },
+        ])
+
+        provider._request = Mock()
+        provider._request.side_effect = [
+            self.empty,  # no zones
+            {
+                'result': {
+                    'id': 42,
+                }
+            },  # zone create
+            None,
+            None,
+        ]
+
+        # Add something and delete something
+        zone = Zone('unit.tests.', [])
+        existing = Record.new(zone, 'a', {
+            'ttl': 300,
+            'type': 'A',
+            # This matches the zone data above, one to swap, one to leave
+            'values': ['1.1.1.1', '2.2.2.2'],
+        })
+        new = Record.new(zone, 'a', {
+            'ttl': 300,
+            'type': 'A',
+            # This leaves one, swaps ones, and adds one
+            'values': ['2.2.2.2', '3.3.3.3', '4.4.4.4'],
+        })
+        change = Update(existing, new)
+        plan = Plan(zone, zone, [change], True)
+        provider._apply(plan)
+
+        provider._request.assert_has_calls([
+            call('GET', '/zones', params={'page': 1}),
+            call('POST', '/zones', data={'jump_start': False,
+                                         'name': 'unit.tests'}),
+            call('PUT', '/zones/ff12ab34cd5611334422ab3322997650/dns_records/'
+                 'fc12ab34cd5611334422ab3322997653',
+                 data={'content': '4.4.4.4', 'type': 'A', 'name':
+                       'a.unit.tests', 'ttl': 300}),
+            call('POST', '/zones/42/dns_records',
+                 data={'content': '3.3.3.3', 'type': 'A',
+                       'name': 'a.unit.tests', 'ttl': 300})
+        ])
+
+    def test_update_delete(self):
+        # We need another run so that we can delete, we can't both add and
+        # delete in one go b/c of swaps
+        provider = CloudflareProvider('test', 'email', 'token')
+
+        provider.zone_records = Mock(return_value=[
+            {
+                "id": "fc12ab34cd5611334422ab3322997653",
+                "type": "NS",
+                "name": "unit.tests",
+                "content": "ns1.foo.bar",
+                "proxiable": True,
+                "proxied": False,
+                "ttl": 300,
+                "locked": False,
+                "zone_id": "ff12ab34cd5611334422ab3322997650",
+                "zone_name": "unit.tests",
+                "modified_on": "2017-03-11T18:01:43.420689Z",
+                "created_on": "2017-03-11T18:01:43.420689Z",
+                "meta": {
+                    "auto_added": False
+                }
+            },
+            {
+                "id": "fc12ab34cd5611334422ab3322997654",
+                "type": "NS",
+                "name": "unit.tests",
+                "content": "ns2.foo.bar",
+                "proxiable": True,
+                "proxied": False,
+                "ttl": 300,
+                "locked": False,
+                "zone_id": "ff12ab34cd5611334422ab3322997650",
+                "zone_name": "unit.tests",
+                "modified_on": "2017-03-11T18:01:43.420689Z",
+                "created_on": "2017-03-11T18:01:43.420689Z",
+                "meta": {
+                    "auto_added": False
+                }
+            },
+        ])
+
+        provider._request = Mock()
+        provider._request.side_effect = [
+            self.empty,  # no zones
+            {
+                'result': {
+                    'id': 42,
+                }
+            },  # zone create
+            None,
+            None,
+        ]
+
+        # Add something and delete something
+        zone = Zone('unit.tests.', [])
+        existing = Record.new(zone, '', {
+            'ttl': 300,
+            'type': 'NS',
+            # This matches the zone data above, one to delete, one to leave
+            'values': ['ns1.foo.bar.', 'ns2.foo.bar.'],
+        })
+        new = Record.new(zone, '', {
+            'ttl': 300,
+            'type': 'NS',
+            # This leaves one and deletes one
+            'value': 'ns2.foo.bar.',
+        })
+        change = Update(existing, new)
+        plan = Plan(zone, zone, [change], True)
+        provider._apply(plan)
+
+        provider._request.assert_has_calls([
+            call('GET', '/zones', params={'page': 1}),
+            call('POST', '/zones',
+                 data={'jump_start': False, 'name': 'unit.tests'}),
+            call('DELETE', '/zones/ff12ab34cd5611334422ab3322997650/'
+                 'dns_records/fc12ab34cd5611334422ab3322997653')
+        ])
+
+    def test_alias(self):
+        provider = CloudflareProvider('test', 'email', 'token')
+
+        # A CNAME for us to transform to ALIAS
+        provider.zone_records = Mock(return_value=[
+            {
+                "id": "fc12ab34cd5611334422ab3322997642",
+                "type": "CNAME",
+                "name": "unit.tests",
+                "content": "www.unit.tests",
+                "proxiable": True,
+                "proxied": False,
+                "ttl": 300,
+                "locked": False,
+                "zone_id": "ff12ab34cd5611334422ab3322997650",
+                "zone_name": "unit.tests",
+                "modified_on": "2017-03-11T18:01:43.420689Z",
+                "created_on": "2017-03-11T18:01:43.420689Z",
+                "meta": {
+                    "auto_added": False
+                }
+            },
+        ])
+
+        zone = Zone('unit.tests.', [])
+        provider.populate(zone)
+        self.assertEquals(1, len(zone.records))
+        record = list(zone.records)[0]
+        self.assertEquals('', record.name)
+        self.assertEquals('unit.tests.', record.fqdn)
+        self.assertEquals('ALIAS', record._type)
+        self.assertEquals('www.unit.tests.', record.value)
+
+        # Make sure we transform back to CNAME going the other way
+        contents = provider._gen_contents(record)
+        self.assertEquals({
+            'content': u'www.unit.tests.',
+            'name': 'unit.tests',
+            'ttl': 300,
+            'type': 'CNAME'
+        }, list(contents)[0])
+
+    def test_cdn(self):
+        provider = CloudflareProvider('test', 'email', 'token', True)
+
+        # A CNAME for us to transform to ALIAS
+        provider.zone_records = Mock(return_value=[
+            {
+                "id": "fc12ab34cd5611334422ab3322997642",
+                "type": "CNAME",
+                "name": "cname.unit.tests",
+                "content": "www.unit.tests",
+                "proxiable": True,
+                "proxied": True,
+                "ttl": 300,
+                "locked": False,
+                "zone_id": "ff12ab34cd5611334422ab3322997650",
+                "zone_name": "unit.tests",
+                "modified_on": "2017-03-11T18:01:43.420689Z",
+                "created_on": "2017-03-11T18:01:43.420689Z",
+                "meta": {
+                    "auto_added": False
+                }
+            },
+            {
+                "id": "fc12ab34cd5611334422ab3322997642",
+                "type": "A",
+                "name": "a.unit.tests",
+                "content": "1.1.1.1",
+                "proxiable": True,
+                "proxied": True,
+                "ttl": 300,
+                "locked": False,
+                "zone_id": "ff12ab34cd5611334422ab3322997650",
+                "zone_name": "unit.tests",
+                "modified_on": "2017-03-11T18:01:43.420689Z",
+                "created_on": "2017-03-11T18:01:43.420689Z",
+                "meta": {
+                    "auto_added": False
+                }
+            },
+            {
+                "id": "fc12ab34cd5611334422ab3322997642",
+                "type": "A",
+                "name": "a.unit.tests",
+                "content": "1.1.1.2",
+                "proxiable": True,
+                "proxied": True,
+                "ttl": 300,
+                "locked": False,
+                "zone_id": "ff12ab34cd5611334422ab3322997650",
+                "zone_name": "unit.tests",
+                "modified_on": "2017-03-11T18:01:43.420689Z",
+                "created_on": "2017-03-11T18:01:43.420689Z",
+                "meta": {
+                    "auto_added": False
+                }
+            },
+            {
+                "id": "fc12ab34cd5611334422ab3322997642",
+                "type": "A",
+                "name": "multi.unit.tests",
+                "content": "1.1.1.3",
+                "proxiable": True,
+                "proxied": True,
+                "ttl": 300,
+                "locked": False,
+                "zone_id": "ff12ab34cd5611334422ab3322997650",
+                "zone_name": "unit.tests",
+                "modified_on": "2017-03-11T18:01:43.420689Z",
+                "created_on": "2017-03-11T18:01:43.420689Z",
+                "meta": {
+                    "auto_added": False
+                }
+            },
+            {
+                "id": "fc12ab34cd5611334422ab3322997642",
+                "type": "AAAA",
+                "name": "multi.unit.tests",
+                "content": "::1",
+                "proxiable": True,
+                "proxied": True,
+                "ttl": 300,
+                "locked": False,
+                "zone_id": "ff12ab34cd5611334422ab3322997650",
+                "zone_name": "unit.tests",
+                "modified_on": "2017-03-11T18:01:43.420689Z",
+                "created_on": "2017-03-11T18:01:43.420689Z",
+                "meta": {
+                    "auto_added": False
+                }
+            },
+        ])
+
+        zone = Zone('unit.tests.', [])
+        provider.populate(zone)
+
+        # the two A records get merged into one CNAME record pointing to
+        # the CDN.
+        self.assertEquals(3, len(zone.records))
+
+        record = list(zone.records)[0]
+        self.assertEquals('multi', record.name)
+        self.assertEquals('multi.unit.tests.', record.fqdn)
+        self.assertEquals('CNAME', record._type)
+        self.assertEquals('multi.unit.tests.cdn.cloudflare.net.', record.value)
+
+        record = list(zone.records)[1]
+        self.assertEquals('cname', record.name)
+        self.assertEquals('cname.unit.tests.', record.fqdn)
+        self.assertEquals('CNAME', record._type)
+        self.assertEquals('cname.unit.tests.cdn.cloudflare.net.', record.value)
+
+        record = list(zone.records)[2]
+        self.assertEquals('a', record.name)
+        self.assertEquals('a.unit.tests.', record.fqdn)
+        self.assertEquals('CNAME', record._type)
+        self.assertEquals('a.unit.tests.cdn.cloudflare.net.', record.value)
+
+        # CDN enabled records can't be updated, we don't know the real values
+        # never point a Cloudflare record to itself.
+        wanted = Zone('unit.tests.', [])
+        wanted.add_record(Record.new(wanted, 'cname', {
+            'ttl': 300,
+            'type': 'CNAME',
+            'value': 'change.unit.tests.cdn.cloudflare.net.'
+        }))
+        wanted.add_record(Record.new(wanted, 'new', {
+            'ttl': 300,
+            'type': 'CNAME',
+            'value': 'new.unit.tests.cdn.cloudflare.net.'
+        }))
+        wanted.add_record(Record.new(wanted, 'created', {
+            'ttl': 300,
+            'type': 'CNAME',
+            'value': 'www.unit.tests.'
+        }))
+
+        plan = provider.plan(wanted)
+        self.assertEquals(1, len(plan.changes))
+
+    def test_cdn_alias(self):
+        provider = CloudflareProvider('test', 'email', 'token', True)
+
+        # A CNAME for us to transform to ALIAS
+        provider.zone_records = Mock(return_value=[
+            {
+                "id": "fc12ab34cd5611334422ab3322997642",
+                "type": "CNAME",
+                "name": "unit.tests",
+                "content": "www.unit.tests",
+                "proxiable": True,
+                "proxied": True,
+                "ttl": 300,
+                "locked": False,
+                "zone_id": "ff12ab34cd5611334422ab3322997650",
+                "zone_name": "unit.tests",
+                "modified_on": "2017-03-11T18:01:43.420689Z",
+                "created_on": "2017-03-11T18:01:43.420689Z",
+                "meta": {
+                    "auto_added": False
+                }
+            },
+        ])
+
+        zone = Zone('unit.tests.', [])
+        provider.populate(zone)
+        self.assertEquals(1, len(zone.records))
+        record = list(zone.records)[0]
+        self.assertEquals('', record.name)
+        self.assertEquals('unit.tests.', record.fqdn)
+        self.assertEquals('ALIAS', record._type)
+        self.assertEquals('unit.tests.cdn.cloudflare.net.', record.value)
+
+        # CDN enabled records can't be updated, we don't know the real values
+        # never point a Cloudflare record to itself.
+        wanted = Zone('unit.tests.', [])
+        wanted.add_record(Record.new(wanted, '', {
+            'ttl': 300,
+            'type': 'ALIAS',
+            'value': 'change.unit.tests.cdn.cloudflare.net.'
+        }))
+
+        plan = provider.plan(wanted)
+        self.assertEquals(False, hasattr(plan, 'changes'))

tests/test_octodns_provider_digitalocean.py

@@ -0,0 +1,243 @@
+#
+#
+#
+
+
+from __future__ import absolute_import, division, print_function, \
+    unicode_literals
+
+from mock import Mock, call
+from os.path import dirname, join
+from requests import HTTPError
+from requests_mock import ANY, mock as requests_mock
+from unittest import TestCase
+
+from octodns.record import Record
+from octodns.provider.digitalocean import DigitalOceanClientNotFound, \
+    DigitalOceanProvider
+from octodns.provider.yaml import YamlProvider
+from octodns.zone import Zone
+
+
+class TestDigitalOceanProvider(TestCase):
+    expected = Zone('unit.tests.', [])
+    source = YamlProvider('test', join(dirname(__file__), 'config'))
+    source.populate(expected)
+
+    # Our test suite differs a bit, add our NS and remove the simple one
+    expected.add_record(Record.new(expected, 'under', {
+        'ttl': 3600,
+        'type': 'NS',
+        'values': [
+            'ns1.unit.tests.',
+            'ns2.unit.tests.',
+        ]
+    }))
+    for record in list(expected.records):
+        if record.name == 'sub' and record._type == 'NS':
+            expected._remove_record(record)
+            break
+
+    def test_populate(self):
+        provider = DigitalOceanProvider('test', 'token')
+
+        # Bad auth
+        with requests_mock() as mock:
+            mock.get(ANY, status_code=401,
+                     text='{"id":"unauthorized",'
+                     '"message":"Unable to authenticate you."}')
+
+            with self.assertRaises(Exception) as ctx:
+                zone = Zone('unit.tests.', [])
+                provider.populate(zone)
+            self.assertEquals('Unauthorized', ctx.exception.message)
+
+        # General error
+        with requests_mock() as mock:
+            mock.get(ANY, status_code=502, text='Things caught fire')
+
+            with self.assertRaises(HTTPError) as ctx:
+                zone = Zone('unit.tests.', [])
+                provider.populate(zone)
+            self.assertEquals(502, ctx.exception.response.status_code)
+
+        # Non-existant zone doesn't populate anything
+        with requests_mock() as mock:
+            mock.get(ANY, status_code=404,
+                     text='{"id":"not_found","message":"The resource you '
+                     'were accessing could not be found."}')
+
+            zone = Zone('unit.tests.', [])
+            provider.populate(zone)
+            self.assertEquals(set(), zone.records)
+
+        # No diffs == no changes
+        with requests_mock() as mock:
+            base = 'https://api.digitalocean.com/v2/domains/unit.tests/' \
+                'records?page='
+            with open('tests/fixtures/digitalocean-page-1.json') as fh:
+                mock.get('{}{}'.format(base, 1), text=fh.read())
+            with open('tests/fixtures/digitalocean-page-2.json') as fh:
+                mock.get('{}{}'.format(base, 2), text=fh.read())
+
+            zone = Zone('unit.tests.', [])
+            provider.populate(zone)
+            self.assertEquals(12, len(zone.records))
+            changes = self.expected.changes(zone, provider)
+            self.assertEquals(0, len(changes))
+
+        # 2nd populate makes no network calls/all from cache
+        again = Zone('unit.tests.', [])
+        provider.populate(again)
+        self.assertEquals(12, len(again.records))
+
+        # bust the cache
+        del provider._zone_records[zone.name]
+
+    def test_apply(self):
+        provider = DigitalOceanProvider('test', 'token')
+
+        resp = Mock()
+        resp.json = Mock()
+        provider._client._request = Mock(return_value=resp)
+
+        domain_after_creation = {
+            "domain_records": [{
+                "id": 11189874,
+                "type": "NS",
+                "name": "@",
+                "data": "ns1.digitalocean.com",
+                "priority": None,
+                "port": None,
+                "ttl": 3600,
+                "weight": None,
+                "flags": None,
+                "tag": None
+            }, {
+                "id": 11189875,
+                "type": "NS",
+                "name": "@",
+                "data": "ns2.digitalocean.com",
+                "priority": None,
+                "port": None,
+                "ttl": 3600,
+                "weight": None,
+                "flags": None,
+                "tag": None
+            }, {
+                "id": 11189876,
+                "type": "NS",
+                "name": "@",
+                "data": "ns3.digitalocean.com",
+                "priority": None,
+                "port": None,
+                "ttl": 3600,
+                "weight": None,
+                "flags": None,
+                "tag": None
+            }, {
+                "id": 11189877,
+                "type": "A",
+                "name": "@",
+                "data": "192.0.2.1",
+                "priority": None,
+                "port": None,
+                "ttl": 3600,
+                "weight": None,
+                "flags": None,
+                "tag": None
+            }],
+            "links": {},
+            "meta": {
+                "total": 4
+            }
+        }
+
+        # non-existant domain, create everything
+        resp.json.side_effect = [
+            DigitalOceanClientNotFound,  # no zone in populate
+            DigitalOceanClientNotFound,  # no domain during apply
+            domain_after_creation
+        ]
+        plan = provider.plan(self.expected)
+
+        # No root NS, no ignored, no excluded, no unsupported
+        n = len(self.expected.records) - 7
+        self.assertEquals(n, len(plan.changes))
+        self.assertEquals(n, provider.apply(plan))
+        self.assertFalse(plan.exists)
+
+        provider._client._request.assert_has_calls([
+            # created the domain
+            call('POST', '/domains', data={'ip_address': '192.0.2.1',
+                                           'name': 'unit.tests'}),
+            # get all records in newly created zone
+            call('GET', '/domains/unit.tests/records', {'page': 1}),
+            # delete the initial A record
+            call('DELETE', '/domains/unit.tests/records/11189877'),
+            # created at least one of the record with expected data
+            call('POST', '/domains/unit.tests/records', data={
+                'name': '_srv._tcp',
+                'weight': 20,
+                'data': 'foo-1.unit.tests.',
+                'priority': 10,
+                'ttl': 600,
+                'type': 'SRV',
+                'port': 30
+            }),
+        ])
+        self.assertEquals(24, provider._client._request.call_count)
+
+        provider._client._request.reset_mock()
+
+        # delete 1 and update 1
+        provider._client.records = Mock(return_value=[
+            {
+                'id': 11189897,
+                'name': 'www',
+                'data': '1.2.3.4',
+                'ttl': 300,
+                'type': 'A',
+            },
+            {
+                'id': 11189898,
+                'name': 'www',
+                'data': '2.2.3.4',
+                'ttl': 300,
+                'type': 'A',
+            },
+            {
+                'id': 11189899,
+                'name': 'ttl',
+                'data': '3.2.3.4',
+                'ttl': 600,
+                'type': 'A',
+            }
+        ])
+
+        # Domain exists, we don't care about return
+        resp.json.side_effect = ['{}']
+
+        wanted = Zone('unit.tests.', [])
+        wanted.add_record(Record.new(wanted, 'ttl', {
+            'ttl': 300,
+            'type': 'A',
+            'value': '3.2.3.4'
+        }))
+
+        plan = provider.plan(wanted)
+        self.assertTrue(plan.exists)
+        self.assertEquals(2, len(plan.changes))
+        self.assertEquals(2, provider.apply(plan))
+        # recreate for update, and delete for the 2 parts of the other
+        provider._client._request.assert_has_calls([
+            call('POST', '/domains/unit.tests/records', data={
+                'data': '3.2.3.4',
+                'type': 'A',
+                'name': 'ttl',
+                'ttl': 300
+            }),
+            call('DELETE', '/domains/unit.tests/records/11189899'),
+            call('DELETE', '/domains/unit.tests/records/11189897'),
+            call('DELETE', '/domains/unit.tests/records/11189898')
+        ], any_order=True)

tests/test_octodns_provider_dnsimple.py

@@ -78,14 +78,14 @@ class TestDnsimpleProvider(TestCase):
 
             zone = Zone('unit.tests.', [])
             provider.populate(zone)
-            self.assertEquals(15, len(zone.records))
+            self.assertEquals(16, len(zone.records))
             changes = self.expected.changes(zone, provider)
             self.assertEquals(0, len(changes))
 
         # 2nd populate makes no network calls/all from cache
         again = Zone('unit.tests.', [])
         provider.populate(again)
-        self.assertEquals(15, len(again.records))
+        self.assertEquals(16, len(again.records))
 
         # bust the cache
         del provider._zone_records[zone.name]
@@ -96,23 +96,23 @@ class TestDnsimpleProvider(TestCase):
                 mock.get(ANY, text=fh.read())
 
             zone = Zone('unit.tests.', [])
-            provider.populate(zone)
+            provider.populate(zone, lenient=True)
             self.assertEquals(set([
                 Record.new(zone, '', {
                     'ttl': 3600,
                     'type': 'SSHFP',
                     'values': []
-                }),
+                }, lenient=True),
                 Record.new(zone, '_srv._tcp', {
                     'ttl': 600,
                     'type': 'SRV',
                     'values': []
-                }),
+                }, lenient=True),
                 Record.new(zone, 'naptr', {
                     'ttl': 600,
                     'type': 'NAPTR',
                     'values': []
-                }),
+                }, lenient=True),
             ]), zone.records)
 
     def test_apply(self):
@@ -129,10 +129,11 @@ class TestDnsimpleProvider(TestCase):
         ]
         plan = provider.plan(self.expected)
 
-        # No root NS, no ignored
-        n = len(self.expected.records) - 2
+        # No root NS, no ignored, no excluded
+        n = len(self.expected.records) - 3
         self.assertEquals(n, len(plan.changes))
         self.assertEquals(n, provider.apply(plan))
+        self.assertFalse(plan.exists)
 
         provider._client._request.assert_has_calls([
             # created the domain
@@ -147,7 +148,7 @@ class TestDnsimpleProvider(TestCase):
             }),
         ])
         # expected number of total calls
-        self.assertEquals(27, provider._client._request.call_count)
+        self.assertEquals(28, provider._client._request.call_count)
 
         provider._client._request.reset_mock()
 
@@ -186,6 +187,7 @@ class TestDnsimpleProvider(TestCase):
         }))
 
         plan = provider.plan(wanted)
+        self.assertTrue(plan.exists)
         self.assertEquals(2, len(plan.changes))
         self.assertEquals(2, provider.apply(plan))
         # recreate for update, and deletes for the 2 parts of the other

tests/test_octodns_provider_dnsmadeeasy.py

@@ -0,0 +1,202 @@
+#
+#
+#
+
+
+from __future__ import absolute_import, division, print_function, \
+    unicode_literals
+
+from mock import Mock, call
+from os.path import dirname, join
+from requests import HTTPError
+from requests_mock import ANY, mock as requests_mock
+from unittest import TestCase
+
+from octodns.record import Record
+from octodns.provider.dnsmadeeasy import DnsMadeEasyClientNotFound, \
+    DnsMadeEasyProvider
+from octodns.provider.yaml import YamlProvider
+from octodns.zone import Zone
+
+import json
+
+
+class TestDnsMadeEasyProvider(TestCase):
+    expected = Zone('unit.tests.', [])
+    source = YamlProvider('test', join(dirname(__file__), 'config'))
+    source.populate(expected)
+
+    # Our test suite differs a bit, add our NS and remove the simple one
+    expected.add_record(Record.new(expected, 'under', {
+        'ttl': 3600,
+        'type': 'NS',
+        'values': [
+            'ns1.unit.tests.',
+            'ns2.unit.tests.',
+        ]
+    }))
+    for record in list(expected.records):
+        if record.name == 'sub' and record._type == 'NS':
+            expected._remove_record(record)
+            break
+
+    def test_populate(self):
+        provider = DnsMadeEasyProvider('test', 'api', 'secret')
+
+        # Bad auth
+        with requests_mock() as mock:
+            mock.get(ANY, status_code=401,
+                     text='{"error": ["API key not found"]}')
+
+            with self.assertRaises(Exception) as ctx:
+                zone = Zone('unit.tests.', [])
+                provider.populate(zone)
+            self.assertEquals('Unauthorized', ctx.exception.message)
+
+        # Bad request
+        with requests_mock() as mock:
+            mock.get(ANY, status_code=400,
+                     text='{"error": ["Rate limit exceeded"]}')
+
+            with self.assertRaises(Exception) as ctx:
+                zone = Zone('unit.tests.', [])
+                provider.populate(zone)
+            self.assertEquals('\n  - Rate limit exceeded',
+                              ctx.exception.message)
+
+        # General error
+        with requests_mock() as mock:
+            mock.get(ANY, status_code=502, text='Things caught fire')
+
+            with self.assertRaises(HTTPError) as ctx:
+                zone = Zone('unit.tests.', [])
+                provider.populate(zone)
+            self.assertEquals(502, ctx.exception.response.status_code)
+
+        # Non-existant zone doesn't populate anything
+        with requests_mock() as mock:
+            mock.get(ANY, status_code=404,
+                     text='<html><head></head><body></body></html>')
+
+            zone = Zone('unit.tests.', [])
+            provider.populate(zone)
+            self.assertEquals(set(), zone.records)
+
+        # No diffs == no changes
+        with requests_mock() as mock:
+            base = 'https://api.dnsmadeeasy.com/V2.0/dns/managed'
+            with open('tests/fixtures/dnsmadeeasy-domains.json') as fh:
+                mock.get('{}{}'.format(base, '/'), text=fh.read())
+            with open('tests/fixtures/dnsmadeeasy-records.json') as fh:
+                mock.get('{}{}'.format(base, '/123123/records'),
+                         text=fh.read())
+
+                zone = Zone('unit.tests.', [])
+                provider.populate(zone)
+                self.assertEquals(13, len(zone.records))
+                changes = self.expected.changes(zone, provider)
+                self.assertEquals(0, len(changes))
+
+        # 2nd populate makes no network calls/all from cache
+        again = Zone('unit.tests.', [])
+        provider.populate(again)
+        self.assertEquals(13, len(again.records))
+
+        # bust the cache
+        del provider._zone_records[zone.name]
+
+    def test_apply(self):
+        # Create provider with sandbox enabled
+        provider = DnsMadeEasyProvider('test', 'api', 'secret', True)
+
+        resp = Mock()
+        resp.json = Mock()
+        provider._client._request = Mock(return_value=resp)
+
+        with open('tests/fixtures/dnsmadeeasy-domains.json') as fh:
+            domains = json.load(fh)
+
+        # non-existant domain, create everything
+        resp.json.side_effect = [
+            DnsMadeEasyClientNotFound,  # no zone in populate
+            DnsMadeEasyClientNotFound,  # no domain during apply
+            domains
+        ]
+        plan = provider.plan(self.expected)
+
+        # No root NS, no ignored, no excluded, no unsupported
+        n = len(self.expected.records) - 5
+        self.assertEquals(n, len(plan.changes))
+        self.assertEquals(n, provider.apply(plan))
+
+        provider._client._request.assert_has_calls([
+            # created the domain
+            call('POST', '/', data={'name': 'unit.tests'}),
+            # get all domains to build the cache
+            call('GET', '/'),
+            # created at least one of the record with expected data
+            call('POST', '/123123/records', data={
+                'name': '_srv._tcp',
+                'weight': 20,
+                'value': 'foo-1.unit.tests.',
+                'priority': 10,
+                'ttl': 600,
+                'type': 'SRV',
+                'port': 30
+            }),
+        ])
+        self.assertEquals(25, provider._client._request.call_count)
+
+        provider._client._request.reset_mock()
+
+        # delete 1 and update 1
+        provider._client.records = Mock(return_value=[
+            {
+                'id': 11189897,
+                'name': 'www',
+                'value': '1.2.3.4',
+                'ttl': 300,
+                'type': 'A',
+            },
+            {
+                'id': 11189898,
+                'name': 'www',
+                'value': '2.2.3.4',
+                'ttl': 300,
+                'type': 'A',
+            },
+            {
+                'id': 11189899,
+                'name': 'ttl',
+                'value': '3.2.3.4',
+                'ttl': 600,
+                'type': 'A',
+            }
+        ])
+
+        # Domain exists, we don't care about return
+        resp.json.side_effect = ['{}']
+
+        wanted = Zone('unit.tests.', [])
+        wanted.add_record(Record.new(wanted, 'ttl', {
+            'ttl': 300,
+            'type': 'A',
+            'value': '3.2.3.4'
+        }))
+
+        plan = provider.plan(wanted)
+        self.assertEquals(2, len(plan.changes))
+        self.assertEquals(2, provider.apply(plan))
+
+        # recreate for update, and deletes for the 2 parts of the other
+        provider._client._request.assert_has_calls([
+            call('POST', '/123123/records', data={
+                'value': '3.2.3.4',
+                'type': 'A',
+                'name': 'ttl',
+                'ttl': 300
+            }),
+            call('DELETE', '/123123/records/11189899'),
+            call('DELETE', '/123123/records/11189897'),
+            call('DELETE', '/123123/records/11189898')
+        ], any_order=True)

tests/test_octodns_provider_dyn.py

@@ -430,6 +430,7 @@ class TestDynProvider(TestCase):
                 update_mock.assert_not_called()
                 provider.apply(plan)
                 update_mock.assert_called()
+                self.assertFalse(plan.exists)
             add_mock.assert_called()
             # Once for each dyn record (8 Records, 2 of which have dual values)
             self.assertEquals(15, len(add_mock.call_args_list))
@@ -474,6 +475,7 @@ class TestDynProvider(TestCase):
                     plan = provider.plan(new)
                     provider.apply(plan)
                     update_mock.assert_called()
+                    self.assertTrue(plan.exists)
                 # we expect 4 deletes, 2 from actual deletes and 2 from
                 # updates which delete and recreate
                 self.assertEquals(4, len(delete_mock.call_args_list))
@@ -491,7 +493,7 @@ class TestDynProviderGeo(TestCase):
         traffic_director_response = loads(fh.read())
 
     @property
-    def traffic_directors_reponse(self):
+    def traffic_directors_response(self):
         return {
             'data': [{
                 'active': 'Y',
@@ -626,7 +628,7 @@ class TestDynProviderGeo(TestCase):
         mock.side_effect = [{'data': []}]
         self.assertEquals({}, provider.traffic_directors)
 
-        # a supported td and an ingored one
+        # a supported td and an ignored one
         response = {
             'data': [{
                 'active': 'Y',
@@ -669,7 +671,7 @@ class TestDynProviderGeo(TestCase):
                           set(tds.keys()))
         self.assertEquals(['A'], tds['unit.tests.'].keys())
         self.assertEquals(['A'], tds['geo.unit.tests.'].keys())
-        provider.log.warn.assert_called_with("Failed to load TraficDirector "
+        provider.log.warn.assert_called_with("Failed to load TrafficDirector "
                                              "'%s': %s", 'something else',
                                              'need more than 1 value to '
                                              'unpack')
@@ -975,7 +977,7 @@ class TestDynProviderGeo(TestCase):
         # only traffic director
         mock.side_effect = [
             # get traffic directors
-            self.traffic_directors_reponse,
+            self.traffic_directors_response,
             # get traffic director
             self.traffic_director_response,
             # get zone
@@ -1026,7 +1028,7 @@ class TestDynProviderGeo(TestCase):
         # both traffic director and regular, regular is ignored
         mock.side_effect = [
             # get traffic directors
-            self.traffic_directors_reponse,
+            self.traffic_directors_response,
             # get traffic director
             self.traffic_director_response,
             # get zone
@@ -1076,7 +1078,7 @@ class TestDynProviderGeo(TestCase):
         # busted traffic director
         mock.side_effect = [
             # get traffic directors
-            self.traffic_directors_reponse,
+            self.traffic_directors_response,
             # get traffic director
             busted_traffic_director_response,
             # get zone
@@ -1130,7 +1132,7 @@ class TestDynProviderGeo(TestCase):
             Delete(geo),
             Delete(regular),
         ]
-        plan = Plan(None, desired, changes)
+        plan = Plan(None, desired, changes, True)
         provider._apply(plan)
         mock.assert_has_calls([
             call('/Zone/unit.tests/', 'GET', {}),
@@ -1149,14 +1151,14 @@ class TestDynProviderGeo(TestCase):
         provider = DynProvider('test', 'cust', 'user', 'pass',
                                traffic_directors_enabled=True)
 
-        # will be tested seperately
+        # will be tested separately
         provider._mod_rulesets = MagicMock()
 
         mock.side_effect = [
             # create traffic director
             self.traffic_director_response,
             # get traffic directors
-            self.traffic_directors_reponse
+            self.traffic_directors_response
         ]
         provider._mod_geo_Create(None, Create(self.geo_record))
         # td now lives in cache

tests/test_octodns_provider_googlecloud.py

@@ -263,7 +263,8 @@ class TestGoogleCloudProvider(TestCase):
         provider.apply(Plan(
             existing=[update_existing_r, delete_r],
             desired=desired,
-            changes=changes
+            changes=changes,
+            exists=True
         ))
 
         calls_mock = gcloud_zone_mock.changes.return_value
@@ -295,7 +296,8 @@ class TestGoogleCloudProvider(TestCase):
             provider.apply(Plan(
                 existing=[update_existing_r, delete_r],
                 desired=desired,
-                changes=changes
+                changes=changes,
+                exists=True
             ))
 
         unsupported_change = Mock()
@@ -357,15 +359,17 @@ class TestGoogleCloudProvider(TestCase):
                          "unit.tests.")
 
         test_zone = Zone('unit.tests.', [])
-        provider.populate(test_zone)
+        exists = provider.populate(test_zone)
+        self.assertTrue(exists)
 
         # test_zone gets fed the same records as zone does, except it's in
         # the format returned by google API, so after populate they should look
-        # excactly the same.
+        # exactly the same.
         self.assertEqual(test_zone.records, zone.records)
 
-        test_zone2 = Zone('nonexistant.zone.', [])
-        provider.populate(test_zone2, False, False)
+        test_zone2 = Zone('nonexistent.zone.', [])
+        exists = provider.populate(test_zone2, False, False)
+        self.assertFalse(exists)
 
         self.assertEqual(len(test_zone2.records), 0,
                          msg="Zone should not get records from wrong domain")
@@ -401,8 +405,8 @@ class TestGoogleCloudProvider(TestCase):
         provider.gcloud_client.list_zones = Mock(
             return_value=DummyIterator([]))
 
-        self.assertIsNone(provider.gcloud_zones.get("nonexistant.xone"),
-                          msg="Check that nonexistant zones return None when"
+        self.assertIsNone(provider.gcloud_zones.get("nonexistent.zone"),
+                          msg="Check that nonexistent zones return None when"
                               "there's no create=True flag")
 
     def test__get_rrsets(self):
@@ -423,7 +427,32 @@ class TestGoogleCloudProvider(TestCase):
         provider.gcloud_client.list_zones = Mock(
             return_value=DummyIterator([]))
 
-        mock_zone = provider._create_gcloud_zone("nonexistant.zone.mock")
+        mock_zone = provider._create_gcloud_zone("nonexistent.zone.mock")
 
         mock_zone.create.assert_called()
         provider.gcloud_client.zone.assert_called()
+
+    def test__create_zone_ip6_arpa(self):
+        def _create_dummy_zone(name, dns_name):
+            return DummyGoogleCloudZone(name=name, dns_name=dns_name)
+
+        provider = self._get_provider()
+
+        provider.gcloud_client = Mock()
+        provider.gcloud_client.zone = Mock(side_effect=_create_dummy_zone)
+
+        mock_zone = \
+            provider._create_gcloud_zone('0.0.0.0.8.b.d.0.1.0.0.2.ip6.arpa')
+
+        self.assertRegexpMatches(mock_zone.name, '^[a-z][a-z0-9-]*[a-z0-9]$')
+        self.assertEqual(len(mock_zone.name), 63)
+
+    def test_semicolon_fixup(self):
+        provider = self._get_provider()
+
+        self.assertEquals({
+            'values': ['abcd\\; ef\\;g', 'hij\\; klm\\;n']
+        }, provider._data_for_TXT(
+            DummyResourceRecordSet(
+                'unit.tests.', 'TXT', 0, ['abcd; ef;g', 'hij\\; klm\\;n'])
+        ))

tests/test_octodns_provider_ns1.py

@@ -30,11 +30,20 @@ class TestNs1Provider(TestCase):
         'ttl': 32,
         'type': 'A',
         'value': '1.2.3.4',
+        'meta': {},
     }))
     expected.add(Record.new(zone, 'foo', {
         'ttl': 33,
         'type': 'A',
         'values': ['1.2.3.4', '1.2.3.5'],
+        'meta': {},
+    }))
+    expected.add(Record.new(zone, 'geo', {
+        'ttl': 34,
+        'type': 'A',
+        'values': ['101.102.103.104', '101.102.103.105'],
+        'geo': {'NA-US-NY': ['201.202.203.204']},
+        'meta': {},
     }))
     expected.add(Record.new(zone, 'cname', {
         'ttl': 34,
@@ -116,6 +125,11 @@ class TestNs1Provider(TestCase):
         'ttl': 33,
         'short_answers': ['1.2.3.4', '1.2.3.5'],
         'domain': 'foo.unit.tests.',
+    }, {
+        'type': 'A',
+        'ttl': 34,
+        'short_answers': ['101.102.103.104', '101.102.103.105'],
+        'domain': 'geo.unit.tests',
     }, {
         'type': 'CNAME',
         'ttl': 34,
@@ -182,23 +196,62 @@ class TestNs1Provider(TestCase):
         load_mock.side_effect = \
             ResourceException('server error: zone not found')
         zone = Zone('unit.tests.', [])
-        provider.populate(zone)
+        exists = provider.populate(zone)
         self.assertEquals(set(), zone.records)
         self.assertEquals(('unit.tests',), load_mock.call_args[0])
+        self.assertFalse(exists)
 
         # Existing zone w/o records
         load_mock.reset_mock()
         nsone_zone = DummyZone([])
         load_mock.side_effect = [nsone_zone]
+        zone_search = Mock()
+        zone_search.return_value = [
+            {
+                "domain": "geo.unit.tests",
+                "zone": "unit.tests",
+                "type": "A",
+                "answers": [
+                    {'answer': ['1.1.1.1'], 'meta': {}},
+                    {'answer': ['1.2.3.4'],
+                     'meta': {'ca_province': ['ON']}},
+                    {'answer': ['2.3.4.5'], 'meta': {'us_state': ['NY']}},
+                    {'answer': ['3.4.5.6'], 'meta': {'country': ['US']}},
+                    {'answer': ['4.5.6.7'],
+                     'meta': {'iso_region_code': ['NA-US-WA']}},
+                ],
+                'ttl': 34,
+            },
+        ]
+        nsone_zone.search = zone_search
         zone = Zone('unit.tests.', [])
         provider.populate(zone)
-        self.assertEquals(set(), zone.records)
+        self.assertEquals(1, len(zone.records))
         self.assertEquals(('unit.tests',), load_mock.call_args[0])
 
         # Existing zone w/records
         load_mock.reset_mock()
         nsone_zone = DummyZone(self.nsone_records)
         load_mock.side_effect = [nsone_zone]
+        zone_search = Mock()
+        zone_search.return_value = [
+            {
+                "domain": "geo.unit.tests",
+                "zone": "unit.tests",
+                "type": "A",
+                "answers": [
+                    {'answer': ['1.1.1.1'], 'meta': {}},
+                    {'answer': ['1.2.3.4'],
+                     'meta': {'ca_province': ['ON']}},
+                    {'answer': ['2.3.4.5'], 'meta': {'us_state': ['NY']}},
+                    {'answer': ['3.4.5.6'], 'meta': {'country': ['US']}},
+                    {'answer': ['4.5.6.7'],
+                     'meta': {'iso_region_code': ['NA-US-WA']}},
+                ],
+                'ttl': 34,
+            },
+        ]
+        nsone_zone.search = zone_search
         zone = Zone('unit.tests.', [])
         provider.populate(zone)
         self.assertEquals(self.expected, zone.records)
@@ -217,6 +270,7 @@ class TestNs1Provider(TestCase):
         # everything except the root NS
         expected_n = len(self.expected) - 1
         self.assertEquals(expected_n, len(plan.changes))
+        self.assertTrue(plan.exists)
 
         # Fails, general error
         load_mock.reset_mock()
@@ -264,11 +318,30 @@ class TestNs1Provider(TestCase):
         }])
         nsone_zone.data['records'][0]['short_answers'][0] = '2.2.2.2'
         nsone_zone.loadRecord = Mock()
+        zone_search = Mock()
+        zone_search.return_value = [
+            {
+                "domain": "geo.unit.tests",
+                "zone": "unit.tests",
+                "type": "A",
+                "answers": [
+                    {'answer': ['1.1.1.1'], 'meta': {}},
+                    {'answer': ['1.2.3.4'],
+                     'meta': {'ca_province': ['ON']}},
+                    {'answer': ['2.3.4.5'], 'meta': {'us_state': ['NY']}},
+                    {'answer': ['3.4.5.6'], 'meta': {'country': ['US']}},
+                    {'answer': ['4.5.6.7'],
+                     'meta': {'iso_region_code': ['NA-US-WA']}},
+                ],
+                'ttl': 34,
+            },
+        ]
+        nsone_zone.search = zone_search
         load_mock.side_effect = [nsone_zone, nsone_zone]
         plan = provider.plan(desired)
-        self.assertEquals(2, len(plan.changes))
+        self.assertEquals(3, len(plan.changes))
         self.assertIsInstance(plan.changes[0], Update)
-        self.assertIsInstance(plan.changes[1], Delete)
+        self.assertIsInstance(plan.changes[2], Delete)
         # ugh, we need a mock record that can be returned from loadRecord for
         # the update and delete targets, we can add our side effects to that to
         # trigger rate limit handling
@@ -276,26 +349,52 @@ class TestNs1Provider(TestCase):
         mock_record.update.side_effect = [
             RateLimitException('one', period=0),
             None,
+            None,
         ]
         mock_record.delete.side_effect = [
             RateLimitException('two', period=0),
             None,
+            None,
         ]
-        nsone_zone.loadRecord.side_effect = [mock_record, mock_record]
+        nsone_zone.loadRecord.side_effect = [mock_record, mock_record,
+                                             mock_record]
         got_n = provider.apply(plan)
-        self.assertEquals(2, got_n)
+        self.assertEquals(3, got_n)
         nsone_zone.loadRecord.assert_has_calls([
             call('unit.tests', u'A'),
+            call('geo', u'A'),
             call('delete-me', u'A'),
         ])
         mock_record.assert_has_calls([
-            call.update(answers=[u'1.2.3.4'], ttl=32),
+            call.update(answers=[{'answer': [u'1.2.3.4'], 'meta': {}}],
+                        filters=[],
+                        ttl=32),
+            call.update(answers=[{u'answer': [u'1.2.3.4'], u'meta': {}}],
+                        filters=[],
+                        ttl=32),
+            call.update(
+                answers=[
+                    {u'answer': [u'101.102.103.104'], u'meta': {}},
+                    {u'answer': [u'101.102.103.105'], u'meta': {}},
+                    {
+                        u'answer': [u'201.202.203.204'],
+                        u'meta': {
+                            u'iso_region_code': [u'NA-US-NY']
+                        },
+                    },
+                ],
+                filters=[
+                    {u'filter': u'shuffle', u'config': {}},
+                    {u'filter': u'geotarget_country', u'config': {}},
+                    {u'filter': u'select_first_n', u'config': {u'N': 1}},
+                ],
+                ttl=34),
+            call.delete(),
             call.delete()
         ])
 
     def test_escaping(self):
         provider = Ns1Provider('test', 'api-key')
-
         record = {
             'ttl': 31,
             'short_answers': ['foo; bar baz; blip']
@@ -326,3 +425,34 @@ class TestNs1Provider(TestCase):
         })
         self.assertEquals(['foo; bar baz; blip'],
                           provider._params_for_TXT(record)['answers'])
+
+    def test_data_for_CNAME(self):
+        provider = Ns1Provider('test', 'api-key')
+
+        # answers from nsone
+        a_record = {
+            'ttl': 31,
+            'type': 'CNAME',
+            'short_answers': ['foo.unit.tests.']
+        }
+        a_expected = {
+            'ttl': 31,
+            'type': 'CNAME',
+            'value': 'foo.unit.tests.'
+        }
+        self.assertEqual(a_expected,
+                         provider._data_for_CNAME(a_record['type'], a_record))
+
+        # no answers from nsone
+        b_record = {
+            'ttl': 32,
+            'type': 'CNAME',
+            'short_answers': []
+        }
+        b_expected = {
+            'ttl': 32,
+            'type': 'CNAME',
+            'value': None
+        }
+        self.assertEqual(b_expected,
+                         provider._data_for_CNAME(b_record['type'], b_record))

tests/test_octodns_provider_ovh.py

@@ -8,7 +8,7 @@ from __future__ import absolute_import, division, print_function, \
 from unittest import TestCase
 
 from mock import patch, call
-from ovh import APIError
+from ovh import APIError, ResourceNotFoundError, InvalidCredential
 
 from octodns.provider.ovh import OvhProvider
 from octodns.record import Record
@@ -17,6 +17,14 @@ from octodns.zone import Zone
 
 class TestOvhProvider(TestCase):
     api_record = []
+    valid_dkim = []
+    invalid_dkim = []
+
+    valid_dkim_key = "p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCxLaG16G4SaE" \
+                     "cXVdiIxTg7gKSGbHKQLm30CHib1h9FzS9nkcyvQSyQj1rMFyqC//" \
+                     "tft3ohx3nvJl+bGCWxdtLYDSmir9PW54e5CTdxEh8MWRkBO3StF6" \
+                     "QG/tAh3aTGDmkqhIJGLb87iHvpmVKqURmEUzJPv5KPJfWLofADI+" \
+                     "q9lQIDAQAB"
 
     zone = Zone('unit.tests.', [])
     expected = set()
@@ -191,14 +199,14 @@ class TestOvhProvider(TestCase):
     api_record.append({
         'fieldType': 'SPF',
         'ttl': 1000,
-        'target': 'v=spf1 include:unit.texts.rerirect ~all',
+        'target': 'v=spf1 include:unit.texts.redirect ~all',
         'subDomain': '',
         'id': 13
     })
     expected.add(Record.new(zone, '', {
         'ttl': 1000,
         'type': 'SPF',
-        'value': 'v=spf1 include:unit.texts.rerirect ~all'
+        'value': 'v=spf1 include:unit.texts.redirect ~all'
     }))
 
     # SSHFP
@@ -233,6 +241,65 @@ class TestOvhProvider(TestCase):
         'value': '1:1ec:1::1',
     }))
 
+    # DKIM
+    api_record.append({
+        'fieldType': 'DKIM',
+        'ttl': 1300,
+        'target': valid_dkim_key,
+        'subDomain': 'dkim',
+        'id': 16
+    })
+    expected.add(Record.new(zone, 'dkim', {
+        'ttl': 1300,
+        'type': 'TXT',
+        'value': valid_dkim_key,
+    }))
+
+    # TXT
+    api_record.append({
+        'fieldType': 'TXT',
+        'ttl': 1400,
+        'target': 'TXT text',
+        'subDomain': 'txt',
+        'id': 17
+    })
+    expected.add(Record.new(zone, 'txt', {
+        'ttl': 1400,
+        'type': 'TXT',
+        'value': 'TXT text',
+    }))
+
+    # LOC
+    # We do not have associated record for LOC, as it's not managed
+    api_record.append({
+        'fieldType': 'LOC',
+        'ttl': 1500,
+        'target': '1 1 1 N 1 1 1 E 1m 1m',
+        'subDomain': '',
+        'id': 18
+    })
+
+    valid_dkim = [valid_dkim_key,
+                  'v=DKIM1 \; %s' % valid_dkim_key,
+                  'h=sha256 \; %s' % valid_dkim_key,
+                  'h=sha1 \; %s' % valid_dkim_key,
+                  's=* \; %s' % valid_dkim_key,
+                  's=email \; %s' % valid_dkim_key,
+                  't=y \; %s' % valid_dkim_key,
+                  't=s \; %s' % valid_dkim_key,
+                  'k=rsa \; %s' % valid_dkim_key,
+                  'n=notes \; %s' % valid_dkim_key,
+                  'g=granularity \; %s' % valid_dkim_key,
+                  ]
+    invalid_dkim = ['p=%invalid%',  # Invalid public key
+                    'v=DKIM1',  # Missing public key
+                    'v=DKIM2 \; %s' % valid_dkim_key,  # Invalid version
+                    'h=sha512 \; %s' % valid_dkim_key,  # Invalid hash algo
+                    's=fake \; %s' % valid_dkim_key,  # Invalid selector
+                    't=fake \; %s' % valid_dkim_key,  # Invalid flag
+                    'u=invalid \; %s' % valid_dkim_key,  # Invalid key
+                    ]
+
     @patch('ovh.Client')
     def test_populate(self, client_mock):
         provider = OvhProvider('test', 'endpoint', 'application_key',
@@ -240,18 +307,40 @@ class TestOvhProvider(TestCase):
 
         with patch.object(provider._client, 'get') as get_mock:
             zone = Zone('unit.tests.', [])
-            get_mock.side_effect = APIError('boom')
+            get_mock.side_effect = ResourceNotFoundError('boom')
             with self.assertRaises(APIError) as ctx:
                 provider.populate(zone)
             self.assertEquals(get_mock.side_effect, ctx.exception)
 
-        with patch.object(provider._client, 'get') as get_mock:
+            get_mock.side_effect = InvalidCredential('boom')
+            with self.assertRaises(APIError) as ctx:
+                provider.populate(zone)
+            self.assertEquals(get_mock.side_effect, ctx.exception)
+
+            zone = Zone('unit.tests.', [])
+            get_mock.side_effect = ResourceNotFoundError('This service does '
+                                                         'not exist')
+            exists = provider.populate(zone)
+            self.assertEquals(set(), zone.records)
+            self.assertFalse(exists)
+
             zone = Zone('unit.tests.', [])
             get_returns = [[record['id'] for record in self.api_record]]
             get_returns += self.api_record
             get_mock.side_effect = get_returns
-            provider.populate(zone)
+            exists = provider.populate(zone)
             self.assertEquals(self.expected, zone.records)
+            self.assertTrue(exists)
+
+    @patch('ovh.Client')
+    def test_is_valid_dkim(self, client_mock):
+        """Test _is_valid_dkim"""
+        provider = OvhProvider('test', 'endpoint', 'application_key',
+                               'application_secret', 'consumer_key')
+        for dkim in self.valid_dkim:
+            self.assertTrue(provider._is_valid_dkim(dkim))
+        for dkim in self.invalid_dkim:
+            self.assertFalse(provider._is_valid_dkim(dkim))
 
     @patch('ovh.Client')
     def test_apply(self, client_mock):
@@ -270,90 +359,91 @@ class TestOvhProvider(TestCase):
                 provider.apply(plan)
             self.assertEquals(get_mock.side_effect, ctx.exception)
 
+        # Records get by API call
         with patch.object(provider._client, 'get') as get_mock:
-            get_returns = [[1, 2], {
-                'fieldType': 'A',
-                'ttl': 600,
-                'target': '5.6.7.8',
-                'subDomain': '',
-                'id': 100
-            }, {'fieldType': 'A',
-                'ttl': 600,
-                'target': '5.6.7.8',
-                'subDomain': 'fake',
-                'id': 101
-                }]
+            get_returns = [
+                [1, 2, 3, 4],
+                {'fieldType': 'A', 'ttl': 600, 'target': '5.6.7.8',
+                 'subDomain': '', 'id': 100},
+                {'fieldType': 'A', 'ttl': 600, 'target': '5.6.7.8',
+                 'subDomain': 'fake', 'id': 101},
+                {'fieldType': 'TXT', 'ttl': 600, 'target': 'fake txt record',
+                 'subDomain': 'txt', 'id': 102},
+                {'fieldType': 'DKIM', 'ttl': 600,
+                 'target': 'v=DKIM1; %s' % self.valid_dkim_key,
+                 'subDomain': 'dkim', 'id': 103}
+            ]
             get_mock.side_effect = get_returns
 
             plan = provider.plan(desired)
 
-            with patch.object(provider._client, 'post') as post_mock:
-                with patch.object(provider._client, 'delete') as delete_mock:
-                    with patch.object(provider._client, 'get') as get_mock:
-                        get_mock.side_effect = [[100], [101]]
-                        provider.apply(plan)
-                        wanted_calls = [
-                            call(u'/domain/zone/unit.tests/record',
-                                 fieldType=u'A',
-                                 subDomain=u'', target=u'1.2.3.4', ttl=100),
-                            call(u'/domain/zone/unit.tests/record',
-                                 fieldType=u'SRV',
-                                 subDomain=u'10 20 30 foo-1.unit.tests.',
-                                 target='_srv._tcp', ttl=800),
-                            call(u'/domain/zone/unit.tests/record',
-                                 fieldType=u'SRV',
-                                 subDomain=u'40 50 60 foo-2.unit.tests.',
-                                 target='_srv._tcp', ttl=800),
-                            call(u'/domain/zone/unit.tests/record',
-                                 fieldType=u'PTR', subDomain='4',
-                                 target=u'unit.tests.', ttl=900),
-                            call(u'/domain/zone/unit.tests/record',
-                                 fieldType=u'NS', subDomain='www3',
-                                 target=u'ns3.unit.tests.', ttl=700),
-                            call(u'/domain/zone/unit.tests/record',
-                                 fieldType=u'NS', subDomain='www3',
-                                 target=u'ns4.unit.tests.', ttl=700),
-                            call(u'/domain/zone/unit.tests/record',
-                                 fieldType=u'SSHFP',
-                                 subDomain=u'1 1 bf6b6825d2977c511a475bbefb88a'
-                                           u'ad54'
-                                           u'a92ac73',
-                                 target=u'', ttl=1100),
-                            call(u'/domain/zone/unit.tests/record',
-                                 fieldType=u'AAAA', subDomain=u'',
-                                 target=u'1:1ec:1::1', ttl=200),
-                            call(u'/domain/zone/unit.tests/record',
-                                 fieldType=u'MX', subDomain=u'',
-                                 target=u'10 mx1.unit.tests.', ttl=400),
-                            call(u'/domain/zone/unit.tests/record',
-                                 fieldType=u'CNAME', subDomain='www2',
-                                 target=u'unit.tests.', ttl=300),
-                            call(u'/domain/zone/unit.tests/record',
-                                 fieldType=u'SPF', subDomain=u'',
-                                 target=u'v=spf1 include:unit.texts.'
-                                        u'rerirect ~all',
-                                 ttl=1000),
-                            call(u'/domain/zone/unit.tests/record',
-                                 fieldType=u'A',
-                                 subDomain='sub', target=u'1.2.3.4', ttl=200),
-                            call(u'/domain/zone/unit.tests/record',
-                                 fieldType=u'NAPTR', subDomain='naptr',
-                                 target=u'10 100 "S" "SIP+D2U" "!^.*$!sip:'
-                                        u'info@bar'
-                                        u'.example.com!" .',
-                                 ttl=500),
-                            call(u'/domain/zone/unit.tests/refresh')]
+            with patch.object(provider._client, 'post') as post_mock, \
+                    patch.object(provider._client, 'delete') as delete_mock:
+                get_mock.side_effect = [[100], [101], [102], [103]]
+                provider.apply(plan)
+                wanted_calls = [
+                    call(u'/domain/zone/unit.tests/record', fieldType=u'TXT',
+                         subDomain='txt', target=u'TXT text', ttl=1400),
+                    call(u'/domain/zone/unit.tests/record', fieldType=u'DKIM',
+                         subDomain='dkim', target=self.valid_dkim_key,
+                         ttl=1300),
+                    call(u'/domain/zone/unit.tests/record', fieldType=u'A',
+                         subDomain=u'', target=u'1.2.3.4', ttl=100),
+                    call(u'/domain/zone/unit.tests/record', fieldType=u'SRV',
+                         subDomain='_srv._tcp',
+                         target=u'10 20 30 foo-1.unit.tests.', ttl=800),
+                    call(u'/domain/zone/unit.tests/record', fieldType=u'SRV',
+                         subDomain='_srv._tcp',
+                         target=u'40 50 60 foo-2.unit.tests.', ttl=800),
+                    call(u'/domain/zone/unit.tests/record', fieldType=u'PTR',
+                         subDomain='4', target=u'unit.tests.', ttl=900),
+                    call(u'/domain/zone/unit.tests/record', fieldType=u'NS',
+                         subDomain='www3', target=u'ns3.unit.tests.', ttl=700),
+                    call(u'/domain/zone/unit.tests/record', fieldType=u'NS',
+                         subDomain='www3', target=u'ns4.unit.tests.', ttl=700),
+                    call(u'/domain/zone/unit.tests/record',
+                         fieldType=u'SSHFP', subDomain=u'', ttl=1100,
+                         target=u'1 1 bf6b6825d2977c511a475bbefb88a'
+                                   u'ad54'
+                                   u'a92ac73',
+                         ),
+                    call(u'/domain/zone/unit.tests/record', fieldType=u'AAAA',
+                         subDomain=u'', target=u'1:1ec:1::1', ttl=200),
+                    call(u'/domain/zone/unit.tests/record', fieldType=u'MX',
+                         subDomain=u'', target=u'10 mx1.unit.tests.', ttl=400),
+                    call(u'/domain/zone/unit.tests/record', fieldType=u'CNAME',
+                         subDomain='www2', target=u'unit.tests.', ttl=300),
+                    call(u'/domain/zone/unit.tests/record', fieldType=u'SPF',
+                         subDomain=u'', ttl=1000,
+                         target=u'v=spf1 include:unit.texts.'
+                                u'redirect ~all',
+                         ),
+                    call(u'/domain/zone/unit.tests/record', fieldType=u'A',
+                         subDomain='sub', target=u'1.2.3.4', ttl=200),
+                    call(u'/domain/zone/unit.tests/record', fieldType=u'NAPTR',
+                         subDomain='naptr', ttl=500,
+                         target=u'10 100 "S" "SIP+D2U" "!^.*$!sip:'
+                                u'info@bar'
+                                u'.example.com!" .'
+                         ),
+                    call(u'/domain/zone/unit.tests/refresh')]
 
-                        post_mock.assert_has_calls(wanted_calls)
+                post_mock.assert_has_calls(wanted_calls)
 
-                        # Get for delete calls
-                        get_mock.assert_has_calls(
-                            [call(u'/domain/zone/unit.tests/record',
-                                  fieldType=u'A', subDomain=u''),
-                             call(u'/domain/zone/unit.tests/record',
-                                  fieldType=u'A', subDomain='fake')]
-                        )
-                        # 2 delete calls, one for update + one for delete
-                        delete_mock.assert_has_calls(
-                            [call(u'/domain/zone/unit.tests/record/100'),
-                             call(u'/domain/zone/unit.tests/record/101')])
+                # Get for delete calls
+                wanted_get_calls = [
+                    call(u'/domain/zone/unit.tests/record', fieldType=u'TXT',
+                         subDomain='txt'),
+                    call(u'/domain/zone/unit.tests/record', fieldType=u'DKIM',
+                         subDomain='dkim'),
+                    call(u'/domain/zone/unit.tests/record', fieldType=u'A',
+                         subDomain=u''),
+                    call(u'/domain/zone/unit.tests/record', fieldType=u'A',
+                         subDomain='fake')]
+                get_mock.assert_has_calls(wanted_get_calls)
+                # 4 delete calls for update and delete
+                delete_mock.assert_has_calls(
+                    [call(u'/domain/zone/unit.tests/record/100'),
+                     call(u'/domain/zone/unit.tests/record/101'),
+                     call(u'/domain/zone/unit.tests/record/102'),
+                     call(u'/domain/zone/unit.tests/record/103')])

tests/test_octodns_provider_powerdns.py

@@ -77,8 +77,8 @@ class TestPowerDnsProvider(TestCase):
         expected = Zone('unit.tests.', [])
         source = YamlProvider('test', join(dirname(__file__), 'config'))
         source.populate(expected)
-        expected_n = len(expected.records) - 1
-        self.assertEquals(15, expected_n)
+        expected_n = len(expected.records) - 2
+        self.assertEquals(16, expected_n)
 
         # No diffs == no changes
         with requests_mock() as mock:
@@ -86,7 +86,7 @@ class TestPowerDnsProvider(TestCase):
 
             zone = Zone('unit.tests.', [])
             provider.populate(zone)
-            self.assertEquals(15, len(zone.records))
+            self.assertEquals(16, len(zone.records))
             changes = expected.changes(zone, provider)
             self.assertEquals(0, len(changes))
 
@@ -99,12 +99,13 @@ class TestPowerDnsProvider(TestCase):
         # No existing records -> creates for every record in expected
         with requests_mock() as mock:
             mock.get(ANY, status_code=200, text=EMPTY_TEXT)
-            # post 201, is reponse to the create with data
+            # post 201, is response to the create with data
             mock.patch(ANY, status_code=201, text=assert_rrsets_callback)
 
             plan = provider.plan(expected)
             self.assertEquals(expected_n, len(plan.changes))
             self.assertEquals(expected_n, provider.apply(plan))
+            self.assertTrue(plan.exists)
 
         # Non-existent zone -> creates for every record in expected
         # OMG this is fucking ugly, probably better to ditch requests_mocks and
@@ -117,12 +118,13 @@ class TestPowerDnsProvider(TestCase):
             mock.get(ANY, status_code=422, text='')
             # patch 422's, unknown zone
             mock.patch(ANY, status_code=422, text=dumps(not_found))
-            # post 201, is reponse to the create with data
+            # post 201, is response to the create with data
             mock.post(ANY, status_code=201, text=assert_rrsets_callback)
 
             plan = provider.plan(expected)
             self.assertEquals(expected_n, len(plan.changes))
             self.assertEquals(expected_n, provider.apply(plan))
+            self.assertFalse(plan.exists)
 
         with requests_mock() as mock:
             # get 422's, unknown zone
@@ -166,7 +168,7 @@ class TestPowerDnsProvider(TestCase):
         expected = Zone('unit.tests.', [])
         source = YamlProvider('test', join(dirname(__file__), 'config'))
         source.populate(expected)
-        self.assertEquals(16, len(expected.records))
+        self.assertEquals(18, len(expected.records))
 
         # A small change to a single record
         with requests_mock() as mock:

tests/test_octodns_provider_rackspace.py

@@ -0,0 +1,866 @@
+#
+#
+#
+
+from __future__ import absolute_import, division, print_function, \
+    unicode_literals
+
+import json
+import re
+from unittest import TestCase
+from urlparse import urlparse
+
+from requests import HTTPError
+from requests_mock import ANY, mock as requests_mock
+
+from octodns.provider.rackspace import RackspaceProvider
+from octodns.record import Record
+from octodns.zone import Zone
+
+EMPTY_TEXT = '''
+{
+  "totalEntries" : 0,
+  "records" : []
+}
+'''
+
+with open('./tests/fixtures/rackspace-auth-response.json') as fh:
+    AUTH_RESPONSE = fh.read()
+
+with open('./tests/fixtures/rackspace-list-domains-response.json') as fh:
+    LIST_DOMAINS_RESPONSE = fh.read()
+
+with open('./tests/fixtures/rackspace-sample-recordset-page1.json') as fh:
+    RECORDS_PAGE_1 = fh.read()
+
+with open('./tests/fixtures/rackspace-sample-recordset-page2.json') as fh:
+    RECORDS_PAGE_2 = fh.read()
+
+
+class TestRackspaceProvider(TestCase):
+    def setUp(self):
+        self.maxDiff = 1000
+        with requests_mock() as mock:
+            mock.post(ANY, status_code=200, text=AUTH_RESPONSE)
+            self.provider = RackspaceProvider('identity', 'test', 'api-key',
+                                              '0')
+            self.assertTrue(mock.called_once)
+
+    def test_bad_auth(self):
+        with requests_mock() as mock:
+            mock.get(ANY, status_code=401, text='Unauthorized')
+
+            with self.assertRaises(Exception) as ctx:
+                zone = Zone('unit.tests.', [])
+                self.provider.populate(zone)
+            self.assertTrue('unauthorized' in ctx.exception.message)
+            self.assertTrue(mock.called_once)
+
+    def test_server_error(self):
+        with requests_mock() as mock:
+            mock.get(ANY, status_code=502, text='Things caught fire')
+
+            with self.assertRaises(HTTPError) as ctx:
+                zone = Zone('unit.tests.', [])
+                self.provider.populate(zone)
+            self.assertEquals(502, ctx.exception.response.status_code)
+            self.assertTrue(mock.called_once)
+
+    def test_nonexistent_zone(self):
+        # Non-existent zone doesn't populate anything
+        with requests_mock() as mock:
+            mock.get(ANY, status_code=404,
+                     json={'error': "Could not find domain 'unit.tests.'"})
+
+            zone = Zone('unit.tests.', [])
+            exists = self.provider.populate(zone)
+            self.assertEquals(set(), zone.records)
+            self.assertTrue(mock.called_once)
+            self.assertFalse(exists)
+
+    def test_multipage_populate(self):
+        with requests_mock() as mock:
+            mock.get(re.compile('domains$'), status_code=200,
+                     text=LIST_DOMAINS_RESPONSE)
+            mock.get(re.compile('records'), status_code=200,
+                     text=RECORDS_PAGE_1)
+            mock.get(re.compile('records.*offset=3'), status_code=200,
+                     text=RECORDS_PAGE_2)
+
+            zone = Zone('unit.tests.', [])
+            self.provider.populate(zone)
+            self.assertEquals(5, len(zone.records))
+
+    def test_plan_disappearing_ns_records(self):
+        expected = Zone('unit.tests.', [])
+        expected.add_record(Record.new(expected, '', {
+            'type': 'NS',
+            'ttl': 600,
+            'values': ['8.8.8.8.', '9.9.9.9.']
+        }))
+        expected.add_record(Record.new(expected, 'sub', {
+            'type': 'NS',
+            'ttl': 600,
+            'values': ['8.8.8.8.', '9.9.9.9.']
+        }))
+        with requests_mock() as mock:
+            mock.get(re.compile('domains$'), status_code=200,
+                     text=LIST_DOMAINS_RESPONSE)
+            mock.get(re.compile('records'), status_code=200, text=EMPTY_TEXT)
+
+            plan = self.provider.plan(expected)
+            self.assertTrue(mock.called)
+            self.assertTrue(plan.exists)
+
+            # OctoDNS does not propagate top-level NS records.
+            self.assertEquals(1, len(plan.changes))
+
+    def test_fqdn_a_record(self):
+        expected = Zone('example.com.', [])
+        # expected.add_record(Record.new(expected, 'foo', '1.2.3.4'))
+
+        with requests_mock() as list_mock:
+            list_mock.get(re.compile('domains$'), status_code=200,
+                          text=LIST_DOMAINS_RESPONSE)
+            list_mock.get(re.compile('records'), status_code=200,
+                          json={'records': [
+                              {'type': 'A',
+                               'name': 'foo.example.com',
+                               'id': 'A-111111',
+                               'data': '1.2.3.4',
+                               'ttl': 300}]})
+            plan = self.provider.plan(expected)
+            self.assertTrue(list_mock.called)
+            self.assertEqual(1, len(plan.changes))
+            self.assertTrue(
+                plan.changes[0].existing.fqdn == 'foo.example.com.')
+
+        with requests_mock() as mock:
+            def _assert_deleting(request, context):
+                parts = urlparse(request.url)
+                self.assertEqual('id=A-111111', parts.query)
+
+            mock.get(re.compile('domains$'), status_code=200,
+                     text=LIST_DOMAINS_RESPONSE)
+            mock.delete(re.compile('domains/.*/records?.*'), status_code=202,
+                        text=_assert_deleting)
+            self.provider.apply(plan)
+            self.assertTrue(mock.called)
+
+    def _test_apply_with_data(self, data):
+        expected = Zone('unit.tests.', [])
+        for record in data.OtherRecords:
+            expected.add_record(
+                Record.new(expected, record['subdomain'], record['data']))
+
+        with requests_mock() as list_mock:
+            list_mock.get(re.compile('domains$'), status_code=200,
+                          text=LIST_DOMAINS_RESPONSE)
+            list_mock.get(re.compile('records'), status_code=200,
+                          json=data.OwnRecords)
+            plan = self.provider.plan(expected)
+            self.assertTrue(list_mock.called)
+            if not data.ExpectChanges:
+                self.assertFalse(plan)
+                return
+
+        with requests_mock() as mock:
+            called = set()
+
+            def make_assert_sending_right_body(expected):
+                def _assert_sending_right_body(request, _context):
+                    called.add(request.method)
+                    if request.method != 'DELETE':
+                        self.assertEqual(request.headers['content-type'],
+                                         'application/json')
+                        self.assertDictEqual(expected,
+                                             json.loads(request.body))
+                    else:
+                        parts = urlparse(request.url)
+                        self.assertEqual(expected, parts.query)
+                    return ''
+
+                return _assert_sending_right_body
+
+            mock.get(re.compile('domains$'), status_code=200,
+                     text=LIST_DOMAINS_RESPONSE)
+            mock.post(re.compile('domains/.*/records$'), status_code=202,
+                      text=make_assert_sending_right_body(
+                          data.ExpectedAdditions))
+            mock.delete(re.compile('domains/.*/records?.*'), status_code=202,
+                        text=make_assert_sending_right_body(
+                            data.ExpectedDeletions))
+            mock.put(re.compile('domains/.*/records$'), status_code=202,
+                     text=make_assert_sending_right_body(data.ExpectedUpdates))
+
+            self.provider.apply(plan)
+            self.assertTrue(data.ExpectedAdditions is None or "POST" in called)
+            self.assertTrue(
+                data.ExpectedDeletions is None or "DELETE" in called)
+            self.assertTrue(data.ExpectedUpdates is None or "PUT" in called)
+
+    def test_apply_no_change_empty(self):
+        class TestData(object):
+            OtherRecords = []
+            OwnRecords = {
+                "totalEntries": 0,
+                "records": []
+            }
+            ExpectChanges = False
+            ExpectedAdditions = None
+            ExpectedDeletions = None
+            ExpectedUpdates = None
+
+        return self._test_apply_with_data(TestData)
+
+    def test_apply_no_change_a_records(self):
+        class TestData(object):
+            OtherRecords = [
+                {
+                    "subdomain": '',
+                    "data": {
+                        'type': 'A',
+                        'ttl': 300,
+                        'values': ['1.2.3.4', '1.2.3.5', '1.2.3.6']
+                    }
+                }
+            ]
+            OwnRecords = {
+                "totalEntries": 3,
+                "records": [{
+                    "name": "unit.tests",
+                    "id": "A-111111",
+                    "type": "A",
+                    "data": "1.2.3.4",
+                    "ttl": 300
+                }, {
+                    "name": "unit.tests",
+                    "id": "A-222222",
+                    "type": "A",
+                    "data": "1.2.3.5",
+                    "ttl": 300
+                }, {
+                    "name": "unit.tests",
+                    "id": "A-333333",
+                    "type": "A",
+                    "data": "1.2.3.6",
+                    "ttl": 300
+                }]
+            }
+            ExpectChanges = False
+            ExpectedAdditions = None
+            ExpectedDeletions = None
+            ExpectedUpdates = None
+
+        return self._test_apply_with_data(TestData)
+
+    def test_apply_no_change_a_records_cross_zone(self):
+        class TestData(object):
+            OtherRecords = [
+                {
+                    "subdomain": 'foo',
+                    "data": {
+                        'type': 'A',
+                        'ttl': 300,
+                        'value': '1.2.3.4'
+                    }
+                },
+                {
+                    "subdomain": 'bar',
+                    "data": {
+                        'type': 'A',
+                        'ttl': 300,
+                        'value': '1.2.3.4'
+                    }
+                }
+            ]
+            OwnRecords = {
+                "totalEntries": 3,
+                "records": [{
+                    "name": "foo.unit.tests",
+                    "id": "A-111111",
+                    "type": "A",
+                    "data": "1.2.3.4",
+                    "ttl": 300
+                }, {
+                    "name": "bar.unit.tests",
+                    "id": "A-222222",
+                    "type": "A",
+                    "data": "1.2.3.4",
+                    "ttl": 300
+                }]
+            }
+            ExpectChanges = False
+            ExpectedAdditions = None
+            ExpectedDeletions = None
+            ExpectedUpdates = None
+
+        return self._test_apply_with_data(TestData)
+
+    def test_apply_one_addition(self):
+        class TestData(object):
+            OtherRecords = [
+                {
+                    "subdomain": '',
+                    "data": {
+                        'type': 'A',
+                        'ttl': 300,
+                        'value': '1.2.3.4'
+                    }
+                },
+                {
+                    "subdomain": 'foo',
+                    "data": {
+                        'type': 'NS',
+                        'ttl': 300,
+                        'value': 'ns.example.com.'
+                    }
+                }
+            ]
+            OwnRecords = {
+                "totalEntries": 0,
+                "records": []
+            }
+            ExpectChanges = True
+            ExpectedAdditions = {
+                "records": [{
+                    "name": "unit.tests",
+                    "type": "A",
+                    "data": "1.2.3.4",
+                    "ttl": 300
+                }, {
+                    "name": "foo.unit.tests",
+                    "type": "NS",
+                    "data": "ns.example.com",
+                    "ttl": 300
+                }]
+            }
+            ExpectedDeletions = None
+            ExpectedUpdates = None
+
+        return self._test_apply_with_data(TestData)
+
+    def test_apply_create_MX(self):
+        class TestData(object):
+            OtherRecords = [
+                {
+                    "subdomain": '',
+                    "data": {
+                        'type': 'MX',
+                        'ttl': 300,
+                        'value': {
+                            'value': 'mail1.example.com.',
+                            'priority': 1,
+                        }
+                    }
+                },
+                {
+                    "subdomain": 'foo',
+                    "data": {
+                        'type': 'MX',
+                        'ttl': 300,
+                        'value': {
+                            'value': 'mail2.example.com.',
+                            'priority': 2
+                        }
+                    }
+                }
+            ]
+            OwnRecords = {
+                "totalEntries": 0,
+                "records": []
+            }
+            ExpectChanges = True
+            ExpectedAdditions = {
+                "records": [{
+                    "name": "foo.unit.tests",
+                    "type": "MX",
+                    "data": "mail2.example.com",
+                    "priority": 2,
+                    "ttl": 300
+                }, {
+                    "name": "unit.tests",
+                    "type": "MX",
+                    "data": "mail1.example.com",
+                    "priority": 1,
+                    "ttl": 300
+                }]
+            }
+            ExpectedDeletions = None
+            ExpectedUpdates = None
+
+        return self._test_apply_with_data(TestData)
+
+    def test_apply_multiple_additions_splatting(self):
+        class TestData(object):
+            OtherRecords = [
+                {
+                    "subdomain": '',
+                    "data": {
+                        'type': 'A',
+                        'ttl': 300,
+                        'values': ['1.2.3.4', '1.2.3.5', '1.2.3.6']
+                    }
+                },
+                {
+                    "subdomain": 'foo',
+                    "data": {
+                        'type': 'NS',
+                        'ttl': 300,
+                        'values': ['ns1.example.com.', 'ns2.example.com.']
+                    }
+                }
+            ]
+            OwnRecords = {
+                "totalEntries": 0,
+                "records": []
+            }
+            ExpectChanges = True
+            ExpectedAdditions = {
+                "records": [{
+                    "name": "unit.tests",
+                    "type": "A",
+                    "data": "1.2.3.4",
+                    "ttl": 300
+                }, {
+                    "name": "unit.tests",
+                    "type": "A",
+                    "data": "1.2.3.5",
+                    "ttl": 300
+                }, {
+                    "name": "unit.tests",
+                    "type": "A",
+                    "data": "1.2.3.6",
+                    "ttl": 300
+                }, {
+                    "name": "foo.unit.tests",
+                    "type": "NS",
+                    "data": "ns1.example.com",
+                    "ttl": 300
+                }, {
+                    "name": "foo.unit.tests",
+                    "type": "NS",
+                    "data": "ns2.example.com",
+                    "ttl": 300
+                }]
+            }
+            ExpectedDeletions = None
+            ExpectedUpdates = None
+
+        return self._test_apply_with_data(TestData)
+
+    def test_apply_multiple_additions_namespaced(self):
+        class TestData(object):
+            OtherRecords = [{
+                "subdomain": 'foo',
+                "data": {
+                    'type': 'A',
+                    'ttl': 300,
+                    'value': '1.2.3.4'
+                }
+            }, {
+                "subdomain": 'bar',
+                "data": {
+                    'type': 'A',
+                    'ttl': 300,
+                    'value': '1.2.3.4'
+                }
+            }, {
+                "subdomain": 'foo',
+                "data": {
+                    'type': 'NS',
+                    'ttl': 300,
+                    'value': 'ns.example.com.'
+                }
+            }]
+            OwnRecords = {
+                "totalEntries": 0,
+                "records": []
+            }
+            ExpectChanges = True
+            ExpectedAdditions = {
+                "records": [{
+                    "name": "bar.unit.tests",
+                    "type": "A",
+                    "data": "1.2.3.4",
+                    "ttl": 300
+                }, {
+                    "name": "foo.unit.tests",
+                    "type": "A",
+                    "data": "1.2.3.4",
+                    "ttl": 300
+                }, {
+                    "name": "foo.unit.tests",
+                    "type": "NS",
+                    "data": "ns.example.com",
+                    "ttl": 300
+                }]
+            }
+            ExpectedDeletions = None
+            ExpectedUpdates = None
+
+        return self._test_apply_with_data(TestData)
+
+    def test_apply_single_deletion(self):
+        class TestData(object):
+            OtherRecords = []
+            OwnRecords = {
+                "totalEntries": 1,
+                "records": [{
+                    "name": "unit.tests",
+                    "id": "A-111111",
+                    "type": "A",
+                    "data": "1.2.3.4",
+                    "ttl": 300
+                }, {
+                    "name": "foo.unit.tests",
+                    "id": "NS-111111",
+                    "type": "NS",
+                    "data": "ns.example.com",
+                    "ttl": 300
+                }]
+            }
+            ExpectChanges = True
+            ExpectedAdditions = None
+            ExpectedDeletions = "id=A-111111&id=NS-111111"
+            ExpectedUpdates = None
+
+        return self._test_apply_with_data(TestData)
+
+    def test_apply_multiple_deletions(self):
+        class TestData(object):
+            OtherRecords = [
+                {
+                    "subdomain": '',
+                    "data": {
+                        'type': 'A',
+                        'ttl': 300,
+                        'value': '1.2.3.5'
+                    }
+                }
+            ]
+            OwnRecords = {
+                "totalEntries": 3,
+                "records": [{
+                    "name": "unit.tests",
+                    "id": "A-111111",
+                    "type": "A",
+                    "data": "1.2.3.4",
+                    "ttl": 300
+                }, {
+                    "name": "unit.tests",
+                    "id": "A-222222",
+                    "type": "A",
+                    "data": "1.2.3.5",
+                    "ttl": 300
+                }, {
+                    "name": "unit.tests",
+                    "id": "A-333333",
+                    "type": "A",
+                    "data": "1.2.3.6",
+                    "ttl": 300
+                }, {
+                    "name": "foo.unit.tests",
+                    "id": "NS-111111",
+                    "type": "NS",
+                    "data": "ns.example.com",
+                    "ttl": 300
+                }]
+            }
+            ExpectChanges = True
+            ExpectedAdditions = None
+            ExpectedDeletions = "id=A-111111&id=A-333333&id=NS-111111"
+            ExpectedUpdates = {
+                "records": [{
+                    "name": "unit.tests",
+                    "id": "A-222222",
+                    "data": "1.2.3.5",
+                    "ttl": 300
+                }]
+            }
+
+        return self._test_apply_with_data(TestData)
+
+    def test_apply_multiple_deletions_cross_zone(self):
+        class TestData(object):
+            OtherRecords = [
+                {
+                    "subdomain": '',
+                    "data": {
+                        'type': 'A',
+                        'ttl': 300,
+                        'value': '1.2.3.4'
+                    }
+                }
+            ]
+            OwnRecords = {
+                "totalEntries": 3,
+                "records": [{
+                    "name": "unit.tests",
+                    "id": "A-111111",
+                    "type": "A",
+                    "data": "1.2.3.4",
+                    "ttl": 300
+                }, {
+                    "name": "foo.unit.tests",
+                    "id": "A-222222",
+                    "type": "A",
+                    "data": "1.2.3.5",
+                    "ttl": 300
+                }, {
+                    "name": "bar.unit.tests",
+                    "id": "A-333333",
+                    "type": "A",
+                    "data": "1.2.3.6",
+                    "ttl": 300
+                }]
+            }
+            ExpectChanges = True
+            ExpectedAdditions = None
+            ExpectedDeletions = "id=A-222222&id=A-333333"
+            ExpectedUpdates = None
+
+        return self._test_apply_with_data(TestData)
+
+    def test_apply_delete_cname(self):
+        class TestData(object):
+            OtherRecords = []
+            OwnRecords = {
+                "totalEntries": 3,
+                "records": [{
+                    "name": "foo.unit.tests",
+                    "id": "CNAME-111111",
+                    "type": "CNAME",
+                    "data": "a.example.com",
+                    "ttl": 300
+                }]
+            }
+            ExpectChanges = True
+            ExpectedAdditions = None
+            ExpectedDeletions = "id=CNAME-111111"
+            ExpectedUpdates = None
+
+        return self._test_apply_with_data(TestData)
+
+    def test_apply_single_update(self):
+        class TestData(object):
+            OtherRecords = [
+                {
+                    "subdomain": '',
+                    "data": {
+                        'type': 'A',
+                        'ttl': 3600,
+                        'value': '1.2.3.4'
+                    }
+                }
+            ]
+            OwnRecords = {
+                "totalEntries": 1,
+                "records": [{
+                    "name": "unit.tests",
+                    "id": "A-111111",
+                    "type": "A",
+                    "data": "1.2.3.4",
+                    "ttl": 300
+                }]
+            }
+            ExpectChanges = True
+            ExpectedAdditions = None
+            ExpectedDeletions = None
+            ExpectedUpdates = {
+                "records": [{
+                    "name": "unit.tests",
+                    "id": "A-111111",
+                    "data": "1.2.3.4",
+                    "ttl": 3600
+                }]
+            }
+
+        return self._test_apply_with_data(TestData)
+
+    def test_apply_update_TXT(self):
+        class TestData(object):
+            OtherRecords = [
+                {
+                    "subdomain": '',
+                    "data": {
+                        'type': 'TXT',
+                        'ttl': 300,
+                        'value': 'othervalue'
+                    }
+                }
+            ]
+            OwnRecords = {
+                "totalEntries": 1,
+                "records": [{
+                    "name": "unit.tests",
+                    "id": "TXT-111111",
+                    "type": "TXT",
+                    "data": "somevalue",
+                    "ttl": 300
+                }]
+            }
+            ExpectChanges = True
+            ExpectedAdditions = {
+                "records": [{
+                    "name": "unit.tests",
+                    "type": "TXT",
+                    "data": "othervalue",
+                    "ttl": 300
+                }]
+            }
+            ExpectedDeletions = 'id=TXT-111111'
+            ExpectedUpdates = None
+
+        return self._test_apply_with_data(TestData)
+
+    def test_apply_update_MX(self):
+        class TestData(object):
+            OtherRecords = [
+                {
+                    "subdomain": '',
+                    "data": {
+                        'type': 'MX',
+                        'ttl': 300,
+                        'value': {u'priority': 50, u'value': 'mx.test.com.'}
+                    }
+                }
+            ]
+            OwnRecords = {
+                "totalEntries": 1,
+                "records": [{
+                    "name": "unit.tests",
+                    "id": "MX-111111",
+                    "type": "MX",
+                    "priority": 20,
+                    "data": "mx.test.com",
+                    "ttl": 300
+                }]
+            }
+            ExpectChanges = True
+            ExpectedAdditions = {
+                "records": [{
+                    "name": "unit.tests",
+                    "type": "MX",
+                    "priority": 50,
+                    "data": "mx.test.com",
+                    "ttl": 300
+                }]
+            }
+            ExpectedDeletions = 'id=MX-111111'
+            ExpectedUpdates = None
+
+        return self._test_apply_with_data(TestData)
+
+    def test_apply_multiple_updates(self):
+        class TestData(object):
+            OtherRecords = [
+                {
+                    "subdomain": '',
+                    "data": {
+                        'type': 'A',
+                        'ttl': 3600,
+                        'values': ['1.2.3.4', '1.2.3.5', '1.2.3.6']
+                    }
+                }
+            ]
+            OwnRecords = {
+                "totalEntries": 3,
+                "records": [{
+                    "name": "unit.tests",
+                    "id": "A-111111",
+                    "type": "A",
+                    "data": "1.2.3.4",
+                    "ttl": 300
+                }, {
+                    "name": "unit.tests",
+                    "id": "A-222222",
+                    "type": "A",
+                    "data": "1.2.3.5",
+                    "ttl": 300
+                }, {
+                    "name": "unit.tests",
+                    "id": "A-333333",
+                    "type": "A",
+                    "data": "1.2.3.6",
+                    "ttl": 300
+                }]
+            }
+            ExpectChanges = True
+            ExpectedAdditions = None
+            ExpectedDeletions = None
+            ExpectedUpdates = {
+                "records": [{
+                    "name": "unit.tests",
+                    "id": "A-222222",
+                    "data": "1.2.3.5",
+                    "ttl": 3600
+                }, {
+                    "name": "unit.tests",
+                    "id": "A-111111",
+                    "data": "1.2.3.4",
+                    "ttl": 3600
+                }, {
+                    "name": "unit.tests",
+                    "id": "A-333333",
+                    "data": "1.2.3.6",
+                    "ttl": 3600
+                }]
+            }
+
+        return self._test_apply_with_data(TestData)
+
+    def test_apply_multiple_updates_cross_zone(self):
+        class TestData(object):
+            OtherRecords = [
+                {
+                    "subdomain": 'foo',
+                    "data": {
+                        'type': 'A',
+                        'ttl': 3600,
+                        'value': '1.2.3.4'
+                    }
+                },
+                {
+                    "subdomain": 'bar',
+                    "data": {
+                        'type': 'A',
+                        'ttl': 3600,
+                        'value': '1.2.3.4'
+                    }
+                }
+            ]
+            OwnRecords = {
+                "totalEntries": 2,
+                "records": [{
+                    "name": "foo.unit.tests",
+                    "id": "A-111111",
+                    "type": "A",
+                    "data": "1.2.3.4",
+                    "ttl": 300
+                }, {
+                    "name": "bar.unit.tests",
+                    "id": "A-222222",
+                    "type": "A",
+                    "data": "1.2.3.4",
+                    "ttl": 300
+                }]
+            }
+            ExpectChanges = True
+            ExpectedAdditions = None
+            ExpectedDeletions = None
+            ExpectedUpdates = {
+                "records": [{
+                    "name": "bar.unit.tests",
+                    "id": "A-222222",
+                    "data": "1.2.3.4",
+                    "ttl": 3600
+                }, {
+                    "name": "foo.unit.tests",
+                    "id": "A-111111",
+                    "data": "1.2.3.4",
+                    "ttl": 3600
+                }]
+            }
+
+        return self._test_apply_with_data(TestData)

tests/test_octodns_provider_route53.py

@@ -324,6 +324,14 @@ class TestRoute53Provider(TestCase):
                     'Value': '0 issue "ca.unit.tests"',
                 }],
                 'TTL': 69,
+            }, {
+                'AliasTarget': {
+                    'HostedZoneId': 'Z119WBBTVP5WFX',
+                    'EvaluateTargetHealth': False,
+                    'DNSName': 'unit.tests.'
+                },
+                'Type': 'A',
+                'Name': 'alias.unit.tests.'
             }],
             'IsTruncated': False,
             'MaxItems': '100',
@@ -342,9 +350,9 @@ class TestRoute53Provider(TestCase):
         stubber.assert_no_pending_responses()
 
         # Populate a zone that doesn't exist
-        noexist = Zone('does.not.exist.', [])
-        provider.populate(noexist)
-        self.assertEquals(set(), noexist.records)
+        nonexistent = Zone('does.not.exist.', [])
+        provider.populate(nonexistent)
+        self.assertEquals(set(), nonexistent.records)
 
     def test_sync(self):
         provider, stubber = self._get_stubbed_provider()
@@ -372,6 +380,7 @@ class TestRoute53Provider(TestCase):
 
         plan = provider.plan(self.expected)
         self.assertEquals(9, len(plan.changes))
+        self.assertTrue(plan.exists)
         for change in plan.changes:
             self.assertIsInstance(change, Create)
         stubber.assert_no_pending_responses()
@@ -604,6 +613,7 @@ class TestRoute53Provider(TestCase):
 
         plan = provider.plan(self.expected)
         self.assertEquals(9, len(plan.changes))
+        self.assertFalse(plan.exists)
         for change in plan.changes:
             self.assertIsInstance(change, Create)
         stubber.assert_no_pending_responses()

tests/test_octodns_provider_yaml.py

@@ -30,7 +30,7 @@ class TestYamlProvider(TestCase):
 
         # without it we see everything
         source.populate(zone)
-        self.assertEquals(16, len(zone.records))
+        self.assertEquals(18, len(zone.records))
 
         # Assumption here is that a clean round-trip means that everything
         # worked as expected, data that went in came back out and could be
@@ -49,12 +49,12 @@ class TestYamlProvider(TestCase):
 
             # We add everything
             plan = target.plan(zone)
-            self.assertEquals(13, len(filter(lambda c: isinstance(c, Create),
+            self.assertEquals(15, len(filter(lambda c: isinstance(c, Create),
                                              plan.changes)))
             self.assertFalse(isfile(yaml_file))
 
             # Now actually do it
-            self.assertEquals(13, target.apply(plan))
+            self.assertEquals(15, target.apply(plan))
             self.assertTrue(isfile(yaml_file))
 
             # There should be no changes after the round trip
@@ -64,15 +64,19 @@ class TestYamlProvider(TestCase):
 
             # A 2nd sync should still create everything
             plan = target.plan(zone)
-            self.assertEquals(13, len(filter(lambda c: isinstance(c, Create),
+            self.assertEquals(15, len(filter(lambda c: isinstance(c, Create),
                                              plan.changes)))
 
             with open(yaml_file) as fh:
                 data = safe_load(fh.read())
 
+                # '' has some of both
+                roots = sorted(data[''], key=lambda r: r['type'])
+                self.assertTrue('values' in roots[0])  # A
+                self.assertTrue('value' in roots[1])   # CAA
+                self.assertTrue('values' in roots[2])  # SSHFP
+
                 # these are stored as plural 'values'
-                for r in data['']:
-                    self.assertTrue('values' in r)
                 self.assertTrue('values' in data['mx'])
                 self.assertTrue('values' in data['naptr'])
                 self.assertTrue('values' in data['_srv._tcp'])

tests/test_octodns_record.py

@@ -96,6 +96,57 @@ class TestRecord(TestCase):
 
             DummyRecord().__repr__()
 
+    def test_values_mixin_data(self):
+        # no values, no value or values in data
+        a = ARecord(self.zone, '', {
+            'type': 'A',
+            'ttl': 600,
+            'values': []
+        })
+        self.assertNotIn('values', a.data)
+
+        # empty value, no value or values in data
+        b = ARecord(self.zone, '', {
+            'type': 'A',
+            'ttl': 600,
+            'values': ['']
+        })
+        self.assertNotIn('value', b.data)
+
+        # empty/None values, no value or values in data
+        c = ARecord(self.zone, '', {
+            'type': 'A',
+            'ttl': 600,
+            'values': ['', None]
+        })
+        self.assertNotIn('values', c.data)
+
+        # empty/None values and valid, value in data
+        c = ARecord(self.zone, '', {
+            'type': 'A',
+            'ttl': 600,
+            'values': ['', None, '10.10.10.10']
+        })
+        self.assertNotIn('values', c.data)
+        self.assertEqual('10.10.10.10', c.data['value'])
+
+    def test_value_mixin_data(self):
+        # unspecified value, no value in data
+        a = AliasRecord(self.zone, '', {
+            'type': 'ALIAS',
+            'ttl': 600,
+            'value': None
+        })
+        self.assertNotIn('value', a.data)
+
+        # unspecified value, no value in data
+        a = AliasRecord(self.zone, '', {
+            'type': 'ALIAS',
+            'ttl': 600,
+            'value': ''
+        })
+        self.assertNotIn('value', a.data)
+
     def test_geo(self):
         geo_data = {'ttl': 42, 'values': ['5.2.3.4', '6.2.3.4'],
                     'geo': {'AF': ['1.1.1.1'],
@@ -294,7 +345,7 @@ class TestRecord(TestCase):
         self.assertEquals(a_data, a.data)
 
         b_value = {
-            'preference': 12,
+            'preference': 0,
             'exchange': 'smtp3.',
         }
         b_data = {'ttl': 30, 'value': b_value}
@@ -379,7 +430,7 @@ class TestRecord(TestCase):
         self.assertEqual(change.new, other)
 
         # full sorting
-        # equivilent
+        # equivalent
         b_naptr_value = b.values[0]
         self.assertEquals(0, b_naptr_value.__cmp__(b_naptr_value))
         # by order
@@ -659,7 +710,7 @@ class TestRecord(TestCase):
             Record.new(self.zone, 'unknown', {})
         self.assertTrue('missing type' in ctx.exception.message)
 
-        # Unkown type
+        # Unknown type
         with self.assertRaises(Exception) as ctx:
             Record.new(self.zone, 'unknown', {
                 'type': 'XXX',
@@ -781,6 +832,16 @@ class TestRecordValidation(TestCase):
             }, lenient=True)
         self.assertEquals(('value',), ctx.exception.args)
 
+        # no exception if we're in lenient mode from config
+        Record.new(self.zone, 'www', {
+            'octodns': {
+                'lenient': True
+            },
+            'type': 'A',
+            'ttl': -1,
+            'value': '1.2.3.4',
+        }, lenient=True)
+
     def test_A_and_values_mixin(self):
         # doesn't blow up
         Record.new(self.zone, '', {
@@ -788,6 +849,13 @@ class TestRecordValidation(TestCase):
             'ttl': 600,
             'value': '1.2.3.4',
         })
+        Record.new(self.zone, '', {
+            'type': 'A',
+            'ttl': 600,
+            'values': [
+                '1.2.3.4',
+            ]
+        })
         Record.new(self.zone, '', {
             'type': 'A',
             'ttl': 600,
@@ -797,13 +865,60 @@ class TestRecordValidation(TestCase):
             ]
         })
 
-        # missing value(s)
+        # missing value(s), no value or value
         with self.assertRaises(ValidationError) as ctx:
             Record.new(self.zone, '', {
                 'type': 'A',
                 'ttl': 600,
             })
         self.assertEquals(['missing value(s)'], ctx.exception.reasons)
+
+        # missing value(s), empty values
+        with self.assertRaises(ValidationError) as ctx:
+            Record.new(self.zone, 'www', {
+                'type': 'A',
+                'ttl': 600,
+                'values': []
+            })
+        self.assertEquals(['missing value(s)'], ctx.exception.reasons)
+
+        # missing value(s), None values
+        with self.assertRaises(ValidationError) as ctx:
+            Record.new(self.zone, 'www', {
+                'type': 'A',
+                'ttl': 600,
+                'values': None
+            })
+        self.assertEquals(['missing value(s)'], ctx.exception.reasons)
+
+        # missing value(s) and empty value
+        with self.assertRaises(ValidationError) as ctx:
+            Record.new(self.zone, 'www', {
+                'type': 'A',
+                'ttl': 600,
+                'values': [None, '']
+            })
+        self.assertEquals(['missing value(s)',
+                           'empty value'], ctx.exception.reasons)
+
+        # missing value(s), None value
+        with self.assertRaises(ValidationError) as ctx:
+            Record.new(self.zone, 'www', {
+                'type': 'A',
+                'ttl': 600,
+                'value': None
+            })
+        self.assertEquals(['missing value(s)'], ctx.exception.reasons)
+
+        # empty value, empty string value
+        with self.assertRaises(ValidationError) as ctx:
+            Record.new(self.zone, 'www', {
+                'type': 'A',
+                'ttl': 600,
+                'value': ''
+            })
+        self.assertEquals(['empty value'], ctx.exception.reasons)
+
         # missing value(s) & ttl
         with self.assertRaises(ValidationError) as ctx:
             Record.new(self.zone, '', {
@@ -960,6 +1075,24 @@ class TestRecordValidation(TestCase):
             })
         self.assertEquals(['missing value'], ctx.exception.reasons)
 
+        # missing value
+        with self.assertRaises(ValidationError) as ctx:
+            Record.new(self.zone, 'www', {
+                'type': 'ALIAS',
+                'ttl': 600,
+                'value': None
+            })
+        self.assertEquals(['missing value'], ctx.exception.reasons)
+
+        # empty value
+        with self.assertRaises(ValidationError) as ctx:
+            Record.new(self.zone, 'www', {
+                'type': 'ALIAS',
+                'ttl': 600,
+                'value': ''
+            })
+        self.assertEquals(['empty value'], ctx.exception.reasons)
+
         # missing trailing .
         with self.assertRaises(ValidationError) as ctx:
             Record.new(self.zone, '', {
@@ -1275,7 +1408,7 @@ class TestRecordValidation(TestCase):
                 'ttl': 600,
                 'value': {
                     'algorithm': 'nope',
-                    'fingerprint_type': 1,
+                    'fingerprint_type': 2,
                     'fingerprint': 'bf6b6825d2977c511a475bbefb88aad54a92ac73'
                 }
             })
@@ -1301,7 +1434,7 @@ class TestRecordValidation(TestCase):
                 'type': 'SSHFP',
                 'ttl': 600,
                 'value': {
-                    'algorithm': 1,
+                    'algorithm': 2,
                     'fingerprint': 'bf6b6825d2977c511a475bbefb88aad54a92ac73'
                 }
             })
@@ -1313,7 +1446,7 @@ class TestRecordValidation(TestCase):
                 'type': 'SSHFP',
                 'ttl': 600,
                 'value': {
-                    'algorithm': 1,
+                    'algorithm': 3,
                     'fingerprint_type': 'yeeah',
                     'fingerprint': 'bf6b6825d2977c511a475bbefb88aad54a92ac73'
                 }

tests/test_octodns_zone.py

@@ -236,3 +236,102 @@ class TestZone(TestCase):
         zone.add_record(cname)
         with self.assertRaises(InvalidNodeException):
             zone.add_record(a)
+
+    def test_excluded_records(self):
+        zone_normal = Zone('unit.tests.', [])
+        zone_excluded = Zone('unit.tests.', [])
+        zone_missing = Zone('unit.tests.', [])
+
+        normal = Record.new(zone_normal, 'www', {
+            'ttl': 60,
+            'type': 'A',
+            'value': '9.9.9.9',
+        })
+        zone_normal.add_record(normal)
+
+        excluded = Record.new(zone_excluded, 'www', {
+            'octodns': {
+                'excluded': ['test']
+            },
+            'ttl': 60,
+            'type': 'A',
+            'value': '9.9.9.9',
+        })
+        zone_excluded.add_record(excluded)
+
+        provider = SimpleProvider()
+
+        self.assertFalse(zone_normal.changes(zone_excluded, provider))
+        self.assertTrue(zone_normal.changes(zone_missing, provider))
+
+        self.assertFalse(zone_excluded.changes(zone_normal, provider))
+        self.assertFalse(zone_excluded.changes(zone_missing, provider))
+
+        self.assertTrue(zone_missing.changes(zone_normal, provider))
+        self.assertFalse(zone_missing.changes(zone_excluded, provider))
+
+    def test_included_records(self):
+        zone_normal = Zone('unit.tests.', [])
+        zone_included = Zone('unit.tests.', [])
+        zone_missing = Zone('unit.tests.', [])
+
+        normal = Record.new(zone_normal, 'www', {
+            'ttl': 60,
+            'type': 'A',
+            'value': '9.9.9.9',
+        })
+        zone_normal.add_record(normal)
+
+        included = Record.new(zone_included, 'www', {
+            'octodns': {
+                'included': ['test']
+            },
+            'ttl': 60,
+            'type': 'A',
+            'value': '9.9.9.9',
+        })
+        zone_included.add_record(included)
+
+        provider = SimpleProvider()
+
+        self.assertFalse(zone_normal.changes(zone_included, provider))
+        self.assertTrue(zone_normal.changes(zone_missing, provider))
+
+        self.assertFalse(zone_included.changes(zone_normal, provider))
+        self.assertTrue(zone_included.changes(zone_missing, provider))
+
+        self.assertTrue(zone_missing.changes(zone_normal, provider))
+        self.assertTrue(zone_missing.changes(zone_included, provider))
+
+    def test_not_included_records(self):
+        zone_normal = Zone('unit.tests.', [])
+        zone_included = Zone('unit.tests.', [])
+        zone_missing = Zone('unit.tests.', [])
+
+        normal = Record.new(zone_normal, 'www', {
+            'ttl': 60,
+            'type': 'A',
+            'value': '9.9.9.9',
+        })
+        zone_normal.add_record(normal)
+
+        included = Record.new(zone_included, 'www', {
+            'octodns': {
+                'included': ['not-here']
+            },
+            'ttl': 60,
+            'type': 'A',
+            'value': '9.9.9.9',
+        })
+        zone_included.add_record(included)
+
+        provider = SimpleProvider()
+
+        self.assertFalse(zone_normal.changes(zone_included, provider))
+        self.assertTrue(zone_normal.changes(zone_missing, provider))
+
+        self.assertFalse(zone_included.changes(zone_normal, provider))
+        self.assertFalse(zone_included.changes(zone_missing, provider))
+
+        self.assertTrue(zone_missing.changes(zone_normal, provider))
+        self.assertFalse(zone_missing.changes(zone_included, provider))