version 0.7

Also supports calling by secondary argument (kind-of busybox
stylekind-of). Drop support for multicall setuptools wrapper – when
installed using setuptools, the user should call explicit function
instead.

.travis.yml

@@ -0,0 +1,16 @@
+before_install:
+  - sudo apt-get install -y bind9utils
+language: python
+python:
+ - "3.5"
+ - "3.6"
+ - "nightly"
+matrix:
+  allow_failures:
+    - python: "nightly"
+install:
+ - pip install -e .
+ - pip install pytest
+script:
+ - pytest
+sudo: false

README.rst

@@ -15,43 +15,80 @@ Main features
 - check if zone file compiles properly using `named-compilezone(8)`_
 - autodetect zone name from file name or ``$ORIGIN`` directive
 - enforce updating serial number when zone content is changed
+- optional ``smudge`` filter to replace ``$UNIXTIME`` directive with current UNIX time
 - both ``pre-commit`` and ``pre-receive``/``update`` hooks to enforce similar checks in the remote repository
 - ``post-receive`` hook to checkout the working copy from a bare repository, generate config snippets for various DNS server software and reload them
-- only Python standard library is used
+- only Python 3.5+ standard library is used
 
 
 Requirements
 ------------
 
 - Python 3.5+
-- `named-compilezone(8)`_ (part of BIND9 package)
+- `named-compilezone(8)`_ (part of `bind9utils` package)
 - git
 
 
-Instalation and usage
----------------------
+Simple instalation (especially for workstations)
+------------------------------------------------
 
-- install required dependencies
-- install ``dzonegit`` package using your favourite tool (``virtualenvwrapper``,
-  ``venv``, ``pipenv``, etc.)
+Since there is no other Python dependency than the standard library, you can
+simply download the `dzonegit.py` file, make it executable and rename/symlink
+it to an appropriate hook location inside the Git repository
+`.git/hooks/pre-commit`. This is especially handy for the end users not
+experienced with Python packaging ecosystem.
+
+
+Full instalation and usage
+--------------------------
+
+- install all the requirements
+- install ``dzonegit`` Python package using your
+  favourite tool (``virtualenvwrapper``, ``venv``, ``pipenv``, etc.)
 - in the local repository, create a symlink for the ``pre-commit`` hook:
-
   ``$ ln -s $(which dzonegit-pre-commit) /path/to/repo/.git/hooks/pre-commit``
-- on the server, install some git repository management software, preferrably Gitolite_
-- on the server, install either ``pre-receive`` or ``update`` hook (both do the same) as
-  well as ``post-receive`` hook. See `Gitolite documentation on how to add custom hooks`_
-- on the server set up the configuration options for each repository
+- on the server, install some git repository management software,
+  preferably Gitolite_
+- on the server, install either ``pre-receive`` or ``update`` hook
+  (both do the same) as  well as the ``post-receive`` hook. See `Gitolite
+  documentation on how to add custom hooks`_
+- on the server, set up the configuration options for each repository
+
+Support for $UNIXTIME directive
+-------------------------------
+
+If you want to use ``$UNIXTIME`` in your zone files instead of serial number,
+you have to install a `smudge` filter on the server, that will replace the
+directive with current unix time on every checkout. First, set up the filter
+in the Git configuration:
+
+.. code-block:: shell
+
+        $ git config --global filter.dzonegit.smudge $(which dzonegit-smudge-serial)
+
+
+Then, apply the filter on all zone files using either ``.git/info/attributes``
+or directly ``.gitattributes`` file inside the repository:
+
+.. code-block::
+
+        *.zone filter=dzonegit
+
 
 Configuration options
 ---------------------
 
-All configuration options are stored in `git-config(1)`_ in section named ``dzonegit``.
-All boolean options default to *False*.
+All configuration options are stored in `git-config(1)`_ in the section
+named ``dzonegit``.  All boolean options default to *False*.
 
 
 *dzonegit.ignorewhitespaceerrors*
   Ignore white space errors in ``pre-commit`` and ``pre-receive``/``update`` hooks.
 
+*dzonegit.allowfancynames*
+  In  ``pre-commit`` and ``pre-receive``/``update`` hooks, do not enforce zone
+  file name to be similar to the name of the zone.
+
 *dzonegit.noserialupdate*
   Do not try to automatically update zone serial number if necessary.
   Valid only in the ``pre-commit`` hook.
@@ -76,7 +113,7 @@ All boolean options default to *False*.
   can be provided by appending single digit from 1 to 9 to this option.
 
 *dzonegit.zonereloadcmd*
-  A command to run for each zone, whose zone file has been modified. Zone
+  A command to run for each zone, where zone file has been modified. Zone
   name is automatically appended as the last argument. Should do something
   like ``rndc reload``. More commands can be provided by appending single digit
   from 1 to 9 to this option.
@@ -84,20 +121,27 @@ All boolean options default to *False*.
 *dzonegit.zoneblacklist*
   Path to a text file containing list of zone names without trailing dots,
   one per line. If zone is found on the blacklist, it is ignored when
-  ``post-receive`` hook generates configuration.
+  ``post-receive`` hook generates configuration. Wildcards can be used as
+  well, see `JSON template`_ below.
 
 *dzonegit.zonewhitelist*
   Path to a text file containing list of zone names without trailing dots,
   one per line. If not empty and zone is not found on the whitelist,
-  it is ignored when ``post-receive`` hook generates configuration.
+  it is ignored when ``post-receive`` hook generates configuration. Wildcards
+  can be used as well, see `JSON template`_ below.
 
 JSON template
 -------------
 
 The DNS server configuration snippets are generated using a simple JSON-based
-template. All keys are optional but please make sure the file is a valid
-JSON file. It is possible to define a zone-specific options, for instance for
-changing DNSSEC parameters per zone.
+template. All keys are optional but please make sure the file is a valid JSON
+file. It is possible to define a zone-specific options, for instance for
+changing DNSSEC parameters per zone. Those zone-specific options allow usage of
+wildcards; if an exact match of zone name is not found, the leftmost label is
+substituted with `*`. If still no match is found, the leftmost label is dropped
+and the second one is again substituted with `*`. In the end, a single `*` is
+checked. Only if even this key is not found, the value of *defaultvar* is used
+as the zone-specific option.
 
 Valid keys are:
 
@@ -112,19 +156,22 @@ Valid keys are:
 
 *defaultvar*
   A string that would template variable ``$zonevar`` expand to if there is not
-  a zone-specific variable defined.
+  a zone-specific variable defined, nor any wildcard matched.
 
 *zonevars*
   An object mapping zone names (without the final dot) to a zone-specific
-  variable to which template variable ``$zonevar`` would expand to.
+  variable to which template variable ``$zonevar`` would expand to. Using
+  wildcards is possible by replacing the leftmost label with `*`. Ultimately,
+  a key with label `*` will match every single zone (making *defaultvar*
+  option litte bit pointless)
 
 In the template strings, these placeholders are supported:
 
 ``$datetime``
-  Current timestamp
+  Current date and time in human readable format
 
 ``$zonename``
-  Zone name, without trailing dot
+  Zone name, without the trailing dot
 
 ``$zonefile``
   Full path to the zone file
@@ -138,12 +185,14 @@ Example JSON template for Knot DNS
 .. code-block:: json
 
     {
-      "header": "# Managed by dzonegit, do not edit.\n",
+      "header": "# Managed by dzonegit, do not edit.\nzone:",
       "footer": "",
-      "item": " - zone: \"$zonename\"\n   file: \"$zonefile\"\n   $zonevar\n",
+      "item": " - domain: \"$zonename\"\n   file: \"$zonefile\"\n   $zonevar\n",
       "defaultvar": "template: default",
       "zonevars": {
-        "example.com": "template: signed"
+        "example.com": "template: signed",
+        "*.cz": "template: czdomains",
+        "*.in-addr.arpa": "template: ipv4reverse"
       }
     }
 

dzonegit.py

@@ -83,14 +83,26 @@ def get_file_contents(path, revision=None):
     return r.stdout
 
 
-def compile_zone(zonename, zonedata):
+def unixtime_directive(zonedata, unixtime=None):
+    """ Filter binary zone data. Replace $UNIXTIME with current unix time. """
+    if unixtime is None:
+        unixtime = int(time.time())
+    return re.sub(
+        br'\$UNIXTIME\b',
+        str(unixtime).encode("ascii"),
+        zonedata,
+        flags=re.IGNORECASE,
+    )
+
+
+def compile_zone(zonename, zonedata, unixtime=None):
     """ Compile the zone. Return tuple with results."""
     CompileResults = namedtuple(
         "CompileResults", "success, serial, zonehash, stderr",
     )
     r = subprocess.run(
         ["/usr/sbin/named-compilezone", "-o", "-", zonename, "/dev/stdin"],
-        input=zonedata,
+        input=unixtime_directive(zonedata, unixtime),
         stdout=subprocess.PIPE,
         stderr=subprocess.PIPE,
     )
@@ -173,15 +185,15 @@ def get_zone_origin(zonedata):
 def get_zone_name(path, zonedata):
     """
     Try to guess zone name from either filename or the first $ORIGIN.
-    Throw a HookException if filename and zone ORIGIN differ more than
-    in slashes.
+    Unless disabled, throw a HookException if filename and zone ORIGIN differ
+    more than in slashes.
     """
     stemname = Path(path).stem.lower()
     originname = get_zone_origin(zonedata)
     if originname:
         tt = str.maketrans("", "", "/_,:-+*%^&#$")
         sn, on = [s.translate(tt) for s in [stemname, originname]]
-        if sn != on:
+        if sn != on and not get_config("dzonegit.allowfancynames", bool):
             raise HookException(
                 "Zone origin {o} differs from zone file.".format(o=originname),
                 fname=path,
@@ -193,13 +205,14 @@ def get_zone_name(path, zonedata):
 
 def check_updated_zones(against, revision=None, autoupdate_serial=False):
     """ Check whether all updated zone files compile. """
+    unixtime = int(time.time())
     for f in get_altered_files(against, "AMCR", revision):
         if not f.suffix == ".zone":
             continue
         print("Checking file {f}".format(f=f))
         zonedata = get_file_contents(f, revision)
         zname = get_zone_name(f, zonedata)
-        rnew = compile_zone(zname, zonedata)
+        rnew = compile_zone(zname, zonedata, unixtime)
         if not rnew.success:
             raise HookException(
                 "New zone version does not compile",
@@ -208,7 +221,7 @@ def check_updated_zones(against, revision=None, autoupdate_serial=False):
         try:
             zonedata = get_file_contents(f, against)
             zname = get_zone_name(f, zonedata)
-            rold = compile_zone(zname, zonedata)
+            rold = compile_zone(zname, zonedata, unixtime-1)
 
             if (rold.success and rold.zonehash != rnew.zonehash and not
                     is_serial_increased(rold.serial, rnew.serial)):
@@ -268,6 +281,23 @@ def replace_serial(path, oldserial, newserial):
     path.write_text(updated)
 
 
+def get_zone_wildcards(name):
+    """ A generator of wildcards out of a zone name.
+    For a DNS name, returns series of:
+     - the name itself
+     - the name with first label substitued as *
+     - the name with first label dropped and second substittuted as *
+     - ...
+     - single *
+"""
+    yield name
+    labels = name.split(".")
+    while labels:
+        labels[0] = "*"
+        yield ".".join(labels)
+        labels.pop(0)
+
+
 def template_config(checkoutpath, template, blacklist=set(), whitelist=set()):
     """ Recursively find all *.zone files and template config file using
     a simple JSON based template like this:
@@ -278,7 +308,9 @@ def template_config(checkoutpath, template, blacklist=set(), whitelist=set()):
       "item": " - zone: \"$zonename\"\n   file: \"$zonefile\"\n   $zonevar\n",
       "defaultvar": "template: default",
       "zonevars": {
-        "example.com": "template: signed"
+        "example.com": "template: signed",
+        "*.com": "template: dotcom",
+        "*": "template: uberdefault"
       }
     }
 
@@ -301,13 +333,16 @@ def template_config(checkoutpath, template, blacklist=set(), whitelist=set()):
     out.append(headertpl.substitute(mapping))
     for f in sorted(Path(checkoutpath).glob("**/*.zone")):
         zonename = get_zone_name(f, f.read_bytes())
-        if whitelist and zonename not in whitelist:
+        if whitelist and not any(
+                n in whitelist
+                for n in get_zone_wildcards(zonename)
+        ):
             print(
                 "WARNING: Ignoring zone {} - not whitelisted for "
                 "this repository.".format(zonename),
             )
             continue
-        if zonename in blacklist:
+        if any(n in blacklist for n in get_zone_wildcards(zonename)):
             print(
                 "WARNING: Ignoring zone {} - blacklisted for "
                 "this repository.".format(zonename),
@@ -323,7 +358,12 @@ def template_config(checkoutpath, template, blacklist=set(), whitelist=set()):
             )
             continue
         zones[zonename] = f.relative_to(checkoutpath)
-        zonevar = zonevars[zonename] if zonename in zonevars else defaultvar
+        for name in get_zone_wildcards(zonename):
+            if name in zonevars:
+                zonevar = zonevars[name]
+                break
+        else:
+            zonevar = defaultvar
         out.append(itemtpl.substitute(
             mapping, zonename=zonename,
             zonefile=str(f), zonevar=zonevar,
@@ -407,6 +447,7 @@ def post_receive(stdin=sys.stdin):
         raise SystemExit("Checkout path not defined. Nothing to do.")
 
     print("Checking out repository into {}…".format(checkoutpath))
+    Path(checkoutpath).mkdir(parents=True, exist_ok=True)
     subprocess.run(
         ["git", "checkout", "-f", "master"],
         check=True,
@@ -438,7 +479,8 @@ def post_receive(stdin=sys.stdin):
         if refname != "refs/heads/master":
             continue
         if against == "0000000000000000000000000000000000000000":
-            against = get_head()  # Empty commit
+            # Empty commit
+            against = "4b825dc642cb6eb9a060e54bf8d69288fbee4904"
         should_reconfig = [
             f for f in get_altered_files(against, "ACDRU", revision)
             if f.suffix == ".zone"
@@ -468,17 +510,36 @@ def post_receive(stdin=sys.stdin):
                     subprocess.run(cmd)
 
 
+def smudge_serial(
+        bstdin=sys.stdin.buffer,
+        bstdout=sys.stdout.buffer,
+        unixtime=None,
+):
+    """Replace all $UNIXTIME directives with current unix time."""
+    bstdout.write(unixtime_directive(bstdin.read(), unixtime))
+
+
+def get_action(argv=sys.argv):
+    name = Path(argv[0]).name
+    if "pre-commit" in name:
+        return pre_commit
+    if "update" in name:
+        return update
+    if "pre-receive" in name:
+        return pre_receive
+    if "post-receive" in name:
+        return post_receive
+    if "smudge" in name:
+        return smudge_serial
+
+
 def main():
-    name = Path(sys.argv[0]).name
-    print(name)
-    if name == "pre-commit":
-        pre_commit()
-    elif name == "update":
-        update()
-    elif name == "pre-receive":
-        pre_receive()
-    elif name == "post-receive":
-        post_receive()
+    action = get_action()
+    if action is None and len(sys.argv) > 1:
+        sys.argv.pop(0)
+        action = get_action()
+    if action:
+        action()
     else:
         sys.exit("No valid command found")
 

setup.py

@@ -5,7 +5,7 @@ readme = Path(__file__).with_name("README.rst").read_text()
 
 setup(
     name="dzonegit",
-    version="0.1",
+    version="0.7",
     description="Git hooks to manage a repository of DNS zones",
     long_description=readme,
     long_description_content_type="text/x-rst",
@@ -19,11 +19,11 @@ setup(
     tests_require=["pytest"],
     entry_points={
             "console_scripts": [
-                "dzonegit = dzonegit:main",
                 "dzonegit-pre-commit = dzonegit:pre_commit",
                 "dzonegit-pre-receive = dzonegit:pre_receive",
                 "dzonegit-post-receive = dzonegit:post_receive",
                 "dzonegit-update = dzonegit:update",
+                "dzonegit-smudge-serial = dzonegit:smudge_serial",
             ],
     },
     classifiers=[

test_dzonegit.py

@@ -4,7 +4,7 @@ import subprocess
 import time
 import datetime
 import os
-from io import StringIO
+from io import StringIO, BytesIO
 from pathlib import Path
 
 import dzonegit
@@ -74,6 +74,33 @@ ns.example.com.      60 IN A 192.0.2.1
     assert r.zonehash == r2.zonehash
 
 
+def test_compile_unsmudged_zone():
+    testzone = b"""
+$ORIGIN example.com.
+@       60 IN SOA ns hostmaster (
+                                $UNIXTIME  ; serial
+                                3600       ; refresh (1 hour)
+                                900        ; retry (15 minutes)
+                                1814400    ; expire (3 weeks)
+                                60         ; minimum (1 minute)
+                                )
+        60 IN NS ns
+ns.example.com.      60 IN A 192.0.2.1
+"""
+    replaced = dzonegit.unixtime_directive(testzone)
+    assert b"$UNIXTIME" not in replaced
+    r = dzonegit.compile_zone("example.com", testzone, 123456)
+    assert r.success
+    assert r.serial == str(123456)
+
+
+def test_smudge_serial():
+    bstdin = BytesIO(b"something $UNIXTIME something")
+    bstdout = BytesIO()
+    dzonegit.smudge_serial(bstdin, bstdout, 123456)
+    assert b"something 123456 something" == bstdout.getvalue()
+
+
 def test_is_serial_increased():
     assert dzonegit.is_serial_increased(1234567890, "2018010100")
     assert dzonegit.is_serial_increased("2018010100", "4018010100")
@@ -131,6 +158,8 @@ ns.example.com.      60 IN A 192.0.2.1
     )
     with pytest.raises(ValueError):
         dzonegit.get_zone_name("zones/example.org.zone", testzone)
+    subprocess.call(["git", "config", "dzonegit.allowfancynames", "TRUE"])
+    dzonegit.get_zone_name("zones/example.org.zone", testzone)
     testzone = b"""
 $ORIGIN 240/28.2.0.192.in-addr.arpa.
 @       60 IN SOA ns hostmaster 1 60 60 60 60
@@ -226,6 +255,29 @@ $ORIGIN dummy.
         dzonegit.check_updated_zones("HEAD", autoupdate_serial=True)
     subprocess.call(["git", "add", "dummy.zone"])
     dzonegit.check_updated_zones(dzonegit.get_head())
+    git_dir.join("dummy.zone").write("""
+$ORIGIN dummy.
+@ 60 IN SOA ns hm $UNIXTIME 61 60 60 60
+  60 NS ns.example.org.
+""")
+    subprocess.call(["git", "add", "dummy.zone"])
+    dzonegit.check_updated_zones(dzonegit.get_head())
+    subprocess.call(["git", "commit", "-m", "dummy.zone with $UNIXTIME"])
+    git_dir.join("dummy.zone").write("""
+$ORIGIN dummy.
+@ 60 IN SOA ns hm 1 60 60 60 60
+  60 NS ns.example.org.
+""")
+    subprocess.call(["git", "add", "dummy.zone"])
+    with pytest.raises(ValueError):
+        dzonegit.check_updated_zones(dzonegit.get_head())
+    git_dir.join("dummy.zone").write("""
+$ORIGIN dummy.
+@ 60 IN SOA ns hm $UNIXTIME 60 60 60 60
+  60 NS ns.example.org.
+""")
+    subprocess.call(["git", "add", "dummy.zone"])
+    dzonegit.check_updated_zones(dzonegit.get_head())
     subprocess.call(["git", "commit", "-m", "final dummy.zone"])
     dzonegit.check_updated_zones("HEAD~", "HEAD")
 
@@ -276,17 +328,18 @@ def test_post_receive(git_dir):
     git_dir.chdir()
     head = dzonegit.get_head()
     revisions = "{} {} refs/heads/master\n".format(
-        "4b825dc642cb6eb9a060e54bf8d69288fbee4904",
+        "0000000000000000000000000000000000000000",
         head,
     )
     stdin = StringIO(revisions)
-    codir = git_dir.mkdir("co")
+    codir = git_dir.join("co")
     subprocess.call(["git", "config", "dzonegit.checkoutpath", str(codir)])
     subprocess.call([
         "git", "config", "dzonegit.reconfigcmd",
         "echo TEST >{}/test".format(codir),
     ])
     dzonegit.post_receive(stdin)
+    dzonegit.post_receive(stdin)  # Check coping with existing codir
     assert codir.join("dummy.zone").check()
     assert codir.join("test").read() == "TEST\n"
 
@@ -298,12 +351,14 @@ def test_template_config(git_dir):
   "item": " - zone: \"$zonename\"\n   file: \"$zonefile\"\n   $zonevar\n",
   "defaultvar": "template: default",
   "zonevars": {
-    "example.com": "template: signed"
+    "example.com": "template: signed",
+    "*": "template: dummy"
   }
 }"""
     output = dzonegit.template_config(str(git_dir), template)
     assert output.startswith("# Managed by dzonegit")
     assert " - zone: \"dummy\"\n   file: \"" in output
+    assert "   template: dummy" in output
     assert output.endswith("# This is the end")
     output = dzonegit.template_config(
         str(git_dir),
@@ -311,9 +366,22 @@ def test_template_config(git_dir):
         whitelist=set("a"),
     )
     assert " - zone: \"dummy\"\n   file: \"" not in output
+    output = dzonegit.template_config(
+        str(git_dir),
+        template,
+        blacklist=set("*"),
+    )
+    assert " - zone: \"dummy\"\n   file: \"" not in output
 
 
 def test_load_set_file(git_dir):
     git_dir.join("dummy").write("dummy\n\n # Comment")
     s = dzonegit.load_set_file("dummy")
     assert s == {"dummy"}
+
+
+def test_get_zone_wildcards():
+    assert list(dzonegit.get_zone_wildcards("a.long.zone.name")) == [
+        "a.long.zone.name", "*.long.zone.name",
+        "*.zone.name", "*.name", "*",
+    ]