Downcase DNS names (#253)

* Downcase DNS names
* Document opinions

docs/index.md

@@ -7,7 +7,7 @@ title: DnsControl
 	<div class="col-md-12">
 		<div>
 			<h1 class="hometitle">DnsControl</h1>
-			<p class="lead">DnsControl is a platform for seamlessly managing your DNS configuration across any number of DNS hosts, both in the cloud or in your own infrastructure. It manages all of the domains for the Stack Overflow network, and can do the same for you!</p>
+			<p class="lead">DnsControl is an <strong><a href="opinions">opinionated</a></strong> platform for seamlessly managing your DNS configuration across any number of DNS hosts, both in the cloud or in your own infrastructure. It manages all of the domains for the Stack Overflow network, and can do the same for you!</p>
 		</div>
 	</div>
 </div>

docs/opinions.md

@@ -0,0 +1,132 @@
+---
+layout: default
+title: An Opinionated System
+---
+
+# DNSControl is an opinionated system
+
+DNSControl is an opinionated system. That means that we have certain
+opinions about how things should work.
+
+This page documents those opinions.
+
+
+# Opinion #1: DNS should be treated like code.
+
+Code is written in a high-level language, version controlled,
+commented, tested, and reviewed by a third party... and all of that
+happens before it goes into production.
+
+DNS information should be stored in a version control system, like
+Git or Mercurial, and receive all the benefits of using VCS.  Changes
+should be in the form of PRs that are approved by someone-other-than-you.
+
+DNS information should be tested for syntax, pass unit tests and
+policy tests, all in an automated CI system that assures all changes
+are made the same way. (We don't provide a CI system, but DNSControl
+makes it easy to use one; and not use one when an emergency update
+is needed.)
+
+Pushing the changes into production should be effortless, not
+requiring people to know which domains are on which providers, or
+that certain providers do things differently that others.  The
+credentials for updates should be controlled such that anyone can
+write a PR, but not everyone has access to the credentials.
+
+
+# Opinion #2: Non-experts should be able to safely make DNS changes.
+
+The goal of DNSControl is to create a system that is set up by DNS
+experts like you, but updates and changes can be made by your
+coworkers who aren't DNS experts.
+
+Things your coworkers should not have to know:
+
+Your coworkers should not have to know obscure DNS technical
+knowledge.  That's your job.
+
+Your coworkers should not have to know what happens in ambiguous
+situations.  That's your job.
+
+Your coworkers should be able to submit PRs to dnsconfig.js for you
+to approve; preferably via a CI system that does rudimentary checks
+before you even have to see the PR.
+
+Your coworkers should be able to figure out the language without
+much training. The system should block them from doing dangerous
+things (even if they are technically legal).
+
+
+# Opinion #3: dnsconfig.js are not zonefiles.
+
+A zonefile can list any kind of DNS record. It has no judgement and
+no morals. It will let you do bad practices as long as the bits are
+RFC-compliant.
+
+dnsconfig.js is a high-level description of your DNS zone data.
+Being high-level permits the code to understand intent, and stop
+bad behavior.
+
+TODO: List an example.
+
+
+# Opinion #4: All DNS is lowercase for languages that have such a concept.
+
+DNSControl downcases all DNS names (domains, labels, and targets).  #sorrynotsorry
+
+When the system reads dnsconfig.js or receives data from DNS providers,
+the DNS names are downcased.
+
+This reduces code complexity, reduces the number of edge-cases that must
+be tested, and makes the system safer to operate.
+
+Yes, we know that DNS is case insensitive.  See Opinion #3.
+
+
+# Opinion #5: Users should state what they want, and DNSControl should do the rest.
+
+When possible, dnsconfig.js lists a high-level description of what
+is desired and the compiler does the hard work for you.
+
+Some examples:
+
+* Macros and iterators permit you to state something once, correctly, and repeat it many places.
+* TXT strings are expressed as JavaScript strings, with no weird DNS-required special escape charactors.  DNSControl does the escaping for you.
+* Domain names with Unicode are listed as real Unicode.  Punycode translation is done for you.
+* IP addresses are expressed as IP addresses; and reversing them to in-addr.arpa addresses is done for you.
+* SPF records are stated in the most verbose way; DNSControl optimizes it for you in a safe, opt-in way.
+
+
+# Opinion #6 If it is ambiguous in DNS, it is forbidden in DNSControl.
+
+When there is ambiguity an expert knows what the system will do.
+Your coworkers should not be expected to be experts. (See Opinion #2).
+
+We would rather DNSControl error out than require users to be DNS experts.
+
+For example:
+
+We know that "bar.com." is a FQDN because it ends with a dot.
+
+Is "bar.com" a FQDN? Well, obviously it is, because it already ends
+with ".com" and we all know that "bar.com.bar.com" is probably not
+what the user intended.
+
+We know that "bar" is *not* an FQDN because it doesn't contain any dots.
+
+Is "meta.xyz" a FQDN?
+
+That's ambiguous.  If the user knows that "xyz" is a top level domain (TLD)
+then it is obvious that it is a FQDN.  However, can anyone really memorize
+all the TLDSs?  There used to be just gov/edu/com/mil/org/net and everyone
+could memorize them easily.  As of 2000, there are many, many, more.
+You can't memorize them all.  In fact, even before 2000 you couldn't
+memorize them all. (In fact, you didn't even realize that we left out "int"!)
+
+"xyz" became a TLD in June 2014.  Thus, after 2014 a system like DNSControl
+would have to act differently.  We don't want to be surprised by changes
+like that.
+
+Therefore, we require all CNAME, MX, and NS targets to be FQDNs (they must
+end with a "."), or to be a shortname (no dots at all).  Everything
+else is ambiguous and therefore an error.

docs/writing-providers.md

@@ -213,6 +213,9 @@ the integration tests to see what works and what doesn't.  Fix any
 bugs and repeat, repeat, repeat until you have all the capabilities
 you want to implement.
 
+FYI: If a provider's capabilities changes, run `go generate` to update
+the documentation.
+
 
 ## Vendoring Dependencies
 

integrationTest/integration_test.go

@@ -112,6 +112,7 @@ func runTests(t *testing.T, prv providers.DNSServiceProvider, domainName string,
 				}
 				dom.Records = append(dom.Records, &rc)
 			}
+			models.Downcase(dom.Records)
 			dom2, _ := dom.Copy()
 			// get corrections for first time
 			corrections, err := prv.GetDomainCorrections(dom)
@@ -392,6 +393,16 @@ func makeTests(t *testing.T) []*TestCase {
 		)
 	}
 
+	// Case
+	tests = append(tests, tc("Empty"),
+		tc("Empty"),
+		tc("Create CAPS", mx("BAR", 5, "BAR.com.")),
+		tc("Downcase label", mx("bar", 5, "BAR.com."), a("decoy", "1.1.1.1")),
+		tc("Downcase target", mx("bar", 5, "bar.com."), a("decoy", "2.2.2.2")),
+		tc("Upcase both", mx("BAR", 5, "BAR.COM."), a("decoy", "3.3.3.3")),
+		// The decoys are required so that there is at least one actual change in each tc.
+	)
+
 	// Test large zonefiles.
 	// Mostly to test paging. Many providers page at 100
 	// Known page sizes:

models/dns.go

@@ -89,12 +89,16 @@ func (r *RecordConfig) String() (content string) {
 
 	content = fmt.Sprintf("%s %s %s %d", r.Type, r.NameFQDN, r.Target, r.TTL)
 	switch r.Type { // #rtype_variations
-	case "A", "AAAA", "CNAME", "PTR", "TXT":
+	case "A", "AAAA", "CNAME", "NS", "PTR", "TXT":
 		// Nothing special.
 	case "MX":
-		content += fmt.Sprintf(" priority=%d", r.MxPreference)
+		content += fmt.Sprintf(" pref=%d", r.MxPreference)
 	case "SOA":
 		content = fmt.Sprintf("%s %s %s %d", r.Type, r.Name, r.Target, r.TTL)
+	case "SRV":
+		content += fmt.Sprintf(" srvpriority=%d srvweight=%d srvport=%d", r.SrvPriority, r.SrvWeight, r.SrvPort)
+	case "TLSA":
+		content += fmt.Sprintf(" tlsausage=%d tlsaselector=%d tlsamatchingtype=%d", r.TlsaUsage, r.TlsaSelector, r.TlsaMatchingType)
 	case "CAA":
 		content += fmt.Sprintf(" caatag=%s caaflag=%d", r.CaaTag, r.CaaFlag)
 	default:
@@ -246,6 +250,25 @@ func (r Records) Grouped() map[RecordKey]Records {
 	return groups
 }
 
+// Downcase converts all labels and targets to lowercase in a list of RecordConfig.
+func Downcase(recs []*RecordConfig) {
+	for _, r := range recs {
+		r.Name = strings.ToLower(r.Name)
+		r.NameFQDN = strings.ToLower(r.NameFQDN)
+		switch r.Type {
+		case "ANAME", "CNAME", "MX", "NS", "PTR":
+			r.Target = strings.ToLower(r.Target)
+		case "A", "AAAA", "ALIAS", "CAA", "IMPORT_TRANSFORM", "SRV", "TLSA", "TXT", "SOA":
+			// Do nothing.
+		default:
+			panic(fmt.Sprintf("Downcase: Unimplemented rtype %v", r.Type))
+			// We panic so that we quickly find any switch statements
+			// that have not been updated for a new RR type.
+		}
+	}
+	return
+}
+
 type RecordKey struct {
 	Name string
 	Type string

models/dns_test.go

@@ -68,3 +68,23 @@ func TestRR(t *testing.T) {
 		t.Errorf("RR expected (%#v) got (%#v)\n", expected, found)
 	}
 }
+
+func TestDowncase(t *testing.T) {
+	dc := DomainConfig{Records: Records{
+		&RecordConfig{Type: "MX", Name: "lower", Target: "targetmx"},
+		&RecordConfig{Type: "MX", Name: "UPPER", Target: "TARGETMX"},
+	}}
+	Downcase(dc.Records)
+	if !dc.HasRecordTypeName("MX", "lower") {
+		t.Errorf("%v: expected (%v) got (%v)\n", dc.Records, false, true)
+	}
+	if !dc.HasRecordTypeName("MX", "upper") {
+		t.Errorf("%v: expected (%v) got (%v)\n", dc.Records, false, true)
+	}
+	if dc.Records[0].Target != "targetmx" {
+		t.Errorf("%v: target0 expected (%v) got (%v)\n", dc.Records, "targetmx", dc.Records[0].Target)
+	}
+	if dc.Records[1].Target != "targetmx" {
+		t.Errorf("%v: target1 expected (%v) got (%v)\n", dc.Records, "targetmx", dc.Records[1].Target)
+	}
+}

pkg/normalize/validate.go

@@ -265,6 +265,7 @@ func NormalizeAndValidateConfig(config *models.DNSConfig) (errs []error) {
 			ns.Name = strings.TrimRight(ns.Name, ".")
 		}
 		// Normalize Records.
+		models.Downcase(domain.Records)
 		for _, rec := range domain.Records {
 			if rec.TTL == 0 {
 				rec.TTL = models.DefaultTTL

providers/activedir/domains.go

@@ -46,6 +46,9 @@ func (c *adProvider) GetDomainCorrections(dc *models.DomainConfig) ([]*models.Co
 		return nil, fmt.Errorf("c.getExistingRecords(%v) failed: %v", dc.Name, err)
 	}
 
+	// Normalize
+	models.Downcase(foundRecords)
+
 	differ := diff.New(dc)
 	_, creates, dels, modifications := differ.IncrementalDiff(foundRecords)
 	// NOTE(tlim): This provider does not delete records.  If

providers/bind/bindProvider.go

@@ -237,6 +237,9 @@ func (c *Bind) GetDomainCorrections(dc *models.DomainConfig) ([]*models.Correcti
 		dc.Records = append(dc.Records, soaRec)
 	}
 
+	// Normalize
+	models.Downcase(foundRecords)
+
 	differ := diff.New(dc)
 	_, create, del, mod := differ.IncrementalDiff(foundRecords)
 

providers/cloudflare/cloudflareProvider.go

@@ -120,6 +120,10 @@ func (c *CloudflareApi) GetDomainCorrections(dc *models.DomainConfig) ([]*models
 		}
 	}
 	checkNSModifications(dc)
+
+	// Normalize
+	models.Downcase(records)
+
 	differ := diff.New(dc, getProxyMetadata)
 	_, create, del, mod := differ.IncrementalDiff(records)
 	corrections := []*models.Correction{}

providers/digitalocean/digitaloceanProvider.go

@@ -101,6 +101,9 @@ func (api *DoApi) GetDomainCorrections(dc *models.DomainConfig) ([]*models.Corre
 		existingRecords[i] = toRc(dc, &records[i])
 	}
 
+	// Normalize
+	models.Downcase(existingRecords)
+
 	differ := diff.New(dc)
 	_, create, delete, modify := differ.IncrementalDiff(existingRecords)
 

providers/dnsimple/dnsimpleProvider.go

@@ -90,6 +90,10 @@ func (c *DnsimpleApi) GetDomainCorrections(dc *models.DomainConfig) ([]*models.C
 		}
 		return true
 	})
+
+	// Normalize
+	models.Downcase(actual)
+
 	differ := diff.New(dc)
 	_, create, delete, modify := differ.IncrementalDiff(actual)
 

providers/gandi/gandiProvider.go

@@ -111,6 +111,10 @@ func (c *GandiApi) GetDomainCorrections(dc *models.DomainConfig) ([]*models.Corr
 		recordsToKeep = append(recordsToKeep, rec)
 	}
 	dc.Records = recordsToKeep
+
+	// Normalize
+	models.Downcase(foundRecords)
+
 	differ := diff.New(dc)
 	_, create, del, mod := differ.IncrementalDiff(foundRecords)
 

providers/gcloud/gcloudProvider.go

@@ -136,6 +136,9 @@ func (g *gcloud) GetDomainCorrections(dc *models.DomainConfig) ([]*models.Correc
 		want.MergeToTarget()
 	}
 
+	// Normalize
+	models.Downcase(existingRecords)
+
 	// first collect keys that have changed
 	differ := diff.New(dc)
 	_, create, delete, modify := differ.IncrementalDiff(existingRecords)

providers/namecheap/namecheapProvider.go

@@ -155,6 +155,9 @@ func (n *Namecheap) GetDomainCorrections(dc *models.DomainConfig) ([]*models.Cor
 		actual = append(actual, rec)
 	}
 
+	// Normalize
+	models.Downcase(actual)
+
 	differ := diff.New(dc)
 	_, create, delete, modify := differ.IncrementalDiff(actual)
 

providers/namedotcom/records.go

@@ -37,6 +37,9 @@ func (n *nameDotCom) GetDomainCorrections(dc *models.DomainConfig) ([]*models.Co
 
 	checkNSModifications(dc)
 
+	// Normalize
+	models.Downcase(actual)
+
 	differ := diff.New(dc)
 	_, create, del, mod := differ.IncrementalDiff(actual)
 	corrections := []*models.Correction{}

providers/ns1/ns1provider.go

@@ -66,6 +66,9 @@ func (n *nsone) GetDomainCorrections(dc *models.DomainConfig) ([]*models.Correct
 	foundGrouped := found.Grouped()
 	desiredGrouped := dc.Records.Grouped()
 
+	//  Normalize
+	models.Downcase(found)
+
 	differ := diff.New(dc)
 	changedGroups := differ.ChangedGroups(found)
 	corrections := []*models.Correction{}

providers/route53/route53Provider.go

@@ -170,6 +170,9 @@ func (r *route53Provider) GetDomainCorrections(dc *models.DomainConfig) ([]*mode
 		want.MergeToTarget()
 	}
 
+	// Normalize
+	models.Downcase(existingRecords)
+
 	//diff
 	differ := diff.New(dc)
 	_, create, delete, modify := differ.IncrementalDiff(existingRecords)

providers/softlayer/softlayerProvider.go

@@ -163,6 +163,9 @@ func (s *SoftLayer) getExistingRecords(domain *datatypes.Dns_Domain) ([]*models.
 		actual = append(actual, recConfig)
 	}
 
+	// Normalize
+	models.Downcase(actual)
+
 	return actual, nil
 }
 

providers/vultr/vultrProvider.go

@@ -97,6 +97,9 @@ func (api *VultrApi) GetDomainCorrections(dc *models.DomainConfig) ([]*models.Co
 		curRecords[i] = r
 	}
 
+	// Normalize
+	models.Downcase(curRecords)
+
 	differ := diff.New(dc)
 	_, create, delete, modify := differ.IncrementalDiff(curRecords)