In the last post, we described three different use cases for custom DNS entries in Kubernetes:
- Making an alias for an external name
- Dynamically adding services to another domain, without running another server
- Adding an arbitrary entry inside the cluster domain
In that post we covered the first two. In this post, we’ll show you how to use the fallthrough option of the kubernetes
plugin to satisfy the third case.
To understand how this works, we first need to look at how CoreDNS processes requests. This was previously addressed in Query Routing, but we’ll go into a bit more detail here.
We all know that CoreDNS chains plugins. But what exactly does that mean? To find out, we’ll dissect a Corefile and see how that translates into CoreDNS internals, and discuss how a query is routed through these internals.
Consider this Corefile:
coredns.io:5300 {
file /etc/coredns/zones/coredns.io.db
}
example.io:53 {
errors
log
file /etc/coredns/zones/example.io.db
}
example.net:53 {
file /etc/coredns/zones/example.net.db
}
.:53 {
errors
log
health
rewrite name foo.example.com foo.default.svc.cluster.local
kubernetes cluster.local 10.0.0.0/24
file /etc/coredns/example.db example.org
forward . /etc/resolv.conf
cache 30
}
Notice here that there are two different ports: 5300 and 53. Internally, each of these ports will
result in a dnsserver.Server.
Even though there are four server blocks (stanzas), we only get two actual servers. CoreDNS will gather up all of the
server blocks associated with the same port and combine them in to the same dnsserver.Server. The server will
multiplex the queries on the port, passing them to the different plugin chains depending upon the zone. It chooses
the most specific matching server block for the zone. If no server block matches, SERVFAIL is returned. This is shown
visually in the diagram below.
So, any specific query will go through exactly one plugin chain. The ordering of the plugin is dictated
at build time, by the plugin.cfg file,
although there is some discussion about making this modifiable
at runtime. This is why even though cache appears at the end of the server block, it is not the end of the
plugin chain.
Notice in the .:53 server block, we define the health plugin, but it doesn’t appear in the diagram. This is
because there are a few different types of plugin. “Normal” plugin perform request handling, and appear in the
plugin chain. However, there are a few plugin that just modify the configuration of the server or server block.
Since they don’t have any request-time logic, they are not inserted in the plugin chain. Some plugin that work
this way are the health, tls, startup, shutdown, and root plugin.
You can divide the plugin that do perform request-time processing into two groups: plugin that
manipulate the request in some way, and backend plugin. Backend plugin provide different sources
of zone and record data. The etcd, file, and kubernetes plugin are all examples of backends.
Plugin that manipulate the request but are not backends - that is, they are not a source of zone data - generally
will pass the query to the next plugin after performing their logic. For example, the rewrite plugin makes
a change to the request, and then passes it on. When the result is returned from the later plugin, it reverts the
question section to the original (so that clients don’t complain), but keeps the response and passes it back to the client.
Any given backend is usually the final word for its zone - it either returns a result, or it returns NXDOMAIN for the
query. However, occasionally this is not the desired behavior, so some of the plugin support a fallthrough option.
When fallthrough is enabled, instead of returning NXDOMAIN when a record is not found, the plugin will pass the
request down the chain. A backend further down the chain then has the opportunity to handle the request.
Coming back to our original discussion of the three use cases in Kubernetes, we can now understand how we can use
fallthrough to meet the third use case. Remember the initial Corefile from that blog:
.:53 {
errors
log
health
kubernetes cluster.local 10.0.0.0/24
forward . /etc/resolv.conf
cache 30
}
This handles the standard in-cluster DNS service discovery. The third use case is to add an arbitrary entry to the existing
cluster domain. To do this, we define another backend for handling the cluster.local zone, and configure the fallthrough
option in the kubernetes plugin. For very dynamic entries, we could use the etcd plugin. But for demonstration
purposes it’s simpler to use the file plugin, so that is what we will do.
Since kubernetes comes before file in plugin.cfg,
using fallthrough in kubernetes will result in file handling any queries that kubernetes does not. This means that we
need to have a zone file as part of our ConfigMap, just like we did to handle the second use case in the previous blog. In this case, though, instead of being configured with a different zone (example.org in the other blog), the file plugin is
configured to use the cluster.local domain:
apiVersion: v1
kind: ConfigMap
metadata:
name: coredns
namespace: kube-system
data:
Corefile: |
.:53 {
errors
log
health
kubernetes cluster.local 10.0.0.0/24 {
fallthrough
}
file /etc/coredns/cluster.db cluster.local
forward . /etc/resolv.conf
cache 30
}
cluster.db: |
cluster.local. IN SOA ns.dns.cluster.local. hostmaster.cluster.local. 2015082541 7200 3600 1209600 3600
something.cluster.local. IN A 10.0.0.1
otherthing.cluster.local. IN CNAME google.com.
Remember to add the cluster.db file to the config-volume for the pod template:
volumes:
- name: config-volume
configMap:
name: coredns
items:
- key: Corefile
path: Corefile
- key: cluster.db
path: cluster.db
and finally to signal CoreDNS to gracefully reload (each pod running):
$ kubectl -n kube-system exec coredns-461002909-7mp96 -- kill -SIGUSR1 1
Now let’s try out our new DNS records.
$ kubectl run -it --rm --restart=Never --image=infoblox/dnstools:latest dnstools
If you don't see a command prompt, try pressing enter.
/ # host kubernetes
kubernetes.default.svc.cluster.local has address 10.0.0.1
/ # host something
something.cluster.local has address 10.0.0.1
/ # host otherthign
Host otherthign not found: 3(NXDOMAIN)
/ # host otherthing
otherthing.cluster.local is an alias for google.com.
google.com has IPv6 address 2607:f8b0:4005:805::200e
google.com mail is handled by 30 alt2.aspmx.l.google.com.
google.com mail is handled by 50 alt4.aspmx.l.google.com.
google.com mail is handled by 40 alt3.aspmx.l.google.com.
google.com mail is handled by 20 alt1.aspmx.l.google.com.
google.com mail is handled by 10 aspmx.l.google.com.
/ #
We can see if we put in an incorrect name, we still get NXDOMAIN as you would expect. However, a correct name will resolve to the record from our zone file. So, we now have a way to create custom entries in the cluster domain.
In the standard CoreDNS release, the kubernetes plugin comes before file and etcd. This means that it gets the
first chance to handle the query. You can rebuild CoreDNS to change that ordering if you wish - take a look at
Miek’s post on How to Add Plugins to CoreDNS
if you want to see how that’s done.